519 Commits (9e4a0e76f6aa9796ad7baa7f623ba98274676c6f)

Author	SHA1	Message	Date
Eugene Yurtsev	616196c620	Docs: Add how to dispatch custom callback events (#24278) ... * Add how-to guide for dispatching custom callback events. * Add links from index to the how to guide * Add link from streaming from within a tool * Update versionadded to correct release https://github.com/langchain-ai/langchain/releases/tag/langchain-core%3D%3D0.2.15	2 months ago
Leonid Ganeline	5ccf8ebfac	core: docstrings `vectorstores` update (#24281) ... Added missed docstrings. Formatted docstrings to the consistent form. --------- Co-authored-by: Erick Friis <erick@langchain.dev>	2 months ago
Bagatur	dc42279eb5	core[patch]: fix Typing.cast import (#24313) ... Fixes #24287	2 months ago
Leonid Ganeline	5fcf2ef7ca	core: docstrings `documents` (#23506) ... Added missed docstrings. Formatted docstrings to the consistent form.	2 months ago
Shenhai Ran	5f2dea2b20	core[patch]: Add encoding options when create prompt template from a file (#24054) ... - Uses default utf-8 encoding for loading prompt templates from file --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	2 months ago
Leonid Ganeline	198b85334f	core[patch]: docstrings `langchain_core/` files update (#24285) ... Added missed docstrings. Formatted docstrings to the consistent form.	2 months ago
Tibor Reiss	1c753d1e81	core[patch]: Update typing for template format to include jinja2 as a Literal (#24144) ... Fixes #23929 via adjusting the typing	2 months ago
JP-Ellis	f77659463a	core[patch]: allow message utils to work with lcel (#23743) ... The functions `convert_to_messages` has had an expansion of the arguments it can take: 1. Previously, it only could take a `Sequence` in order to iterate over it. This has been broadened slightly to an `Iterable` (which should have no other impact). 2. Support for `PromptValue` and `BaseChatPromptTemplate` has been added. These are generated when combining messages using the overloaded `+` operator. Functions which rely on `convert_to_messages` (namely `filter_messages`, `merge_message_runs` and `trim_messages`) have had the type of their arguments similarly expanded. Resolves #23706. <!-- If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17. --> --------- Signed-off-by: JP-Ellis <josh@jpellis.me> Co-authored-by: Bagatur <baskaryan@gmail.com>	2 months ago
Harold Martin	ccdaf14eff	docs: Spell check fixes (#24217) ... **Description:** Spell check fixes for docs, comments, and a couple of strings. No code change e.g. variable names. **Issue:** none **Dependencies:** none **Twitter handle:** hmartin	2 months ago
Leonid Ganeline	cacdf96f9c	core docstrings `tracers` update (#24211) ... Added missed docstrings. Formatted docstrings to the consistent form.	2 months ago
Leonid Ganeline	36ee083753	core: docstrings `utils` update (#24213) ... Added missed docstrings. Formatted docstrings to the consistent form.	2 months ago
ccurme	888fbc07b5	core[patch]: support passing `args_schema` through `as_tool` (#24269) ... Note: this allows the schema to be passed in positionally. ```python from langchain_core.pydantic_v1 import BaseModel, Field from langchain_core.runnables import RunnableLambda class Add(BaseModel): """Add two integers together.""" a: int = Field(..., description="First integer") b: int = Field(..., description="Second integer") def add(input: dict) -> int: return input["a"] + input["b"] runnable = RunnableLambda(add) as_tool = runnable.as_tool(Add) as_tool.args_schema.schema() ``` ``` {'title': 'Add', 'description': 'Add two integers together.', 'type': 'object', 'properties': {'a': {'title': 'A', 'description': 'First integer', 'type': 'integer'}, 'b': {'title': 'B', 'description': 'Second integer', 'type': 'integer'}}, 'required': ['a', 'b']} ```	2 months ago
Bagatur	0da5078cad	langchain[minor]: Generic configurable model (#23419) ... alternative to [23244](https://github.com/langchain-ai/langchain/pull/23244). allows you to use chat model declarative methods 	2 months ago
Bagatur	d0728b0ba0	core[patch]: add tool name to tool message (#24243) ... Copying current ToolNode behavior	2 months ago
Bagatur	65321bf975	core[patch]: fix ToolCall "type" when streaming (#24218)	2 months ago
Eugene Yurtsev	8d82a0d483	core[patch]: Mark GraphVectorStore as beta (#24195) ... * This PR marks graph vectorstore as beta	2 months ago
Bagatur	6166ea67a8	core[minor]: rename ToolMessage.raw_output -> artifact (#24185)	2 months ago
Leonid Ganeline	aa3e3cfa40	core[patch]: docstrings `runnables` update (#24161) ... Added missed docstrings. Formatted docstrings to the consistent form.	2 months ago
Nuno Campos	1d37aa8403	core: Remove extra newline (#24157)	2 months ago
Bagatur	8d100c58de	core[patch]: Tool accept RunnableConfig (#24143) ... Relies on #24038 --------- Co-authored-by: Erick Friis <erick@langchain.dev>	2 months ago
Bagatur	5fd1e67808	core[minor], integrations...[patch]: Support ToolCall as Tool input and ToolMessage as Tool output (#24038) ... Changes: - ToolCall, InvalidToolCall and ToolCallChunk can all accept a "type" parameter now - LLM integration packages add "type" to all the above - Tool supports ToolCall inputs that have "type" specified - Tool outputs ToolMessage when a ToolCall is passed as input - Tools can separately specify ToolMessage.content and ToolMessage.raw_output - Tools emit events for validation errors (using on_tool_error and on_tool_end) Example: ```python @tool("structured_api", response_format="content_and_raw_output") def _mock_structured_tool_with_raw_output( arg1: int, arg2: bool, arg3: Optional[dict] = None ) -> Tuple[str, dict]: """A Structured Tool""" return f"{arg1} {arg2}", {"arg1": arg1, "arg2": arg2, "arg3": arg3} def test_tool_call_input_tool_message_with_raw_output() -> None: tool_call: Dict = { "name": "structured_api", "args": {"arg1": 1, "arg2": True, "arg3": {"img": "base64string..."}}, "id": "123", "type": "tool_call", } expected = ToolMessage("1 True", raw_output=tool_call["args"], tool_call_id="123") tool = _mock_structured_tool_with_raw_output actual = tool.invoke(tool_call) assert actual == expected tool_call.pop("type") with pytest.raises(ValidationError): tool.invoke(tool_call) actual_content = tool.invoke(tool_call["args"]) assert actual_content == expected.content ``` --------- Co-authored-by: Erick Friis <erick@langchain.dev>	2 months ago
Nuno Campos	03fba07d15	core[patch]: Update styles for mermaid graphs (#24147)	2 months ago
ccurme	8ee8ca7c83	core[patch]: propagate `parse_docstring` to tool decorator (#24123) ... Disabled by default. ```python from langchain_core.tools import tool @tool(parse_docstring=True) def foo(bar: str, baz: int) -> str: """The foo. Args: bar: this is the bar baz: this is the baz """ return bar foo.args_schema.schema() ``` ```json { "title": "fooSchema", "description": "The foo.", "type": "object", "properties": { "bar": { "title": "Bar", "description": "this is the bar", "type": "string" }, "baz": { "title": "Baz", "description": "this is the baz", "type": "integer" } }, "required": [ "bar", "baz" ] } ```	2 months ago
ccurme	122e80e04d	core[patch]: add versionadded to `as_tool` (#24138)	2 months ago
Nuno Campos	2428984205	core: Add metadata to graph json repr (#24131) ... Thank you for contributing to LangChain! - [ ] **PR title**: "package: description" - Where "package" is whichever of langchain, community, core, experimental, etc. is being modified. Use "docs: ..." for purely docs changes, "templates: ..." for template changes, "infra: ..." for CI changes. - Example: "community: add foobar LLM" - [ ] **PR message**: ***Delete this entire checklist*** and replace with - **Description:** a description of the change - **Issue:** the issue # it fixes, if applicable - **Dependencies:** any dependencies required for this change - **Twitter handle:** if your PR gets announced, and you'd like a mention, we'll gladly shout you out! - [ ] **Add tests and docs**: If you're adding a new integration, please include 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. - [ ] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ Additional guidelines: - Make sure optional dependencies are imported within a function. - Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests. - Most PRs should not touch more than one package. - Changes should be backwards compatible. - If you are adding something to community, do not re-import it in langchain. If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17.	2 months ago
Nuno Campos	3e454d7568	core: fix docstring (#24129)	2 months ago
Nuno Campos	ee3fe20af4	core: mermaid: Render metadata key-value pairs when drawing mermaid graph (#24103) ... - if node is runnable binding with metadata attached	2 months ago
Eugene Yurtsev	dc131ac42a	core[minor]: Add dispatching for custom events (#24080) ... This PR allows dispatching adhoc events for a given run. # Context This PR allows users to send arbitrary data to the callback system and to the astream events API from within a given runnable. This can be extremely useful to surface custom information to end users about progress etc. Integration with langsmith tracer will be done separately since the data cannot be currently visualized. It'll be accommodated using the events attribute of the Run # Examples with astream events ```python from langchain_core.callbacks import adispatch_custom_event from langchain_core.tools import tool @tool async def foo(x: int) -> int: """Foo""" await adispatch_custom_event("event1", {"x": x}) await adispatch_custom_event("event2", {"x": x}) return x + 1 async for event in foo.astream_events({'x': 1}, version='v2'): print(event) ``` ```python {'event': 'on_tool_start', 'data': {'input': {'x': 1}}, 'name': 'foo', 'tags': [], 'run_id': 'fd6fb7a7-dd37-4191-962c-e43e245909f6', 'metadata': {}, 'parent_ids': []} {'event': 'on_custom_event', 'run_id': 'fd6fb7a7-dd37-4191-962c-e43e245909f6', 'name': 'event1', 'tags': [], 'metadata': {}, 'data': {'x': 1}, 'parent_ids': []} {'event': 'on_custom_event', 'run_id': 'fd6fb7a7-dd37-4191-962c-e43e245909f6', 'name': 'event2', 'tags': [], 'metadata': {}, 'data': {'x': 1}, 'parent_ids': []} {'event': 'on_tool_end', 'data': {'output': 2}, 'run_id': 'fd6fb7a7-dd37-4191-962c-e43e245909f6', 'name': 'foo', 'tags': [], 'metadata': {}, 'parent_ids': []} ``` ```python from langchain_core.callbacks import adispatch_custom_event from langchain_core.runnables import RunnableLambda @RunnableLambda async def foo(x: int) -> int: """Foo""" await adispatch_custom_event("event1", {"x": x}) await adispatch_custom_event("event2", {"x": x}) return x + 1 async for event in foo.astream_events(1, version='v2'): print(event) ``` ```python {'event': 'on_chain_start', 'data': {'input': 1}, 'name': 'foo', 'tags': [], 'run_id': 'ce2beef2-8608-49ea-8eba-537bdaafb8ec', 'metadata': {}, 'parent_ids': []} {'event': 'on_custom_event', 'run_id': 'ce2beef2-8608-49ea-8eba-537bdaafb8ec', 'name': 'event1', 'tags': [], 'metadata': {}, 'data': {'x': 1}, 'parent_ids': []} {'event': 'on_custom_event', 'run_id': 'ce2beef2-8608-49ea-8eba-537bdaafb8ec', 'name': 'event2', 'tags': [], 'metadata': {}, 'data': {'x': 1}, 'parent_ids': []} {'event': 'on_chain_stream', 'run_id': 'ce2beef2-8608-49ea-8eba-537bdaafb8ec', 'name': 'foo', 'tags': [], 'metadata': {}, 'data': {'chunk': 2}, 'parent_ids': []} {'event': 'on_chain_end', 'data': {'output': 2}, 'run_id': 'ce2beef2-8608-49ea-8eba-537bdaafb8ec', 'name': 'foo', 'tags': [], 'metadata': {}, 'parent_ids': []} ``` # Examples with handlers This is copy pasted from unit tests ```python class CustomCallbackManager(BaseCallbackHandler): def __init__(self) -> None: self.events: List[Any] = [] def on_custom_event( self, name: str, data: Any, *, run_id: UUID, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any, ) -> None: assert kwargs == {} self.events.append( ( name, data, run_id, tags, metadata, ) ) callback = CustomCallbackManager() run_id = uuid.UUID(int=7) @RunnableLambda def foo(x: int, config: RunnableConfig) -> int: dispatch_custom_event("event1", {"x": x}) dispatch_custom_event("event2", {"x": x}, config=config) return x foo.invoke(1, {"callbacks": [callback], "run_id": run_id}) assert callback.events == [ ("event1", {"x": 1}, UUID("00000000-0000-0000-0000-000000000007"), [], {}), ("event2", {"x": 1}, UUID("00000000-0000-0000-0000-000000000007"), [], {}), ] ```	2 months ago
ccurme	975b6129f6	core[patch]: support conversion of runnables to tools (#23992) ... Open to other thoughts on UX. string input: ```python as_tool = retriever.as_tool() as_tool.invoke("cat") # [Document(...), ...] ``` typed dict input: ```python class Args(TypedDict): key: int def f(x: Args) -> str: return str(x["key"] * 2) as_tool = RunnableLambda(f).as_tool( name="my tool", description="description", # name, description are inferred if not supplied ) as_tool.invoke({"key": 3}) # "6" ``` for untyped dict input, allow specification of parameters + types ```python def g(x: Dict[str, Any]) -> str: return str(x["key"] * 2) as_tool = RunnableLambda(g).as_tool(arg_types={"key": int}) result = as_tool.invoke({"key": 3}) # "6" ``` Passing the `arg_types` is slightly awkward but necessary to ensure tool calls populate parameters correctly: ```python from typing import Any, Dict from langchain_core.runnables import RunnableLambda from langchain_openai import ChatOpenAI def f(x: Dict[str, Any]) -> str: return str(x["key"] * 2) runnable = RunnableLambda(f) as_tool = runnable.as_tool(arg_types={"key": int}) llm = ChatOpenAI().bind_tools([as_tool]) result = llm.invoke("Use the tool on 3.") tool_call = result.tool_calls[0] args = tool_call["args"] assert args == {"key": 3} as_tool.run(args) ``` Contrived (?) example with langgraph agent as a tool: ```python from typing import List, Literal from typing_extensions import TypedDict from langchain_openai import ChatOpenAI from langgraph.prebuilt import create_react_agent llm = ChatOpenAI(temperature=0) def magic_function(input: int) -> int: """Applies a magic function to an input.""" return input + 2 agent_1 = create_react_agent(llm, [magic_function]) class Message(TypedDict): role: Literal["human"] content: str agent_tool = agent_1.as_tool( arg_types={"messages": List[Message]}, name="Jeeves", description="Ask Jeeves.", ) agent_2 = create_react_agent(llm, [agent_tool]) ``` --------- Co-authored-by: Erick Friis <erick@langchain.dev>	3 months ago
Bagatur	6928f4c438	core[minor]: Add ToolMessage.raw_output (#23994) ... Decisions to discuss: 1. is a new attr needed or could additional_kwargs be used for this 2. is raw_output a good name for this attr 3. should raw_output default to {} or None 4. should raw_output be included in serialization 5. do we need to update repr/str to exclude raw_output	3 months ago
William FH	1e1fd30def	[Core] Fix fstring in logger warning (#24043)	3 months ago
Nuno Campos	859e434932	core: Speed up json parse for large strings (#24036) ... for a large string: - old 4.657918874989264 - new 0.023724667000351474	3 months ago
Nuno Campos	160fc7f246	core: Move json parsing in base chat model / output parser to bg thread (#24031) ... - add version of AIMessageChunk.__add__ that can add many chunks, instead of only 2 - In agenerate_from_stream merge and parse chunks in bg thread - In output parse base classes do more work in bg threads where appropriate --------- Co-authored-by: William FH <13333726+hinthornw@users.noreply.github.com>	3 months ago
Eugene Yurtsev	f765e8fa9d	core[minor],community[patch],standard-tests[patch]: Move InMemoryImplementation to langchain-core (#23986) ... This PR moves the in memory implementation to langchain-core. * The implementation remains importable from langchain-community. * Supporting utilities are marked as private for now.	3 months ago
Eugene Yurtsev	2c180d645e	core[minor],community[minor]: Upgrade all @root_validator() to @pre_init (#23841) ... This PR introduces a @pre_init decorator that's a @root_validator(pre=True) but with all the defaults populated!	3 months ago
Eugene Yurtsev	e0186df56b	core[patch]: Clarify upsert response semantics (#23921)	3 months ago
Eugene Yurtsev	5b7d5f7729	core[patch]: Add comment to clarify aadd_documents (#23920) ... Add comment to clarify how add documents works	3 months ago
ccurme	74c7198906	core, anthropic[patch]: support streaming tool calls when function has no arguments (#23915) ... resolves https://github.com/langchain-ai/langchain/issues/23911 When an AIMessageChunk is instantiated, we attempt to parse tool calls off of the tool_call_chunks. Here we add a special-case to this parsing, where `""` will be parsed as `{}`. This is a reaction to how Anthropic streams tool calls in the case where a function has no arguments: ``` {'id': 'toolu_01J8CgKcuUVrMqfTQWPYh64r', 'input': {}, 'name': 'magic_function', 'type': 'tool_use', 'index': 1} {'partial_json': '', 'type': 'tool_use', 'index': 1} ``` The `partial_json` does not accumulate to a valid json string-- most other providers tend to emit `"{}"` in this case.	3 months ago
Christophe Bornet	42d049f618	core[minor]: Add Graph Store component (#23092) ... This PR introduces a GraphStore component. GraphStore extends VectorStore with the concept of links between documents based on document metadata. This allows linking documents based on a variety of techniques, including common keywords, explicit links in the content, and other patterns. This works with existing Documents, so it’s easy to extend existing VectorStores to be used as GraphStores. The interface can be implemented for any Vector Store technology that supports metadata, not only graph DBs. When retrieving documents for a given query, the first level of search is done using classical similarity search. Next, links may be followed using various traversal strategies to get additional documents. This allows documents to be retrieved that aren’t directly similar to the query but contain relevant information. 2 retrieving methods are added to the VectorStore ones : * traversal_search which gets all linked documents up to a certain depth * mmr_traversal_search which selects linked documents using an MMR algorithm to have more diverse results. If a depth of retrieval of 0 is used, GraphStore is effectively a VectorStore. It enables an easy transition from a simple VectorStore to GraphStore by adding links between documents as a second step. An implementation for Apache Cassandra is also proposed. See https://github.com/datastax/ragstack-ai/blob/main/libs/knowledge-store/notebooks/astra_support.ipynb for a notebook explaining how to use GraphStore and that shows that it can answer correctly to questions that a simple VectorStore cannot. **Twitter handle:** _cbornet	3 months ago
Leonid Ganeline	77f5fc3d55	core: docstrings `load` (#23787) ... Added missed docstrings. Formatted docstrings to the consistent form.	3 months ago
Eugene Yurtsev	6f08e11d7c	core[minor]: add upsert, streaming_upsert, aupsert, astreaming_upsert methods to the VectorStore abstraction (#23774) ... This PR rolls out part of the new proposed interface for vectorstores (https://github.com/langchain-ai/langchain/pull/23544) to existing store implementations. The PR makes the following changes: 1. Adds standard upsert, streaming_upsert, aupsert, astreaming_upsert methods to the vectorstore. 2. Updates `add_texts` and `aadd_texts` to be non required with a default implementation that delegates to `upsert` and `aupsert` if those have been implemented. The original `add_texts` and `aadd_texts` methods are problematic as they spread object specific information across document and **kwargs. (e.g., ids are not a part of the document) 3. Adds a default implementation to `add_documents` and `aadd_documents` that delegates to `upsert` and `aupsert` respectively. 4. Adds standard unit tests to verify that a given vectorstore implements a correct read/write API. A downside of this implementation is that it creates `upsert` with a very similar signature to `add_documents`. The reason for introducing `upsert` is to: * Remove any ambiguities about what information is allowed in `kwargs`. Specifically kwargs should only be used for information common to all indexed data. (e.g., indexing timeout). *Allow inheriting from an anticipated generalized interface for indexing that will allow indexing `BaseMedia` (i.e., allow making a vectorstore for images/audio etc.) `add_documents` can be deprecated in the future in favor of `upsert` to make sure that users have a single correct way of indexing content. --------- Co-authored-by: ccurme <chester.curme@gmail.com>	3 months ago
G Sreejith	3c752238c5	core[patch]: Fix typo in docstring (graphm -> graph) (#23910) ... Changes has been as per the request Replaced graphm with graph	3 months ago
Leonid Ganeline	12c92b6c19	core: docstrings `outputs` (#23889) ... Added missed docstrings. Formatted docstrings to the consistent form.	3 months ago
Leonid Ganeline	1eca98ec56	core: docstrings `prompts` (#23890) ... Added missed docstrings. Formatted docstrings to the consistent form.	3 months ago
Mohammad Mohtashim	2274d2b966	core[patch]: Accounting for Optional Input Variables in BasePromptTemplate (#22851) ... **Description**: After reviewing the prompts API, it is clear that the only way a user can explicitly mark an input variable as optional is through the `MessagePlaceholder.optional` attribute. Otherwise, the user must explicitly pass in the `input_variables` expected to be used in the `BasePromptTemplate`, which will be validated upon execution. Therefore, to semantically handle a `MessagePlaceholder` `variable_name` as optional, we will treat the `variable_name` of `MessagePlaceholder` as a `partial_variable` if it has been marked as optional. This approach aligns with how the `variable_name` of `MessagePlaceholder` is already handled [here](https://github.com/keenborder786/langchain/blob/optional_input_variables/libs/core/langchain_core/prompts/chat.py#L991). Additionally, an attribute `optional_variable` has been added to `BasePromptTemplate`, and the `variable_name` of `MessagePlaceholder` is also made part of `optional_variable` when marked as optional. Moreover, the `get_input_schema` method has been updated for `BasePromptTemplate` to differentiate between optional and non-optional variables. **Issue**: #22832, #21425 --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	3 months ago
Eugene Yurtsev	9ccc4b1616	core[patch]: Fix logic in BaseChatModel that processes the llm string that is used as a key for caching chat models responses (#23842) ... This PR should fix the following issue: https://github.com/langchain-ai/langchain/issues/23824 Introduced as part of this PR: https://github.com/langchain-ai/langchain/pull/23416 I am unable to reproduce the issue locally though it's clear that we're getting a `serialized` object which is not a dictionary somehow. The test below passes for me prior to the PR as well ```python def test_cache_with_sqllite() -> None: from langchain_community.cache import SQLiteCache from langchain_core.globals import set_llm_cache cache = SQLiteCache(database_path=".langchain.db") set_llm_cache(cache) chat_model = FakeListChatModel(responses=["hello", "goodbye"], cache=True) assert chat_model.invoke("How are you?").content == "hello" assert chat_model.invoke("How are you?").content == "hello" ```	3 months ago
Vadym Barda	9bb623381b	core[minor]: update conversion utils to handle RemoveMessage (#23840)	3 months ago
Théo Deschamps	39b19cf764	core[patch]: extract input variables for `path` and `detail` keys in order to format an `ImagePromptTemplate` (#22613) ... - Description: Add support for `path` and `detail` keys in `ImagePromptTemplate`. Previously, only variables associated with the `url` key were considered. This PR allows for the inclusion of a local image path and a detail parameter as input to the format method. - Issues: - fixes #20820 - related to #22024 - Dependencies: None - Twitter handle: @DeschampsTho5 --------- Co-authored-by: tdeschamps <tdeschamps@kameleoon.com> Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com> Co-authored-by: Eugene Yurtsev <eugene@langchain.dev>	3 months ago
Leonid Ganeline	55f6f91f17	core[patch]: docstrings `output_parsers` (#23825) ... Added missed docstrings. Formatted docstrings to the consistent form.	3 months ago
Bagatur	a0c2281540	infra: update mypy 1.10, ruff 0.5 (#23721) ... ```python """python scripts/update_mypy_ruff.py""" import glob import tomllib from pathlib import Path import toml import subprocess import re ROOT_DIR = Path(__file__).parents[1] def main(): for path in glob.glob(str(ROOT_DIR / "libs/**/pyproject.toml"), recursive=True): print(path) with open(path, "rb") as f: pyproject = tomllib.load(f) try: pyproject["tool"]["poetry"]["group"]["typing"]["dependencies"]["mypy"] = ( "^1.10" ) pyproject["tool"]["poetry"]["group"]["lint"]["dependencies"]["ruff"] = ( "^0.5" ) except KeyError: continue with open(path, "w") as f: toml.dump(pyproject, f) cwd = "/".join(path.split("/")[:-1]) completed = subprocess.run( "poetry lock --no-update; poetry install --with typing; poetry run mypy . --no-color", cwd=cwd, shell=True, capture_output=True, text=True, ) logs = completed.stdout.split("\n") to_ignore = {} for l in logs: if re.match("^(.*)\:(\d+)\: error:.*\[(.*)\]", l): path, line_no, error_type = re.match( "^(.*)\:(\d+)\: error:.*\[(.*)\]", l ).groups() if (path, line_no) in to_ignore: to_ignore[(path, line_no)].append(error_type) else: to_ignore[(path, line_no)] = [error_type] print(len(to_ignore)) for (error_path, line_no), error_types in to_ignore.items(): all_errors = ", ".join(error_types) full_path = f"{cwd}/{error_path}" try: with open(full_path, "r") as f: file_lines = f.readlines() except FileNotFoundError: continue file_lines[int(line_no) - 1] = ( file_lines[int(line_no) - 1][:-1] + f" # type: ignore[{all_errors}]\n" ) with open(full_path, "w") as f: f.write("".join(file_lines)) subprocess.run( "poetry run ruff format .; poetry run ruff --select I --fix .", cwd=cwd, shell=True, capture_output=True, text=True, ) if __name__ == "__main__": main() ```	3 months ago