1404 Commits (e669f9d731af3d673885dcddc46be4cbef5959b9)

Author	SHA1	Message	Date
April	c14a8df2ee	wrap confluence attachment processing with a try-except block (#11503) ... Prevents document loading from erroring out when an attachment is not found at the url. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
eajechiloae	4ba2c8ba75	Fix ClearML callback (#11472) ... Handle different field names in dicts/dataframes, fixing the ClearML callback. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Lawrence Wu	93bb19f69a	Fix chains/loading.py error messages (#11688) ... - **Description:** make the error messages consistent in chains/loading.py - **Dependencies:** None	9 months ago
Harrison Chase	18ebce2032	fix tool async (#11689)	9 months ago
sudranga	9beb03e771	11474 (#11519) ... No relevant documents may be found for a given question. In some use cases, we could directly respond with a fixed message instead of doing an LLM call with an empty context. This PR exposes this as an option: response_if_no_docs_found. --------- Co-authored-by: Sudharsan Rangarajan <sudranga@nile-global.com>	9 months ago
Joaquin Menendez	ef99b06362	feature: add metadata information into the embedding file before uplo… (#11553) ... Replace this entire comment with: - **Description:** In this modified version of the function, if the metadatas parameter is not None, the function includes the corresponding metadata in the JSON object for each text. This allows the metadata to be stored alongside the text's embedding in the vector store. - - **Issue:** #10924 - **Dependencies:** None - **Tag maintainer:** @hwchase17 @agola11 - **Twitter handle:** @MelliJoaco --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Marcin Wątroba	51a3a86022	#11655 Add SQLAlchemyMd5Cache implementation (#11660) ... - **Description:** Add SQLAlchemyMd5Cache implementation, - **Issue:** the issue # #11655, - **Dependencies:** no deps, - **Tag maintainer:** @markowanga --------- Co-authored-by: Marcin Wątroba <marcin.watroba@pwr.edu.pl> Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Suresh Kumar Ponnusamy	70f7558db2	langchain-experimental: Add allow_list support in experimental/data_anonymizer (#11597) ... - **Description:** Add allow_list support in langchain experimental data-anonymizer package - **Issue:** no - **Dependencies:** no - **Tag maintainer:** @hwchase17 - **Twitter handle:**	9 months ago
wemysschen	2363c02cf3	Bos loader (#11525) ... **Description:** Add BaiduCloud BOS document loader. --------- Co-authored-by: chenweixu01 <chenweixu01@baidu.com> Co-authored-by: root <root@icoding-cwx.bcc-szzj.baidu.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Kwanghoon Choi	fbb82608cd	Fixed a bug in reporting Python code validation (#11522) ... - **Description:** fixed a bug in pal-chain when it reports Python code validation errors. When node.func does not have any ids, the original code tried to print node.func.id in raising ValueError. - **Issue:** n/a, - **Dependencies:** no dependencies, - **Tag maintainer:** @hazzel-cn, @eyurtsev - **Twitter handle:** @lazyswamp --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Harrison Chase	9f39c23a13	add input type for convo retrieval chain (#11679)	9 months ago
zhaozhiming	d5e762d328	fix: Change the docs of JSONAgentOutputParser (#11594) ... I am merely making some minor adjustments to the function documentation. I hope to provide a small assistance to LangChain. - **Description:** Change the docs of JSONAgentOutputParser. It will be `JSON` better, - **Issue:** no, - **Dependencies:** no, - **Tag maintainer:** @hwchase17, - **Twitter handle:** Not worth mentioning.	9 months ago
Vinay Kakade	dd0cd98861	Add support for ChatOpenAI models in Infino callback handler (#11608) ... **Description:** This PR adds support for ChatOpenAI models in the Infino callback handler. In particular, this PR implements `on_chat_model_start` callback, so that ChatOpenAI models are supported. With this change, Infino callback handler can be used to track latency, errors, and prompt tokens for ChatOpenAI models too (in addition to the support for OpenAI and other non-chat models it has today). The existing example notebook is updated to show how to use this integration as well. cc/ @naman-modi @savannahar68 **Issue:** https://github.com/langchain-ai/langchain/issues/11607 **Dependencies:** None **Tag maintainer:** @hwchase17 **Twitter handle:** [@vkakade](https://twitter.com/vkakade)	9 months ago
Israel Ekpo	d0603c86b6	Add Support for Azure Cosmos DB MongoDB vCore Vector Store #11627 (#11632) ... This PR adds support for the Azure Cosmos DB MongoDB vCore Vector Store https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/ https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search Summary: - **Description:** added vector store integration for Azure Cosmos DB MongoDB vCore Vector Store, - **Issue:** the issue # it fixes #11627, - **Dependencies:** pymongo dependency, - **Tag maintainer:** @hwchase17, - **Twitter handle:** @izzyacademy --------- Co-authored-by: Israel Ekpo <israel.ekpo@gmail.com> Co-authored-by: Israel Ekpo <44282278+izzyacademy@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Erick Friis	28ee6a7c12	Track ChatFireworks time to first_token (#11672)	9 months ago
Eugene Yurtsev	539941281d	Fix output types for BaseChatModel (#11670) ... * Should use non chunked messages for Invoke/Batch * After this PR, stream output type is not represented, do we want to use the union? --------- Co-authored-by: Erick Friis <erick@langchain.dev>	9 months ago
Eugene Yurtsev	99adcdb1c9	Add dedicated `type` attribute to be used solely for serialization purposes (#11585) ... Adds standard `type` field for all messages that will be serialized/validated by pydantic. * The presence of `type` makes it easier for developers consuming schemas to write client code to serialize/deserialize. * In LangServe `type` will be used for both validation and will appear in the generated openapi specs	9 months ago
eryk-dsai	06d5971be9	Fix issue #10985 - Skip model.to(device) if it is instantiated with bitsandbytes config (#11009) ... Preventing error caused by attempting to move the model that was already loaded on the GPU using the Accelerate module to the same or another device. It is not possible to load model with Accelerate/PEFT to CPU for now Addresses: [#10985](https://github.com/langchain-ai/langchain/issues/10985)	9 months ago
Nuno Campos	64969bc8ae	Add patch_config(configurable=) arg, make with_config(configurable=) merge it with existing (#11662) ... <!-- Thank you for contributing to LangChain! Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes (if applicable), - **Dependencies:** any dependencies required for this change, - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below), - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. -->	9 months ago
Harrison Chase	ce0019b646	make utils conditional (#11646)	9 months ago
Harrison Chase	8f06085b24	make tools conditional (#11647)	9 months ago
Bassem Yacoube	5451b724fc	Adds support for llama2 and fixes MPT-7b url (#11465) ... - **Description:** This is an update to OctoAI LLM provider that adds support for llama2 endpoints hosted on OctoAI and updates MPT-7b url with the current one. @baskaryan Thanks! --------- Co-authored-by: ML Wiz <bassemgeorgi@gmail.com>	9 months ago
Todd Kerpelman	0bff399af1	Make metadata from the url_selenium loader match that of the web_base loader (#11617) ... **Description:** I noticed the metadata returned by the url_selenium loader was missing several values included by the web_base loader. (The former returned `{source: ...}`, the latter returned `{source: ..., title: ..., description: ..., language: ...}`.) This change fixes it so both loaders return all 4 key value pairs. Files have been properly formatted and all tests are passing. Note, however, that I am not much of a python expert, so that whole "Adding the imports inside the code so that tests pass" thing seems weird to me. Please LMK if I did anything wrong.	9 months ago
Tarun Thotakura	c9d4d53545	Fixed the assignment of custom_llm_provider argument (#11628) ... - **Description:** Assigning the custom_llm_provider to the default params function so that it will be passed to the litellm - **Issue:** Even though the custom_llm_provider argument is being defined it's not being assigned anywhere in the code and hence its not being passed to litellm, therefore any litellm call which uses the custom_llm_provider as required parameter is being failed. This parameter is mainly used by litellm when we are doing inference via Custom API server. https://docs.litellm.ai/docs/providers/custom_openai_proxy - **Dependencies:** No dependencies are required @krrishdholakia , @baskaryan --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Leonid Ganeline	db67ccb0bb	docstrings cleanup (#11640) ... Added missed docstrings. Some reformatting.	9 months ago
Yang, Bo	3a82bd7bdb	Use raise from statement so that users can find detailed error message (#11461) ... - **Description:** Use `raise from` statement so that users can find detailed error message - **Tag maintainer:** @baskaryan, @eyurtsev, @hwchase17	9 months ago
Nuno Campos	9a0ed75a95	Add configurable fields with options (#11601) ... <!-- Thank you for contributing to LangChain! Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes (if applicable), - **Dependencies:** any dependencies required for this change, - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below), - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. -->	9 months ago
Bagatur	7232e082de	bump 312 (#11621)	9 months ago
Eugene Yurtsev	58220cda72	Remove LLM Bash and related bash utilities (#11619) ... Deprecate LLMBash and related bash utilities	9 months ago
Shubham Kushwaha	49de862076	Arcee.ai LLM & Retriever integration (#11579) ... - **Description:** This PR introduces a new LLM and Retriever API to https://arcee.ai for the python client - **Issue:** implements the integrations as requested in #11578 , - **Dependencies:** no dependencies are required, - **Tag maintainer:** @hwchase17 - **Twitter handle:** shwooobham **✅ `make format`, `make lint` and `make test` runs locally.** ```shell =========== 1245 passed, 277 skipped, 20 warnings in 16.26s =========== ./scripts/check_pydantic.sh . ./scripts/check_imports.sh poetry run ruff . [ "." = "" ] || poetry run black . --check All done! ✨ 🍰 ✨ 1818 files would be left unchanged. [ "." = "" ] || poetry run mypy . Success: no issues found in 1815 source files [ "." = "" ] || poetry run black . All done! ✨ 🍰 ✨ 1818 files left unchanged. [ "." = "" ] || poetry run ruff --select I --fix . poetry run codespell --toml pyproject.toml poetry run codespell --toml pyproject.toml -w ``` **Contributions** 1. Arcee (langchain/llms), ArceeRetriever (langchain/retrievers), ArceeWrapper (langchain/utilities) 2. docs for Arcee (llms/arcee.py) and ArceeRetriever(retrievers/arcee.py) 3. cc: @jacobsolawetz @ben-epstein --------- Co-authored-by: Shubham <shubham@sORo.local>	9 months ago
Eugene Yurtsev	b56ca0c2a4	Deprecate LLMSymbolicMath from langchain core (#11615) ... Deprecate LLMSymbolicMath from langchain core package.	9 months ago
Eugene Yurtsev	c9bce5bbfb	Add version to langchain_experimental (#11613) ... Add version to langchain experimental	9 months ago
Predrag Gruevski	22abeb9f6c	Disable loading jinja2 `PromptTemplate` from file. (#10252) ... jinja2 templates are not sandboxed and are at risk for arbitrary code execution. To mitigate this risk: - We no longer support loading jinja2-formatted prompt template files. - `PromptTemplate` with jinja2 may still be constructed manually, but the class carries a security warning reminding the user to not pass untrusted input into it. Resolves #4394.	9 months ago
Nuno Campos	c7c03d4709	Fix mutation bugs in callback manager configure (#11603) ... <!-- Thank you for contributing to LangChain! Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes (if applicable), - **Dependencies:** any dependencies required for this change, - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below), - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. -->	9 months ago
cccs-eric	e2a9072b80	Fix CohereRerank configuration (#11583) ... **Description:** CohereRerank is missing `cohere_api_key` as a field and since extras are forbidden, it is not possible to pass-in the key. The only way is to use an env variable named `COHERE_API_KEY`. For example, if trying to create a compressor like this: ```python cohere_api_key = "......Cohere api key......" compressor = CohereRerank(cohere_api_key=cohere_api_key) ``` you will get the following error: ``` File "/langchain/.venv/lib/python3.10/site-packages/pydantic/v1/main.py", line 341, in __init__ raise validation_error pydantic.v1.error_wrappers.ValidationError: 1 validation error for CohereRerank cohere_api_key extra fields not permitted (type=value_error.extra) ```	9 months ago
Anar	55fef4b64b	implemented add files method in LLMRails (#11518) ... This PR provides add files method with LLMRails. Implemented here are: docs/extras/integrations/vectorstores/llm-rails.ipynb --------- Co-authored-by: Anar Aliyev <aaliyev@mgmt.cloudnet.services>	9 months ago
Stephen Hankinson	316dddc7cd	fix wording of query_sql_database_tool_description (#11530) ... - **Description:** Fixes minor typo for the query_sql_database_tool_description in the db toolkit - **Issue:** N/A - **Dependencies:** N/A - **Tag maintainer:** @nfcampos - **Twitter handle:** N/A	9 months ago
Ash Vardanian	1acfe86353	Accelerating Math Utils with SimSIMD (#11566) ... LangChain relies on NumPy to compute cosine distances, which becomes a bottleneck with the growing dimensionality and number of embeddings. To avoid this bottleneck, in our libraries at [Unum](https://github.com/unum-cloud), we have created a specialized package - [SimSIMD](https://github.com/ashvardanian/simsimd), that knows how to use newer hardware capabilities. Compared to SciPy and NumPy, it reaches 3x-200x performance for various data types. Since publication, several LangChain users have asked me if I can integrate it into LangChain to accelerate their workflows, so here I am 🤗 ## Benchmarking To conduct benchmarks locally, run this in your Jupyter: ```py import numpy as np import scipy as sp import simsimd as simd import timeit as tt def cosine_similarity_np(X: np.ndarray, Y: np.ndarray) -> np.ndarray: X_norm = np.linalg.norm(X, axis=1) Y_norm = np.linalg.norm(Y, axis=1) with np.errstate(divide="ignore", invalid="ignore"): similarity = np.dot(X, Y.T) / np.outer(X_norm, Y_norm) similarity[np.isnan(similarity) | np.isinf(similarity)] = 0.0 return similarity def cosine_similarity_sp(X: np.ndarray, Y: np.ndarray) -> np.ndarray: return 1 - sp.spatial.distance.cdist(X, Y, metric='cosine') def cosine_similarity_simd(X: np.ndarray, Y: np.ndarray) -> np.ndarray: return 1 - simd.cdist(X, Y, metric='cosine') X = np.random.randn(1, 1536).astype(np.float32) Y = np.random.randn(1, 1536).astype(np.float32) repeat = 1000 print("NumPy: {:,.0f} ops/s, SciPy: {:,.0f} ops/s, SimSIMD: {:,.0f} ops/s".format( repeat / tt.timeit(lambda: cosine_similarity_np(X, Y), number=repeat), repeat / tt.timeit(lambda: cosine_similarity_sp(X, Y), number=repeat), repeat / tt.timeit(lambda: cosine_similarity_simd(X, Y), number=repeat), )) ``` ## Results I ran this on an M2 Pro Macbook for various data types and different number of rows in `X` and reformatted the results as a table for readability: | Data Type | NumPy | SciPy | SimSIMD | | :--- | ---: | ---: | ---: | | `f32, 1` | 59,114 ops/s | 80,330 ops/s | 475,351 ops/s | | `f16, 1` | 32,880 ops/s | 82,420 ops/s | 650,177 ops/s | | `i8, 1` | 47,916 ops/s | 115,084 ops/s | 866,958 ops/s | | `f32, 10` | 40,135 ops/s | 24,305 ops/s | 185,373 ops/s | | `f16, 10` | 7,041 ops/s | 17,596 ops/s | 192,058 ops/s | | `f16, 10` | 21,989 ops/s | 25,064 ops/s | 619,131 ops/s | | `f32, 100` | 3,536 ops/s | 3,094 ops/s | 24,206 ops/s | | `f16, 100` | 900 ops/s | 2,014 ops/s | 23,364 ops/s | | `i8, 100` | 5,510 ops/s | 3,214 ops/s | 143,922 ops/s | It's important to note that SimSIMD will underperform if both matrices are huge. That, however, seems to be an uncommon usage pattern for LangChain users. You can find a much more detailed performance report for different hardware models here: - [Apple M2 Pro](https://ashvardanian.com/posts/simsimd-faster-scipy/#appendix-1-performance-on-apple-m2-pro). - [4th Gen Intel Xeon Platinum](https://ashvardanian.com/posts/simsimd-faster-scipy/#appendix-2-performance-on-4th-gen-intel-xeon-platinum-8480). - [AWS Graviton 3](https://ashvardanian.com/posts/simsimd-faster-scipy/#appendix-3-performance-on-aws-graviton-3). ## Additional Notes 1. Previous version used `X = np.array(X)`, to repackage lists of lists. It's an anti-pattern, as it will use double-precision floating-point numbers, which are slow on both CPUs and GPUs. I have replaced it with `X = np.array(X, dtype=np.float32)`, but a more selective approach should be discussed. 2. In numerical computations, it's recommended to explicitly define tolerance levels, which were previously avoided in `np.allclose(expected, actual)` calls. For now, I've set absolute tolerance to distance computation errors as 0.01: `np.allclose(expected, actual, atol=1e-2)`. --- - **Dependencies:** adds `simsimd` dependency - **Tag maintainer:** @hwchase17 - **Twitter handle:** @ashvardanian --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
benchello	5de64e6d60	Add option to specify metadata columns in CSV loader (#11576) ... #### Description This PR adds the option to specify additional metadata columns in the CSVLoader beyond just `Source`. The current CSV loader includes all columns in `page_content` and if we want to have columns specified for `page_content` and `metadata` we have to do something like the below.: ``` csv = pd.read_csv( "path_to_csv" ).to_dict("records") documents = [ Document( page_content=doc["content"], metadata={ "last_modified_by": doc["last_modified_by"], "point_of_contact": doc["point_of_contact"], } ) for doc in csv ] ``` #### Usage Example Usage: ``` csv_test = CSVLoader( file_path="path_to_csv", metadata_columns=["last_modified_by", "point_of_contact"] ) ``` Example CSV: ``` content, last_modified_by, point_of_contact "hello world", "Person A", "Person B" ``` Example Result: ``` Document { page_content: "hello world" metadata: { row: '0', source: 'path_to_csv', last_modified_by: 'Person A', point_of_contact: 'Person B', } ``` --------- Co-authored-by: Ben Chello <bchello@dropbox.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Stephen Hankinson	447a523662	fix comments in output format (#11536) ... - **Description:** Fixes the comments in the ConvoOutputParser. Because the \\\\ is escaping a single \\, they render something like: `"action_input": string \ The input to the action` in the prompt. Changing this to \\\\\\\\ lets it escape two slashes so that it renders a proper comment: `"action_input": string \\ The input to the action` - **Issue:** N/A - **Dependencies:** - **Tag maintainer:** @hwchase17 - **Twitter handle:**	9 months ago
Michael Landis	8e45f720a8	feat: add momento vector index as a vector store provider (#11567) ... **Description**: - Added Momento Vector Index (MVI) as a vector store provider. This includes an implementation with docstrings, integration tests, a notebook, and documentation on the docs pages. - Updated the Momento dependency in pyproject.toml and the lock file to enable access to MVI. - Refactored the Momento cache and chat history session store to prefer using "MOMENTO_API_KEY" over "MOMENTO_AUTH_TOKEN" for consistency with MVI. This change is backwards compatible with the previous "auth_token" variable usage. Updated the code and tests accordingly. **Dependencies**: - Updated Momento dependency in pyproject.toml. **Testing**: - Run the integration tests with a Momento API key. Get one at the [Momento Console](https://console.gomomento.com) for free. MVI is available in AWS us-west-2 with a superuser key. - `MOMENTO_API_KEY=<your key> poetry run pytest tests/integration_tests/vectorstores/test_momento_vector_index.py` **Tag maintainer:** @eyurtsev **Twitter handle**: Please mention @momentohq for this addition to langchain. With the integration of Momento Vector Index, Momento caching, and session store, Momento provides serverless support for the core langchain data needs. Also mention @mlonml for the integration.	9 months ago
Eugene Yurtsev	ca2eed36b7	LangChain cli fix a few bugs (#11573) ... Code was assuming that `git` and `poetry` exist. In addition, it was not ignoring pycache files that get generated during run time	9 months ago
Hugues Chocart	258ae1ba5f	[LLMonitor Callback Handler]: Add error handling (#11563) ... Wraps every callback handler method in error handlers to avoid breaking users' programs when an error occurs inside the handler. Thanks @valdo99 for the suggestion 🙂	9 months ago
Eugene Yurtsev	2aabfafe1e	Module documentation for langchain runnables (#11550) ... Add in code documentation for langchain runnables module.	9 months ago
Eugene Yurtsev	d8fa94e6fa	RunnablePassthrough: In code documentation (#11552) ... Add in code documentation for a runnable passthrough	9 months ago
Eugene Yurtsev	b42f218cfc	RunnableLambda: Add in code docs (#11521) ... Add in code docs for Runnable Lambda	9 months ago
maks-operlejn-ds	f64522fbaf	Reset deanonymizer mapping (#11559) ... @hwchase17 @baskaryan	9 months ago
maks-operlejn-ds	b14b65d62a	Support all presidio entities (#11558) ... https://microsoft.github.io/presidio/supported_entities/ @baskaryan @hwchase17	9 months ago
maks-operlejn-ds	4d62def9ff	Better deanonymizer matching strategy (#11557) ... @baskaryan, @hwchase17	9 months ago
Ash Vardanian	a992b9670d	Fix: Missing DuckDuckGo package version (#11535) ... [The `duckduckgo-search` v3.9.2 was removed from PyPi](https://pypi.org/project/duckduckgo-search/#history). That breaks the build. - **Description:** refreshes the Poetry dependency to v3.9.3 - **Tag maintainer:** @baskaryan - **Twitter handle:** @ashvardanian	9 months ago
Bagatur	8932ed3f07	bump 311 (#11555)	9 months ago
Bagatur	e7a0def1bc	QoL improvements to query constructor (#11504) ... updating query constructor and self query retriever to - make it easier to pass in examples - validate attributes used in query - remove invalid parts of query - make it easier to get + edit prompt - make query constructor a runnable - make self query retriever use as runnable	9 months ago
Taikono-Himazin	eec53fa294	Added autodetect_encoding option to csvLoader (#11327)	9 months ago
Holt Skinner	09c66fe04f	feat: Update Google Document AI Parser (#11413) ... - **Description:** Code Refactoring, Documentation Improvements for Google Document AI PDF Parser - Adds Online (synchronous) processing option. - Adds default field mask to limit payload size. - Skips Human review by default. - **Issue:** Fixes #10589 --------- Co-authored-by: Erick Friis <erick@langchain.dev>	9 months ago
Nuno Campos	628cc4cce8	Rename RunnableMap to RunnableParallel (#11487) ... - keep alias for RunnableMap - update docs to use RunnableParallel and RunnablePassthrough.assign <!-- Thank you for contributing to LangChain! Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes (if applicable), - **Dependencies:** any dependencies required for this change, - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below), - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. -->	9 months ago
Eugene Yurtsev	6a10e8ef31	Add documentation to Runnable (#11516)	9 months ago
William FH	eb572f41a6	Add LangSmith Run Chat Loader (#11458)	9 months ago
David Duong	484947c492	Fetch up-to-date attributes for env-pulled kwargs during serialisation of OpenAI classes (#11499)	9 months ago
Bagatur	5470e730d2	raise openapi import error (#11495)	9 months ago
Erick Friis	29f5f70415	Rename some last hwchase17/langchain links (#11494)	9 months ago
Fabrice Pont	872836c541	feat: add markdown list parser (#11411) ... **Description:** add `MarkdownListOutputParser` as a new `ListOutputParser` **Issue:** #11410	9 months ago
Erick Friis	8f50b616c5	Remove optional from vectara source (#11493) ... fyi @ofermend --------- Co-authored-by: Ofer Mendelevitch <ofer@vectara.com> Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com>	9 months ago
Bagatur	53887242a1	bump 310 (#11486)	9 months ago
Jesús Vélez Santiago	a1c7532298	Add async sql record manager and async indexing API (#10726) ... - **Description:** Add support for a SQLRecordManager in async environments. It includes the creation of `RecorManagerAsync` abstract class. - **Issue:** None - **Dependencies:** Optional `aiosqlite`. - **Tag maintainer:** @nfcampos - **Twitter handle:** @jvelezmagic --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	9 months ago
Qihui Xie	57ade13b2b	fix llm_inputs duplication problem in intermediate_steps in SQLDatabaseChain (#10279) ... Use `.copy()` to fix the bug that the first `llm_inputs` element is overwritten by the second `llm_inputs` element in `intermediate_steps`. ***Problem description:*** In [line 127]( c732d8fffd/libs/experimental/langchain_experimental/sql/base.py (L127C17-L127C17)), the `llm_inputs` of the sql generation step is appended as the first element of `intermediate_steps`: ``` intermediate_steps.append(llm_inputs) # input: sql generation ``` However, `llm_inputs` is a mutable dict, it is updated in [line 179](https://github.com/langchain-ai/langchain/blob/master/libs/experimental/langchain_experimental/sql/base.py#L179) for the final answer step: ``` llm_inputs["input"] = input_text ``` Then, the updated `llm_inputs` is appended as another element of `intermediate_steps` in [line 180](c732d8fffd/libs/experimental/langchain_experimental/sql/base.py (L180)): ``` intermediate_steps.append(llm_inputs) # input: final answer ``` As a result, the final `intermediate_steps` returned in [line 189](c732d8fffd/libs/experimental/langchain_experimental/sql/base.py (L189C43-L189C43)) actually contains two same `llm_inputs` elements, i.e., the `llm_inputs` for the sql generation step overwritten by the one for final answer step by mistake. Users are not able to get the actual `llm_inputs` for the sql generation step from `intermediate_steps` Simply calling `.copy()` when appending `llm_inputs` to `intermediate_steps` can solve this problem.	9 months ago
Florian	d78f418c0d	Extract abstracts from Pubmed articles, even if they have no extra label (#10245) ... ### Description This pull request involves modifications to the extraction method for abstracts/summaries within the PubMed utility. A condition has been added to verify the presence of unlabeled abstracts. Now an abstract will be extracted even if it does not have a subtitle. In addition, the extraction of the abstract was extended to books. ### Issue The PubMed utility occasionally returns an empty result when extracting abstracts from articles, despite the presence of an abstract for the paper on PubMed. This issue arises due to the varying structure of articles; some articles follow a "subtitle/label: text" format, while others do not include subtitles in their abstracts. An example of the latter case can be found at: [https://pubmed.ncbi.nlm.nih.gov/37666905/](url) --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Viktor Zhemchuzhnikov	fd9da60aea	Add async support to SelfQueryRetriever (#10175) ... ### Description SelfQueryRetriever is missing async support, so I am adding it. I also removed deprecated predict_and_parse method usage here, and added some tests. ### Issue N/A ### Tag maintainer Not yet ### Twitter handle N/A	9 months ago
Theron Tau	35297ca0d3	Add feature for extracting images from pdf and recognizing text from images. (#10653) ... **Description** It is for #10423 that it will be a useful feature if we can extract images from pdf and recognize text on them. I have implemented it with `PyPDFLoader`, `PyPDFium2Loader`, `PyPDFDirectoryLoader`, `PyMuPDFLoader`, `PDFMinerLoader`, and `PDFPlumberLoader`. [RapidOCR](https://github.com/RapidAI/RapidOCR.git) is used to recognize text on extracted images. It is time-consuming for ocr so a boolen parameter `extract_images` is set to control whether to extract and recognize. I have tested the time usage for each parser on my own laptop thinkbook 14+ with AMD R7-6800H by unit test and the result is: | extract_images | PyPDFParser | PDFMinerParser | PyMuPDFParser | PyPDFium2Parser | PDFPlumberParser | | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | | False | 0.27s | 0.39s | 0.06s | 0.08s | 1.01s | | True | 17.01s | 20.67s | 20.32s | 19,75s | 20.55s | **Issue** #10423 **Dependencies** rapidocr_onnxruntime in [RapidOCR](https://github.com/RapidAI/RapidOCR/tree/main) --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Bagatur	8e3fbc97ca	Add vowpal_wabbit RL chain (#11462)	9 months ago
Haris Wang	f1269830a0	Fix bug in MarkdownHeaderTextSplitter for codeblock (#10262) ... - Description: The previous version of the MarkdownHeaderTextSplitter did not take into account the possibility of '#' appearing within code blocks, which caused segmentation anomalies in these situations. This PR has fixed this issue. - Issue: - Dependencies: No - Tag maintainer: - Twitter handle: cc @baskaryan @eyurtsev @rlancemartin --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Eddie Cohen	656d2303f7	add in, nin for pinecone (#10303) ... Description: Adds the in and nin comparators for pinecone seen [here](https://docs.pinecone.io/docs/metadata-filtering#metadata-query-language) --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Bagatur	a3a2ce623e	Revise vowpal_wabbit notebook	9 months ago
Bagatur	8fafa1af91	merge	9 months ago
olgavrou	3b07c0cf3d	RL Chain with VowpalWabbit (#10242) ... - Description: This PR adds a new chain `rl_chain.PickBest` for learned prompt variable injection, detailed description and usage can be found in the example notebook added. It essentially adds a [VowpalWabbit](https://github.com/VowpalWabbit/vowpal_wabbit) layer before the llm call in order to learn or personalize prompt variable selections. Most of the code is to make the API simple and provide lots of defaults and data wrangling that is needed to use Vowpal Wabbit, so that the user of the chain doesn't have to worry about it. - Dependencies: [vowpal-wabbit-next](https://pypi.org/project/vowpal-wabbit-next/), - sentence-transformers (already a dep) - numpy (already a dep) - tagging @ataymano who contributed to this chain - Tag maintainer: @baskaryan - Twitter handle: @olgavrou Added example notebook and unit tests	9 months ago
Manikanta5112	56048b909f	added ContentFormatter escape special characters for message content (#10319) ... --------- Co-authored-by: Manikanta5112 <42089393+mani5112@users.noreply.github.com>	9 months ago
Leonid Ganeline	d17416ec79	docstrings `callbacks` (#11456) ... Added missed docstrings to the `callbacks/` --------- Co-authored-by: Erick Friis <erick@langchain.dev>	9 months ago
Ofer Mendelevitch	3c7653bf0f	"source" argument in constructor of Vectara (#11454) ... Replace this entire comment with: - **Description:** minor update to constructor to allow for specification of "source" - **Tag maintainer:** @baskaryan - **Twitter handle:** @ofermend	9 months ago
Eugene Yurtsev	d9018ae5f1	Improve CLI ux (#11452) ... Improve UX for cli	9 months ago
Jaikanth J	9f85f7c543	fix(cache): use dumps for RedisCache (#10408) ... # Description Attempts to fix RedisCache for ChatGenerations using `loads` and `dumps` used in SQLAlchemy cache by @hwchase17 . this is better than pickle dump, because this won't execute any arbitrary code during de-serialisation. # Issues #7722 & #8666 # Dependencies None, but removes the warning introduced in #8041 by @baskaryan Handle: @jaikanthjay46	9 months ago
rodrigo-clickup	5944c1851b	Add ClickUp Toolkit (#10662) ... - **Description:** Adds a toolkit to interact with the [ClickUp](https://clickup.com/) [Public API](https://clickup.com/api/) - **Dependencies:** None - **Tag maintainer:** @rodrigo-georgian, @rodrigo-clickup, @aiswaryasankarwork - **Twitter handle:** - Aiswarya (https://twitter.com/Aiswarya_Sankar, https://www.linkedin.com/in/sankaraiswarya/) - Rodrigo (https://www.linkedin.com/in/rodrigo-ceballos-lentini/) --------- Co-authored-by: Aiswarya Sankar <aiswaryasankar@Aiswaryas-MacBook-Pro.local> Co-authored-by: aiswaryasankarwork <143119412+aiswaryasankarwork@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
John Reynolds	68901e1e40	Update output_parser.py (#10430) ... - Description: Updated output parser for mrkl to remove any hallucination actions after the final answer; this was encountered when using Anthropic claude v2 for planning; reopening PR with updated unit tests - Issue: #10278 - Dependencies: N/A - Twitter handle: @johnreynolds	9 months ago
Joshua Sundance Bailey	790010703b	ArcGISLoader: Limit number of results in query (#10615) ... Description: this PR changes the `ArcGISLoader` to set `return_all_records` to `False` when `result_record_count` is provided as a keyword argument. Previously, `return_all_records` was `True` by default and this made the API ignore `result_record_count`. Issue: `ArcGISLoader` would ignore `result_record_count` unless user also passed `return_all_records=False`.	9 months ago
mrbean	9903a70379	Add youdotcom retriever (#11304) ... --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
ashish-dahal	1655ff2ded	Fix PyMuPDFLoader kwargs (#11434) ... - **Description:** Fix the `PyMuPDFLoader` to accept `loader_kwargs` from the document loader's `loader_kwargs` option. This provides more flexibility in formatting the output from documents. - **Issue:** The `loader_kwargs` is not passed into the `load` method from the document loader, which limits configuration options. - **Dependencies:** None --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Leonid Kuligin	e4a46747dc	integration test for DocAI parser (#11424) ... - **Description:** added an integration test - **Issue:** #11407 @baskaryan	9 months ago
Aashish Saini	2abbdc6ecb	Update bageldb.py (#11421) ... I have restructured the code to ensure uniform handling of ImportError. In place of previously used ValueError, I've adopted the standard practice of raising ImportError with explanatory messages. This modification enhances code readability and clarifies that any problems stem from module importation.	9 months ago
maks-operlejn-ds	2aae1102b0	Instance anonymization (#10501) ... ### Description Add instance anonymization - if `John Doe` will appear twice in the text, it will be treated as the same entity. The difference between `PresidioAnonymizer` and `PresidioReversibleAnonymizer` is that only the second one has a built-in memory, so it will remember anonymization mapping for multiple texts: ``` >>> anonymizer = PresidioAnonymizer() >>> anonymizer.anonymize("My name is John Doe. Hi John Doe!") 'My name is Noah Rhodes. Hi Noah Rhodes!' >>> anonymizer.anonymize("My name is John Doe. Hi John Doe!") 'My name is Brett Russell. Hi Brett Russell!' ``` ``` >>> anonymizer = PresidioReversibleAnonymizer() >>> anonymizer.anonymize("My name is John Doe. Hi John Doe!") 'My name is Noah Rhodes. Hi Noah Rhodes!' >>> anonymizer.anonymize("My name is John Doe. Hi John Doe!") 'My name is Noah Rhodes. Hi Noah Rhodes!' ``` ### Twitter handle @deepsense_ai / @MaksOpp ### Tag maintainer @baskaryan @hwchase17 @hinthornw --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Kyle Pancamo	203258b4d6	Update pdf.py comment for PyPDFLoader (#10495) ... PyPDF does not chunk at the character level to my understanding. Description: PyPDF does not chunk at the character level, but instead breaks up content by page. Fixup comment --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Juan Daza	4236ae3851	Added Streaming Capability to SageMaker LLMs (#10535) ... This PR adds the ability to declare a Streaming response in the SageMaker LLM by leveraging the `invoke_endpoint_with_response_stream` capability in `boto3`. It is heavily based on the AWS Blog Post announcement linked [here](https://aws.amazon.com/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/). It does not add any additional dependencies since it uses the existing `boto3` version. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Laurentiu Piciu	d9670a5945	openai_functions_multi_agent: solved the case when the "arguments" is valid JSON but it does not contain `actions` key (#10543) ... Description: There are cases when the output from the LLM comes fine (i.e. function_call["arguments"] is a valid JSON object), but it does not contain the key "actions". So I split the validation in 2 steps: loading arguments as JSON and then checking for "actions" in it. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
Eugene Yurtsev	fcccde406d	Add SymbolicMathChain to experiment in preparation for deprecation (#11129) ... Move symbolic math chain to experimental	9 months ago
Holt Skinner	9f73fec057	fix: Update Google Cloud Enterprise Search to Vertex AI Search (#10513) ... - Description: Google Cloud Enterprise Search was renamed to Vertex AI Search - https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-search-and-conversation-is-now-generally-available - This PR updates the documentation and Retriever class to use the new terminology. - Changed retriever class from `GoogleCloudEnterpriseSearchRetriever` to `GoogleVertexAISearchRetriever` - Updated documentation to specify that `extractive_segments` requires the new [Enterprise edition](https://cloud.google.com/generative-ai-app-builder/docs/about-advanced-features#enterprise-features) to be enabled. - Fixed spelling errors in documentation. - Change parameter for Retriever from `search_engine_id` to `data_store_id` - When this retriever was originally implemented, there was no distinction between a data store and search engine, but now these have been split. - Fixed an issue blocking some users where the api_endpoint can't be set	9 months ago
Patrick Randell	1d678f805f	Additional Weaviate Filter Comparators (#10522) ... ### Description When using Weaviate Self-Retrievers, certain common filter comparators generated by user queries were unimplemented, resulting in errors. This PR implements some of them. All linting and format commands have been run and tests passed. ### Issue #10474 ### Dependencies timestamp module --------- Co-authored-by: Patrick Randell <prandell@deloitte.com.au>	9 months ago
Nuno Campos	79011f835f	Remove str() from RunnableConfigurableAlternatives (#11446)	9 months ago
Harrison Chase	31d5bd84d7	make vectorstores optional (#11393)	9 months ago
Eugene Yurtsev	8aa545901a	Update agent type docs (#11137) ... In code docs for agent types	9 months ago
Eugene Yurtsev	3e31d6e35f	Start deprecation of LLMBashChain (#11300) ... In preparation for migration LLMBashChain and related tools add a derprecation warning to the code.	9 months ago
Bagatur	8b6b8bf68c	bump 309 (#11443)	9 months ago
billytrend-cohere	2ff91a46c0	Add cohere /chat integration (#11389) ... Add cohere /chat integration and an iPython notebook to demonstrate the addition. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	9 months ago
adrienohana	ca346011b7	added interactive login for azure cognitive search vector store (#11360) ... **Description:** Previously if the access to Azure Cognitive Search was not done via an API key, the default credential was called which doesn't allow to use an interactive login. I simply added the option to use "INTERACTIVE" as a key name, and this will launch a login window upon initialization of the AzureSearch object.	9 months ago