151 Commits (0b33abc8b1fdd0eb269b776f6390d6f59d1f1432)

Author	SHA1	Message	Date
Nejc Habjan	b4fa847a90	community[minor]: add exclude parameter to DirectoryLoader (#17316) ... - **Description:** adds an `exclude` parameter to the DirectoryLoader class, based on similar behavior in GenericLoader - **Issue:** discussed in https://github.com/langchain-ai/langchain/discussions/9059 and I think in some other issues that I cannot find at the moment 🙇 - **Dependencies:** None - **Twitter handle:** don't have one sorry! Just https://github/nejch --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	7 months ago
Bagatur	8f14234afb	infra: ignore flakey lua test (#17618)	7 months ago
morgana	9d7ca7df6e	community[patch]: update copy of metadata in rockset vectorstore integration (#17612) ... - **Description:** This fixes an issue with working with RecordManager. RecordManager was generating new hashes on documents because `add_texts` was modifying the metadata directly. Additionally moved some tests to unit tests since that was a more appropriate home. - **Issue:** N/A - **Dependencies:** N/A - **Twitter handle:** `@_morgan_adams_`	7 months ago
Moshe Berchansky	20a56fe0a2	community[minor]: Add QuantizedEmbedders (#17391) ... **Description:** * adding Quantized embedders using optimum-intel and intel-extension-for-pytorch. * added mdx documentation and example notebooks * added embedding import testing. **Dependencies:** optimum = {extras = ["neural-compressor"], version = "^1.14.0", optional = true} intel_extension_for_pytorch = {version = "^2.2.0", optional = true} Dependencies have been added to pyproject.toml for the community lib. **Twitter handle:** @peter_izsak --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
Amir Karbasi	bccc9241ea	community[patch]: Resolve KuzuQAChain API Changes (#16885) ... - **Description:** Updates to the Kuzu API had broken this functionality. These updates resolve those issues and add a new test to demonstrate the updates. - **Issue:** #11874 - **Dependencies:** No new dependencies - **Twitter handle:** @amirk08 Test results: ``` tests/integration_tests/graphs/test_kuzu.py::TestKuzu::test_query_no_params PASSED [ 33%] tests/integration_tests/graphs/test_kuzu.py::TestKuzu::test_query_params PASSED [ 66%] tests/integration_tests/graphs/test_kuzu.py::TestKuzu::test_refresh_schema PASSED [100%] =================================================== slowest 5 durations =================================================== 0.53s call tests/integration_tests/graphs/test_kuzu.py::TestKuzu::test_refresh_schema 0.34s call tests/integration_tests/graphs/test_kuzu.py::TestKuzu::test_query_no_params 0.28s call tests/integration_tests/graphs/test_kuzu.py::TestKuzu::test_query_params 0.03s teardown tests/integration_tests/graphs/test_kuzu.py::TestKuzu::test_refresh_schema 0.02s teardown tests/integration_tests/graphs/test_kuzu.py::TestKuzu::test_query_params ==================================================== 3 passed in 1.27s ==================================================== ```	7 months ago
Christophe Bornet	789cd5198d	community[patch]: Use astrapy built-in pagination prefetch in AstraDBLoader (#17569)	7 months ago
Christophe Bornet	ff1f985a2a	community: Fix some mypy types in cassandra doc loader (#17570) ... Thank you!	7 months ago
nvpranak	91bcc9c5c9	community[minor]: Nemo embeddings(#16206) ... This PR is adding support for NVIDIA NeMo embeddings issue #16095. --------- Co-authored-by: Praveen Nakshatrala <pnakshatrala@gmail.com> Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
wulixuan	c776cfc599	community[minor]: integrate with model Yuan2.0 (#15411) ... 1. integrate with [`Yuan2.0`](https://github.com/IEIT-Yuan/Yuan-2.0/blob/main/README-EN.md) 2. update `langchain.llms` 3. add a new doc for [Yuan2.0 integration](docs/docs/integrations/llms/yuan2.ipynb) --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
volodymyr-memsql	e36bc379f2	community[patch]: Add vector index support to SingleStoreDB VectorStore (#17308) ... This pull request introduces support for various Approximate Nearest Neighbor (ANN) vector index algorithms in the VectorStore class, starting from version 8.5 of SingleStore DB. Leveraging this enhancement enables users to harness the power of vector indexing, significantly boosting search speed, particularly when handling large sets of vectors. --------- Co-authored-by: Volodymyr Tkachuk <vtkachuk-ua@singlestore.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
Kate Silverstein	0bc4a9b3fc	community[minor]: Adds Llamafile as an LLM (#17431) ... * **Description:** Adds a simple LLM implementation for interacting with [llamafile](https://github.com/Mozilla-Ocho/llamafile)-based models. * **Dependencies:** N/A * **Issue:** N/A **Detail** [llamafile](https://github.com/Mozilla-Ocho/llamafile) lets you run LLMs locally from a single file on most computers without installing any dependencies. To use the llamafile LLM implementation, the user needs to: 1. Download a llamafile e.g. https://huggingface.co/jartine/TinyLlama-1.1B-Chat-v1.0-GGUF/resolve/main/TinyLlama-1.1B-Chat-v1.0.Q5_K_M.llamafile?download=true 2. Make the file executable. 3. Run the llamafile in 'server mode'. (All llamafiles come packaged with a lightweight server; by default, the server listens at `http://localhost:8080`.) ```bash wget https://url/of/model.llamafile chmod +x model.llamafile ./model.llamafile --server --nobrowser ``` Now, the user can invoke the LLM via the LangChain client: ```python from langchain_community.llms.llamafile import Llamafile llm = Llamafile() llm.invoke("Tell me a joke.") ```	7 months ago
Qihui Xie	5738143d4b	add mongodb_store (#13801) ... # Add MongoDB storage - **Description:** Add MongoDB Storage as an option for large doc store. Example usage: ```Python # Instantiate the MongodbStore with a MongoDB connection from langchain.storage import MongodbStore mongo_conn_str = "mongodb://localhost:27017/" mongodb_store = MongodbStore(mongo_conn_str, db_name="test-db", collection_name="test-collection") # Set values for keys doc1 = Document(page_content='test1') doc2 = Document(page_content='test2') mongodb_store.mset([("key1", doc1), ("key2", doc2)]) # Get values for keys values = mongodb_store.mget(["key1", "key2"]) # [doc1, doc2] # Iterate over keys for key in mongodb_store.yield_keys(): print(key) # Delete keys mongodb_store.mdelete(["key1", "key2"]) ``` - **Dependencies:** Use `mongomock` for integration test. --------- Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	7 months ago
wulixuan	5d06797905	community[minor]: integrate chat models with Yuan2.0 (#16575) ... 1. integrate chat models with [`Yuan2.0`](https://github.com/IEIT-Yuan/Yuan-2.0/blob/main/README-EN.md) 2. add a new doc for [Yuan2.0 integration](docs/docs/integrations/llms/yuan2.ipynb) Yuan2.0 is a new generation Fundamental Large Language Model developed by IEIT System. We have published all three models, Yuan 2.0-102B, Yuan 2.0-51B, and Yuan 2.0-2B. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
Ian Gregory	e5472b5eb8	Framework for supporting more languages in LanguageParser (#13318) ... ## Description I am submitting this for a school project as part of a team of 5. Other team members are @LeilaChr, @maazh10, @Megabear137, @jelalalamy. This PR also has contributions from community members @Harrolee and @Mario928. Initial context is in the issue we opened (#11229). This pull request adds: - Generic framework for expanding the languages that `LanguageParser` can handle, using the [tree-sitter](https://github.com/tree-sitter/py-tree-sitter#py-tree-sitter) parsing library and existing language-specific parsers written for it - Support for the following additional languages in `LanguageParser`: - C - C++ - C# - Go - Java (contributed by @Mario928 https://github.com/ThatsJustCheesy/langchain/pull/2) - Kotlin - Lua - Perl - Ruby - Rust - Scala - TypeScript (contributed by @Harrolee https://github.com/ThatsJustCheesy/langchain/pull/1) Here is the [design document](https://docs.google.com/document/d/17dB14cKCWAaiTeSeBtxHpoVPGKrsPye8W0o_WClz2kk) if curious, but no need to read it. ## Issues - Closes #11229 - Closes #10996 - Closes #8405 ## Dependencies `tree_sitter` and `tree_sitter_languages` on PyPI. We have tried to add these as optional dependencies. ## Documentation We have updated the list of supported languages, and also added a section to `source_code.ipynb` detailing how to add support for additional languages using our framework. ## Maintainer - @hwchase17 (previously reviewed https://github.com/langchain-ai/langchain/pull/6486) Thanks!! ## Git commits We will gladly squash any/all of our commits (esp merge commits) if necessary. Let us know if this is desirable, or if you will be squash-merging anyway. <!-- Thank you for contributing to LangChain! Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes (if applicable), - **Dependencies:** any dependencies required for this change, - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below), - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --> --------- Co-authored-by: Maaz Hashmi <mhashmi373@gmail.com> Co-authored-by: LeilaChr <87657694+LeilaChr@users.noreply.github.com> Co-authored-by: Jeremy La <jeremylai511@gmail.com> Co-authored-by: Megabear137 <zubair.alnoor27@gmail.com> Co-authored-by: Lee Harrold <lhharrold@sep.com> Co-authored-by: Mario928 <88029051+Mario928@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	7 months ago
Sridhar Ramaswamy	9f1cbbc6ed	community[minor]: Add pebblo safe document loader (#16862) ... - **Description:** Pebblo opensource project enables developers to safely load data to their Gen AI apps. It identifies semantic topics and entities found in the loaded data and summarizes them in a developer-friendly report. - **Dependencies:** none - **Twitter handle:** srics @hwchase17	7 months ago
mhavey	1bbb64d956	community[minor], langchian[minor]: Add Neptune Rdf graph and chain (#16650) ... **Description**: This PR adds a chain for Amazon Neptune graph database RDF format. It complements the existing Neptune Cypher chain. The PR also includes a Neptune RDF graph class to connect to, introspect, and query a Neptune RDF graph database from the chain. A sample notebook is provided under docs that demonstrates the overall effect: invoking the chain to make natural language queries against Neptune using an LLM. **Issue**: This is a new feature **Dependencies**: The RDF graph class depends on the AWS boto3 library if using IAM authentication to connect to the Neptune database. --------- Co-authored-by: Piyush Jain <piyushjain@duck.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
Andreas Motl	1fdd9bd980	community/SQLDatabase: Generalize and trim software tests (#16659) ... - **Description:** Improve test cases for `SQLDatabase` adapter component, see [suggestion](https://github.com/langchain-ai/langchain/pull/16655#pullrequestreview-1846749474). - **Depends on:** GH-16655 - **Addressed to:** @baskaryan, @cbornet, @eyurtsev _Remark: This PR is stacked upon GH-16655, so that one will need to go in first._ Edit: Thank you for bringing in GH-17191, @eyurtsev. This is a little aftermath, improving/streamlining the corresponding test cases.	7 months ago
Chris	f9f5626ca4	community[patch]: Fix github search issues and PRs PaginatedList has no len() error (#16806) ... **Description:** Bugfix: Langchain_community's GitHub Api wrapper throws a TypeError when searching for issues and/or PRs (the `search_issues_and_prs` method). This is because PyGithub's PageinatedList type does not support the len() method. See https://github.com/PyGithub/PyGithub/issues/1476  **Dependencies:** None **Twitter handle**: @ChrisKeoghNZ I haven't registered an issue as it would take me longer to fill the template out than to make the fix, but I'm happy to if that's deemed essential. I've added a simple integration test to cover this as there were no existing unit tests and it was going to be tricky to set them up. Co-authored-by: Chris Keogh <chris.keogh@xero.com>	7 months ago
morgana	722aae4fd1	community: add delete method to rocksetdb vectorstore to support recordmanager (#17030) ... - **Description:** This adds a delete method so that rocksetdb can be used with `RecordManager`. - **Issue:** N/A - **Dependencies:** N/A - **Twitter handle:** `@_morgan_adams_` --------- Co-authored-by: Rockset API Bot <admin@rockset.io>	7 months ago
yin1991	37ef6ac113	community[patch]: Add Pagination to GitHubIssuesLoader for Efficient GitHub Issues Retrieval (#16934) ... - **Description:** Add Pagination to GitHubIssuesLoader for Efficient GitHub Issues Retrieval - **Issue:** [the issue # it fixes if applicable,](https://github.com/langchain-ai/langchain/issues/16864) --------- Co-authored-by: root <root@ip-172-31-46-160.ap-southeast-1.compute.internal> Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
Spencer Kelly	54fa78c887	community[patch]: fixed vector similarity filtering (#16967) ... **Description:** changed filtering so that failed filter doesn't add document to results. Currently filtering is entirely broken and all documents are returned whether or not they pass the filter. fixes issue introduced in https://github.com/langchain-ai/langchain/pull/16190	7 months ago
Abhijeeth Padarthi	584b647b96	community[minor]: AWS Athena Document Loader (#15625) ... - **Description:** Adds the document loader for [AWS Athena](https://aws.amazon.com/athena/), a serverless and interactive analytics service. - **Dependencies:** Added boto3 as a dependency	7 months ago
david-tempelmann	93da18b667	community[minor]: Add mmr and similarity_score_threshold retrieval to DatabricksVectorSearch (#16829) ... - **Description:** This PR adds support for `search_types="mmr"` and `search_type="similarity_score_threshold"` to retrievers using `DatabricksVectorSearch`, - **Issue:** - **Dependencies:** - **Twitter handle:** --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
Erick Friis	3a2eb6e12b	infra: add print rule to ruff (#16221) ... Added noqa for existing prints. Can slowly remove / will prevent more being intro'd	8 months ago
Quang Hoa	54c1fb3f25	community[patch]: Make some functions work with Milvus (#10695) ... **Description** Make some functions work with Milvus: 1. get_ids: Get primary keys by field in the metadata 2. delete: Delete one or more entities by ids 3. upsert: Update/Insert one or more entities **Issue** None **Dependencies** None **Tag maintainer:** @hwchase17 **Twitter handle:** None --------- Co-authored-by: HoaNQ9 <hoanq.1811@gmail.com> Co-authored-by: Erick Friis <erick@langchain.dev>	8 months ago
kYLe	c9999557bf	community[patch]: Modify LLMs/Anyscale work with OpenAI API v1 (#14206) ... <!-- Thank you for contributing to LangChain! Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes (if applicable), - **Dependencies:** any dependencies required for this change, - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below), - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --> - **Description:** 1. Modify LLMs/Anyscale to work with OAI v1 2. Get rid of openai_ prefixed variables in Chat_model/ChatAnyscale 3. Modify `anyscale_api_base` to `anyscale_base_url` to follow OAI name convention (reverted) --------- Co-authored-by: Erick Friis <erick@langchain.dev>	8 months ago
Bagatur	65e97c9b53	infra: mv SQLDatabase tests to community (#17276)	8 months ago
Armin Stepanyan	641efcf41c	community: add runtime kwargs to HuggingFacePipeline (#17005) ... This PR enables changing the behaviour of huggingface pipeline between different calls. For example, before this PR there's no way of changing maximum generation length between different invocations of the chain. This is desirable in cases, such as when we want to scale the maximum output size depending on a dynamic prompt size. Usage example: ```python from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline model_id = "gpt2" tokenizer = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained(model_id) pipe = pipeline("text-generation", model=model, tokenizer=tokenizer) hf = HuggingFacePipeline(pipeline=pipe) hf("Say foo:", pipeline_kwargs={"max_new_tokens": 42}) ``` --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	8 months ago
Scott Nath	a32798abd7	community: Add you.com utility, update you retriever integration docs (#17014) ... <!-- Thank you for contributing to LangChain! Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified. Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes if applicable, - **Dependencies:** any dependencies required for this change, - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --> - **Description: changes to you.com files** - general cleanup - adds community/utilities/you.py, moving bulk of code from retriever -> utility - removes `snippet` as endpoint - adds `news` as endpoint - adds more tests <s>**Description: update community MAKE file** - adds `integration_tests` - adds `coverage`</s> - **Issue:** the issue # it fixes if applicable, - [For New Contributors: Update Integration Documentation](https://github.com/langchain-ai/langchain/issues/15664#issuecomment-1920099868) - **Dependencies:** n/a - **Twitter handle:** @scottnath - **Mastodon handle:** scottnath@mastodon.social --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	8 months ago
Liang Zhang	7306600e2f	community[patch]: Support SerDe transform functions in Databricks LLM (#16752) ... **Description:** Databricks LLM does not support SerDe the transform_input_fn and transform_output_fn. After saving and loading, the LLM will be broken. This PR serialize these functions into a hex string using pickle, and saving the hex string in the yaml file. Using pickle to serialize a function can be flaky, but this is a simple workaround that unblocks many use cases. If more sophisticated SerDe is needed, we can improve it later. Test: Added a simple unit test. I did manual test on Databricks and it works well. The saved yaml looks like: ``` llm: _type: databricks cluster_driver_port: null cluster_id: null databricks_uri: databricks endpoint_name: databricks-mixtral-8x7b-instruct extra_params: {} host: e2-dogfood.staging.cloud.databricks.com max_tokens: null model_kwargs: null n: 1 stop: null task: null temperature: 0.0 transform_input_fn: 80049520000000000000008c085f5f6d61696e5f5f948c0f7472616e73666f726d5f696e7075749493942e transform_output_fn: null ``` @baskaryan ```python from langchain_community.embeddings import DatabricksEmbeddings from langchain_community.llms import Databricks from langchain.chains import RetrievalQA from langchain.document_loaders import TextLoader from langchain.text_splitter import CharacterTextSplitter from langchain.vectorstores import FAISS import mlflow embeddings = DatabricksEmbeddings(endpoint="databricks-bge-large-en") def transform_input(**request): request["messages"] = [ { "role": "user", "content": request["prompt"] } ] del request["prompt"] return request llm = Databricks(endpoint_name="databricks-mixtral-8x7b-instruct", transform_input_fn=transform_input) persist_dir = "faiss_databricks_embedding" # Create the vector db, persist the db to a local fs folder loader = TextLoader("state_of_the_union.txt") documents = loader.load() text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) docs = text_splitter.split_documents(documents) db = FAISS.from_documents(docs, embeddings) db.save_local(persist_dir) def load_retriever(persist_directory): embeddings = DatabricksEmbeddings(endpoint="databricks-bge-large-en") vectorstore = FAISS.load_local(persist_directory, embeddings) return vectorstore.as_retriever() retriever = load_retriever(persist_dir) retrievalQA = RetrievalQA.from_llm(llm=llm, retriever=retriever) with mlflow.start_run() as run: logged_model = mlflow.langchain.log_model( retrievalQA, artifact_path="retrieval_qa", loader_fn=load_retriever, persist_dir=persist_dir, ) # Load the retrievalQA chain loaded_model = mlflow.pyfunc.load_model(logged_model.model_uri) print(loaded_model.predict([{"query": "What did the president say about Ketanji Brown Jackson"}])) ```	8 months ago
Neli Hateva	9bb5157a3d	langchain[patch], community[patch]: Fixes in the Ontotext GraphDB Graph and QA Chain (#17239) ... - **Description:** Fixes in the Ontotext GraphDB Graph and QA Chain related to the error handling in case of invalid SPARQL queries, for which `prepareQuery` doesn't throw an exception, but the server returns 400 and the query is indeed invalid - **Issue:** N/A - **Dependencies:** N/A - **Twitter handle:** @OntotextGraphDB	8 months ago
ByeongUk Choi	b88329e9a5	community[patch]: Implement Unique ID Enforcement in FAISS (#17244) ... **Description:** Implemented unique ID validation in the FAISS component to ensure all document IDs are distinct. This update resolves issues related to non-unique IDs, such as inconsistent behavior during deletion processes.	8 months ago
Luiz Ferreira	34d2daffb3	community[patch]: Fix chat openai unit test (#17124) ... - **Description:** Actually the test named `test_openai_apredict` isn't testing the apredict method from ChatOpenAI. - **Twitter handle:** https://twitter.com/OAlmofadas	8 months ago
Frank	ef082c77b1	community[minor]: add github file loader to load any github file content b… (#15305) ... ### Description support load any github file content based on file extension. Why not use [git loader](https://python.langchain.com/docs/integrations/document_loaders/git#load-existing-repository-from-disk) ? git loader clones the whole repo even only interested part of files, that's too heavy. This GithubFileLoader only downloads that you are interested files. ### Twitter handle my twitter: @shufanhaotop --------- Co-authored-by: Hao Fan <h_fan@apple.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	8 months ago
Ryan Kraus	f027696b5f	community: Added new Utility runnables for NVIDIA Riva. (#15966) ... **Please tag this issue with `nvidia_genai`** - **Description:** Added new Runnables for integration NVIDIA Riva into LCEL chains for Automatic Speech Recognition (ASR) and Text To Speech (TTS). - **Issue:** N/A - **Dependencies:** To use these runnables, the NVIDIA Riva client libraries are required. It they are not installed, an error will be raised instructing how to install them. The Runnables can be safely imported without the riva client libraries. - **Twitter handle:** N/A All of the Riva Runnables are inside a single folder in the Utilities module. In this folder are four files: - common.py - Contains all code that is common to both TTS and ASR - stream.py - Contains a class representing an audio stream that allows the end user to put data into the stream like a queue. - asr.py - Contains the RivaASR runnable - tts.py - Contains the RivaTTS runnable The following Python function is an example of creating a chain that makes use of both of these Runnables: ```python def create( config: Configuration, audio_encoding: RivaAudioEncoding, sample_rate: int, audio_channels: int = 1, ) -> Runnable[ASRInputType, TTSOutputType]: """Create a new instance of the chain.""" _LOGGER.info("Instantiating the chain.") # create the riva asr client riva_asr = RivaASR( url=str(config.riva_asr.service.url), ssl_cert=config.riva_asr.service.ssl_cert, encoding=audio_encoding, audio_channel_count=audio_channels, sample_rate_hertz=sample_rate, profanity_filter=config.riva_asr.profanity_filter, enable_automatic_punctuation=config.riva_asr.enable_automatic_punctuation, language_code=config.riva_asr.language_code, ) # create the prompt template prompt = PromptTemplate.from_template("{user_input}") # model = ChatOpenAI() model = ChatNVIDIA(model="mixtral_8x7b") # type: ignore # create the riva tts client riva_tts = RivaTTS( url=str(config.riva_asr.service.url), ssl_cert=config.riva_asr.service.ssl_cert, output_directory=config.riva_tts.output_directory, language_code=config.riva_tts.language_code, voice_name=config.riva_tts.voice_name, ) # construct and return the chain return {"user_input": riva_asr} | prompt | model | riva_tts # type: ignore ``` The following code is an example of creating a new audio stream for Riva: ```python input_stream = AudioStream(maxsize=1000) # Send bytes into the stream for chunk in audio_chunks: await input_stream.aput(chunk) input_stream.close() ``` The following code is an example of how to execute the chain with RivaASR and RivaTTS ```python output_stream = asyncio.Queue() while not input_stream.complete: async for chunk in chain.astream(input_stream): output_stream.put(chunk) ``` Everything should be async safe and thread safe. Audio data can be put into the input stream while the chain is running without interruptions. --------- Co-authored-by: Hayden Wolff <hwolff@nvidia.com> Co-authored-by: Hayden Wolff <hwolff@Haydens-Laptop.local> Co-authored-by: Hayden Wolff <haydenwolff99@gmail.com> Co-authored-by: Erick Friis <erick@langchain.dev>	8 months ago
Bagatur	6e2ed9671f	infra: fix breebs test lint (#17075)	8 months ago
Alex Boury	334b6ebdf3	community[minor]: Breebs docs retriever (#16578) ... - **Description:** Implementation of breeb retriever with integration tests -> libs/community/tests/integration_tests/retrievers/test_breebs.py and documentation (notebook) -> docs/docs/integrations/retrievers/breebs.ipynb. - **Dependencies:** None	8 months ago
Harrison Chase	4eda647fdd	infra: add -p to mkdir in lint steps (#17013) ... Previously, if this did not find a mypy cache then it wouldnt run this makes it always run adding mypy ignore comments with existing uncaught issues to unblock other prs --------- Co-authored-by: Erick Friis <erick@langchain.dev> Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	8 months ago
Killinsun - Ryota Takeuchi	bcfce146d8	community[patch]: Correct the calling to collection_name in qdrant (#16920) ... ## Description In #16608, the calling `collection_name` was wrong. I made a fix for it. Sorry for the inconvenience! ## Issue https://github.com/langchain-ai/langchain/issues/16962 ## Dependencies N/A <!-- Thank you for contributing to LangChain! Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified. Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes if applicable, - **Dependencies:** any dependencies required for this change, - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --> --------- Co-authored-by: Kumar Shivendu <kshivendu1@gmail.com> Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	8 months ago
Erick Friis	b1a847366c	community: revert SQL Stores (#16912) ... This reverts commit cfc225ecb3. https://github.com/langchain-ai/langchain/pull/15909#issuecomment-1922418097 These will have existed in langchain-community 0.0.16 and 0.0.17.	8 months ago
Christophe Bornet	af8c5c185b	langchain[minor],community[minor]: Add async methods in BaseLoader (#16634) ... Adds: * methods `aload()` and `alazy_load()` to interface `BaseLoader` * implementation for class `MergedDataLoader ` * support for class `BaseLoader` in async function `aindex()` with unit tests Note: this is compatible with existing `aload()` methods that some loaders already had. **Twitter handle:** @cbornet_ --------- Co-authored-by: Eugene Yurtsev <eugene@langchain.dev>	8 months ago
Christophe Bornet	744070ee85	Add async methods for the AstraDB VectorStore (#16391) ... - **Description**: fully async versions are available for astrapy 0.7+. For older astrapy versions or if the user provides a sync client without an async one, the async methods will call the sync ones wrapped in `run_in_executor` - **Twitter handle:** cbornet_	8 months ago
baichuan-assistant	f8f2649f12	community: Add Baichuan LLM to community (#16724) ... Replace this entire comment with: - **Description:** Add Baichuan LLM to integration/llm, also updated related docs. Co-authored-by: BaiChuanHelper <wintergyc@WinterGYCs-MacBook-Pro.local>	8 months ago
thiswillbeyourgithub	1d082359ee	community: add support for callable filters in FAISS (#16190) ... - **Description:** Filtering in a FAISS vectorstores is very inflexible and doesn't allow that many use case. I think supporting callable like this enables a lot: regular expressions, condition on multiple keys etc. **Note** I had to manually alter a test. I don't understand if it was falty to begin with or if there is something funky going on. - **Issue:** None - **Dependencies:** None - **Twitter handle:** None Signed-off-by: thiswillbeyourgithub <26625900+thiswillbeyourgithub@users.noreply.github.com>	8 months ago
Volodymyr Machula	32c5be8b73	community[minor]: Connery Tool and Toolkit (#14506) ... ## Summary This PR implements the "Connery Action Tool" and "Connery Toolkit". Using them, you can integrate Connery actions into your LangChain agents and chains. Connery is an open-source plugin infrastructure for AI. With Connery, you can easily create a custom plugin with a set of actions and seamlessly integrate them into your LangChain agents and chains. Connery will handle the rest: runtime, authorization, secret management, access management, audit logs, and other vital features. Additionally, Connery and our community offer a wide range of ready-to-use open-source plugins for your convenience. Learn more about Connery: - GitHub: https://github.com/connery-io/connery-platform - Documentation: https://docs.connery.io - Twitter: https://twitter.com/connery_io ## TODOs - [x] API wrapper - [x] Integration tests - [x] Connery Action Tool - [x] Docs - [x] Example - [x] Integration tests - [x] Connery Toolkit - [x] Docs - [x] Example - [x] Formatting (`make format`) - [x] Linting (`make lint`) - [x] Testing (`make test`)	8 months ago
Harrison Chase	8457c31c04	community[patch]: activeloop ai tql deprecation (#14634) ... Co-authored-by: AdkSarsen <adilkhan@activeloop.ai>	8 months ago
Neli Hateva	c95facc293	langchain[minor], community[minor]: Implement Ontotext GraphDB QA Chain (#16019) ... - **Description:** Implement Ontotext GraphDB QA Chain - **Issue:** N/A - **Dependencies:** N/A - **Twitter handle:** @OntotextGraphDB	8 months ago
Jael Gu	a1aa3a657c	community[patch]: Milvus supports add & delete texts by ids (#16256) ... # Description To support [langchain indexing](https://python.langchain.com/docs/modules/data_connection/indexing) as requested by users, vectorstore Milvus needs to support: - document addition by id (`add_documents` method with `ids` argument) - delete by id (`delete` method with `ids` argument) Example usage: ```python from langchain.indexes import SQLRecordManager, index from langchain.schema import Document from langchain_community.vectorstores import Milvus from langchain_openai import OpenAIEmbeddings collection_name = "test_index" embedding = OpenAIEmbeddings() vectorstore = Milvus(embedding_function=embedding, collection_name=collection_name) namespace = f"milvus/{collection_name}" record_manager = SQLRecordManager( namespace, db_url="sqlite:///record_manager_cache.sql" ) record_manager.create_schema() doc1 = Document(page_content="kitty", metadata={"source": "kitty.txt"}) doc2 = Document(page_content="doggy", metadata={"source": "doggy.txt"}) index( [doc1, doc1, doc2], record_manager, vectorstore, cleanup="incremental", # None, "incremental", or "full" source_id_key="source", ) ``` # Fix issues Fix https://github.com/milvus-io/milvus/issues/30112 --------- Signed-off-by: Jael Gu <mengjia.gu@zilliz.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	8 months ago
Benito Geordie	f3fdc5c5da	community: Added integrations for ThirdAI's NeuralDB with Retriever and VectorStore frameworks (#15280) ... **Description:** Adds ThirdAI NeuralDB retriever and vectorstore integration. NeuralDB is a CPU-friendly and fine-tunable text retrieval engine.	8 months ago
Christophe Bornet	2e3af04080	Use Postponed Evaluation of Annotations in Astra and Cassandra doc loaders (#16694) ... Minor/cosmetic change	8 months ago