langchain

Commit Graph

Author	SHA1	Message	Date
Harrison Chase	794ff2dae8	Harrison/hf lru (#10154 ) Co-authored-by: Pascal Bro <git@pascalbrokmeier.de> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Stanko Kuveljic	4765c09703	Pinecone upsert parallelization (#9859 ) Issue: closes #9855 * consolidates `from_texts` and `add_texts` functions for pinecone upsert * adds two types of batching (one for embeddings and one for index upsert) * adds thread pool size when instantiating pinecone index	1 year ago
Lorenzo	00a7c31ffd	Fix: Nested Dicts Handling of Document Metadata (#9880 ) ## Description When the `MultiQueryRetriever` is used to get the list of documents relevant according to a query, inside a vector store, and at least one of these contain metadata with nested dictionaries, a `TypeError: unhashable type: 'dict'` exception is thrown. This is caused by the `unique_union` function which, to guarantee the uniqueness of the returned documents, tries, unsuccessfully, to hash the nested dictionaries and use them as a part of key. ```python unique_documents_dict = { (doc.page_content, tuple(sorted(doc.metadata.items()))): doc for doc in documents } ``` ## Issue #9872 (MultiQueryRetriever (get_relevant_documents) raises TypeError: unhashable type: 'dict' with dic metadata) ## Solution A possible solution is to dump the metadata dict to a string and use it as a part of hashed key. ```python unique_documents_dict = { (doc.page_content, json.dumps(doc.metadata, sort_keys=True)): doc for doc in documents } ``` --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Davide Menini	b8baead70c	fix (Html2TextTransformer): allow configuration of html2text (#9914 ) Hi, this PR enables configuring the html2text package, instead of being bound to use the hardcoded values. While simply passing `ignore_links` and `ignore_images` to the `transform_documents` method was possible, I preferred passing them to the `__init__` method for 2 reasons: 1. It is more efficient in case of subsequent calls to `transform_documents`. 2. It allows to move the "complexity" to the instantiation, keeping the actual execution simple and general enough. IMO the transformers should all follow this pattern, allowing something like this: ```python # Instantiate transformers transformers = [ TransformerA(foo='bar'), TransformerB(bar='foo'), # others ] # During execution, call them sequentially documents = ... for tr in transformers: documents = tr.transform_documents(documents) ``` Thanks for the reviews! --------- Co-authored-by: taamedag <Davide.Menini@swisscom.com>	1 year ago
Frédéric Lepied	4dc47bd3ac	time_weighted_retriever: use a timestamp if needed (#9906 ) If last_accessed_at metadata is a float use it as a timestamp. This allows to support vector stores that do not store datetime objects like ChromaDb. Fixes: https://github.com/langchain-ai/langchain/issues/3685 <!-- Thank you for contributing to LangChain! Replace this entire comment with: - Description: a description of the change, - Issue: the issue # it fixes (if applicable), - Dependencies: any dependencies required for this change, - Tag maintainer: for a quicker response, tag the relevant maintainer (see below), - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. These live is docs/extras directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin. -->	1 year ago
Josh White	bc8cceebf7	Extend DynamoDBChatMessageHistory to support composite keys (#9896 ) - Description: Adds two optional parameters to the DynamoDBChatMessageHistory class to enable users to pass in a name for their PrimaryKey, or a Key object itself to enable the use of composite keys, a common DynamoDB paradigm. [AWS DynamoDB Key docs](https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/) - Issue: N/A - Dependencies: N/A - Twitter handle: N/A --------- Co-authored-by: Josh White <josh@ctrlstack.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Programmers Emperor	872d829201	Update __init__.py (#9955 ) Add SQLDatabaseSequentialChain Class to __init__.py so it can be accessed and used <!-- Thank you for contributing to LangChain! Replace this entire comment with: - Description: SQLDatabaseSequentialChain is not found when importing Langchain_experimental package, when I open __init__.py Langchain_expermental.sql, I found that SQLDatabaseSequentialChain is imported and add to __all__ list - Issue: SQLDatabaseSequentialChain is not found in Langchain_experimental package - Dependencies: None, - Tag maintainer: None, - Twitter handle: None, Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. These live is docs/extras directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin. -->	1 year ago
Lucas Rodrigues Pereira	5c7afe8aae	Fix json parsing error of MULTI_PROMPT_ROUTER_TEMPLATE (#9944 ) The output at times lacks the closing markdown code block. The prompt is changed to explicitly request the closing backticks. <!-- Thank you for contributing to LangChain! Replace this entire comment with: - Description: a description of the change, - Issue: the issue # it fixes (if applicable), - Dependencies: any dependencies required for this change, - Tag maintainer: for a quicker response, tag the relevant maintainer (see below), - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. These live is docs/extras directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin. -->	1 year ago
Lance Martin	387813bfb2	Sort by most recent chatIDs (#9946 ) When we `lazy_load` iMessage chats, return chats w/ most recent msg first (matches what is visualized in app).	1 year ago
German Martin	cf5a50469f	TextGen is missing async methods. (#9986 ) Adding _acall and _astream method that were missing. Preventing streaming during async executions. @rlancemartin.	1 year ago
Blake (Yung Cher Ho)	f4bed8a04c	Takeoff baseurl support (#10091 ) ## Description This PR introduces a minor change to the TitanTakeoff integration. Instead of specifying a port on localhost, this PR will allow users to specify a baseURL instead. This will allow users to use the integration if they have TitanTakeoff deployed externally (not on localhost). This removes the hardcoded reference to localhost "http://localhost:{port}". ### Info about Titan Takeoff Titan Takeoff is an inference server created by [TitanML](https://www.titanml.co/) that allows you to deploy large language models locally on your hardware in a single command. Most generative model architectures are included, such as Falcon, Llama 2, GPT2, T5 and many more. Read more about Titan Takeoff here: - [Blog](https://medium.com/@TitanML/introducing-titan-takeoff-6c30e55a8e1e) - [Docs](https://docs.titanml.co/docs/titan-takeoff/getting-started) ### Dependencies No new dependencies are introduced. However, users will need to install the titan-iris package in their local environment and start the Titan Takeoff inferencing server in order to use the Titan Takeoff integration. Thanks for your help and please let me know if you have any questions. cc: @hwchase17 @baskaryan --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	1 year ago
Eddie Cohen	565c021730	Add ne comparator (#10006 ) Description: Adds the not comparator and operator to pinecone, chroma and deeplake. Issue: Not a registered issue but when using a selfqueryretriever with pinecone I got this error + stacktrace when I entered a query that asked to not include specific data: > raised following `error:` > Received unrecognized function ne. Valid functions are [<Operator.AND: 'and'>, <Operator.OR: 'or'>, <Operator.NOT: 'not'>, <Comparator.EQ: 'eq'>, <Comparator.GT: 'gt'>, <Comparator.GTE: 'gte'>, <Comparator.LT: 'lt'>, <Comparator.LTE: 'lte'>] I noticed that chroma and deeplake also support not equals/not filtering so I added it there as well [pinecone](https://docs.pinecone.io/docs/metadata-filtering#metadata-query-language) [chroma](https://docs.trychroma.com/usage-guide#filtering-by-metadata) [deeplake](https://docs.activeloop.ai/enterprise-features/compute-engine/querying-datasets/query-syntax#and-or-not)	1 year ago
Leonid Ganeline	2221194450	`Yahoo Finance News` tool (#10014 ) Added: - the `Yahoo Finance News` tool - Ut-s - An example	1 year ago
Lars von Wedel	6d82503eb1	Add parser and loader for Azure document intelligence service. (#10136 ) Hi, this PR contains loader / parser for Azure Document intelligence which is a ML-based service to ingest arbitrary PDFs / images, even if scanned. The loader generates Documents by pages of the original document. This is my first contribution to LangChain. Unfortunately I could not find the correct place for test cases. Happy to add one if you can point me to the location, but as this is a cloud-based service, a test would require network access and credentials - so might be of limited help. Dependencies: The needed dependency was already part of pyproject.toml, no change. Twitter: feel free to mention @LarsAC on the announcement	1 year ago
Harrison Chase	4abe85be57	Harrison/string inplace (#10153 ) Co-authored-by: Wrick Talukdar <wrick.talukdar@gmail.com> Co-authored-by: Anjan Biswas <anjanavb@amazon.com> Co-authored-by: Jha <nikjha@amazon.com> Co-authored-by: Lucky-Lance <77819606+Lucky-Lance@users.noreply.github.com> Co-authored-by: 陆徐东 <luxudong@MacBook-Pro.local>	1 year ago
Harrison Chase	f5af756397	fake messages list model (#10152 ) create a fake chat model that you can configure with list of messages	1 year ago
Harrison Chase	9e6cc7b236	make hub push public by default (#10138 )	1 year ago
Bagatur	0e4c5dd176	bump 13 (#10130 )	1 year ago
Bagatur	42582adb66	bump 280 (#10117 )	1 year ago
Bagatur	9e196cb470	rm sqlite3 import (#10115 )	1 year ago
Arpan Pokharel	f8bca156d4	Add where filter in weaviate similarity search with score (#9978 ) - Description: Add where filter in weaviate similarity search with score - Issue: #9853 - Dependencies: - - Tag maintainer: - - Twitter handle: -	1 year ago
Leonid Kuligin	30239b3025	added support for inference from Model Garden (#9367 ) #8850 --------- Co-authored-by: Leonid Kuligin <kuligin@google.com>	1 year ago
Benjamin Matson	58d7d86e51	feat: add bedrock chat model (#8017 ) Replace this comment with: - Description: Add Bedrock implementation of Anthropic Claude for Chat - Tag maintainer: @hwchase17, @baskaryan - Twitter handle: @bwmatson --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Massimiliano Pronesti	a7c9bd30d4	feat(llms): add missing params to huggingface text-generation (#9724 ) This small PR aims at supporting the following missing parameters in the `HuggingfaceTextGen` LLM: - `return_full_text` - sometimes useful for completion tasks - `do_sample` - quite handy to control the randomness of the model. - `watermark` @hwchase17 @baskaryan --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
KyrianC	491089754d	EdenAI LLM update. Add models name option (#8963 ) This PR follows the Eden AI (LLM + embeddings) integration. #8633 We added an optional parameter to choose different AI models for providers (like 'text-bison' for provider 'google', 'text-davinci-003' for provider 'openai', etc.). Usage: ```python llm = EdenAI( feature="text", provider="google", params={ "model": "text-bison", # new "temperature": 0.2, "max_tokens": 250, }, ) ``` You can also change the provider + model after initialization ```python llm = EdenAI( feature="text", provider="google", params={ "temperature": 0.2, "max_tokens": 250, }, ) prompt = """ hi """ llm(prompt, providers='openai', model='text-davinci-003') # change provider & model ``` The jupyter notebook as been updated with an example well. Ping: @hwchase17, @baskaryan --------- Co-authored-by: RedhaWassim <rwasssim@gmail.com> Co-authored-by: sam <melaine.samy@gmail.com>	1 year ago
maks-operlejn-ds	b5a74fb973	Temporarily remove language selection (#10097 ) Adapting Microsoft Presidio to other languages requires a bit more work, so for now it will be good idea to remove the language option to choose, so as not to cause errors and confusion. https://microsoft.github.io/presidio/analyzer/languages/ I will handle different languages after the weekend 😄	1 year ago
Bagatur	71c418725f	index rename delete_mode -> cleanup (#10103 )	1 year ago
Nuno Campos	427f696fb0	Nc/runnables seqmap tags (#9753 )	1 year ago
Harrison Chase	d7bf7dc412	add repr for not serializable (#10071 ) Co-authored-by: Nuno Campos <nuno@boringbits.io>	1 year ago
Bagatur	355ff09cce	bump 279 (#10098 )	1 year ago
Pihplipe Oegr	3dafbd852e	Add sqlite-vss as a vector database (#10047 ) This adds sqlite-vss as an option for a vector database. Contains the code and a few tests. Tests are passing and the library sqlite-vss is added as optional as explained in the contributing guidelines. I adjusted the code for lint/black/ and mypy. It looks that everything is currently passing. Adding sqlite-vss was mentioned in this issue: https://github.com/langchain-ai/langchain/issues/1019. Also mentioned here in the sqlite-vss repo for the curious: https://github.com/asg017/sqlite-vss/issues/66 Maintainer tag: @baskaryan --------- Co-authored-by: Philippe Oger <philippe.oger@adevinta.com>	1 year ago
KyrianC	c7a5504789	Add EdenAI Tools (#9764 ) This PR follows the Eden AI (LLM + embeddings) integration. #8633 We added different Tools to empower agents with new capabilities : - text: explicit content detection - image: explicit content detection - image: object detection - OCR: invoice parsing - OCR: ID parsing - audio: speech to text - audio: text to speech We plan to add more in the future (like translation, language detection, + others). Usage: ```python llm=EdenAI(feature="text",provider="openai", params={"temperature" : 0.2,"max_tokens" : 250}) tools = [ EdenAiTextModerationTool(providers=["openai"],language="en"), EdenAiObjectDetectionTool(providers=["google","api4ai"]), EdenAiTextToSpeechTool(providers=["amazon"],language="en",voice="MALE"), EdenAiExplicitImageTool(providers=["amazon","google"]), EdenAiSpeechToTextTool(providers=["amazon"]), EdenAiParsingIDTool(providers=["amazon","klippa"],language="en"), EdenAiParsingInvoiceTool(providers=["amazon","google"],language="en"), ] agent_chain = initialize_agent( tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True, return_intermediate_steps=True, ) result = agent_chain(""" i have this text : 'i want to slap you' first : i want to know if this text contains explicit content or not . second : if it does contain explicit content i want to know what is the explicit content in this text, third : i want to make the text into speech . if there is URL in the observations , you will always put it in the output (final answer) . """) ``` output: > Entering new AgentExecutor chain... > I need to extract the information from the ID and then convert it to text and then to speech > Action: edenai_identity_parsing > Action Input: "https://www.citizencard.com/images/citizencard-uk-id-card-2023.jpg" > Observation: last_name : > value : ANGELA > given_names : > value : GREENE > birth_place : > birth_date : > value : 2000-11-09 > issuance_date : > expire_date : > document_id : > issuing_state : > address : > age : > country : > document_type : > value : DRIVER LICENSE FRONT > gender : > image_id : > image_signature : > mrz : > nationality : > Thought: I now need to convert the information to text and then to speech > Action: edenai_text_to_speech > Action Input: "Welcome Angela Greene!" > Observation: https://d14uq1pz7dzsdq.cloudfront.net/0c494819-0bbc-4433-bfa4-6e99bd9747ea_.mp3?Expires=1693316851&Signature=YcMoVQgPuIMEOuSpFuvhkFM8JoBMSoGMcZb7MVWdqw7JEf5~67q9dEI90o5todE5mYXB5zSYoib6rGrmfBl4Rn5~yqDwZ~Tmc24K75zpQZIEyt5~ZSnHuXy4IFWGmlIVuGYVGMGKxTGNeCRNUXDhT6TXGZlr4mwa79Ei1YT7KcNyc1dsTrYB96LphnsqOERx4X9J9XriSwxn70X8oUPFfQmLcitr-syDhiwd9Wdpg6J5yHAJjf657u7Z1lFTBMoXGBuw1VYmyno-3TAiPeUcVlQXPueJ-ymZXmwaITmGOfH7HipZngZBziofRAFdhMYbIjYhegu5jS7TxHwRuox32A__&Key-Pair-Id=K1F55BTI9AHGIK > Thought: I now know the final answer > Final Answer: https://d14uq1pz7dzsdq.cloudfront.net/0c494819-0bbc-4433-bfa4-6e99bd9747ea_.mp3?Expires=1693316851&Signature=YcMoVQgPuIMEOuSpFuvhkFM8JoBMSoGMcZb7MVWdqw7JEf5~67q9dEI90o5todE5mYXB5zSYoib6rGrmfBl4Rn5~yqDwZ~Tmc24K75zpQZIEyt5~ZSnHuXy4IFWGmlIVuGYVGMGKxTGNeCRNUXDhT6TXGZlr4mwa79Ei1YT7KcNyc1dsTrYB96LphnsqOERx4X9J9XriSwxn70X8oUPFfQmLcitr-syDhiwd9Wdpg6J5y > > Finished chain. Other examples are available in the jupyter notebook. This PR is made in parallel with EdenAI LLM update #8963 I apologize for the messy PR. While working in implementing Tools we realized there was a few problems we needed to fix on LLM as well. Ping: @hwchase17, @baskaryan --------- Co-authored-by: RedhaWassim <rwasssim@gmail.com>	1 year ago
Nuno Campos	5569385ee1	Lint	1 year ago
Nuno Campos	e17275ee57	Add root run wrapping call to RunnableEach()	1 year ago
Nuno Campos	63306899a2	PR review suggestions	1 year ago
Nuno Campos	7966af1e9c	Lint	1 year ago
Nuno Campos	4c0e1e501c	Re-implement retry, adding a root run, and implement return_exception for batch() and abatch()	1 year ago
Nuno Campos	0eba80912f	Lint	1 year ago
Nuno Campos	af2e4ce2cd	Use a non-inheritable tag	1 year ago
Nuno Campos	85088dc5df	Lint	1 year ago
Nuno Campos	4eecf90f33	Lint	1 year ago
Nuno Campos	2242e2160f	Lint	1 year ago
Nuno Campos	b2ac835466	Add .with_retry() to Runnables	1 year ago
Nuno Campos	81ebcc161e	Lint	1 year ago
Nuno Campos	fc42726ea0	Styling	1 year ago
Nuno Campos	897f791940	Remove run_id from patch	1 year ago
William Fu-Hinthorn	4d7cd6db5f	add cm	1 year ago
Nuno Campos	f9a845b382	Lint	1 year ago
Nuno Campos	06e89c1caa	Lint	1 year ago
Nuno Campos	738d93215d	Allow patching run_name and max_concurrency	1 year ago
Nuno Campos	9a07032055	Lint	1 year ago
Nuno Campos	5426712311	Adjust merge logic	1 year ago
Nuno Campos	f95bd0bcd9	Fix issue	1 year ago
Nuno Campos	f69155b4f7	Add run_id, run_name to RunnableConfig	1 year ago
Nuno Campos	a3c69cf41d	Add .with_config() method to Runnables which allows binding any config values to a Runnable	1 year ago
jmhayes3	324c86acd5	fix typo in web_research.py (#10076 ) fix spelling	1 year ago
Davide Menini	3f8f3de28e	fix (parsers/json): do not escape double quotes if already escaped (#9916 ) This PR fixes an issues I found when upgrading to a more recent version of Langchain. I was using 0.0.142 before, and this issue popped up already when the `_custom_parser` was added to `output_parsers/json`. Anyway, the issue is that the parser tries to escape quotes when they are double-escaped (e.g. `\\"`), leading to OutputParserException. This is particularly undesired in my app, because I have an Agent that uses a single input Tool, which expects as input a JSON string with the structure: ```python { "foo": string, "bar": string } ``` The LLM (GPT3.5) response is (almost) always something like `"action_input": "{\\"foo\\": \\"bar\\", \\"bar\\": \\"foo\\"}"` and since the upgrade this is not correctly parsed. --------- Co-authored-by: taamedag <Davide.Menini@swisscom.com>	1 year ago
Harrison Chase	566ce06f4a	add async support for tools (#10058 )	1 year ago
Jiří Moravčík	86646ec555	feat: Add `ApifyWrapper` class (#10067 ) If you look at documentation https://python.langchain.com/docs/integrations/tools/apify (or the actual file https://github.com/langchain-ai/langchain/blob/master/docs/extras/integrations/tools/apify.ipynb ), there's a class `ApifyWrapper` mentioned. It seems it got lost in some refactoring, i.e. it does not exist in the codebase ATM. I just propose to add it back. It would fix issues e.g. https://github.com/langchain-ai/langchain/issues/8307 or https://github.com/langchain-ai/langchain/issues/8201 To add, Apify is a wanted integration, e.g. see https://twitter.com/hwchase17/status/1695490295914545626 or https://twitter.com/hwchase17/status/1695470765343461756 Lastly, I offer taking ownership of the Apify-related parts of the codebase, so you can tag me if anything is needed.	1 year ago
Robert Perrotta	02e51f4217	update_forward_refs for Run (#9969 ) Adds a call to Pydantic's `update_forward_refs` for the `Run` class (in addition to the `ChainRun` and `ToolRun` classes, for which that method is already called). Without it, the self-reference of child classes (type `List[Run]`) is problematic. For example: ```python from langchain.callbacks import StdOutCallbackHandler from langchain.chains import LLMChain from langchain.llms import OpenAI from langchain.prompts import PromptTemplate from wandb.integration.langchain import WandbTracer llm = OpenAI() prompt = PromptTemplate.from_template("1 + {number} = ") chain = LLMChain(llm=llm, prompt=prompt, callbacks=[StdOutCallbackHandler(), WandbTracer()]) print(chain.run(number=2)) ``` results in the following output before the change ``` WARNING:root:Error in on_chain_start callback: field "child_runs" not yet prepared so type is still a ForwardRef, you might need to call Run.update_forward_refs(). > Entering new LLMChain chain... Prompt after formatting: 1 + 2 = WARNING:root:Error in on_chain_end callback: No chain Run found to be traced > Finished chain. 3 ``` but afterwards the callback error messages are gone.	1 year ago
Eugene Yurtsev	74fcfed4e2	lint for pydantic imports (#9937 ) Catch pydantic imports	1 year ago
Zizhong Zhang	641b71e2cd	refactor: rename to OpaquePrompts (#10013 ) Renamed to OpaquePrompts cc @baskaryan Thanks in advance!	1 year ago
Bagatur	19400ba253	bump 278 (#10052 )	1 year ago
Bagatur	29270e0378	fix #3117 (#9957 ) fix #3117	1 year ago
Bagatur	5b913003e0	bump	1 year ago
Bagatur	4b15328767	Add indexing support for postgresql (#9933 ) Add support to postgresql for the SQL Manager Record This code was tested locally. I'm looking at how to add testing with postgres in a separate PR.	1 year ago
Bagatur	e60e1cdf23	fixed openai_functions api_response format args err (#9968 ) root cause: args may not have a key (params) resulting in an error	1 year ago
Bagatur	3efab8d3df	implement vectorstores by tencent vectordb (#9989 ) Hi there！ I'm excited to open this PR to add support for using 'Tencent Cloud VectorDB' as a vector store. Tencent Cloud VectorDB is a fully-managed, self-developed, enterprise-level distributed database service designed for storing, retrieving, and analyzing multi-dimensional vector data. The database supports multiple index types and similarity calculation methods, with a single index supporting vector scales up to 1 billion and capable of handling millions of QPS with millisecond-level query latency. Tencent Cloud VectorDB not only provides external knowledge bases for large models to improve their accuracy, but also has wide applications in AI fields such as recommendation systems, NLP services, computer vision, and intelligent customer service. The PR includes: Implementation of Vectorstore. I have read your [contributing guidelines](`72b7d76d79/.github/CONTRIBUTING.md`). And I have passed the tests below make format make lint make coverage make test	1 year ago
Bagatur	d43a36c32a	Bagatur/dereference tool schema (#10007 ) fix for #9375	1 year ago
Bagatur	6b5a970949	refactor(document_loaders): abstract page evaluation logic in PlaywrightURLLoader (#9995 ) This PR brings structural updates to `PlaywrightURLLoader`, aiming at making the code more readable and extensible through the abstraction of page evaluation logic. These changes also align this implementation with a similar structure used in LangChain.js. The key enhancements include: 1. Introduction of 'PlaywrightEvaluator', an abstract base class for all evaluators. 2. Creation of 'UnstructuredHtmlEvaluator', a concrete class implementing 'PlaywrightEvaluator', which uses `unstructured` library for processing page's HTML content. 3. Extension of 'PlaywrightURLLoader' constructor to optionally accept an evaluator of the type 'PlaywrightEvaluator'. It defaults to 'UnstructuredHtmlEvaluator' if no evaluator is provided. 4. Refactoring of 'load' and 'aload' methods to use the 'evaluate' and 'evaluate_async' methods of the provided 'PageEvaluator' for page content handling. This update brings flexibility to 'PlaywrightURLLoader' as it can now utilize different evaluators for page processing depending on the requirement. The abstraction also improves code maintainability and readability. Twitter: @ywkim	1 year ago
Bagatur	b1644bc9ad	cr	1 year ago
Hunsmore	13fef1e5d3	add bloomz_7b, llama-2-7b, llama-2-13b, llama-2-70b to ErnieBotChat (#10024 ) - Description: Add bloomz_7b, llama-2-7b, llama-2-13b, llama-2-70b to ErnieBotChat, which only supported ERNIE-Bot-turbo and ERNIE-Bot. - Issue: #10022, - Dependencies: no extra dependencies --------- Co-authored-by: hetianfeng <hetianfeng@meituan.com>	1 year ago
skspark	52a3e8a261	Add integration TCs on bing search (#8068 ) (#10021 ) ## Description Added integration TCs on bing search utility ## Issue #8068 ## Dependencies None	1 year ago
William FH	5341b04d68	Update error message (#9970 ) in evals	1 year ago
William FH	b82ad19ed2	Check memory address (#9971 ) Don't want to dup the collector but can have multiple	1 year ago
Bagatur	e805f8e263	add tests	1 year ago
Bagatur	1f5c579ef4	add	1 year ago
Bagatur	240cc289e6	wip	1 year ago
maks-operlejn-ds	a8f804a618	Add data anonymizer (#9863 ) ### Description The feature for anonymizing data has been implemented. In order to protect private data, such as when querying external APIs (OpenAI), it is worth pseudonymizing sensitive data to maintain full privacy. Anonynization consists of two steps: 1. Identification: Identify all data fields that contain personally identifiable information (PII). 2. Replacement: Replace all PIIs with pseudo values or codes that do not reveal any personal information about the individual but can be used for reference. We're not using regular encryption, because the language model won't be able to understand the meaning or context of the encrypted data. We use Microsoft Presidio together with Faker framework for anonymization purposes because of the wide range of functionalities they provide. The full implementation is available in `PresidioAnonymizer`. ### Future works - deanonymization - add the ability to reverse anonymization. For example, the workflow could look like this: `anonymize -> LLMChain -> deanonymize`. By doing this, we will retain anonymity in requests to, for example, OpenAI, and then be able restore the original data. - instance anonymization - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object. ### Twitter handle @deepsense_ai / @MaksOpp --------- Co-authored-by: MaksOpp <maks.operlejn@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Bagatur	b3e3a31240	bump 277 (#9997 )	1 year ago
Bagatur	9828701de1	mv base cache to schema (#9953 ) if you remove all other imports from langchain.init it exposes a circular dep	1 year ago
Christophe Bornet	9870bfb9cd	Add bucket and object key to metadata in S3 loader (#9317 ) - Description: this PR adds `s3_object_key` and `s3_bucket` to the doc metadata when loading an S3 file. This is particularly useful when using `S3DirectoryLoader` to remove the files from the dir once they have been processed (getting the object keys from the metadata `source` field seems brittle) - Dependencies: N/A - Tag maintainer: ? - Twitter handle: _cbornet --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	1 year ago
Eugene Yurtsev	6da158388b	Merge branch 'master' into ywkim/master	1 year ago
Guy Korland	24c0b01c38	Extend the FalkorDB QA demo (#9992 ) - Description: Extend the FalkorDB QA demo - Tag maintainer: @baskaryan	1 year ago
Eugene Yurtsev	588237ef30	Make document serializable, create utility to create a docstore (#9674 ) This PR makes the following changes: 1. Documents become serializable using langhchain serialization 2. Make a utility to create a docstore kw store Will help to address issue here: https://github.com/langchain-ai/langchain/issues/9345	1 year ago
Eugene Yurtsev	e8f29be350	x	1 year ago
Buckler89	a28e888b36	fix call _get_keys for custom_evaluator (#9763 ) In the function _load_run_evaluators the function _get_keys was not called if only custom_evaluators parameter is used - Description: In the function _load_run_evaluators the function _get_keys was not called if only custom_evaluators parameter is used, - Issue: no issue created for this yet, - Dependencies: None, - Tag maintainer: @vowelparrot, - Twitter handle: Buckler89 --------- Co-authored-by: ddroghini <d.droghini@mflgroup.com>	1 year ago
Eugene Yurtsev	cafce9ed23	x	1 year ago
wlleiiwang	8c4e29240c	implement vectorstores by tencent vectordb	1 year ago
Bagatur	2d2b097fab	mv chat history (#9725 )	1 year ago
Bagatur	d762a6b51f	rm mutable defaults (#9974 )	1 year ago
Arjun Aravindan	6a51672164	Update SeleniumURLLoader to use webdriver Service in favor of deprecated executable_path parameter (#9814 ) Description: This commit uses the new Service object in Selenium webdriver as executable_path has been [deprecated and removed in selenium version 4.11.2](`9f5801c82f`) Issue: https://github.com/langchain-ai/langchain/issues/9808 Tag Maintainer: @eyurtsev	1 year ago
William FH	c844aaa7a6	Weakref to tracer (#9954 ) Prevent memory/thread leakage	1 year ago
Jurik-001	a05fed9369	Fix add callbacks to spark_sql due to depreciation of callback_manager (#9831 ) Description: Due to depreciation (regarding to line 109 in [langchain/libs/langchain/langchain/chains/base.py](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/chains/base.py) of callback_manager i replaced several parts Issue: None Dependencies: Maintainer: @baskaryan --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
dafu	c26deb6b38	fixed openai_functions api_response format args err root cause: args may not have a key (params) resulting in an error	1 year ago
axiangcoding	ffa5625134	feat(llms): improve ERNIE-Bot chat model (#9833 ) - Description: improve ERNIE-Bot chat model, add request timeout and more testcases. - Issue: None - Dependencies: None - Tag maintainer: @baskaryan --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Bagatur	d966ba63e2	fixed GoogleCloudEnterpriseSearchRetriever returning an empty array (#9858 ) `GoogleCloudEnterpriseSearchRetriever` returned an empty array of documents earlier, fixed	1 year ago
Bagatur	ec362ecbe2	Fixed regex bug in RetrievalQAWithSources in previous update (#9898 ) - Description: In my previous PR, I had modified the code to catch all kinds of [SOURCES, sources, Source, Sources]. However, this change included checking for a colon or a white space which should actually have been only checking for a colon. - Issue: the issue # it fixes (if applicable), - Dependencies: any dependencies required for this change,	1 year ago
Nikhil Suresh	56a0165a4e	cleaned up unit test example	1 year ago
William FH	cedfad541d	don't emit none from eval config (#9963 )	1 year ago
Nikhil Suresh	b31475c622	minor updates to regex	1 year ago
Bagatur	8fb0a9594c	Add LLMonitor Callback Handler Integration - open-source observability & analytics (#9870 ) Adds support for [llmonitor](https://llmonitor.com) callbacks. It enables: - Requests tracking / logging / analytics - Error debugging - Cost analytics - User tracking Let me know if anythings neds to be changed for merge. Thank you!	1 year ago
William FH	d799963870	Wfh/async tool (#9878 ) Co-authored-by: Daniel Brenot <dbrenot@pelmorex.com> Co-authored-by: Daniel <daniel.alexander.brenot@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Bagatur	16eb935469	Fix for similarity_search_with_score (#9903 ) - Description: the implementation for similarity_search_with_score did not actually include a score or logic to filter. Now fixed. - Tag maintainer: @rlancemartin - Twitter handle: @ofermend	1 year ago
Bagatur	c70bb0ec28	Activeloopai runtime arg (#9961 )	1 year ago
Bagatur	0f85671630	fmt	1 year ago
Bagatur	78c014399f	fmt	1 year ago
Eugene Yurtsev	5cce6529a4	Speed up openai tests (#9943 ) Saves ~8-10 seconds from total unit tests times --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Guy Korland	7cbe872af8	Add support for Falkordb (ex-RedisGraph) (#9821 ) Replace this entire comment with: - Description: Add support for Falkordb (ex-RedisGraph) - Tag maintainer: @hwchase17 - Twitter handle: @g_korland	1 year ago
Bagatur	9f2d908316	cr	1 year ago
Bagatur	3c1547925a	fix	1 year ago
William FH	fbd792ac7c	Fix import (#9945 )	1 year ago
Zizhong Zhang	8bd7a9d18e	feat: PromptGuard takes a list of str (#9948 ) Recently we made the decision that PromptGuard takes a list of strings instead of a string. @ggroode implemented the integration change. --------- Co-authored-by: ggroode <ggroode@berkeley.edu> Co-authored-by: ggroode <46691276+ggroode@users.noreply.github.com>	1 year ago
Predrag Gruevski	8dbf4cbe80	Add notice about security-sensitive experimental code to experimental README. (#9936 ) It renders like this: https://github.com/langchain-ai/langchain/tree/pg/experimental-readme/libs/experimental ![image](https://github.com/langchain-ai/langchain/assets/2348618/a5f9569d-96f6-44c6-8559-921adb3e337d)	1 year ago
Predrag Gruevski	b5cd1e0fed	Add security notices on PAL and CPAL experimental chains. (#9938 ) Clearly document that the PAL and CPAL techniques involve generating code, and that such code must be properly sandboxed and given appropriate narrowly-scoped credentials in order to ensure security. While our implementations include some mitigations, Python and SQL sandboxing is well-known to be a very hard problem and our mitigations are no replacement for proper sandboxing and permissions management. The implementation of such techniques must be performed outside the scope of the Python process where this package's code runs, so its correct setup and administration must therefore be the responsibility of the user of this code.	1 year ago
Jan-Luca Barthel	f5faac8859	addition of cosine distance function for faiss (#9939 ) - Description: added the _cosine_relevance_score_fn to _select_relevance_score_fn of faiss.py to enable the use of cosine distance for similarity for this vector store and to comply with the Error Message, that implies, that cosine should be a valid distance strategy - Issue: no relevant Issue found, but needed this function myself and tested it in a private repo - Dependencies: none	1 year ago
Eugene Yurtsev	880bf06290	x	1 year ago
Eugene Yurtsev	9efc29e3d1	x	1 year ago
Bagatur	d6957921f0	bump 276 (#9931 )	1 year ago
Tomaz Bratanic	db13fba7ea	Add neo4j vector support (#9770 ) Neo4j has added vector index integration just recently. To allow both ingestion and integrating it as vector RAG applications, I wrapped it as a vector store as the implementation is completely different from `GraphCypherQAChain`. Here, we are not generating any Cypher statements at query time, we are simply doing the vector similarity search using the new vector index as if we were dealing with a vector database. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Bagatur	49ebbe4bcd	fix pydantic import (#9930 )	1 year ago
Mike Nitsenko	c80e406e95	Cube semantic loader: allow cubes processing (#9927 ) We've started to receive feedback (after launch) that using only views is confusing. We're considering this as a good practice, as a view serves as a "facade" for your data - however, we decided to let users decide this on their own. Solves the questions from: - https://github.com/cube-js/cube/issues/7028 - https://github.com/langchain-ai/langchain/pull/9690	1 year ago
Nikhil Suresh	dd10cf945c	fixed minor linting issues	1 year ago
adilkhan	bbae8cb88f	Added runtime argument	1 year ago
Ofer Mendelevitch	4454204455	reformat black	1 year ago
Ofer Mendelevitch	318a21e267	fixed typo in spelling	1 year ago
hughcrt	e71f4760db	Change multiline comment width	1 year ago
Ofer Mendelevitch	a5450be32e	fixed lint	1 year ago
Ofer Mendelevitch	8b8d2a6535	fixed similarity_search_with_score to really use a score updated unit test with a test for score threshold Updated demo notebook	1 year ago
hughcrt	7979cef06a	Replace `\|` by `Union`	1 year ago
Nikhil Suresh	23ef836b48	matches colon and any number of white spaces after colon	1 year ago
Nikhil Suresh	64eb5a6082	removed unnecessary white space in regex that breaks qa with sources chain	1 year ago
Nikhil Suresh	8a4670e127	updated formatting changes	1 year ago
Nikhil Suresh	b1f649bca5	fixed issue with white space and added unit tests	1 year ago
Nikhil Suresh	6d3485e798	fixed regex to match sources for all cases, also includes source	1 year ago
Predrag Gruevski	47499c6db4	Avoid `type: ignore` suppression by adding mypy type hint. (#9881 ) Mypy was not able to determine a good type for `type_to_loader_dict`, since the values in the dict are functions whose return types are related to each other in a complex way. One can see this by adding a line like `reveal_type(type_to_loader_dict)` and running mypy, which will get mypy to show what type it has inferred for that value. Adding an explicit type hint to help out mypy avoids the need for a mypy suppression and allows the code to type-check cleanly.	1 year ago
maks-operlejn-ds	f327535eda	Add conftest file to langchain experimental (#9886 ) In order to use `requires` marker in langchain-experimental, there's a need for conftest.py file inside. Everything is identical to the main langchain module. Co-authored-by: maks-operlejn-ds <maks.operlejn@gmail.com>	1 year ago
William FH	907c57e324	Add collect_runs callback (#9885 )	1 year ago
William FH	3103f07e03	Use existing required args obj if specified (#9883 ) We always overwrote the required args but we infer them by default. Doing it only the old way makes it so the llm guesses even if an arg is optional (e.g., for uuids)	1 year ago
William FH	b14d74dd4d	iMessage loader (#9832 ) Add an iMessage chat loader	1 year ago
Predrag Gruevski	eb3d1fa93c	Add security warning to experimental `SQLDatabaseChain` class. (#9867 ) The most reliable way to not have a chain run an undesirable SQL command is to not give it database permissions to run that command. That way the database itself performs the rule enforcement, so it's much easier to configure and use properly than anything we could add in ourselves.	1 year ago
hughcrt	97741d41c5	Add LLMonitorCallbackHandler	1 year ago
eryk-dsai	7f5713b80a	feat: grammar-based sampling in llama-cpp (#9712 ) ## Description The following PR enables the [grammar-based sampling](https://github.com/ggerganov/llama.cpp/tree/master/grammars) in llama-cpp LLM. In short, loading file with formal grammar definition will constrain model outputs. For instance, one can force the model to generate valid JSON or generate only python lists. In the follow-up PR we will add: * docs with some description why it is cool and how it works * maybe some code sample for some task such as in llama repo --------- Co-authored-by: Lance Martin <lance@langchain.dev> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
William FH	cb642ef658	Return feedback (#9629 ) Return the feedback values in an eval run result Also made a helper method to display as a dataframe but it may be overkill	1 year ago
Bagatur	5e2d0cf54e	bump 275 (#9860 )	1 year ago
Leonid Kuligin	00baddf34c	fixed enterprise search returning an empty array	1 year ago
Eugene Yurtsev	5edf819524	Qdrant Client: Expose instance for creating client (#9706 ) Expose classmethods to convenient initialize the vectostore. The purpose of this PR is to make it easy for users to initialize an empty vectorstore that's properly pre-configured without having to index documents into it via `from_documents`. This will make it easier for users to rely on the following indexing code: https://github.com/langchain-ai/langchain/pull/9614 to help manage data in the qdrant vectorstore.	1 year ago
Harrison Chase	610f46d83a	accept openai terms (#9826 )	1 year ago
Harrison Chase	c1badc1fa2	add gmail loader (#9810 )	1 year ago
Bagatur	0d01cede03	bump 274 (#9805 )	1 year ago

1 2 3 4 5 ...

833 Commits (f7f3c025855e89aac8849b8d90b0e58018a0c78e)