langchain

Commit Graph

Author	SHA1	Message	Date
Harrison Chase	88d3ce12b8	Harrison/diffbot (#2984 ) Co-authored-by: Manuel Saelices <msaelices@gmail.com>	1 year ago
vowelparrot	5ca7ce77cd	Remove pythonrepl from LLM-MathChain (#2943 ) Use numexpr evaluate instead of the python REPL to avoid malicious code injection. Tested against the (limited) math dataset and got the same score as before. For more permissive tools (like the REPL tool itself), other approaches ought to be provided (some combination of Sanitizer + Restricted python + unprivileged-docker + ...), but for a calculator tool, only mathematical expressions should be permitted. See https://github.com/hwchase17/langchain/issues/814	1 year ago
Daniel Nouri	2a0f65f7af	tiktoken: Relax Python version check (#2966 ) tiktoken supports Python >= 3.8, see here: `e1c661edf3/pyproject.toml (L10)` Also works fine when trying locally!	1 year ago
Chetanya Rastogi	aead062a70	Add an example tutorial for using PDFMinerPDFasHTMLLoader (#2960 ) Last week I added the `PDFMinerPDFasHTMLLoader`. I am adding some example code in the notebook to serve as a tutorial for how that loader can be used to create snippets of a pdf that are structured within sections. All the other loaders only provide the `Document` objects segmented by pages but that's pretty loose given the amount of other metadata that can be extracted. With the new loader, one can leverage font-size of the text to decide when a new sections starts and can segment the text more semantically as shown in the tutorial notebook. The cell shows that we are able to find the content of entire section under Related Work for the example pdf which is spread across 2 pages and hence is stored as two separate documents by other loaders	1 year ago
Tim Asp	51894ddd98	allow tokentextsplitters to use model name to select encoder (#2963 ) Fixes a bug I was seeing when the `TokenTextSplitter` was correctly splitting text under the gpt3.5-turbo token limit, but when firing the prompt off too openai, it'd come back with an error that we were over the context limit. gpt3.5-turbo and gpt-4 use `cl100k_base` tokenizer, and so the counts are just always off with the default `gpt-2` encoder. It's possible to pass along the encoding to the `TokenTextSplitter`, but it's much simpler to pass the model name of the LLM. No more concern about keeping the tokenizer and llm model in sync :)	1 year ago
Alex Iribarren	706ebd8f9c	Enforce maximum Wikipedia query length (#2969 ) I got the following stacktrace when the agent was trying to search Wikipedia with a huge query: ``` Thought:{ "action": "Wikipedia", "action_input": "Outstanding is a song originally performed by the Gap Band and written by member Raymond Calhoun. The song originally appeared on the group's platinum-selling 1982 album Gap Band IV. It is one of their signature songs and biggest hits, reaching the number one spot on the U.S. R&B Singles Chart in February 1983. \"Outstanding\" peaked at number 51 on the Billboard Hot 100." } Traceback (most recent call last): File "/usr/src/app/tests/chat.py", line 121, in <module> answer = agent_chain.run(input=question) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain/chains/base.py", line 216, in run return self(kwargs)[self.output_keys[0]] ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain/chains/base.py", line 116, in __call__ raise e File "/usr/local/lib/python3.11/site-packages/langchain/chains/base.py", line 113, in __call__ outputs = self._call(inputs) ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain/agents/agent.py", line 828, in _call next_step_output = self._take_next_step( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain/agents/agent.py", line 725, in _take_next_step observation = tool.run( ^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain/tools/base.py", line 73, in run raise e File "/usr/local/lib/python3.11/site-packages/langchain/tools/base.py", line 70, in run observation = self._run(tool_input) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain/agents/tools.py", line 17, in _run return self.func(tool_input) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/langchain/utilities/wikipedia.py", line 40, in run search_results = self.wiki_client.search(query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/wikipedia/util.py", line 28, in __call__ ret = self._cache[key] = self.fn(args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/wikipedia/wikipedia.py", line 109, in search raise WikipediaException(raw_results['error']['info']) wikipedia.exceptions.WikipediaException: An unknown error occured: "Search request is longer than the maximum allowed length. (Actual: 373; allowed: 300)". Please report it on GitHub! ``` This commit limits the maximum size of the query passed to Wikipedia to avoid this issue.	1 year ago
Nahin Khan	9a03f00e6c	Fix typos (#2977 )	1 year ago
Altay Sansal	9d8ab28837	Add `top_k` and `filter` fields to `ChatGPTPluginRetriever` (#2852 ) This allows to adjust the number of results to retrieve and filter documents based on metadata. --------- Co-authored-by: Altay Sansal <altay.sansal@tgs.com>	1 year ago
vowelparrot	4ffc58e07b	Add similarity_search_with_normalized_similarities (#2916 ) Add a method that exposes a similarity search with corresponding normalized similarity scores. Implement only for FAISS now. ### Motivation: Some memory definitions combine `relevance` with other scores, like recency , importance, etc. While many (but not all) of the `VectorStore`'s expose a `similarity_search_with_score` method, they don't all interpret the units of that score (depends on the distance metric and whether or not the the embeddings are normalized). This PR proposes a `similarity_search_with_normalized_similarities` method that lets consumers of the vector store not have to worry about the metric and embedding scale. Most providers default to euclidean distance, with Pinecone being one exception (defaults to cosine _similarity_). --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	1 year ago
Tim Asp	b9db20481f	Fix wrong token counts from `get_num_tokens` from openai llms (#2952 ) The encoding fetch was out of date. Luckily OpenAI has a nice[ `encoding_for_model`](`46287bfa49/tiktoken/model.py`) function in `tiktoken` we can use now.	1 year ago
Tim Asp	fea5619ce9	Add title, lang, description to Web loader document metadata (#2955 ) Title, lang and description are on almost every web page, and are incredibly useful pieces of information that currently isn't captured with the current web base loader I thought about adding the title and description to the content of the document, as that content could be useful in search, but I left it out for right now. If you think it'd be worth adding, happy to add it. I've found it's nice to have the title/description in the metadata to have some structured data when retrieving rows from vectordbs for use with summary and source citation, so if we do want to add it to the `page_content`, i'd advocate for it to also be included in metadata.	1 year ago
Maciej Pióro	f7bf917baf	Fix missing docker-compose (#2899 ) Fix missing `docker-compose` command if only `docker compose` (note space) is available.	1 year ago
Harrison Chase	b634489b2e	bump version to 141 (#2950 )	1 year ago
Harrison Chase	274b25c010	SVM retriever (#2947 ) (#2949 ) Add SVM retriever class, based on https://github.com/karpathy/randomfun/blob/master/knn_vs_svm.ipynb. Testing still WIP, but the logic is correct (I have a local implementation outside of Langchain working). --------- Co-authored-by: Lance Martin <122662504+PineappleExpress808@users.noreply.github.com> Co-authored-by: rlm <31treehaus@31s-MacBook-Pro.local>	1 year ago
Harrison Chase	baf350e32b	parametrize redis (#2946 )	1 year ago
dev2049	36aa7f30e4	Move PythonRepl -> langchain.utilities (#2917 )	1 year ago
dev2049	7c73e9df5d	Add kwargs to VectorStore.maximum_marginal_relevance (#2921 ) Same as similarity_search, allows child classes to add vector store-specific args (this was technically already happening in couple places but now typing is correct).	1 year ago
Davit Buniatyan	b3a5b51728	[minor] Deep Lake auth improvements in docs, kwargs pass, faster tests (#2927 ) Minor cosmetic changes - Activeloop environment cred authentication in notebooks with `getpass.getpass` (instead of CLI which not always works) - much faster tests with Deep Lake pytest mode on - Deep Lake kwargs pass Notes - I put pytest environment creds inside `vectorstores/conftest.py`, but feel free to suggest a better location. For context, if I put in `test_deeplake.py`, `ruff` doesn't let me to set them before import deeplake --------- Co-authored-by: Davit Buniatyan <d@activeloop.ai>	1 year ago
Harrison Chase	c4ae8c1d24	bump ver to 140 (#2895 )	1 year ago
Nahin Khan	ad3973a3b8	Fix typo (#2942 )	1 year ago
Harrison Chase	cf2789d86d	delete antropic chat notebook (#2945 )	1 year ago
Hai Nguyen Mau	0aa828b1dc	typo fix (#2937 ) missing w in link	1 year ago
Ankush Gola	ec59e9d886	Fix ChatAnthropic stop_sequences error (#2919 ) (#2920 ) Note to self: Always run integration tests, even on "that last minute change you thought would be safe" :) --------- Co-authored-by: Mike Lambert <mike.lambert@anthropic.com>	1 year ago
Akash NP	13a0ed064b	add encoding to avoid UnicodeDecodeError (#2908 ) About Specify encoding to avoid UnicodeDecodeError when reading .txt for users who are following the tutorial. Reference ``` return codecs.charmap_decode(input,self.errors,decoding_table)[0] UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d in position 1205: character maps to <undefined> ``` Environment OS: Win 11 Python: 3.8	1 year ago
Mike Lambert	392f1b3218	Add Anthropic ChatModel to langchain (#2293 ) * Adds an Anthropic ChatModel * Factors out common code in our LLMModel and ChatModel * Supports streaming llm-tokens to the callbacks on a delta basis (until a future V2 API does that for us) * Some fixes	1 year ago
Kwuang Tang	66bef1d7ed	Ignore files from .gitignore in Git loader (#2909 ) fixes #2905 extends #2851	1 year ago
Boris Feld	7ee87eb0c8	Comet callback updates (#2889 ) I'm working with @DN6 and I made some small fixes and improvements after playing with the integration.	1 year ago
dev2049	634358db5e	Fix OpenAI LLM docstring (#2910 )	1 year ago
pranjaldoshi96	30573b2e30	Correct instruction to use openweathermap utility in docstring (#2906 ) Co-authored-by: Pranjal Doshi <pranjald@nvidia.com>	1 year ago
Kwuang Tang	a508afa91c	Add file filter param to Git loader (#2904 ) Allows users to specify what files should be loaded instead of indiscriminately loading the entire repo. extends #2851 NOTE: for reviewers, `hide whitespace` option recommended since I changed the indentation of an if-block to use `continue` instead so it looks less like a Christmas tree :)	1 year ago
Ismail Pelaseyed	7e525a3b91	Add link to repo for deploying LangChain to Digitalocean App Platform (#2894 ) This PR adds a link to a minimal example of deploying `LangChain` to `Digitalocean App Platform`.	1 year ago
Peter Stolz	ccacf804a8	Fix format string in pinecone error handling (#2897 )	1 year ago
Francis Felici	86189cdcf9	Update load_qa_chain() docstring (#2900 ) Seems to be missing `map_rerank` as a potential argument of `chain_type`	1 year ago
Harrison Chase	8fef69296d	nits (#2873 )	1 year ago
Harrison Chase	0a38bbc750	updates to vectorstore memory (#2875 )	1 year ago
Ikko Eltociear Ashimine	203c0eb2ae	docs: update getting_started.ipynb (#2883 ) HuggingFace -> Hugging Face	1 year ago
ecneladis	1a44b71ddf	Fix Baby AGI notebooks (#2882 ) - fix broken notebook cell in `ae485b623d` - Python Black formatting	1 year ago
Nicolas	3c7204d604	docs: Quick fix to Mendable Search (#2876 ) Fixed a small issue on the icon UI when using in Safari.	1 year ago
Harrison Chase	1e9378d0a8	Harrison/weaviate fixes (#2872 ) Co-authored-by: cs0lar <cristiano.solarino@gmail.com> Co-authored-by: cs0lar <cristiano.solarino@brightminded.com>	1 year ago
Harrison Chase	07d7096de6	Harrison/playwright (#2871 ) Co-authored-by: Manuel Saelices <msaelices@gmail.com>	1 year ago
Jon Luo	5565f56273	Use SQL dialect-specific prompts for SQLDatabaseChain (#2748 ) Mentioned the idea here initially: https://github.com/hwchase17/langchain/pull/2106#issuecomment-1487509106 Since there have been dialect-specific issues, we should use dialect-specific prompts. This way, each prompt can be separately modified to best suit each dialect as needed. This adds a prompt for each dialect supported in sqlalchemy (mssql, mysql, mariadb, postgres, oracle, sqlite). For this initial implementation, the only differencse between the prompts is the instruction for the clause to use to limit the number of rows queried for, and the instruction for wrapping column names using each dialect's identifier quote character.	1 year ago
drod	9907cb0485	Refactor similarity_search function in elastic_vector_search.py (#2761 ) Optimization :Limit search results when k < 10 Fix issue when k > 10: Elasticsearch will return only 10 docs [default-search-result](https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html) By default, searches return the top 10 matching hits Add size parameter to the search request to limit the number of returned results from Elasticsearch. Remove slicing of the hits list, since the response will already contain the desired number of results.	1 year ago
rafael	1cc7ea333c	chat_models.openai: Set tenacity timeout to openai's recommendation (#2768 ) [OpenAI's cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_handle_rate_limits.ipynb) suggest a tenacity backoff between 1 and 60 seconds. Currently langchain's backoff is between 4 and 10 seconds, which causes frequent timeout errors on my end. This PR changes the timeout to the suggested values.	1 year ago
Harrison Chase	705596b46a	Harrison/fix create sql agent (#2870 ) Co-authored-by: Timothé Pearce <timothe.pearce@gmail.com>	1 year ago
Harrison Chase	8a98e5b50b	Harrison/index name (#2869 ) Co-authored-by: Mesum Raza Hemani <mes.javacca@gmail.com>	1 year ago
Andrey Vasnetsov	dcb17503f2	Update qdrant.py (#2750 ) At the moment of upload we should already know the format of data, therefore we can skip the costly pydantic validation.	1 year ago
ecneladis	74abeb8c53	Update output in Git notebook (#2868 ) Supplemental to https://github.com/hwchase17/langchain/pull/2851. Updates one notebook cell that I forgot to commit before.	1 year ago
Nicolas	0226b375d9	docs: Mendable Search integration (#2803 ) Mendable Seach Integration is Finally here! Hey yall, After various requests for Mendable in Python docs, we decided to get our hands dirty and try to implement it. Here is a version where we implement our floating button that sits on the bottom right of the screen that once triggered (via press or CMD K) will work the same as the js langchain docs. Super excited about this and hopefully the community will be too. @hwchase17 will send you the admin details via dm etc. The anon_key is fine to be public. Let me know if you need any further customization. I added the langchain logo to it.	1 year ago
sergerdn	04c458a270	feat: improve pinecone tests (#2806 ) Improve the integration tests for Pinecone by adding an `.env.example` file for local testing. Additionally, add some dev dependencies specifically for integration tests. This change also helps me understand how Pinecone deals with certain things, see related issues https://github.com/hwchase17/langchain/issues/2484 https://github.com/hwchase17/langchain/issues/2816	1 year ago
ecneladis	016738e676	Add GitLoader (#2851 )	1 year ago

1 2 3 4 5 ...

1332 Commits (88d3ce12b8de8f461e8b95dc8a7f50f4feb4c0f1) All Branches Search

1332 Commits (88d3ce12b8de8f461e8b95dc8a7f50f4feb4c0f1)

All Branches