langchain

Commit Graph

Author	SHA1	Message	Date
Harrison Chase	656efe6ef3	Harrison/fix nb (#1678 )	2 years ago
Matt Robinson	63aa28e2a6	feat: allow the unstructured kwargs to be passed in to Unstructured document loaders (#1667 ) ### Summary Allows users to pass in `**unstructured_kwargs` to Unstructured document loaders. Implemented with the `strategy` kwargs in mind, but will pass in other kwargs like `include_page_breaks` as well. The two currently supported strategies are `"hi_res"`, which is more accurate but takes longer, and `"fast"`, which processes faster but with lower accuracy. The `"hi_res"` strategy is the default. For PDFs, if `detectron2` is not available and the user selects `"hi_res"`, the loader will fallback to using the `"fast"` strategy. ### Testing #### Make sure the `strategy` kwarg works Run the following in iPython to verify that the `"fast"` strategy is indeed faster. ```python from langchain.document_loaders import UnstructuredFileLoader loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements") %timeit loader.load() loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements") %timeit loader.load() ``` On my system I get: ```python In [3]: from langchain.document_loaders import UnstructuredFileLoader In [4]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements") In [5]: %timeit loader.load() 247 ms ± 369 µs per loop (mean ± std. dev. of 7 runs, 1 loop each) In [6]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements") In [7]: %timeit loader.load() 2.45 s ± 31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ``` #### Make sure older versions of `unstructured` still work Run `pip install unstructured==0.5.3` and then verify the following runs without error: ```python from langchain.document_loaders import UnstructuredFileLoader loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements") loader.load() ```	2 years ago
Matthias Kern	c3dfbdf0da	Remove outdated code from Chat VectorDB QA example (#1670 )	2 years ago
Bilel MEDIMEGH	a2280f321f	Docs: Fix typo in memory/key_concepts.md (#1671 ) dialouge -> dialogue	2 years ago
Xin Qiu	4e13cef05a	feat: add redisearch vectorstore (#1307 ) # Description Add `RediSearch` vectorstore for LangChain RediSearch: [RediSearch quick start](https://redis.io/docs/stack/search/quick_start/) # How to use ``` from langchain.vectorstores.redisearch import RediSearch rds = RediSearch.from_documents(docs, embeddings,redisearch_url="redis://localhost:6379") ```	2 years ago
Harrison Chase	2d098e8869	Harrison/agent eval (#1620 ) Co-authored-by: jerwelborn <jeremy.welborn@gmail.com>	2 years ago
Harrison Chase	7cf46b3fee	Harrison/convo agent (#1642 )	2 years ago
Jon Luo	0a1b1806e9	sql: do not hard code the LIMIT clause in the table_info section (#1563 ) Seeing a lot of issues in Discord in which the LLM is not using the correct LIMIT clause for different SQL dialects. ie, it's using `LIMIT` for mssql instead of `TOP`, or instead of `ROWNUM` for Oracle, etc. I think this could be due to us specifying the LIMIT statement in the example rows portion of `table_info`. So the LLM is seeing the `LIMIT` statement used in the prompt. Since we can't specify each dialect's method here, I think it's fine to just replace the `SELECT... LIMIT 3;` statement with `3 rows from table_name table:`, and wrap everything in a block comment directly following the `CREATE` statement. The Rajkumar et al paper wrapped the example rows and `SELECT` statement in a block comment as well anyway. Thoughts @fpingham?	2 years ago
Tim Asp	b3234bf3b0	cleanup: unify 3 different pdf loaders, rename PagedPDFSplitter (#1615 ) `OnlinePDFLoader` and `PagedPDFSplitter` lived separate from the rest of the pdf loaders. Because they're all similar, I propose moving all to `pdy.py` and the same docs/examples page. Additionally, `PagedPDFSplitter` naming doesn't match the pattern the rest of the loaders follow, so I renamed to `PyPDFLoader` and had it inherit from `BasePDFLoader` so it can now load from remote file sources.	2 years ago
Harrison Chase	56aff797c0	docs req (#1647 )	2 years ago
Harrison Chase	d53ff270e0	bump version to 109 (#1646 )	2 years ago
Harrison Chase	df6c33d4b3	Harrison/new output parser (#1617 )	2 years ago
Eugene Yurtsev	bd4a2a670b	Add copy button to sphinx notebooks (#1622 ) This adds a copy button at the top right corner of all notebook cells in sphinx notebooks.	2 years ago
Ikko Eltociear Ashimine	6e98ab01e1	Fix typo in vectorstore.ipynb (#1614 ) Initalize -> Initialize	2 years ago
yakigac	acd86d33bc	Add read only shared memory (#1491 ) Provide shared memory capability for the Agent. Inspired by #1293 . ## Problem If both Agent and Tools (i.e., LLMChain) use the same memory, both of them will save the context. It can be annoying in some cases. ## Solution Create a memory wrapper that ignores the save and clear, thereby preventing updates from Agent or Tools.	2 years ago
Harrison Chase	c9b5a30b37	move output parsing (#1605 )	2 years ago
Harrison Chase	15de3e8137	Harrison/docs footer (#1600 ) Co-authored-by: Albert Avetisian <albert.avetisian@gmail.com>	2 years ago
Harrison Chase	9f78717b3c	Harrison/callbacks (#1587 )	2 years ago
Harrison Chase	90846dcc28	fix chat agent (#1586 )	2 years ago
Zach Schillaci	624c72c266	Add wikipedia tool doc (#1579 )	2 years ago
Tim Asp	30383abb12	Add CSVLoader document loader (#1573 ) Simple CSV document loader which wraps `csv` reader, and preps the file with a single `Document` per row. The column header is prepended to each value for context which is useful for context with embedding and semantic search	2 years ago
Andriy Mulyar	c9189d354a	AtlasDB vector store documentation updates. (#1572 ) - Updated errors in the AtlasDB vector store documentation - Removed extraneous output logs in example notebook.	2 years ago
Matt Robinson	7018806a92	feat: document loader for markdown files (#1558 ) ### Summary Adds a document loader for handling markdown files. This document loader requires `unstructured>=0.4.16`. ### Testing ```python from langchain.document_loaders import UnstructuredMarkdownLoader loader = UnstructuredMarkdownLoader("README.md") loader.load() ```	2 years ago
Harrison Chase	bd335ffd64	bump version to 106 (#1562 )	2 years ago
Harrison Chase	a094c49153	add chat agent (#1509 )	2 years ago
Brenton Wheeler	99fe023496	docs: fix typo in modules/indexes/chain_examples/question_answering (#1551 ) docs: fix typo in modules/indexes/chain_examples/question_answering ![image](https://user-images.githubusercontent.com/11394076/224007874-3a52adf6-ff7a-4f22-9dbf-18c83d08167f.png)	2 years ago
Harrison Chase	3ee32a01ea	Harrison/prompt layer (#1547 ) Co-authored-by: Jonathan Pedoeem <jonathanped@gmail.com> Co-authored-by: AbuBakar <abubakarsohail123@gmail.com>	2 years ago
Harrison Chase	cc423f40f1	Harrison/youtube loader (#1545 ) Co-authored-by: Julian Wustl <57504258+Julianwustl@users.noreply.github.com>	2 years ago
Harrison Chase	523ad8d2e2	Harrison/chat history formatter1 (#1538 ) Co-authored-by: Youssef A. Abukwaik <yousseb@users.noreply.github.com>	2 years ago
Graham Neubig	31303d0b11	Added other evaluation metrics for data-augmented QA (#1521 ) This PR adds additional evaluation metrics for data-augmented QA, resulting in a report like this at the end of the notebook: ![Screen Shot 2023-03-08 at 8 53 23 AM](https://user-images.githubusercontent.com/398875/223731199-8eb8e77f-5ff3-40a2-a23e-f3bede623344.png) The score calculation is based on the [Critique](https://docs.inspiredco.ai/critique/) toolkit, an API-based toolkit (like OpenAI) that has minimal dependencies, so it should be easy for people to run if they choose. The code could further be simplified by actually adding a chain that calls Critique directly, but that probably should be saved for another PR if necessary. Any comments or change requests are welcome!	2 years ago
gidler	494c9d341a	[DOCS] Assorted wording, punctuation, and consistency revisions (#1443 ) Contributing some small fixes I noticed while reading through the documentation. Thank you for a creating and maintaining this project!	2 years ago
Harrison Chase	c4a557bdd4	add concept of prompt collection (#1507 )	2 years ago
Ivan	97e3666e0d	changed requests.run to requests.get (#1485 ) This pull request proposes an update to the Lightweight wrapper library's documentation. The current documentation provides an example of how to use the library's requests.run method, as follows: requests.run("https://www.google.com"). However, this example does not work for the 0.0.102 version of the library. Testing: The changes have been tested locally to ensure they are working as intended. Thank you for considering this pull request.	2 years ago
Tom Dyson	e3354404ad	Fix link to Pinecone notebook (#1492 )	2 years ago
Harrison Chase	3610ef2830	add fake embeddings class (#1503 )	2 years ago
Harrison Chase	4f41e20f09	memory docs (#1501 )	2 years ago
Harrison Chase	f276bfad8e	Harrison/chat memory (#1495 )	2 years ago
Harrison Chase	7bec461782	Harrison/memory refactor (#1478 ) moves memory to own module, factors out common stuff	2 years ago
Harrison Chase	0e21463f07	(rfc) chat models (#1424 ) Co-authored-by: Ankush Gola <ankush.gola@gmail.com>	2 years ago
Harrison Chase	63a5614d23	Harrison/simple memory (#1435 ) Co-authored-by: Tim Asp <707699+timothyasp@users.noreply.github.com>	2 years ago
Harrison Chase	a1b9dfc099	Harrison/similarity search chroma (#1434 ) Co-authored-by: shibuiwilliam <shibuiyusuke@gmail.com>	2 years ago
Ikko Eltociear Ashimine	b8a7828d1f	Update huggingface_datasets.ipynb (#1417 ) HuggingFace -> Hugging Face	2 years ago
Tim Asp	23231d65a9	Add PyMuPDF PDF loader (#1426 ) Different PDF libraries have different strengths and weaknesses. PyMuPDF does a good job at extracting the most amount of content from the doc, regardless of the source quality, extremely fast (especially compared to Unstructured). https://pymupdf.readthedocs.io/en/latest/index.html	2 years ago
blob42	3d54b05863	searx: add install instructions, update doc and notebooks (#1420 ) - Added instructions on setting up self hosted searx - Add notebook example with agent - Use `localhost:8888` as example url to stay consistent since public instances are not really usable. Co-authored-by: blob42 <spike@w530>	2 years ago
Tim Asp	bca0935d90	[docs] fix minor import error (#1425 )	2 years ago
JonLuca De Caro	443992c4d5	[Docs] Add missing word from prompt docs (#1406 ) The prompt in the first example of the quickstart guide was missing `for `	2 years ago
Jason Gill	1989e7d4c2	Update examples to prevent confusing missing _type warning (#1391 ) The YAML and JSON examples of prompt serialization now give a strange `No '_type' key found, defaulting to 'prompt'` message when you try to run them yourself or copy the format of the files. The reason for this harmless warning is that the _type key was not in the config files, which means they are parsed as a standard prompt. This could be confusing to new users (like it was confusing to me after upgrading from 0.0.85 to 0.0.86+ for my few_shot prompts that needed a _type added to the example_prompt config), so this update includes the _type key just for clarity. Obviously this is not critical as the warning is harmless, but it could be confusing to track down or be interpreted as an error by a new user, so this update should resolve that.	2 years ago
Harrison Chase	dda5259f68	bump version to 0.0.99 (#1390 )	2 years ago
Kacper Łukawski	9ac442624c	Add Qdrant named arguments (#1386 ) This PR: - Increases `qdrant-client` version to 1.0.4 - Introduces custom content and metadata keys (as requested in #1087) - Moves all the `QdrantClient` parameters into the method parameters to simplify code completion	2 years ago
Ankush Gola	fe30be6fba	add async and streaming support to `OpenAIChat` (#1378 ) title says it all	2 years ago
Lakshya Agarwal	cfed0497ac	Minor grammatical fixes (#1325 ) Fixed typos and links in a few places across documents	2 years ago
Harrison Chase	1cd8996074	Harrison/summarizer chain (#1356 ) Co-authored-by: Tim Asp <707699+timothyasp@users.noreply.github.com>	2 years ago
Harrison Chase	4b5e850361	chatgpt wrapper (#1367 )	2 years ago
Harrison Chase	4d4b43cf5a	fix doc names (#1354 )	2 years ago
Harrison Chase	fe7dbecfe6	pandas and csv agents (#1353 )	2 years ago
Harrison Chase	02ec72df87	improve docs (#1351 )	2 years ago
Jon Luo	92ab27e4b8	sql doc formatting (#1350 ) My bad, missed a few tabs between the two PRs	2 years ago
Ankush Gola	82baecc892	Add a SQL agent for interacting with SQL Databases and JSON Agent for interacting with large JSON blobs (#1150 ) This PR adds * `ZeroShotAgent.as_sql_agent`, which returns an agent for interacting with a sql database. This builds off of `SQLDatabaseChain`. The main advantages are 1) answering general questions about the db, 2) access to a tool for double checking queries, and 3) recovering from errors * `ZeroShotAgent.as_json_agent` which returns an agent for interacting with json blobs. * Several examples in notebooks --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	2 years ago
Jon Luo	35f1e8f569	separate columns by tabs instead of single space in sql sample rows (#1348 ) Use tabs to separate columns instead of a single space - confusing when there are spaces in a cell	2 years ago
James Brotchie	3574418a40	Fix link in summarization.md (#1344 ) "Utilities for working with Documents" was linking to a non-useful page. Re-linked to the utils page that includes info about working with docs.	2 years ago
Jon Luo	5bf8772f26	add option to use user-defined SQL table info (#1347 ) Currently, table information is gathered through SQLAlchemy as complete table DDL and a user-selected number of sample rows from each table. This PR adds the option to use user-defined table information instead of automatically collecting it. This will use the provided table information and fall back to the automatic gathering for tables that the user didn't provide information for. Off the top of my head, there are a few cases where this can be quite useful: - The first n rows of a table are uninformative, or very similar to one another. In this case, hand-crafting example rows for a table such that they provide the good, diverse information can be very helpful. Another approach we can think about later is getting a random sample of n rows instead of the first n rows, but there are some performance considerations that need to be taken there. Even so, hand-crafting the sample rows is useful and can guarantee the model sees informative data. - The user doesn't want every column to be available to the model. This is not an elegant way to fulfill this specific need since the user would have to provide the table definition instead of a simple list of columns to include or ignore, but it does work for this purpose. - For the developers, this makes it a lot easier to compare/benchmark the performance of different prompting structures for providing table information in the prompt. These are cases I've run into myself (particularly cases 1 and 3) and I've found these changes useful. Personally, I keep custom table info for a few tables in a yaml file for versioning and easy loading. Definitely open to other opinions/approaches though!	2 years ago
Harrison Chase	786852e9e6	partial variables (#1308 )	2 years ago
Tim Asp	72ef69d1ba	Add new iFixit document loader (#1333 ) iFixit is a wikipedia-like site that has a huge amount of open content on how to fix things, questions/answers for common troubleshooting and "things" related content that is more technical in nature. All content is licensed under CC-BY-SA-NC 3.0 Adding docs from iFixit as context for user questions like "I dropped my phone in water, what do I do?" or "My macbook pro is making a whining noise, what's wrong with it?" can yield significantly better responses than context free response from LLMs.	2 years ago
Matt Robinson	1aa41b5741	feat: document loader for image files (#1330 ) ### Summary Adds a document loader for image files such as `.jpg` and `.png` files. ### Testing Run the following using the example document from the [`unstructured` repo](https://github.com/Unstructured-IO/unstructured/tree/main/example-docs). ```python from langchain.document_loaders.image import UnstructuredImageLoader loader = UnstructuredImageLoader("layout-parser-paper-fast.jpg") loader.load() ```	2 years ago
Eugene Yurtsev	c14cff60d0	Documentation: Minor typo fixes (#1327 ) Fixing a few minor typos in the documentation (and likely introducing other ones in the process).	2 years ago
Harrison Chase	f61858163d	bump version to 0.0.95 (#1324 )	2 years ago
Harrison Chase	0824d65a5c	Harrison/indexing pipeline (#1317 )	2 years ago
Akshay	a0bf856c70	Update agent_vectorstore.ipynb (#1318 ) nitpicking but just thought i'd add this typo which I found when going through the How-to 😄 (unless it was intentional) also, it's amazing that you added ReAct to LangChain!	2 years ago
Harrison Chase	166cda2cc6	Harrison/deeplake (#1316 ) Co-authored-by: Davit Buniatyan <d@activeloop.ai>	2 years ago
Harrison Chase	aaad6cc954	Harrison/atlas db (#1315 ) Co-authored-by: Brandon Duderstadt <brandonduderstadt@gmail.com>	2 years ago
Marc Puig	3989c793fd	Making it possible to use "certainty" as a parameter for the weaviate similarity_search (#1218 ) Checking if weaviate similarity_search kwargs contains "certainty" and use it accordingly. The minimal level of certainty must be a float, and it is computed by normalized distance.	2 years ago
Harrison Chase	81abcae91a	Harrison/banana fix (#1311 ) Co-authored-by: Erik Dunteman <44653944+erik-dunteman@users.noreply.github.com>	2 years ago
Casey A. Fitzpatrick	648b3b3909	Fix use case sentence for bash util doc (#1295 ) Thanks for all your hard work! I noticed a small typo in the bash util doc so here's a quick update. Additionally, my formatter caught some spacing in the `.md` as well. Happy to revert that if it's an issue. The main change is just ``` - A common use case this is for letting it interact with your local file system. + A common use case for this is letting the LLM interact with your local file system. ``` ## Testing `make docs_build` succeeds locally and the changes show as expected ✌️ <img width="704" alt="image" src="https://user-images.githubusercontent.com/17773666/221376160-e99e59a6-b318-49d1-a1d7-89f5c17cdab4.png">	2 years ago
Ingo Kleiber	fd9975dad7	add CoNLL-U document loader (#1297 ) I've added a simple [CoNLL-U](https://universaldependencies.org/format.html) document loader. CoNLL-U is a common format for NLP tasks and is used, for example, in the Universal Dependencies treebank corpora. The loader reads a single file in standard CoNLL-U format and returns a document.	2 years ago
Harrison Chase	d29f74114e	copy paste loader (#1302 )	2 years ago
Harrison Chase	ce441edd9c	improve docs (#1309 )	2 years ago
Harrison Chase	6f30d68581	add example of using agent with vectorstores (#1285 )	2 years ago
Matt Robinson	2f15c11b87	feat: document loader for MS Word documents (#1282 ) ### Summary Adds a document loader for MS Word Documents. Works with both `.docx` and `.doc` files as longer as the user has installed `unstructured>=0.4.11`. ### Testing The follow workflow test the loader for both `.doc` and `.docx` files using example docs from the `unstructured` repo. #### `.docx` ```python from langchain.document_loaders import UnstructuredWordDocumentLoader filename = "../unstructured/example-docs/fake.docx" loader = UnstructuredWordDocumentLoader(filename) loader.load() ``` #### `.doc` ```python from langchain.document_loaders import UnstructuredWordDocumentLoader filename = "../unstructured/example-docs/fake.doc" loader = UnstructuredWordDocumentLoader(filename) loader.load() ```	2 years ago
Harrison Chase	96db6ed073	cleanup (#1274 )	2 years ago
Harrison Chase	42167a1e24	Harrison/fb loader (#1277 ) Co-authored-by: Vairo Di Pasquale <vairo.dp@gmail.com>	2 years ago
Klein Tahiraj	8a0751dadd	adding .ipynb loader and documentation Fixes #1248 (#1252 ) `NotebookLoader.load()` loads the `.ipynb` notebook file into a `Document` object. Parameters: * `include_outputs` (bool): whether to include cell outputs in the resulting document (default is False). * `max_output_length` (int): the maximum number of characters to include from each cell output (default is 10). * `remove_newline` (bool): whether to remove newline characters from the cell sources and outputs (default is False). * `traceback` (bool): whether to include full traceback (default is False).	2 years ago
Enrico Shippole	9becdeaadf	Add Writer, Banana, Modal, StochasticAI (#1270 ) Add LLM wrappers and examples for Banana, Writer, Modal, Stochastic AI Added rigid json format for Banana and Modal	2 years ago
Matt Robinson	10e73a3723	docs: remove nltk download steps (#1253 ) ### Summary Updates the docs to remove the `nltk` download steps from `unstructured`. As of `unstructured` `0.4.14`, this is handled automatically in the relevant modules within `unstructured`.	2 years ago
Justin Torre	5bc6dc076e	added caching and properties docs (#1255 )	2 years ago
Iskren Ivov Chernev	8e3cd3e0dd	Add DeepInfra LLM support (#1232 ) DeepInfra is an Inference-as-a-Service provider. Add a simple wrapper using HTTPS requests.	2 years ago
Dmitri Melikyan	b7765a95a0	docs: add Graphsignal ecosystem page (#1228 ) Adds a Graphsignal ecosystem page	2 years ago
Harrison Chase	6085fe18d4	add ifttt tool (#1244 )	2 years ago
Harrison Chase	71709ad5d5	Update key_concepts.md (#1209 ) (#1237 ) Link for easier navigation (it's not immediately clear where to find more info on SimpleSequentialChain (3 clicks away) --------- Co-authored-by: Larry Fisherman <l4rryfisherman@protonmail.com>	2 years ago
Dennis Antela Martinez	53c67e04d4	add aleph alpha llm (#1207 ) Integrate Aleph Alpha's client into Langchain to provide access to the luminous models - more info on latest benchmarks here: https://www.aleph-alpha.com/luminous-performance-benchmarks	2 years ago
Ikko Eltociear Ashimine	334b553260	Update petals.md (#1225 ) Huggingface -> Hugging Face	2 years ago
Sason	cc7d2e5621	Correct typo in "Question Answering" How-To Guide (#1221 )	2 years ago
Matt Robinson	3d5f56a8a1	docs: add quotes to `unstructured[local-inference]` install instructions (#1208 ) ### Summary Corrects the install instruction for local inference to `pip install "unstructured[local-inference]"`	2 years ago
Harrison Chase	047231840d	add docs for chroma persistance (#1202 )	2 years ago
Harrison Chase	5bdb8dd6fe	Harrison/unstructured io (#1200 )	2 years ago
Harrison Chase	d90a287d8f	Harrison/updating docs (#1196 )	2 years ago
Dennis Antela Martinez	23243ae69c	add gitbook document loader (#1180 ) Added a GitBook document loader. It lets you both, (1) fetch text from any single GitBook page, or (2) fetch all relative paths and return their respective content in Documents. I've modified the `scrape` method in the `WebBaseLoader` to accept custom web paths if given, but happy to remove it and move that logic into the `GitbookLoader` itself.	2 years ago
Naveen Tatikonda	0118706fd6	Add Support for OpenSearch Vector database (#1191 ) ### Description This PR adds a wrapper which adds support for the OpenSearch vector database. Using opensearch-py client we are ingesting the embeddings of given text into opensearch cluster using Bulk API. We can perform the `similarity_search` on the index using the 3 popular searching methods of OpenSearch k-NN plugin: - `Approximate k-NN Search` use approximate nearest neighbor (ANN) algorithms from the [nmslib](https://github.com/nmslib/nmslib), [faiss](https://github.com/facebookresearch/faiss), and [Lucene](https://lucene.apache.org/) libraries to power k-NN search. - `Script Scoring` extends OpenSearch’s script scoring functionality to execute a brute force, exact k-NN search. - `Painless Scripting` adds the distance functions as painless extensions that can be used in more complex combinations. Also, supports brute force, exact k-NN search like Script Scoring. ### Issues Resolved https://github.com/hwchase17/langchain/issues/1054 --------- Signed-off-by: Naveen Tatikonda <navtat@amazon.com>	2 years ago
Harrison Chase	926c121b98	Harrison/text splitter docs (#1188 )	2 years ago
Harrison Chase	91446a5e9b	clean up text splitting docs (#1184 )	2 years ago
Harrison Chase	5a954efdd7	update gallery with slack bot (#1177 )	2 years ago

1 2 3 4 5 ...

425 Commits (4f364db9a9a695e0667d8e4442bd04e611766fea)