langchain

Commit Graph

Author	SHA1	Message	Date
Frank	bf5805bb32	Add quip loader (#12259 ) - Description: implement [quip](https://quip.com) loader - Issue: https://github.com/langchain-ai/langchain/issues/10352 - Dependencies: No - pass make format, make lint, make test --------- Co-authored-by: Hao Fan <h_fan@apple.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	11 months ago
Bagatur	88ab69c288	mv docs extras (#11399 )	12 months ago
Anatolii Kmetiuk	34a64101cc	Add explanations to GoogleDriveLoader how to avoid errors (#11335 ) - Description: add a paragraph to the GoogleDriveLoader doc on how to bypass errors on authentication. For some reason, specifying credential path via `credentials_path` constructor parameter when creating `GoogleDriveLoader` makes it so that the oAuth screen is never showing up when first using GoogleDriveLoader. Instead, the `RefreshError: ('invalid_grant: Bad Request', {'error': 'invalid_grant', 'error_description': 'Bad Request'})` error happens. Setting it via `os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ...` solves the problem. Also, `token_path` constructor parameter is mandatory, otherwise another error happens when trying to `load()` for the first time. These errors are tricky and time-consuming to figure out, so I believe it's good to mention them in the docs. --------- Co-authored-by: Erick Friis <erick@langchain.dev>	12 months ago
Kazuki Maeda	a363ab5292	rename repo namespace to langchain-ai (#11259 ) ### Description renamed several repository links from `hwchase17` to `langchain-ai`. ### Why I discovered that the README file in the devcontainer contains an old repository name, so I took the opportunity to rename the old repository name in all files within the repository, excluding those that do not require changes. ### Dependencies none ### Tag maintainer @baskaryan ### Twitter handle [kzk_maeda](https://twitter.com/kzk_maeda)	12 months ago
Jon Saginaw	715ffda28b	mongodb doc loader init (#10645 ) - Description: A Document Loader for MongoDB - Issue: n/a - Dependencies: Motor, the async driver for MongoDB - Tag maintainer: n/a - Twitter handle: pigpenblue Note that an initial mongodb document loader was created 4 months ago, but the [PR ](https://github.com/langchain-ai/langchain/pull/4285)was never pulled in. @leo-gan had commented on that PR, but given it is extremely far behind the master branch and a ton has changed in Langchain since then (including repo name and structure), I rewrote the branch and issued a new PR with the expectation that the old one can be closed. Please reference that old PR for comments/context, but it can be closed in favor of this one. Thanks! --------- Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	12 months ago
Leonid Ganeline	21199cc7b4	📖 docs: fixed `integrations/document loaders` toc (#9281 ) Fixed navbar: - renamed several files, so ToC is sorted correctly - made ToC items consistent: formatted several Titles - added several links - reformatted several docs to a consistent format - renamed several files (removed `_example` suffix) - added renamed files to the `docs/docs_skeleton/vercel.json`	12 months ago
Palau	89ef440c14	Kay retriever (#10657 ) - Description: Adding retrievers for [kay.ai](https://kay.ai) and SEC filings powered by Kay and Cybersyn. Kay provides context as a service: it's an API built for RAG. - Issue: N/A - Dependencies: Just added a dep to the [kay](https://pypi.org/project/kay/) package - Tag maintainer: @baskaryan @hwchase17 Discussed in slack - Twtter handle: [@vishalrohra_](https://twitter.com/vishalrohra_) --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	12 months ago
Taqi Jaffri	54763a61f8	fix broken link in docugami loader docs (#10753 ) Just fixing the link to the self query retriever in docugami loader docs Co-authored-by: Taqi Jaffri <tjaffri@docugami.com>	1 year ago
Bagatur	2ae568dcf5	Separate platforms integrations docs (#10609 )	1 year ago
Joshua Sundance Bailey	85e05fa5d6	ArcGISLoader: add keyword arguments, error handling, and better tests (#10558 ) * More clarity around how geometry is handled. Not returned by default; when returned, stored in metadata. This is because it's usually a waste of tokens, but it should be accessible if needed. * User can supply layer description to avoid errors when layer properties are inaccessible due to passthrough access. * Enhanced testing * Updated notebook --------- Co-authored-by: Connor Sutton <connor.sutton@swca.com> Co-authored-by: connorsutton <135151649+connorsutton@users.noreply.github.com>	1 year ago
Christophe Bornet	803d0d9656	Add the possibility to configure boto3 in the S3 loaders (#9304 ) - Description: this PR adds the possibility to configure boto3 in the S3 loaders. Any named argument you add will be used to create the Boto3 session. This is useful when the AWS credentials can't be passed as env variables or can't be read from the credentials file. - Issue: N/A - Dependencies: N/A - Tag maintainer: ? - Twitter handle: cbornet_ --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Philippe PRADOS	f59e5d48ed	Google drive integration (lite) (#9999 ) My other [pull-request](https://github.com/langchain-ai/langchain/pull/5135) is too big to be acceptable. I propose another 'lite' version. I update only notebook to propose an integration with the external project [`langchain-googledrive`](https://github.com/pprados/langchain-googledrive). --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	1 year ago
Lars von Wedel	6d82503eb1	Add parser and loader for Azure document intelligence service. (#10136 ) Hi, this PR contains loader / parser for Azure Document intelligence which is a ML-based service to ingest arbitrary PDFs / images, even if scanned. The loader generates Documents by pages of the original document. This is my first contribution to LangChain. Unfortunately I could not find the correct place for test cases. Happy to add one if you can point me to the location, but as this is a cloud-based service, a test would require network access and credentials - so might be of limited help. Dependencies: The needed dependency was already part of pyproject.toml, no change. Twitter: feel free to mention @LarsAC on the announcement	1 year ago
Bagatur	b485c3048b	rm base64 images from docs (#10110 ) Causing problems indexing docs and notebook images don't render after markdown conversion anyways	1 year ago
Christophe Bornet	9870bfb9cd	Add bucket and object key to metadata in S3 loader (#9317 ) - Description: this PR adds `s3_object_key` and `s3_bucket` to the doc metadata when loading an S3 file. This is particularly useful when using `S3DirectoryLoader` to remove the files from the dir once they have been processed (getting the object keys from the metadata `source` field seems brittle) - Dependencies: N/A - Tag maintainer: ? - Twitter handle: _cbornet --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	1 year ago
Mike Nitsenko	c80e406e95	Cube semantic loader: allow cubes processing (#9927 ) We've started to receive feedback (after launch) that using only views is confusing. We're considering this as a good practice, as a view serves as a "facade" for your data - however, we decided to let users decide this on their own. Solves the questions from: - https://github.com/cube-js/cube/issues/7028 - https://github.com/langchain-ai/langchain/pull/9690	1 year ago
Mazhar (Taha) Mumbaiwala	e80834d783	docs: Fix spelling mistakes in Etherscan.ipynb (#9845 )	1 year ago
Philippe PRADOS	7fdb7439e0	Update google drive notebooks (#9851 ) Update google drive doc loader and retriever notebooks. Show how to use with langchain-googledrive package. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Patrick Loeber	6bedfdf25a	Fix docs for AssemblyAIAudioTranscriptLoader (shorter import path) (#9687 ) Uses the shorter import path `from langchain.document_loaders import` instead of the full path `from langchain.document_loaders.assemblyai` Applies those changes to the docs and the unit test. See #9667 that adds this new loader.	1 year ago
Patrick Loeber	5990651070	Add new document_loader: AssemblyAIAudioTranscriptLoader (#9667 ) This PR adds a new document loader `AssemblyAIAudioTranscriptLoader` that allows to transcribe audio files with the [AssemblyAI API](https://www.assemblyai.com) and loads the transcribed text into documents. - Add new document_loader with class `AssemblyAIAudioTranscriptLoader` - Add optional dependency `assemblyai` - Add unit tests (using a Mock client) - Add docs notebook This is the equivalent to the JS integration already available in LangChain.js. See the [LangChain JS docs AssemblyAI page](https://js.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_loaders/assemblyai_audio_transcription). At its simplest, you can use the loader to get a transcript back from an audio file like this: ```python from langchain.document_loaders.assemblyai import AssemblyAIAudioTranscriptLoader loader = AssemblyAIAudioTranscriptLoader(file_path="./testfile.mp3") docs = loader.load() ``` To use it, it needs the `assemblyai` python package installed, and the environment variable `ASSEMBLYAI_API_KEY` set with your API key. Alternatively, the API key can also be passed as an argument. Twitter handles to shout out if so kindly 🙇 [@AssemblyAI](https://twitter.com/AssemblyAI) and [@patloeber](https://twitter.com/patloeber) --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	1 year ago
seamusp	25f2c82ae8	docs:misc fixes (#9671 ) Improve internal consistency in LangChain documentation - Change occurrences of eg and eg. to e.g. - Fix headers containing unnecessary capital letters. - Change instances of "few shot" to "few-shot". - Add periods to end of sentences where missing. - Minor spelling and grammar fixes.	1 year ago
Bagatur	e2e582f1f6	Fixed source key name for docugami loader (#8598 ) The Docugami loader was not returning the source metadata key. This was triggering this exception when used with retrievers, per https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/schema/prompt_template.py#L193C1-L195C41 The fix is simple and just updates the metadata key name for the document each chunk is sourced from, from "name" to "source" as expected. I tested by running the python notebook that has an end to end scenario in it. Tagging DataLoader maintainers @rlancemartin @eyurtsev	1 year ago
toddkim95	fba29f203a	Add to support polars (#9610 ) ### Description Polars is a DataFrame interface on top of an OLAP Query Engine implemented in Rust. Polars is faster to read than pandas, so I'm looking forward to seeing it added to the document loader. ### Dependencies polars (https://pola-rs.github.io/polars-book/user-guide/) --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
José Ferraz Neto	f116e10d53	Add SharePoint Loader (#4284 ) - Added a loader (`SharePointLoader`) that can pull documents (`pdf`, `docx`, `doc`) from the [SharePoint Document Library](https://support.microsoft.com/en-us/office/what-is-a-document-library-3b5976dd-65cf-4c9e-bf5a-713c10ca2872). - Added a Base Loader (`O365BaseLoader`) to be used for all Loaders that use [O365](https://github.com/O365/python-o365) Package - Code refactoring on `OneDriveLoader` to use the new `O365BaseLoader`. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Martin Schade	0c8a88b3fa	AmazonTextractPDFLoader documentation updates (#9415 ) Description: Updating documentation to add AmazonTextractPDFLoader according to [comment](https://github.com/langchain-ai/langchain/pull/8661#issuecomment-1666572992) from [baskaryan](https://github.com/baskaryan) Adding one notebook and instructions to the modules/data_connection/document_loaders/pdf.mdx --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com> Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	1 year ago
Taqi Jaffri	069c0a041f	comment update for poetry install	1 year ago
Taqi Jaffri	5cd244e9b7	CR feedback	1 year ago
Matt Robinson	83d2a871eb	fix: apply unstructured preprocess functions (#9473 ) ### Summary Fixes a bug from #7850 where post processing functions in Unstructured loaders were not apply. Adds a assertion to the test to verify the post processing function was applied and also updates the explanation in the example notebook.	1 year ago
Joshua Sundance Bailey	ef0664728e	ArcGISLoader update (#9240 ) Small bug fixes and added metadata based on user feedback. This PR is from the author of https://github.com/langchain-ai/langchain/pull/8873 .	1 year ago
Joshua Sundance Bailey	eaa505fb09	Create ArcGISLoader & example notebook (#8873 ) - Description: Adds the ArcGISLoader class to `langchain.document_loaders` - Allows users to load data from ArcGIS Online, Portal, and similar - Users can authenticate with `arcgis.gis.GIS` or retrieve public data anonymously - Uses the `arcgis.features.FeatureLayer` class to retrieve the data - Defines the most relevant keywords arguments and accepts `**kwargs` - Dependencies: Using this class requires `arcgis` and, optionally, `bs4.BeautifulSoup`. Tagging maintainers: - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Hai The Dude	e4418d1b7e	Added new use case docs for Web Scraping, Chromium loader, BS4 transformer (#8732 ) - Description: Added a new use case category called "Web Scraping", and a tutorial to scrape websites using OpenAI Functions Extraction chain to the docs. - Tag maintainer:@baskaryan @hwchase17 , - Twitter handle: https://www.linkedin.com/in/haiphunghiem/ (I'm on LinkedIn mostly) --------- Co-authored-by: Lance Martin <lance@langchain.dev>	1 year ago
Luca Foppiano	dfb93dd2b5	Improved grobid documentation (#9025 ) - Description: Improvement in the Grobid loader documentation, typos and suggesting to use the docker image instead of installing Grobid in local (the documentation was also limited to Mac, while docker allow running in any platform) - Tag maintainer: @rlancemartin, @eyurtsev - Twitter handle: @whitenoise	1 year ago
Taqi Jaffri	5919c0f4a2	notebook cleanup	1 year ago
Taqi Jaffri	bcdf3be530	Merge branch 'master' into tjaffri/docugami_loader_source	1 year ago
Joe Reuter	8f0cd91d57	Airbyte based loaders (#8586 ) This PR adds 8 new loaders: * `AirbyteCDKLoader` This reader can wrap and run all python-based Airbyte source connectors. * Separate loaders for the most commonly used APIs: * `AirbyteGongLoader` * `AirbyteHubspotLoader` * `AirbyteSalesforceLoader` * `AirbyteShopifyLoader` * `AirbyteStripeLoader` * `AirbyteTypeformLoader` * `AirbyteZendeskSupportLoader` ## Documentation and getting started I added the basic shape of the config to the notebooks. This increases the maintenance effort a bit, but I think it's worth it to make sure people can get started quickly with these important connectors. This is also why I linked the spec and the documentation page in the readme as these two contain all the information to configure a source correctly (e.g. it won't suggest using oauth if that's avoidable even if the connector supports it). ## Document generation The "documents" produced by these loaders won't have a text part (instead, all the record fields are put into the metadata). If a text is required by the use case, the caller needs to do custom transformation suitable for their use case. ## Incremental sync All loaders support incremental syncs if the underlying streams support it. By storing the `last_state` from the reader instance away and passing it in when loading, it will only load updated records. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Leonid Ganeline	33a2f58fbf	`tensoflow_datasets` document loader (#8721 ) This PR adds `tensoflow_datasets` document loader	1 year ago
Leonid Ganeline	2d078c7767	`PubMed` document loader (#8893 ) - added `PubMed Document Loader` artifacts; ut-s; examples - fixed `PubMed utility`; ut-s @hwchase17	1 year ago
Leonid Kuligin	b52a3785c9	Allow to specify a custom loader for GcsFileLoader (#8868 ) Co-authored-by: Leonid Kuligin <kuligin@google.com>	1 year ago
Zend	bd4865b6fe	Async Recursive URL loader (#8502 ) Description: This PR improves the function of recursive_url_loader, such as limiting the depth of the access, and customizable extractors(from the raw webpage to the text of the Document object), so that users can use other tools to extract the webpage. This PR also includes the document and test for the new loader. Old PR closed due to project structure change. #7756 Because socket requests are not allowed, the old unit test was removed. Issue: N/A Dependencies: asyncio, aiohttp Tag maintainer: @rlancemartin Twitter handle: @ Zend_Nihility --------- Co-authored-by: Lance Martin <lance@langchain.dev>	1 year ago
Bagatur	d7b613a293	Bagatur/revert revert nuclia (#8833 )	1 year ago
Bagatur	2f309a4ce6	Revert "Bagatur/nuclia (#8404 )" (#8832 )	1 year ago
Bagatur	9fc9018951	Bagatur/nuclia (#8404 ) Co-authored-by: Eric BREHAULT <ebrehault@gmail.com>	1 year ago
Bagatur	8c35fcb571	update rss doc (#8761 )	1 year ago
ruze	8ef7e14a85	RSS Feed / OPML loader (#8694 ) Replace this comment with: - Description: added a document loader for a list of RSS feeds or OPML. It iterates through the list and uses NewsURLLoader to load each article. - Issue: N/A - Dependencies: feedparser, listparser - Tag maintainer: @rlancemartin, @eyurtsev - Twitter handle: @ruze --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
ruze	71f98db2fe	Newspaper (#8647 ) - Description: Added newspaper3k based news article loader. Provide a list of urls. - Issue: N/A - Dependencies: newspaper3k, - Tag maintainer: @rlancemartin , @eyurtsev - Twitter handle: @ruze --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Comendeiro	5c516945d0	Add local support for audio models (PR #7329 ) (#7591 ) - Description: run the poetry dependencies - Issue: #7329 - Dependencies: any dependencies required for this change, - Tag maintainer: @rlancemartin --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Taqi Jaffri	4806504ebc	Fixed one last key name	1 year ago
Matt Robinson	8961c720b8	docs: update `unstructured` install instructions (#8596 ) ### Summary Updates the `unstructured` install instructions. For `unstructured>=0.9.0`, dependencies are broken out by document type and the base `unstructured` package includes fewer dependencies. `pip install "unstructured[local-inference]"` has been replace by `pip install "unstructured[all-docs]"`, though the `local-inference` extra is still supported for the time being. ### Reviewers - @rlancemartin - @eyurtsev - @hwchase17	1 year ago
Taqi Jaffri	96843f3bd4	Fixed source key name for docugami loader	1 year ago
mpb159753	7df2dfc4c2	Add Support for Loading Documents from Huawei OBS (#8573 ) Description: This PR adds support for loading documents from Huawei OBS (Object Storage Service) in Langchain. OBS is a cloud-based object storage service provided by Huawei Cloud. With this enhancement, Langchain users can now easily access and load documents stored in Huawei OBS directly into the system. Key Changes: - Added a new document loader module specifically for Huawei OBS integration. - Implemented the necessary logic to authenticate and connect to Huawei OBS using access credentials. - Enabled the loading of individual documents from a specified bucket and object key in Huawei OBS. - Provided the option to specify custom authentication information or obtain security tokens from Huawei Cloud ECS for easy access. How to Test: 1. Ensure the required package "esdk-obs-python" is installed. 2. Configure the endpoint, access key, secret key, and bucket details for Huawei OBS in the Langchain settings. 3. Load documents from Huawei OBS using the updated document loader module. 4. Verify that documents are successfully retrieved and loaded into Langchain for further processing. Please review this PR and let us know if any further improvements are needed. Your feedback is highly appreciated! @rlancemartin, @eyurtsev --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago

1 2

63 Commits (a9b70baef975130e6f3fe255a05f6b795f81b610)