langchain

Commit Graph

Author	SHA1	Message	Date
yin1991	c454dc36fc	community[proxy]: Enhancement/add proxy support playwrighturlloader 16751 (#16822 ) - Description: Enhancement/add proxy support playwrighturlloader 16751 - Issue: [Enhancement: Add Proxy Support to PlaywrightURLLoader Class](https://github.com/langchain-ai/langchain/issues/16751) - Dependencies: - Twitter handle: @ootR77013489 --------- Co-authored-by: root <root@ip-172-31-46-160.ap-southeast-1.compute.internal> Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
yin1991	37ef6ac113	community[patch]: Add Pagination to GitHubIssuesLoader for Efficient GitHub Issues Retrieval (#16934 ) - Description: Add Pagination to GitHubIssuesLoader for Efficient GitHub Issues Retrieval - Issue: [the issue # it fixes if applicable,](https://github.com/langchain-ai/langchain/issues/16864) --------- Co-authored-by: root <root@ip-172-31-46-160.ap-southeast-1.compute.internal> Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
Robby	ece4b43a81	community[patch]: doc loaders mypy fixes (#17368 ) Description: Fixed `type: ignore`'s for mypy for some document_loaders. Issue: [Remove "type: ignore" comments #17048 ](https://github.com/langchain-ai/langchain/issues/17048) --------- Co-authored-by: Robby <h0rv@users.noreply.github.com> Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	7 months ago
Bagatur	f7e453971d	community[patch]: remove print (#17435 )	7 months ago
Abhijeeth Padarthi	584b647b96	community[minor]: AWS Athena Document Loader (#15625 ) - Description: Adds the document loader for [AWS Athena](https://aws.amazon.com/athena/), a serverless and interactive analytics service. - Dependencies: Added boto3 as a dependency	7 months ago
Erick Friis	3a2eb6e12b	infra: add print rule to ruff (#16221 ) Added noqa for existing prints. Can slowly remove / will prevent more being intro'd	7 months ago
Leonid Ganeline	932c52c333	community[patch]: docstrings (#16810 ) - added missed docstrings - formated docstrings to the consistent form	7 months ago
Bagatur	af74301ab9	core[patch], community[patch]: link extraction continue on failure (#17200 )	7 months ago
Frank	ef082c77b1	community[minor]: add github file loader to load any github file content b… (#15305 ) ### Description support load any github file content based on file extension. Why not use [git loader](https://python.langchain.com/docs/integrations/document_loaders/git#load-existing-repository-from-disk) ? git loader clones the whole repo even only interested part of files, that's too heavy. This GithubFileLoader only downloads that you are interested files. ### Twitter handle my twitter: @shufanhaotop --------- Co-authored-by: Hao Fan <h_fan@apple.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	7 months ago
Harrison Chase	4eda647fdd	infra: add -p to mkdir in lint steps (#17013 ) Previously, if this did not find a mypy cache then it wouldnt run this makes it always run adding mypy ignore comments with existing uncaught issues to unblock other prs --------- Co-authored-by: Erick Friis <erick@langchain.dev> Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	7 months ago
Christophe Bornet	9d458d089a	community: Factorize AstraDB components constructors (#16779 ) * Adds `AstraDBEnvironment` class and use it in `AstraDBLoader`, `AstraDBCache`, `AstraDBSemanticCache`, `AstraDBBaseStore` and `AstraDBChatMessageHistory` * Create an `AsyncAstraDB` if we only have an `AstraDB` and vice-versa so: * we always have an instance of `AstraDB` * we always have an instance of `AsyncAstraDB` for recent versions of astrapy * Create collection if not exists in `AstraDBBaseStore` * Some typing improvements Note: `AstraDB` `VectorStore` not using `AstraDBEnvironment` at the moment. This will be done after the `langchain-astradb` package is out.	7 months ago
Christophe Bornet	af8c5c185b	langchain[minor],community[minor]: Add async methods in BaseLoader (#16634 ) Adds: * methods `aload()` and `alazy_load()` to interface `BaseLoader` * implementation for class `MergedDataLoader ` * support for class `BaseLoader` in async function `aindex()` with unit tests Note: this is compatible with existing `aload()` methods that some loaders already had. Twitter handle: @cbornet_ --------- Co-authored-by: Eugene Yurtsev <eugene@langchain.dev>	7 months ago
Raphael	bf9068516e	community[minor]: add the ability to load existing transcripts from AssemblyAI by their id. (#16051 ) - Description: the existing AssemblyAI API allows to pass a path or an url to transcribe an audio file and turn in into Langchain Documents, this PR allows to get existing transcript by their transcript id and turn them into Documents. - Issue: not related to an existing issue - Dependencies: requests --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	8 months ago
Alexander Conway	4acd2654a3	Report which file was errored on in DirectoryLoader (#16790 ) The current implementation leaves it up to the particular file loader implementation to report the file on which an error was encountered - in my case pdfminer was simply saying it could not parse a file as a PDF, but I didn't know which of my hundreds of files it was failing on. No reason not to log the particular item on which an error was encountered, and it should be an immense debugging assistant. <!-- Thank you for contributing to LangChain! Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified. Replace this entire comment with: - Description: a description of the change, - Issue: the issue # it fixes if applicable, - Dependencies: any dependencies required for this change, - Twitter handle: we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. -->	8 months ago
Christophe Bornet	2e3af04080	Use Postponed Evaluation of Annotations in Astra and Cassandra doc loaders (#16694 ) Minor/cosmetic change	8 months ago
Christophe Bornet	36e432672a	community[minor]: Add async methods to AstraDBLoader (#16652 )	8 months ago
João Carlos Ferra de Almeida	3e87b67a3c	community[patch]: Add Cookie Support to Fetch Method (#16673 ) - Description: This change allows the `_fetch` method in the `WebBaseLoader` class to utilize cookies from an existing `requests.Session`. It ensures that when the `fetch` method is used, any cookies in the provided session are included in the request. This enhancement maintains compatibility with existing functionality while extending the utility of the `fetch` method for scenarios where cookie persistence is necessary. - Issue: Not applicable (new feature), - Dependencies: Requires `aiohttp` and `requests` libraries (no new dependencies introduced), - Twitter handle: N/A Co-authored-by: Joao Almeida <joao.almeida@mercedes-benz.io>	8 months ago
Christophe Bornet	4915c3cd86	[Fix] Fix Cassandra Document loader default page content mapper (#16273 ) We can't use `json.dumps` by default as many types returned by the cassandra driver are not serializable. It's safer to use `str` and let users define their own custom `page_content_mapper` if needed.	8 months ago
Pasha	4e189cd89a	community[patch]: youtube loader transcript format (#16625 ) - Description: YoutubeLoader right now returns one document that contains the entire transcript. I think it would be useful to add an option to return multiple documents, where each document would contain one line of transcript with the start time and duration in the metadata. For example, [AssemblyAIAudioTranscriptLoader](https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/document_loaders/assemblyai.py) is implemented in a similar way, it allows you to choose between the format to use for the document loader.	8 months ago
bachr	b3ed98dec0	community[patch]: avoid KeyError when language not in LANGUAGE_SEGMENTERS (#15212 ) Description: Handle unsupported languages in same way as when none is provided Issue: The following line will throw a KeyError if the language is not supported. ```python self.Segmenter = LANGUAGE_SEGMENTERS[language] ``` E.g. when using `Language.CPP` we would get `KeyError: <Language.CPP: 'cpp'>` --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	8 months ago
Karim Lalani	4c7755778d	community[patch]: SurrealDB fix for asyncio (#16092 ) Code fix for asyncio	8 months ago
Raunak	476bf8b763	community[patch]: Load list of files using UnstructuredFileLoader (#16216 ) - Description: Updated `_get_elements()` function of `UnstructuredFileLoader `class to check if the argument self.file_path is a file or list of files. If it is a list of files then it iterates over the list of file paths, calls the partition function for each one, and appends the results to the elements list. If self.file_path is not a list, it calls the partition function as before. - Issue: Fixed #15607, - Dependencies: NA - Twitter handle: NA Co-authored-by: H161961 <Raunak.Raunak@Honeywell.com>	8 months ago
Florian MOREL	4b7969efc5	community[minor]: New documents loader for visio files (with extension .vsdx) (#16171 ) Description : New documents loader for visio files (with extension .vsdx) A [visio file](https://fr.wikipedia.org/wiki/Microsoft_Visio) (with extension .vsdx) is associated with Microsoft Visio, a diagram creation software. It stores information about the structure, layout, and graphical elements of a diagram. This format facilitates the creation and sharing of visualizations in areas such as business, engineering, and computer science. A Visio file can contain multiple pages. Some of them may serve as the background for others, and this can occur across multiple layers. This loader extracts the textual content from each page and its associated pages, enabling the extraction of all visible text from each page, similar to what an OCR algorithm would do. Dependencies : xmltodict package	8 months ago
Alireza Kashani	d1b4ead87c	community[patch]: Update grobid.py (#16298 ) there is a case where "coords" does not exist in the "sentence" therefore, the "split(";")" will lead to error. we can fix that by adding "if sentence.get("coords") is not None:" the resulting empty "sbboxes" from this scenario will raise error at "sbboxes[0]["page"]" because sbboxes are empty. the PDF from https://pubmed.ncbi.nlm.nih.gov/23970373/ can replicate those errors.	8 months ago
Christophe Bornet	3ccbe11363	community[minor]: Add Cassandra document loader (#16215 ) - Description: document loader for Apache Cassandra - Twitter handle: cbornet_	8 months ago
andrijdavid	d196646811	community[patch]: Refactor OpenAIWhisperParserLocal (#15150 ) This PR addresses an issue in OpenAIWhisperParserLocal where requesting CUDA without availability leads to an AttributeError #15143 Changes: - Refactored Logic for CUDA Availability: The initialization now includes a check for CUDA availability. If CUDA is not available, the code falls back to using the CPU. This ensures seamless operation without manual intervention. - Parameterizing Batch Size and Chunk Size: The batch_size and chunk_size are now configurable parameters, offering greater flexibility and optimization options based on the specific requirements of the use case. --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	8 months ago
Karim Lalani	14244bd7e5	community[minor]: Added document loader for SurrealDB (#15995 ) Added a simple document loader to work with SurrealDB.	8 months ago
Edwin Wenink	9fb09c1c30	community: fix the "page" mode in the AzureAIDocumentIntelligenceParser (bug) (#15958 ) Description: the "page" mode in the AzureAIDocumentIntelligenceParser is not accessible due to a wrong membership test. The mode argument can only be a string (also see the assertion in the `__init__`: `assert self.mode in ["single", "page", "object", "markdown"]`, so the check `elif self.mode == ["page"]:` always fails. As a result, effectively the "object" mode is used when selecting the "page" mode, which may lead to errors. The docstring of the `AzureAIDocumentIntelligenceLoader` also ommitted the `mode` parameter alltogether, so I added it. Issue: I could not find a related issue (this class is only 3 weeks old anyways) Dependencies: this PR does not introduce or affect dependencies. The current demo notebook and examples are not affected because they all use the default markdown mode.	8 months ago
Christophe Bornet	1f5f6381ec	Add doc for AstraDB document loader (#15703 ) <!-- Thank you for contributing to LangChain! Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified. Replace this entire comment with: - Description: a description of the change, - Issue: the issue # it fixes if applicable, - Dependencies: any dependencies required for this change, - Twitter handle: we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --> See preview : https://langchain-git-fork-cbornet-astra-loader-doc-langchain.vercel.app/docs/integrations/document_loaders/astradb	8 months ago
chyroc	a17a3638b5	Docs: fix excel document loader typo (#15470 )	8 months ago
chyroc	9ae901c5e6	Feat: add CHM file loader (#15519 ) fix https://github.com/langchain-ai/langchain/issues/15469	8 months ago
Chad Norvell	f6226d464e	community: Include PDF ID in MathPix metadata (#15629 ) - Description: Includes the PDF ID in the MathPix document metadata. This is useful in case you need to re-request a processed PDF from the MathPix API later.	8 months ago
Chad Norvell	d2a686b165	community: Provide more actionable errors in the MathPix PDF loader (#15630 ) - Description: The `error_info['id']` can be cross-referenced with the MathPix API documentation to get very specific information about why an error occurred.	8 months ago
Bagatur	fa5d49f2c1	docs, experimental[patch], langchain[patch], community[patch]: update storage imports (#15429 ) ran ```bash g grep -l "langchain.vectorstores" \| xargs -L 1 sed -i '' "s/langchain\.vectorstores/langchain_community.vectorstores/g" g grep -l "langchain.document_loaders" \| xargs -L 1 sed -i '' "s/langchain\.document_loaders/langchain_community.document_loaders/g" g grep -l "langchain.chat_loaders" \| xargs -L 1 sed -i '' "s/langchain\.chat_loaders/langchain_community.chat_loaders/g" g grep -l "langchain.document_transformers" \| xargs -L 1 sed -i '' "s/langchain\.document_transformers/langchain_community.document_transformers/g" g grep -l "langchain\.graphs" \| xargs -L 1 sed -i '' "s/langchain\.graphs/langchain_community.graphs/g" g grep -l "langchain\.memory\.chat_message_histories" \| xargs -L 1 sed -i '' "s/langchain\.memory\.chat_message_histories/langchain_community.chat_message_histories/g" gco master libs/langchain/tests/unit_tests//test_imports.py gco master libs/langchain/tests/unit_tests/*/test_public_api.py ```	8 months ago
Christophe Bornet	e2a8962ba6	Add AstraDB document loader (#14747 ) - Description: this adds the AstraDB document loader and an integration test - Twitter handle: cbornet_	8 months ago
Christian Janiake	be578f32be	community:Lazy load wikipedia dump file (#15111 ) Description: the MWDumpLoader implementation currently does not support the lazy_load method, and the files are usually very large. We are proposing refactoring the load function, extracting two private functions with the functionality of loading the dump file and parsing a single page, to reuse the code in the lazy_load implementation.	8 months ago
Phill Zarfos	35896faab7	community: correct spelling mistakes of "Suffle" and "reporoducibility" (#15172 ) - Description: Correct spelling mistakes of "Suffle" and "reporoducibility" in `DirectoryLoader` class - Issue: N/A - Dependencies: N/A - Twitter handle: N/A	9 months ago
Archan Ghosh	affa3e755a	Update arxiv.py with get_summaries_as_docs inside of Arxivloader (#14953 ) Added the call function get_summaries_as_docs inside of Arxivloader - Description: Added a function that returns the documents from get_summaries_as_docs, as the call signature is present in the parent file but never used from Arxivloader, this can be used from Arxivloader itself just like .load() as both the signatures are same. - Issue: Reduces time to load papers as no pdf is processed only metadata is pulled from Arxiv allowing users for faster load times on bulk loads. Users can then choose one or more paper and use ID directly with .load() to load pdf thereby loading all the contents of the paper.	9 months ago
Grzegorz Sajko	dc71fcfabf	corrected outdated link (#15053 ) <!-- Thank you for contributing to LangChain! Please title your PR "<package>: <description>", where <package> is whichever of langchain, community, core, experimental, etc. is being modified. Replace this entire comment with: - Description: a description of the change, - Issue: the issue # it fixes if applicable, - Dependencies: any dependencies required for this change, - Twitter handle: we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. -->	9 months ago
Ran	c3f8733aef	fix: correct spelling mistakes of "seperate, intialise, pre-defined" (#14647 ) fix spellings seperate -> separate: found more occurrences, see https://github.com/langchain-ai/langchain/pull/14602 initialise -> intialize: the latter is more common in the repo pre-defined > predefined: adding a comma after a prefix is a delicate matter, but this is a generally accepted word also, another word that appears in the repo is "fs" (stands for filesystem), e.g., in `libs/core/langchain_core/prompts/loading.py` ` """Unified method for loading a prompt from LangChainHub or local fs."""` Isn't "filesystem" better?	9 months ago
QIAN Zifei	2460f977c5	community[minor]: Azure DocumentIntelligenceLoader/Parser support update with latest SDK (#14389 ) - Description: Add DocumentIntelligenceLoader & DocumentIntelligenceParser implementation using the latest Azure Document Intelligence SDK with markdown support. The core logic resides in DocumentIntelligenceParser and DocumentIntelligenceLoader is a mere wrapper of the parser. The parser will takes api_endpoint and api_key and creates DocumentIntelligenceClient for the user. 4 parsing modes are supported: 1. Markdown (default) 2. Single 3. Page 4. Object UT and notebook are also updated accordingly. - Dependencies: Azure Document Intelligence SDK: azure-ai-documentintelligence [azure-sdk-for-python/sdk/documentintelligence/azure-ai-documentintelligence at 7c42462ac662522a6fd21b17d2a20f4cd40d0356 · Azure/azure-sdk-for-python (github.com)](https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Fazure-sdk-for-python%2Ftree%2F7c42462ac662522a6fd21b17d2a20f4cd40d0356%2Fsdk%2Fdocumentintelligence%2Fazure-ai-documentintelligence&data=05%7C01%7CZifei.Qian%40microsoft.com%7C298225aa3e31468a863108dbf07374ff%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638368150928704292%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=oE0Sl4HERnMKdbkV9KgBV46Z2xytcQAShdTWf7ZNl%2Bs%3D&reserved=0). --------- Co-authored-by: Erick Friis <erick@langchain.dev>	9 months ago
Leonid Ganeline	b2fd41331e	docs: docstrings `langchain_community` update (#14889 ) Addded missed docstrings. Fixed inconsistency in docstrings. Note CC @efriis There were PR errors on `langchain_experimental/prompt_injection_identifier/hugging_face_identifier.py` But, I didn't touch this file in this PR! Can it be some cache problems? I fixed this error.	9 months ago
Bagatur	ed58eeb9c5	community[major], core[patch], langchain[patch], experimental[patch]: Create langchain-community (#14463 ) Moved the following modules to new package langchain-community in a backwards compatible fashion: ``` mv langchain/langchain/adapters community/langchain_community mv langchain/langchain/callbacks community/langchain_community/callbacks mv langchain/langchain/chat_loaders community/langchain_community mv langchain/langchain/chat_models community/langchain_community mv langchain/langchain/document_loaders community/langchain_community mv langchain/langchain/docstore community/langchain_community mv langchain/langchain/document_transformers community/langchain_community mv langchain/langchain/embeddings community/langchain_community mv langchain/langchain/graphs community/langchain_community mv langchain/langchain/llms community/langchain_community mv langchain/langchain/memory/chat_message_histories community/langchain_community mv langchain/langchain/retrievers community/langchain_community mv langchain/langchain/storage community/langchain_community mv langchain/langchain/tools community/langchain_community mv langchain/langchain/utilities community/langchain_community mv langchain/langchain/vectorstores community/langchain_community mv langchain/langchain/agents/agent_toolkits community/langchain_community mv langchain/langchain/cache.py community/langchain_community mv langchain/langchain/adapters community/langchain_community mv langchain/langchain/callbacks community/langchain_community/callbacks mv langchain/langchain/chat_loaders community/langchain_community mv langchain/langchain/chat_models community/langchain_community mv langchain/langchain/document_loaders community/langchain_community mv langchain/langchain/docstore community/langchain_community mv langchain/langchain/document_transformers community/langchain_community mv langchain/langchain/embeddings community/langchain_community mv langchain/langchain/graphs community/langchain_community mv langchain/langchain/llms community/langchain_community mv langchain/langchain/memory/chat_message_histories community/langchain_community mv langchain/langchain/retrievers community/langchain_community mv langchain/langchain/storage community/langchain_community mv langchain/langchain/tools community/langchain_community mv langchain/langchain/utilities community/langchain_community mv langchain/langchain/vectorstores community/langchain_community mv langchain/langchain/agents/agent_toolkits community/langchain_community mv langchain/langchain/cache.py community/langchain_community ``` Moved the following to core ``` mv langchain/langchain/utils/json_schema.py core/langchain_core/utils mv langchain/langchain/utils/html.py core/langchain_core/utils mv langchain/langchain/utils/strings.py core/langchain_core/utils cat langchain/langchain/utils/env.py >> core/langchain_core/utils/env.py rm langchain/langchain/utils/env.py ``` See .scripts/community_split/script_integrations.sh for all changes	9 months ago

43 Commits (cd00a87db732687d0d6a9ffdc1eb9f6f5ebc49b6)