178 Commits (03178ee74ffe23aa05f3ba6e9a0253f57b59efc9)

Author	SHA1	Message	Date
Dristy Srivastava	ef3df45d9d	community[minor]: Updating payload for pebblo discover API (#22309) ... **Description:** Updating response for pebblo discover API. Also updating filed name case type **Documentation:** N/A **Unit tests:** N/A	4 months ago
Brandon Sharp	56e5aa4dd9	community[patch]: Airtable to allow for addtl params (#22092) ... - [X] **PR title**: "community: added optional params to Airtable table.all()" - [X] **PR message**: - **Description:** Add's **kwargs to AirtableLoader to allow for kwargs: https://pyairtable.readthedocs.io/en/latest/api.html#pyairtable.Table.all - **Issue:** N/A - **Dependencies:** N/A - **Twitter handle:** parakoopa88 - [X] **Add tests and docs**: If you're adding a new integration, please include 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. - [X] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17. --------- Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	4 months ago
Will Higgins	83d10df78d	community[patch]: Update firecrawl api key name (#22183) ... Change 'FIREWALL' to 'FIRECRAWL' as I believe this may have been in error. Other docs refer to 'FIRECRAWL_API_KEY'. Thank you for contributing to LangChain! - [ ] **PR title**: "package: description" - Where "package" is whichever of langchain, community, core, experimental, etc. is being modified. Use "docs: ..." for purely docs changes, "templates: ..." for template changes, "infra: ..." for CI changes. - Example: "community: add foobar LLM" - [ ] **PR message**: ***Delete this entire checklist*** and replace with - **Description:** a description of the change - **Issue:** the issue # it fixes, if applicable - **Dependencies:** any dependencies required for this change - **Twitter handle:** if your PR gets announced, and you'd like a mention, we'll gladly shout you out! - [ ] **Add tests and docs**: If you're adding a new integration, please include 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. - [ ] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ Additional guidelines: - Make sure optional dependencies are imported within a function. - Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests. - Most PRs should not touch more than one package. - Changes should be backwards compatible. - If you are adding something to community, do not re-import it in langchain. If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, ccurme, vbarda, hwchase17. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	4 months ago
Rahul Triptahi	1a485f59b9	community[patch]: Put authorized identities behind a feature flag in SharepointLoader (#22125) ... Description: Put authorised identities behind a feature flag, load_auth. Documentation: N/A Unit tests: N/A --------- Signed-off-by: Rahul Tripathi <rauhl.psit.ec@gmail.com> Co-authored-by: Rahul Tripathi <rauhl.psit.ec@gmail.com>	4 months ago
Eugene Yurtsev	d3db83abe3	community[major]: lint for usage of xml library (#22132) ... * Lint for usage of standard xml library * Add forced opt-in for quip client * Actual security issue is with underlying QuipClient not LangChain integration (since the client is doing the parsing), but adding enforcement at the LangChain level.	4 months ago
Martin Triska	2df8ac402a	community[minor]: Added propagation of document metadata from O365BaseLoader (#20663) ... **Description:** - Added propagation of document metadata from O365BaseLoader to FileSystemBlobLoader (O365BaseLoader uses FileSystemBlobLoader under the hood). - This is done by passing dictionary `metadata_dict`: key=filename and value=dictionary containing document's metadata - Modified `FileSystemBlobLoader` to accept the `metadata_dict`, use `mimetype` from it (if available) and pass metadata further into blob loader. **Issue:** - `O365BaseLoader` under the hood downloads documents to temp folder and then uses `FileSystemBlobLoader` on it. - However metadata about the document in question is lost in this process. In particular: - `mime_type`: `FileSystemBlobLoader` guesses `mime_type` from the file extension, but that does not work 100% of the time. - `web_url`: this is useful to keep around since in RAG LLM we might want to provide link to the source document. In order to work well with document parsers, we pass the `web_url` as `source` (`web_url` is ignored by parsers, `source` is preserved) **Dependencies:** None **Twitter handle:** @martintriska1 Please review @baskaryan --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	4 months ago
Eugene Yurtsev	e5541d1da7	community[patch]: Update doc-string in CloudBlobLoader (#22069) ... Update doc-string	4 months ago
Philippe PRADOS	6dd621d636	community[minor]: Add CloudBlobLoader that supports loading data from cloud buckets (#21957) ... Thank you for contributing to LangChain! - [ ] **PR title**: "Add CloudBlobLoader" - community: Add CloudBlobLoader - [ ] **PR message**: Add cloud blob loader - **Description:** Langchain provides several approaches to read different file formats: Specific loaders (`CVSLoader`) or blob-compatible loaders (`FileSystemBlobLoader`). The only implementation proposed for BlobLoader is `FileSystemBlobLoader`. Many projects retrieve files from cloud storage. We propose a new implementation of `BlobLoader` to read files from the three cloud storage systems. The interface is strictly identical to `FileSystemBlobLoader`. The only difference is the constructor, which takes a cloud "url" object such as `s3://my-bucket`, `az://my-bucket`, or `gs://my-bucket`. By streamlining the process, this novel implementation eliminates the requirement to pre-download files from cloud storage to local temporary files (which are seldom removed). The code relies on the [CloudPathLib](https://cloudpathlib.drivendata.org/stable/) library to interpret cloud URLs. This has been added as an optional dependency. ```Python loader = CloudBlobLoader("s3://mybucket/id") for blob in loader.yield_blobs(): print(blob) ``` - [X] **Dependencies:** CloudPathLib - [X] **Twitter handle:** pprados - [X] **Add tests and docs**: Add unit test, but it's easy to convert to integration test, with some files in a cloud storage (see `test_cloud_blob_loader.py`) - [X] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. Hello from Paris @hwchase17. Can you review this PR? --------- Co-authored-by: Eugene Yurtsev <eugene@langchain.dev>	4 months ago
Christophe Bornet	74947ec894	community[minor]: Add Cassandra ByteStore (#22064)	4 months ago
Bagatur	50186da0a1	infra: rm unused # noqa violations (#22049) ... Updating #21137	4 months ago
arpitkumar980	444c2a3d9f	community[patch]: sharepoint loader identity enabled (#21176) ... Thank you for contributing to LangChain! - [ ] **PR title**: "package: description" - Where "package" is whichever of langchain, community, core, experimental, etc. is being modified. Use "docs: ..." for purely docs changes, "templates: ..." for template changes, "infra: ..." for CI changes. - Example: "community: add foobar LLM" - [ ] **PR message**: ***Delete this entire checklist*** and replace with - **Description:** a description of the change - **Issue:** the issue # it fixes, if applicable - **Dependencies:** any dependencies required for this change - **Twitter handle:** if your PR gets announced, and you'd like a mention, we'll gladly shout you out! - [ ] **Add tests and docs**: If you're adding a new integration, please include 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. - [ ] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/ Additional guidelines:https://github.com/arpitkumar980/langchain.git - Make sure optional dependencies are imported within a function. - Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests. - Most PRs should not touch more than one package. - Changes should be backwards compatible. - If you are adding something to community, do not re-import it in langchain. If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17. --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	4 months ago
Mazen Ramadan	3c1d77dd64	community[minor]: Add Scrapfly Loader community integration (#22036) ... Added [Scrapfly](https://scrapfly.io/) Web Loader integration. Scrapfly is a web scraping API that allows extracting web page data into accessible markdown or text datasets. - __Description__: Added Scrapfly web loader for retrieving web page data as markdown or text. - Dependencies: scrapfly-sdk - Twitter: @thealchemi1st --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	4 months ago
maang-h	fc93bed8c4	community: Fix CSVLoader columns is None (#20701) ... - **Bug code**: In langchain_community/document_loaders/csv_loader.py:100 - **Description**: currently, when 'CSVLoader' reads the column as None in the 'csv' file, it will report an error because the 'CSVLoader' does not verify whether the column is of str type and does not consider how to handle the corresponding 'row_data' when the column is' None 'in the csv. This pr provides a solution. - **Issue:** Fix #20699 - **thinking:** 1. Refer to the processing method for 'langchain_community/document_loaders/csv_loader.py:100' when **'v'** equals'None', and apply the same method to '**k**'. (Reference`csv.DictReader` ,**'k'** will only be None when ` len(columns) < len(number_row_data)` is established) 2. **‘k’** equals None only holds when it is the last column, and its corresponding **'v'** type is a list. Therefore, I referred to the data format in 'Document' and used ',' to concatenated the elements in the list.(But I'm not sure if you accept this form, if you have any other ideas, communicate) --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	4 months ago
Sihan Chen	1f81277b9b	community[minor]: allow enabling proxy in aiohttp session in AsyncHTML (#19499) ... Allow enabling proxy in aiohttp session async html	4 months ago
Eugene Yurtsev	36813d2f00	community[patch]: Fix remaining __inits__ in community (#22037) ... Fixes the __init__ files in community to use __all__ which is statically defined.	4 months ago
Kefan You	24b5c27bb1	community[patch]: raise_for_status logic missing in async _fetch of WebBaseLoader (#21948) ... ## 'raise_for_status' parameter of WebBaseLoader works in sync load but not in async load. In webBaseLoader: Sync load is calling `_scrape` and has `raise_for_status` properly handled. ``` def _scrape( self, url: str, parser: Union[str, None] = None, bs_kwargs: Optional[dict] = None, ) -> Any: from bs4 import BeautifulSoup if parser is None: if url.endswith(".xml"): parser = "xml" else: parser = self.default_parser self._check_parser(parser) html_doc = self.session.get(url, **self.requests_kwargs) if self.raise_for_status: html_doc.raise_for_status() if self.encoding is not None: html_doc.encoding = self.encoding elif self.autoset_encoding: html_doc.encoding = html_doc.apparent_encoding return BeautifulSoup(html_doc.text, parser, **(bs_kwargs or {})) ``` Async load is calling `_fetch` but missing `raise_for_status` logic. ``` async def _fetch( self, url: str, retries: int = 3, cooldown: int = 2, backoff: float = 1.5 ) -> str: async with aiohttp.ClientSession() as session: for i in range(retries): try: async with session.get( url, headers=self.session.headers, ssl=None if self.session.verify else False, cookies=self.session.cookies.get_dict(), ) as response: return await response.text() ``` Co-authored-by: kefan.you <darkfss@sina.com>	4 months ago
Rahul Triptahi	96bd0b0844	community[patch]: Remove redundant pebblo cloud api call (#21589) ... Description: removed redundant pebblo cloud api call. Changed classified `doc` key to `ai_apps_data`. Documentation: N/A Unit tests: N/A	4 months ago
Jiří Spilka	6499897c87	community[patch]: update apify integration to attribute API activity to langchain (#21909) ... **Description:** Add `Origin/langchain` to Apify's client's user-agent to attribute API activity to LangChain (at Apify, we aim to monitor our integrations to evaluate whether we should invest more in the LangChain integration regarding functionality and content) **Issue:** None **Dependencies:** None **Twitter handle:** None	4 months ago
WilliamEspegren	30bca57aae	doc list not empty (#21208) ... Make sure the doc list is not empty, and set Metadata: true in param, to enable the user to disable metadata for slightly faster crawls.	4 months ago
Mish Ushakov	d77e60a7f4	community: updated Browserbase loader (#21757) ... Thank you for contributing to LangChain! - [x] **PR title**: "community: updated Browserbase loader" - [x] **PR message**: Updates the Browserbase loader with more options and improved docs. - [x] **Lint and test**: Run `make format`, `make lint` and `make test` from the root of the package(s) you've modified. See contribution guidelines for more: https://python.langchain.com/docs/contributing/	4 months ago
Eugene Yurtsev	25fbe356b4	community[patch]: upgrade to recent version of mypy (#21616) ... This PR upgrades community to a recent version of mypy. It inserts type: ignore on all existing failures.	4 months ago
Christophe Bornet	bcf53f93e1	[community]: Add missing docstring param to CassandraLoader (#21611)	4 months ago
Christophe Bornet	e6fa4547b1	community[minor]: Add alazy_load to AsyncHtmlLoader (#21536) ... Also fixes a bug that `_scrape` was called and was doing a second HTTP request synchronously. **Twitter handle:** cbornet_	4 months ago
roiperlman	9992beaff9	community: Add arguments to whisper parser (#20378) ... **Description:** Added a few additional arguments to the whisper parser, which can be consumed by the underlying API. The prompt is especially important to fine-tune transcriptions. --------- Co-authored-by: Roi Perlman <roi@fivesigmalabs.com> Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	4 months ago
Rohan Aggarwal	8021d2a2ab	community[minor]: Oraclevs integration (#21123) ... Thank you for contributing to LangChain! - Oracle AI Vector Search Oracle AI Vector Search is designed for Artificial Intelligence (AI) workloads that allows you to query data based on semantics, rather than keywords. One of the biggest benefit of Oracle AI Vector Search is that semantic search on unstructured data can be combined with relational search on business data in one single system. This is not only powerful but also significantly more effective because you don't need to add a specialized vector database, eliminating the pain of data fragmentation between multiple systems. - Oracle AI Vector Search is designed for Artificial Intelligence (AI) workloads that allows you to query data based on semantics, rather than keywords. One of the biggest benefit of Oracle AI Vector Search is that semantic search on unstructured data can be combined with relational search on business data in one single system. This is not only powerful but also significantly more effective because you don't need to add a specialized vector database, eliminating the pain of data fragmentation between multiple systems. This Pull Requests Adds the following functionalities Oracle AI Vector Search : Vector Store Oracle AI Vector Search : Document Loader Oracle AI Vector Search : Document Splitter Oracle AI Vector Search : Summary Oracle AI Vector Search : Oracle Embeddings - We have added unit tests and have our own local unit test suite which verifies all the code is correct. We have made sure to add guides for each of the components and one end to end guide that shows how the entire thing runs. - We have made sure that make format and make lint run clean. Additional guidelines: - Make sure optional dependencies are imported within a function. - Please do not add dependencies to pyproject.toml files (even optional ones) unless they are required for unit tests. - Most PRs should not touch more than one package. - Changes should be backwards compatible. - If you are adding something to community, do not re-import it in langchain. If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17. --------- Co-authored-by: skmishraoracle <shailendra.mishra@oracle.com> Co-authored-by: hroyofc <harichandan.roy@oracle.com> Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	5 months ago
ccurme	6da3d92b42	(all): update removal in deprecation warnings from 0.2 to 0.3 (#21265) ... We are pushing out the removal of these to 0.3. `find . -type f -name "*.py" -exec sed -i '' 's/removal="0\.2/removal="0.3/g' {} +`	5 months ago
Eugene Yurtsev	0e5bf16d00	langchain[patch]: Migrate document loaders to use optional langchain community imports (#21095)	5 months ago
Noah	45ddf4d26f	community[patch]: Update comments for lazy_load method (#21063) ... - [ ] **PR message**: - **Description:** Refactored the lazy_load method to use asynchronous execution for improved performance. The method now initiates scraping of all URLs simultaneously using asyncio.gather, enhancing data fetching efficiency. Each Document object is yielded immediately once its content becomes available, streamlining the entire process. - **Issue:** N/A - **Dependencies:** Requires the asyncio library for handling asynchronous tasks, which should already be part of standard Python libraries in Python 3.7 and above. - **Email:** [r73327118@gmail.com](mailto:r73327118@gmail.com) --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	5 months ago
fubuki8087	f1c3687aa5	community[patch]: Using the right encoding to parse the web page in RecursiveUrlLoader (#20632) ... As shown in #13749 , `RecursiveUrlLoader` has encoding issue. This PR is to solve this. --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	5 months ago
Charlie Marsh	8f38b7a725	multiple: Remove unnecessary Ruff suppression comments (#21050) ... ## Summary I ran `ruff check --extend-select RUF100 -n` to identify `# noqa` comments that weren't having any effect in Ruff, and then `ruff check --extend-select RUF100 -n --fix` on select files to remove all of the unnecessary `# noqa: F401` violations. It's possible that these were needed at some point in the past, but they're not necessary in Ruff v0.1.15 (used by LangChain) or in the latest release. Co-authored-by: Erick Friis <erick@langchain.dev>	5 months ago
Kuro Denjiro	fa4124b821	community[minor]: add mintbase loader to langchain (#20089) ... - [x] **Add Near NFT loader**: "community: Load NFT near block chain using mintbase graph API" - [x] **PR message**: - **Description:** a description of the change - **Twitter handle:**Kurodenjiro --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	5 months ago
Rahul Triptahi	c172611647	community[patch]: Add classifier_url argument in PebbloSafeLoader and documentation update. (#21030) ... Description: Add classifier_url argument in PebbloSafeLoader. Documentation: Updated PebbloSafeLoader documentation with above change and new links for pebblo github pages. --------- Signed-off-by: Rahul Tripathi <rauhl.psit.ec@gmail.com> Co-authored-by: Rahul Tripathi <rauhl.psit.ec@gmail.com>	5 months ago
Leonid Ganeline	85094cbb3a	docs: community docstring updates (#21040) ... Added missed docstrings. Updated docstrings to consistent format.	5 months ago
Pengcheng Liu	1fad39be1c	community[minor]: Add LarkSuite wiki document loader. (#21016) ... **Description:** Add LarkSuite wiki document loader. Refer to [LarkSuite api document ](https://open.feishu.cn/document/server-docs/docs/wiki-v2/space-node/list)for details. **Issue:** None **Dependencies:** None **Twitter handle:** None	5 months ago
Leonid Ganeline	dc7c06bc07	community[minor]: import fix (#20995) ... Issue: When the third-party package is not installed, whenever we need to `pip install <package>` the ImportError is raised. But sometimes, the `ValueError` or `ModuleNotFoundError` is raised. It is bad for consistency. Change: replaced the `ValueError` or `ModuleNotFoundError` with `ImportError` when we raise an error with the `pip install <package>` message. Note: Ideally, we replace all `try: import... except... raise ... `with helper functions like `import_aim` or just use the existing [langchain_core.utils.utils.guard_import](https://api.python.langchain.com/en/latest/utils/langchain_core.utils.utils.guard_import.html#langchain_core.utils.utils.guard_import) But it would be much bigger refactoring. @baskaryan Please, advice on this.	5 months ago
WilliamEspegren	804390ba4b	community: Spider integration (#20937) ... Added the [Spider.cloud](https://spider.cloud) document loader. [Spider](https://github.com/spider-rs/spider) is the [fastest](https://github.com/spider-rs/spider/blob/main/benches/BENCHMARKS.md) and cheapest crawler that returns LLM-ready data. ``` - **Description:** Adds Spider data loader - **Dependencies:** spider-client - **Twitter handle:** @WilliamEspegren ``` --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: = <=> Co-authored-by: Chester Curme <chester.curme@gmail.com>	5 months ago
Rahul Triptahi	955cf186d2	community[patch]: Ingest source, owner and full_path if present in Document's metadata. (#20949) ... Description: The PebbloSafeLoader should first check for owner, full_path and size in metadata before implementing its own logic. Dependencies: None Documentation: NA. Signed-off-by: Rahul Tripathi <rauhl.psit.ec@gmail.com> Co-authored-by: Rahul Tripathi <rauhl.psit.ec@gmail.com>	5 months ago
Dristy Srivastava	5f1d1666e3	community[patch]: Add support for pebblo server and client version (#20269) ... **Description**: _PebbloSafeLoader_: Add support for pebblo server and client version **Documentation:** NA **Unit test:** NA **Issue:** NA **Dependencies:** None --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	5 months ago
am-kinetica	b54b19ba1c	community[minor]: Implemented Kinetica Document Loader and added notebooks (#20002) ... - [ ] **Kinetica Document Loader**: "community: a class to load Documents from Kinetica" - [ ] **Kinetica Document Loader**: - **Description:** implemented KineticaLoader in `kinetica_loader.py` - **Dependencies:** install the Kinetica API using `pip install gpudb==7.2.0.1 `	5 months ago
Rahul Triptahi	dc921f0823	community[patch]: Add semantic info to metadata, classified by pebblo-server. (#20468) ... Description: Add support for Semantic topics and entities. Classification done by pebblo-server is not used to enhance metadata of Documents loaded by document loaders. Dependencies: None Documentation: Updated. Signed-off-by: Rahul Tripathi <rauhl.psit.ec@gmail.com> Co-authored-by: Rahul Tripathi <rauhl.psit.ec@gmail.com>	5 months ago
Lei Zhang	748a6ae609	community[patch]: add HTTP response headers Content-Type to metadata of RecursiveUrlLoader document (#20875) ... **Description:** The RecursiveUrlLoader loader offers a link_regex parameter that can filter out URLs. However, this filtering capability is limited, and if the internal links of the website change, unexpected resources may be loaded. These resources, such as font files, can cause problems in subsequent embedding processing. > https://blog.langchain.dev/assets/fonts/source-sans-pro-v21-latin-ext_latin-regular.woff2?v=0312715cbf We can add the Content-Type in the HTTP response headers to the document metadata so developers can choose which resources to use. This allows developers to make their own choices. For example, the following may be a good choice for text knowledge. - text/plain - simple text file - text/html - HTML web page - text/xml - XML format file - text/json - JSON format data - application/pdf - PDF file - application/msword - Word document and ignore the following - text/css - CSS stylesheet - text/javascript - JavaScript script - application/octet-stream - binary data - image/jpeg - JPEG image - image/png - PNG image - image/gif - GIF image - image/svg+xml - SVG image - audio/mpeg - MPEG audio files - video/mp4 - MP4 video file - application/font-woff - WOFF font file - application/font-ttf - TTF font file - application/zip - ZIP compressed file - application/octet-stream - binary data **Twitter handle:** @coolbeevip --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	5 months ago
Mish Ushakov	6ccecf2363	community[minor]: added Browserbase loader (#20478)	5 months ago
hulitaitai	7d0a008744	community[minor]: Add audio-parser "faster-whisper" in audio.py (#20012) ... faster-whisper is a reimplementation of OpenAI's Whisper model using CTranslate2, which is up to 4 times faster than enai/whisper for the same accuracy while using less memory. The efficiency can be further improved with 8-bit quantization on both CPU and GPU. It can automatically detect the following 14 languages and transcribe the text into their respective languages: en, zh, fr, de, ja, ko, ru, es, th, it, pt, vi, ar, tr. The gitbub repository for faster-whisper is : https://github.com/SYSTRAN/faster-whisper --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	5 months ago
Christophe Bornet	d2d01370bc	community[minor]: Add async methods to CassandraLoader (#20609) ... Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	5 months ago
Marco Perini	11c9ed3362	community[patch]: exposing headless flag parameter to AsyncChromiumLoader class (#20424) ... - **Description:** added the headless parameter as optional argument to the langchain_community.document_loaders AsyncChromiumLoader class - **Dependencies:** None - **Twitter handle:** @perinim_98 If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17. --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	5 months ago
Tomer Cagan	463160c3f6	community: fix `DirectoryLoader` progress bar (#19821) ... **Description:** currently, the `DirectoryLoader` progress-bar maximum value is based on an incorrect number of files to process In langchain_community/document_loaders/directory.py:127: ```python paths = p.rglob(self.glob) if self.recursive else p.glob(self.glob) items = [ path for path in paths if not (self.exclude and any(path.match(glob) for glob in self.exclude)) ] ``` `paths` returns both files and directories. `items` is later used to determine the maximum value of the progress-bar which gives an incorrect progress indication.	5 months ago
Rahul Triptahi	2cbfc94bcb	community[patch]: Add support for authorized identities in PebbloSafeLoader. (#20055) ... Description: Add support for authorized identities in PebbloSafeLoader. Now with this change, PebbloSafeLoader will extract authorized_identities from metadata and send it to pebblo server Dependencies: None Documentation: None Signed-off-by: Rahul Tripathi <rauhl.psit.ec@gmail.com> Co-authored-by: Rahul Tripathi <rauhl.psit.ec@gmail.com>	5 months ago
Hyeongchan Kim	7824291252	community[patch]: Fix not to cast to str type when `file_path` is None (#20057) ... From `langchain_community 0.0.30`, there's a bug that cannot send a file-like object via `file` parameter instead of `file path` due to casting the `file_path` to str type even if `file_path` is None. which means that when I call the `partition_via_api()`, exactly one of `filename` and `file` must be specified by the following error message. however, from `langchain_community 0.0.30`, `file_path` is casted into `str` type even `file_path` is None in `get_elements_from_api()` and got an error at `exactly_one(filename=filename, file=file)`. here's an error message ``` ---> 51 exactly_one(filename=filename, file=file) 53 if metadata_filename and file_filename: 54 raise ValueError( 55 "Only one of metadata_filename and file_filename is specified. " 56 "metadata_filename is preferred. file_filename is marked for deprecation.", 57 ) File /opt/homebrew/lib/python3.11/site-packages/unstructured/partition/common.py:441, in exactly_one(**kwargs) 439 else: 440 message = f"{names[0]} must be specified." --> 441 raise ValueError(message) ValueError: Exactly one of filename and file must be specified. ``` So, I simply made a change that casting to str type when `file_path` is not None. I use `UnstructuredAPIFileLoader` like below. ``` from langchain_community.document_loaders.unstructured import UnstructuredAPIFileLoader documents: list = UnstructuredAPIFileLoader( file_path=None, file=file, # file-like object, io.BytesIO type mode='elements', url='http://127.0.0.1:8000/general/v0/general', content_type='application/pdf', metadata_filename='asdf.pdf', ).load_and_split() ```	5 months ago
MacanPN	bce69ae43d	community[patch]: Changes to base_o365 and sharepoint document loaders (#20373) ... ## Description: The PR introduces 3 changes: 1. added `recursive` property to `O365BaseLoader`. (To keep the behavior unchanged, by default is set to `False`). When `recursive=True`, `_load_from_folder()` also recursively loads all nested folders. 2. added `folder_id` to SharePointLoader.(similar to (this PR)[https://github.com/langchain-ai/langchain/pull/10780] ) This provides an alternative to `folder_path` that doesn't seem to reliably work. 3. when none of `document_ids`, `folder_id`, `folder_path` is provided, the loader fetches documets from root folder. Combined with `recursive=True` this provides an easy way of loading all compatible documents from SharePoint. The PR contains the same logic as [this stale PR](https://github.com/langchain-ai/langchain/pull/10780) by @WaleedAlfaris. I'd like to ask his blessing for moving forward with this one. ## Issue: - As described in https://github.com/langchain-ai/langchain/issues/19938 and https://github.com/langchain-ai/langchain/pull/10780 the sharepoint loader often does not seem to work with folder_path. - Recursive loading of subfolders is a missing functionality ## Dependecies: None Twitter handle: @martintriska1 @WRhetoric This is my first PR here, please be gentle :-) Please review @baskaryan --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	5 months ago
Ravindu Somawansa	5acc7ba622	community[minor]: Add glue catalog loader (#20220) ... Add Glue Catalog loader	5 months ago