19 Commits (70fd7cda1446e80f439e1230dfe92e8616576c87)

Author	SHA1	Message	Date
Eugene Yurtsev	d3300bd799	YouTube Loader: Replace regexp with built-in parsing (#4729)	1 year ago
Eugene Yurtsev	3c490b5ba3	Docugami DataLoader (#4727) ... ### Adds a document loader for Docugami Specifically: 1. Adds a data loader that talks to the [Docugami](http://docugami.com) API to download processed documents as semantic XML 2. Parses the semantic XML into chunks, with additional metadata capturing chunk semantics 3. Adds a detailed notebook showing how you can use additional metadata returned by Docugami for techniques like the [self-querying retriever](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query_retriever.html) 4. Adds an integration test, and related documentation Here is an example of a result that is not possible without the capabilities added by Docugami (from the notebook): <img width="1585" alt="image" src="https://github.com/hwchase17/langchain/assets/749277/bb6c1ce3-13dc-4349-a53b-de16681fdd5b"> --------- Co-authored-by: Taqi Jaffri <tjaffri@docugami.com> Co-authored-by: Taqi Jaffri <tjaffri@gmail.com>	1 year ago
KNiski	c2761aa8f4	Improve video_id extraction in YoutubeLoader (#4452) ... # Improve video_id extraction in `YoutubeLoader` `YoutubeLoader.from_youtube_url` can only deal with one specific url format. I've introduced `YoutubeLoader.extract_video_id` which can extract video id from common YT urls. Fixes #4451 @eyurtsev --------- Co-authored-by: Kamil Niski <kamil.niski@gmail.com>	1 year ago
Lester Yang	cd3f9865f3	Feature: pdfplumber PDF loader with BaseBlobParser (#4552) ... # Feature: pdfplumber PDF loader with BaseBlobParser * Adds pdfplumber as a PDF loader * Adds pdfplumber as a blob parser.	1 year ago
Li Yuanzheng	3b6206af49	Respect User-Specified User-Agent in WebBaseLoader (#4579) ... # Respect User-Specified User-Agent in WebBaseLoader This pull request modifies the `WebBaseLoader` class initializer from the `langchain.document_loaders.web_base` module to preserve any User-Agent specified by the user in the `header_template` parameter. Previously, even if a User-Agent was specified in `header_template`, it would always be overridden by a random User-Agent generated by the `fake_useragent` library. With this change, if a User-Agent is specified in `header_template`, it will be used. Only in the case where no User-Agent is specified will a random User-Agent be generated and used. This provides additional flexibility when using the `WebBaseLoader` class, allowing users to specify their own User-Agent if they have a specific need or preference, while still providing a reasonable default for cases where no User-Agent is specified. This change has no impact on existing users who do not specify a User-Agent, as the behavior in this case remains the same. However, for users who do specify a User-Agent, their choice will now be respected and used for all subsequent requests made using the `WebBaseLoader` class. Fixes #4167 ## Before submitting ============================= test session starts ============================== collecting ... collected 1 item test_web_base.py::TestWebBaseLoader::test_respect_user_specified_user_agent ============================== 1 passed in 3.64s =============================== PASSED [100%] ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: @eyurtsev --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	1 year ago
Harrison Chase	cdc20d1203	Harrison/json loader fix (#4686) ... Co-authored-by: Triet Le <112841660+triet-lq-holistics@users.noreply.github.com>	1 year ago
Paresh Mathur	e2bc836571	Fix #4087 by setting the correct csv dialect (#4103) ... The error in #4087 was happening because of the use of csv.Dialect.* which is just an empty base class. we need to make a choice on what is our base dialect. I usually use excel so I put it as excel, if maintainers have other preferences do let me know. Open Questions: 1. What should be the default dialect? 2. Should we rework all tests to mock the open function rather than the csv.DictReader? 3. Should we make a separate input for `dialect` like we have for `encoding`? --------- Co-authored-by: = <=>	1 year ago
Eugene Yurtsev	08ed927c32	Turn on extended tests (#4588) ... # Turn on strict extended tests This PR turns on strict testing for extended tests.	1 year ago
Eugene Yurtsev	80558b5b27	Add workflow for testing with all deps (#4410) ... # Add action to test with all dependencies installed PR adds a custom action for setting up poetry that allows specifying a cache key: https://github.com/actions/setup-python/issues/505#issuecomment-1273013236 This makes it possible to run 2 types of unit tests: (1) unit tests with only core dependencies (2) unit tests with extended dependencies (e.g., those that rely on an optional pdf parsing library) As part of this PR, we're moving some pdf parsing tests into the unit-tests section and making sure that these unit tests get executed when running with extended dependencies.	1 year ago
Eugene Yurtsev	ae0c3382dd	Add MimeType based parser (#4376) ... # Add MimeType Based Parser This PR adds a MimeType Based Parser. The parser inspects the mime-type of the blob it is parsing and based on the mime-type can delegate to the sub parser. ## Before submitting Waiting on adding notebooks until more implementations are landed. ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: @hwchase17 @vowelparrot	1 year ago
Eugene Yurtsev	aa11f7c89b	Add progress bar to filesystemblob loader, update pytest config for unit tests (#4212) ... This PR adds: * Option to show a tqdm progress bar when using the file system blob loader * Update pytest run configuration to be stricter * Adding a new marker that checks that required pkgs exist	1 year ago
Eugene Yurtsev	423f497168	Add BlobParser abstraction (#3979) ... This PR adds the BlobParser abstraction. It follows the proposal described here: https://github.com/hwchase17/langchain/pull/2833#issuecomment-1509097756	1 year ago
Eugene Yurtsev	7cce68a051	Add minimal file system blob loader (#3669) ... This adds a minimal file system blob loader. If looks good, this PR will be merged and a few additional enhancements will be made.	1 year ago
Harrison Chase	bd7e0a534c	Harrison/csv loader (#3771) ... Co-authored-by: mrT23 <tal.r@codium.ai>	1 year ago
Eugene Yurtsev	708787dddb	Blob: Add validator and use future annotations (#3650) ... Minor changes to the Blob schema. --------- Co-authored-by: Zander Chase <130414180+vowelparrot@users.noreply.github.com>	1 year ago
Eugene Yurtsev	5d02010763	Introduce Blob and Blob Loader interface (#3603) ... This PR introduces a Blob data type and a Blob loader interface. This is the first of a sequence of PRs that follows this proposal: https://github.com/hwchase17/langchain/pull/2833 The primary goals of these abstraction are: * Decouple content loading from content parsing code. * Help duplicated content loading code from document loaders. * Make lazy loading a default for langchain.	1 year ago
Harrison Chase	f281033362	rm pandas dependency (#2102)	2 years ago
Harrison Chase	30e3b31b04	Harrison/document cleanup (#2062) ... Co-authored-by: Delip Rao <delip@users.noreply.github.com>	2 years ago
Harrison Chase	cc423f40f1	Harrison/youtube loader (#1545) ... Co-authored-by: Julian Wustl <57504258+Julianwustl@users.noreply.github.com>	2 years ago