7 Commits (4e42c737f835ae202f78f3737d9a92cf8f2f0d8e)

Author	SHA1	Message	Date
Matt Robinson	c51dec5101	feat: add Unstructured API loaders (#3906) ... ### Summary Adds `UnstructuredAPIFileLoaders` and `UnstructuredAPIFIleIOLoaders` that partition documents through the Unstructured API. Defaults to the URL for hosted Unstructured API, but can switch to a self hosted or locally running API using the `url` kwarg. Currently, the Unstructured API is open and does not require an API, but it will soon. A note was added about that to the Unstructured ecosystem page. ### Testing ```python from langchain.document_loaders import UnstructuredAPIFileIOLoader filename = "fake-email.eml" with open(filename, "rb") as f: loader = UnstructuredAPIFileIOLoader(file=f, file_filename=filename) docs = loader.load() docs[0] ``` ```python from langchain.document_loaders import UnstructuredAPIFileLoader filename = "fake-email.eml" loader = UnstructuredAPIFileLoader(file_path=filename, mode="elements") docs = loader.load() docs[0] ```	1 year ago
qued	5b34931948	docs: update unstructured detectron install instructions (#2498) ... Updated recommended `detectron2` version to install for use with `unstructured`. Should now match version in [Unstructured README](https://github.com/Unstructured-IO/unstructured/blob/main/README.md#eight_pointed_black_star-quick-start).	1 year ago
Matt Robinson	3dfe1cf60e	feat: document loader for epublications (#2202) ... ### Summary Adds a new document loader for processing e-publications. Works with `unstructured>=0.5.4`. You need to have [`pandoc`](https://pandoc.org/installing.html) installed for this loader to work. ### Testing ```python from langchain.document_loaders import UnstructuredEPubLoader loader = UnstructuredEPubLoader("winter-sports.epub", mode="elements") data = loader.load() data[0] ```	1 year ago
Matt Robinson	63aa28e2a6	feat: allow the unstructured kwargs to be passed in to Unstructured document loaders (#1667) ... ### Summary Allows users to pass in `**unstructured_kwargs` to Unstructured document loaders. Implemented with the `strategy` kwargs in mind, but will pass in other kwargs like `include_page_breaks` as well. The two currently supported strategies are `"hi_res"`, which is more accurate but takes longer, and `"fast"`, which processes faster but with lower accuracy. The `"hi_res"` strategy is the default. For PDFs, if `detectron2` is not available and the user selects `"hi_res"`, the loader will fallback to using the `"fast"` strategy. ### Testing #### Make sure the `strategy` kwarg works Run the following in iPython to verify that the `"fast"` strategy is indeed faster. ```python from langchain.document_loaders import UnstructuredFileLoader loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements") %timeit loader.load() loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements") %timeit loader.load() ``` On my system I get: ```python In [3]: from langchain.document_loaders import UnstructuredFileLoader In [4]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements") In [5]: %timeit loader.load() 247 ms ± 369 µs per loop (mean ± std. dev. of 7 runs, 1 loop each) In [6]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements") In [7]: %timeit loader.load() 2.45 s ± 31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) ``` #### Make sure older versions of `unstructured` still work Run `pip install unstructured==0.5.3` and then verify the following runs without error: ```python from langchain.document_loaders import UnstructuredFileLoader loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements") loader.load() ```	1 year ago
Matt Robinson	10e73a3723	docs: remove nltk download steps (#1253) ... ### Summary Updates the docs to remove the `nltk` download steps from `unstructured`. As of `unstructured` `0.4.14`, this is handled automatically in the relevant modules within `unstructured`.	1 year ago
Matt Robinson	3d5f56a8a1	docs: add quotes to `unstructured[local-inference]` install instructions (#1208) ... ### Summary Corrects the install instruction for local inference to `pip install "unstructured[local-inference]"`	1 year ago
Matt Robinson	b956070f08	docs: add an unstructured section to the ecosystem page (#1125) ... ### Summary Adds an Unstructured section to the ecosystem page.	1 year ago