2023-02-17 21:02:23 +00:00
|
|
|
# Unstructured
|
|
|
|
|
|
|
|
|
|
This page covers how to use the [`unstructured`](https://github.com/Unstructured-IO/unstructured)
|
|
|
|
|
ecosystem within LangChain. The `unstructured` package from
|
|
|
|
|
[Unstructured.IO](https://www.unstructured.io/) extracts clean text from raw source documents like
|
|
|
|
|
PDFs and Word documents.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
This page is broken into two parts: installation and setup, and then references to specific
|
|
|
|
|
`unstructured` wrappers.
|
|
|
|
|
|
|
|
|
|
## Installation and Setup
|
2023-05-02 03:37:35 +00:00
|
|
|
|
|
|
|
|
If you are using a loader that runs locally, use the following steps to get `unstructured` and
|
|
|
|
|
its dependencies running locally.
|
|
|
|
|
|
2023-02-21 16:06:43 +00:00
|
|
|
- Install the Python SDK with `pip install "unstructured[local-inference]"`
|
2023-02-17 21:02:23 +00:00
|
|
|
- Install the following system dependencies if they are not already available on your system.
|
|
|
|
|
Depending on what document types you're parsing, you may not need all of these.
|
2023-03-31 03:45:31 +00:00
|
|
|
- `libmagic-dev` (filetype detection)
|
|
|
|
|
- `poppler-utils` (images and PDFs)
|
|
|
|
|
- `tesseract-ocr`(images and PDFs)
|
|
|
|
|
- `libreoffice` (MS Office docs)
|
|
|
|
|
- `pandoc` (EPUBs)
|
feat: allow the unstructured kwargs to be passed in to Unstructured document loaders (#1667)
### Summary
Allows users to pass in `**unstructured_kwargs` to Unstructured document
loaders. Implemented with the `strategy` kwargs in mind, but will pass
in other kwargs like `include_page_breaks` as well. The two currently
supported strategies are `"hi_res"`, which is more accurate but takes
longer, and `"fast"`, which processes faster but with lower accuracy.
The `"hi_res"` strategy is the default. For PDFs, if `detectron2` is not
available and the user selects `"hi_res"`, the loader will fallback to
using the `"fast"` strategy.
### Testing
#### Make sure the `strategy` kwarg works
Run the following in iPython to verify that the `"fast"` strategy is
indeed faster.
```python
from langchain.document_loaders import UnstructuredFileLoader
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")
%timeit loader.load()
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
%timeit loader.load()
```
On my system I get:
```python
In [3]: from langchain.document_loaders import UnstructuredFileLoader
In [4]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")
In [5]: %timeit loader.load()
247 ms ± 369 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [6]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
In [7]: %timeit loader.load()
2.45 s ± 31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
#### Make sure older versions of `unstructured` still work
Run `pip install unstructured==0.5.3` and then verify the following runs
without error:
```python
from langchain.document_loaders import UnstructuredFileLoader
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
loader.load()
```
2023-03-15 01:15:28 +00:00
|
|
|
- If you are parsing PDFs using the `"hi_res"` strategy, run the following to install the `detectron2` model, which
|
2023-02-17 21:02:23 +00:00
|
|
|
`unstructured` uses for layout detection:
|
2023-04-06 19:48:19 +00:00
|
|
|
- `pip install "detectron2@git+https://github.com/facebookresearch/detectron2.git@e2ce8dc#egg=detectron2"`
|
feat: allow the unstructured kwargs to be passed in to Unstructured document loaders (#1667)
### Summary
Allows users to pass in `**unstructured_kwargs` to Unstructured document
loaders. Implemented with the `strategy` kwargs in mind, but will pass
in other kwargs like `include_page_breaks` as well. The two currently
supported strategies are `"hi_res"`, which is more accurate but takes
longer, and `"fast"`, which processes faster but with lower accuracy.
The `"hi_res"` strategy is the default. For PDFs, if `detectron2` is not
available and the user selects `"hi_res"`, the loader will fallback to
using the `"fast"` strategy.
### Testing
#### Make sure the `strategy` kwarg works
Run the following in iPython to verify that the `"fast"` strategy is
indeed faster.
```python
from langchain.document_loaders import UnstructuredFileLoader
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")
%timeit loader.load()
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
%timeit loader.load()
```
On my system I get:
```python
In [3]: from langchain.document_loaders import UnstructuredFileLoader
In [4]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", strategy="fast", mode="elements")
In [5]: %timeit loader.load()
247 ms ± 369 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
In [6]: loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
In [7]: %timeit loader.load()
2.45 s ± 31 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
#### Make sure older versions of `unstructured` still work
Run `pip install unstructured==0.5.3` and then verify the following runs
without error:
```python
from langchain.document_loaders import UnstructuredFileLoader
loader = UnstructuredFileLoader("layout-parser-paper-fast.pdf", mode="elements")
loader.load()
```
2023-03-15 01:15:28 +00:00
|
|
|
- If `detectron2` is not installed, `unstructured` will fallback to processing PDFs
|
|
|
|
|
using the `"fast"` strategy, which uses `pdfminer` directly and doesn't require
|
|
|
|
|
`detectron2`.
|
2023-02-17 21:02:23 +00:00
|
|
|
|
2023-05-02 03:37:35 +00:00
|
|
|
If you want to get up and running with less set up, you can
|
|
|
|
|
simply run `pip install unstructured` and use `UnstructuredAPIFileLoader` or
|
|
|
|
|
`UnstructuredAPIFileIOLoader`. That will process your document using the hosted Unstructured API.
|
|
|
|
|
Note that currently (as of 1 May 2023) the Unstructured API is open, but it will soon require
|
|
|
|
|
an API. The [Unstructured documentation page](https://unstructured-io.github.io/) will have
|
|
|
|
|
instructions on how to generate an API key once they're available. Check out the instructions
|
|
|
|
|
[here](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image)
|
|
|
|
|
if you'd like to self-host the Unstructured API or run it locally.
|
|
|
|
|
|
2023-02-17 21:02:23 +00:00
|
|
|
## Wrappers
|
|
|
|
|
|
|
|
|
|
### Data Loaders
|
|
|
|
|
|
|
|
|
|
The primary `unstructured` wrappers within `langchain` are data loaders. The following
|
|
|
|
|
shows how to use the most basic unstructured data loader. There are other file-specific
|
|
|
|
|
data loaders available in the `langchain.document_loaders` module.
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
from langchain.document_loaders import UnstructuredFileLoader
|
|
|
|
|
|
|
|
|
|
loader = UnstructuredFileLoader("state_of_the_union.txt")
|
|
|
|
|
loader.load()
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
If you instantiate the loader with `UnstructuredFileLoader(mode="elements")`, the loader
|
|
|
|
|
will track additional metadata like the page number and text type (i.e. title, narrative text)
|
|
|
|
|
when that information is available.
|
