Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-06 03:20:49 +00:00
Code Issues Packages Projects Releases Wiki Activity
bc66b3fb8d
langchain/docs/integrations/unstructured.md

50 lines
2.2 KiB
Markdown
Raw Normal View History

docs: add an unstructured section to the ecosystem page (#1125) ### Summary Adds an Unstructured section to the ecosystem page.
2023-02-17 21:02:23 +00:00
# Unstructured
docs: cleaning (#5413) # docs cleaning Changed docs to consistent format (probably, we need an official doc integration template): - ClearML - added product descriptions; changed title/headers - Rebuff - added product descriptions; changed title/headers - WhyLabs - added product descriptions; changed title/headers - Docugami - changed title/headers/structure - Airbyte - fixed title - Wolfram Alpha - added descriptions, fixed title - OpenWeatherMap - - added product descriptions; changed title/headers - Unstructured - changed description ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: @hwchase17 @dev2049
2023-05-30 20:58:16 +00:00
>The `unstructured` package from
docs: add an unstructured section to the ecosystem page (#1125) ### Summary Adds an Unstructured section to the ecosystem page.
2023-02-17 21:02:23 +00:00
[Unstructured.IO](https://www.unstructured.io/) extracts clean text from raw source documents like
PDFs and Word documents.
docs: cleaning (#5413) # docs cleaning Changed docs to consistent format (probably, we need an official doc integration template): - ClearML - added product descriptions; changed title/headers - Rebuff - added product descriptions; changed title/headers - WhyLabs - added product descriptions; changed title/headers - Docugami - changed title/headers/structure - Airbyte - fixed title - Wolfram Alpha - added descriptions, fixed title - OpenWeatherMap - - added product descriptions; changed title/headers - Unstructured - changed description ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: @hwchase17 @dev2049
2023-05-30 20:58:16 +00:00
This page covers how to use the [`unstructured`](https://github.com/Unstructured-IO/unstructured)
docs: unstructured no longer requires installing detectron2 from source (#5524) # Update Unstructured docs to remove the `detectron2` install instructions Removes `detectron2` installation instructions from the Unstructured docs because installing `detectron2` is no longer required for `unstructured>=0.7.0`. The `detectron2` model now runs using the ONNX runtime. ## Who can review? @hwchase17 @eyurtsev
2023-05-31 22:03:21 +00:00
ecosystem within LangChain.
docs: add an unstructured section to the ecosystem page (#1125) ### Summary Adds an Unstructured section to the ecosystem page.
2023-02-17 21:02:23 +00:00
## Installation and Setup
feat: add Unstructured API loaders (#3906) ### Summary Adds `UnstructuredAPIFileLoaders` and `UnstructuredAPIFIleIOLoaders` that partition documents through the Unstructured API. Defaults to the URL for hosted Unstructured API, but can switch to a self hosted or locally running API using the `url` kwarg. Currently, the Unstructured API is open and does not require an API, but it will soon. A note was added about that to the Unstructured ecosystem page. ### Testing ```python from langchain.document_loaders import UnstructuredAPIFileIOLoader filename = "fake-email.eml" with open(filename, "rb") as f: loader = UnstructuredAPIFileIOLoader(file=f, file_filename=filename) docs = loader.load() docs[0] ``` ```python from langchain.document_loaders import UnstructuredAPIFileLoader filename = "fake-email.eml" loader = UnstructuredAPIFileLoader(file_path=filename, mode="elements") docs = loader.load() docs[0] ```
2023-05-02 03:37:35 +00:00
If you are using a loader that runs locally, use the following steps to get `unstructured` and
its dependencies running locally.
docs: add quotes to `unstructured[local-inference]` install instructions (#1208) ### Summary Corrects the install instruction for local inference to `pip install "unstructured[local-inference]"`
2023-02-21 16:06:43 +00:00
- Install the Python SDK with `pip install "unstructured[local-inference]"`
docs: add an unstructured section to the ecosystem page (#1125) ### Summary Adds an Unstructured section to the ecosystem page.
2023-02-17 21:02:23 +00:00
- Install the following system dependencies if they are not already available on your system.
Depending on what document types you're parsing, you may not need all of these.
feat: document loader for epublications (#2202) ### Summary Adds a new document loader for processing e-publications. Works with `unstructured>=0.5.4`. You need to have [`pandoc`](https://pandoc.org/installing.html) installed for this loader to work. ### Testing ```python from langchain.document_loaders import UnstructuredEPubLoader loader = UnstructuredEPubLoader("winter-sports.epub", mode="elements") data = loader.load() data[0] ```
2023-03-31 03:45:31 +00:00
- `libmagic-dev` (filetype detection)
- `poppler-utils` (images and PDFs)
- `tesseract-ocr`(images and PDFs)
- `libreoffice` (MS Office docs)
- `pandoc` (EPUBs)
docs: add an unstructured section to the ecosystem page (#1125) ### Summary Adds an Unstructured section to the ecosystem page.
2023-02-17 21:02:23 +00:00
feat: add Unstructured API loaders (#3906) ### Summary Adds `UnstructuredAPIFileLoaders` and `UnstructuredAPIFIleIOLoaders` that partition documents through the Unstructured API. Defaults to the URL for hosted Unstructured API, but can switch to a self hosted or locally running API using the `url` kwarg. Currently, the Unstructured API is open and does not require an API, but it will soon. A note was added about that to the Unstructured ecosystem page. ### Testing ```python from langchain.document_loaders import UnstructuredAPIFileIOLoader filename = "fake-email.eml" with open(filename, "rb") as f: loader = UnstructuredAPIFileIOLoader(file=f, file_filename=filename) docs = loader.load() docs[0] ``` ```python from langchain.document_loaders import UnstructuredAPIFileLoader filename = "fake-email.eml" loader = UnstructuredAPIFileLoader(file_path=filename, mode="elements") docs = loader.load() docs[0] ```
2023-05-02 03:37:35 +00:00
If you want to get up and running with less set up, you can
simply run `pip install unstructured` and use `UnstructuredAPIFileLoader` or
`UnstructuredAPIFileIOLoader`. That will process your document using the hosted Unstructured API.
Note that currently (as of 1 May 2023) the Unstructured API is open, but it will soon require
an API. The [Unstructured documentation page](https://unstructured-io.github.io/) will have
instructions on how to generate an API key once they're available. Check out the instructions
[here](https://github.com/Unstructured-IO/unstructured-api#dizzy-instructions-for-using-the-docker-image)
if you'd like to self-host the Unstructured API or run it locally.
docs: add an unstructured section to the ecosystem page (#1125) ### Summary Adds an Unstructured section to the ecosystem page.
2023-02-17 21:02:23 +00:00
## Wrappers
### Data Loaders
The primary `unstructured` wrappers within `langchain` are data loaders. The following
shows how to use the most basic unstructured data loader. There are other file-specific
data loaders available in the `langchain.document_loaders` module.
```python
from langchain.document_loaders import UnstructuredFileLoader
loader = UnstructuredFileLoader("state_of_the_union.txt")
loader.load()
```
If you instantiate the loader with `UnstructuredFileLoader(mode="elements")`, the loader
will track additional metadata like the page number and text type (i.e. title, narrative text)
when that information is available.
Reference in New Issue Copy Permalink