blob42/langchain
1
0
Fork 0
forked from Archives/langchain
Code Issues Pull Requests Packages Projects Releases Wiki Activity
e2d7677526
langchain/docs/integrations/unstructured.md
Leonid Ganeline e2d7677526
docs: compound ecosystem and integrations (#4870) 
# Docs: compound ecosystem and integrations

**Problem statement:** We have a big overlap between the
References/Integrations and Ecosystem/LongChain Ecosystem pages. It
confuses users. It creates a situation when new integration is added
only on one of these pages, which creates even more confusion.
- removed References/Integrations page (but move all its information
into the individual integration pages - in the next PR).
- renamed Ecosystem/LongChain Ecosystem into Integrations/Integrations.
I like the Ecosystem term. It is more generic and semantically richer
than the Integration term. But it mentally overloads users. The
`integration` term is more concrete.
UPDATE: after discussion, the Ecosystem is the term.
Ecosystem/Integrations is the page (in place of Ecosystem/LongChain
Ecosystem).

As a result, a user gets a single place to start with the individual
integration.
2023-05-18 09:29:57 -07:00

2.7 KiB
Raw Blame History

Unstructured

This page covers how to use the unstructured ecosystem within LangChain. The unstructured package from Unstructured.IO extracts clean text from raw source documents like PDFs and Word documents.

This page is broken into two parts: installation and setup, and then references to specific unstructured wrappers.

Installation and Setup

If you are using a loader that runs locally, use the following steps to get unstructured and its dependencies running locally.

  • Install the Python SDK with pip install "unstructured[local-inference]"
  • Install the following system dependencies if they are not already available on your system. Depending on what document types you're parsing, you may not need all of these.
    • libmagic-dev (filetype detection)
    • poppler-utils (images and PDFs)
    • tesseract-ocr(images and PDFs)
    • libreoffice (MS Office docs)
    • pandoc (EPUBs)
  • If you are parsing PDFs using the "hi_res" strategy, run the following to install the detectron2 model, which unstructured uses for layout detection:
    • pip install "detectron2@git+https://github.com/facebookresearch/detectron2.git@e2ce8dc#egg=detectron2"
    • If detectron2 is not installed, unstructured will fallback to processing PDFs using the "fast" strategy, which uses pdfminer directly and doesn't require detectron2.

If you want to get up and running with less set up, you can simply run pip install unstructured and use UnstructuredAPIFileLoader or UnstructuredAPIFileIOLoader. That will process your document using the hosted Unstructured API. Note that currently (as of 1 May 2023) the Unstructured API is open, but it will soon require an API. The Unstructured documentation page will have instructions on how to generate an API key once they're available. Check out the instructions here if you'd like to self-host the Unstructured API or run it locally.

Wrappers

Data Loaders

The primary unstructured wrappers within langchain are data loaders. The following shows how to use the most basic unstructured data loader. There are other file-specific data loaders available in the langchain.document_loaders module.

from langchain.document_loaders import UnstructuredFileLoader

loader = UnstructuredFileLoader("state_of_the_union.txt")
loader.load()

If you instantiate the loader with UnstructuredFileLoader(mode="elements"), the loader will track additional metadata like the page number and text type (i.e. title, narrative text) when that information is available.