Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-06 03:20:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

Unstructured example notebook: add a pdf, related deps (#1011)

Browse Source 
Updates the Unstructured example notebook with a PDF example. Includes
additional dependencies for PDF processing (and images, etc).
This commit is contained in:
cragwolfe 2023-02-12 14:56:48 -08:00 committed by GitHub
parent 03e5794978
commit 05d8969c79
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 75 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

77
docs/modules/document_loaders/examples/unstructured_file.ipynb
View File

@ -17,7 +17,9 @@
"outputs": [],
"source": [
"# # Install package\n",
"# !pip install unstructured"
"!pip install unstructured[local-inference]\n",
"!pip install \"detectron2@git+https://github.com/facebookresearch/detectron2.git@v0.6#egg=detectron2\"\n",
"!pip install layoutparser[layoutmodels,tesseract]"
]
},
{
@ -29,7 +31,12 @@
"source": [
"# # Install other dependencies\n",
"# # https://github.com/Unstructured-IO/unstructured/blob/main/docs/source/installing.rst\n",
"# !brew install libmagic"
"# !brew install libmagic\n",
"# !brew install poppler\n",
"# !brew install tesseract\n",
"# # If parsing xml / html documents:\n",
"# !brew install libxml2\n",
"# !brew install libxslt"
]
},
{
@ -149,6 +156,72 @@
"docs[:5]"
]
},
{
"cell_type": "markdown",
"id": "7874d01d",
"metadata": {},
"source": [
"## PDF Example\n",
"\n",
"Processing PDF documents works exactly the same way. Unstructured detects the file type and extracts the same types of `elements`. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "8ca8a648",
"metadata": {},
"outputs": [],
"source": [
"!wget https://raw.githubusercontent.com/Unstructured-IO/unstructured/main/example-docs/layout-parser-paper.pdf -P \"../../\""
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "686e5eb4",
"metadata": {},
"outputs": [],
"source": [
"loader = UnstructuredFileLoader(\"../../layout-parser-paper.pdf\", mode=\"elements\")"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "f8348ca0",
"metadata": {},
"outputs": [],
"source": [
"docs = loader.load()"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "6ec859d8",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='LayoutParser : A Unified Toolkit for Deep Learning Based Document Image Analysis', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0),\n",
" Document(page_content='Zejiang Shen 1 ( (ea)\\n ), Ruochen Zhang 2 , Melissa Dell 3 , Benjamin Charles Germain Lee 4 , Jacob Carlson 3 , and Weining Li 5', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0),\n",
" Document(page_content='Allen Institute for AI shannons@allenai.org', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0),\n",
" Document(page_content='Brown University ruochen zhang@brown.edu', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0),\n",
" Document(page_content='Harvard University { melissadell,jacob carlson } @fas.harvard.edu', lookup_str='', metadata={'source': '../../layout-parser-paper.pdf'}, lookup_index=0)]"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"docs[:5]"
]
},
{
"cell_type": "code",
"execution_count": null,