mirror of https://github.com/hwchase17/langchain
docs: Custom Document Loaders (#19935)
Add information that shows how to create custom document loaderspull/19971/head^2
Eugene Yurtsev
6 months ago
committed by
GitHub
parent
83f62fdacf
commit
ea276d6547
@ -0,0 +1,778 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "raw",
|
||||
"id": "c5990f4f-4430-4bbb-8d25-9703d7d8e95c",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"---\n",
|
||||
"title: Custom Document Loader\n",
|
||||
"sidebar_position: 10\n",
|
||||
"---"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4be0aa7c-aee3-4e11-b7f4-059611ab8626",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Custom Document Loader\n",
|
||||
"\n",
|
||||
"## Overview\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"Applications based on LLMs frequently entail extracting data from databases or files, like PDFs, and converting it into a format that LLMs can utilize. In LangChain, this usually involves creating Document objects, which encapsulate the extracted text (`page_content`) along with metadata—a dictionary containing details about the document, such as the author's name or the date of publication.\n",
|
||||
"\n",
|
||||
"`Document` objects are often formatted into prompts that are fed into an LLM, allowing the LLM to use the information in the `Document` to generate a desired response (e.g., summarizing the document).\n",
|
||||
"`Documents` can be either used immediately or indexed into a vectorstore for future retrieval and use.\n",
|
||||
"\n",
|
||||
"The main abstractions for Document Loading are:\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"| Component | Description |\n",
|
||||
"|--------------------|--------------------------------|\n",
|
||||
"| Document | Contains `text` and `metadata` |\n",
|
||||
"| BaseDocumentLoader | Use to convert raw data into `Documents` |\n",
|
||||
"| Blob | A representation of binary data thta's located either in a file or in memory |\n",
|
||||
"| BaseBlobParser | Logic to parse a `Blob` to yield `Document` objects |\n",
|
||||
"\n",
|
||||
"This guide will demonstrate how to write custom document loading and file parsing logic; specifically, we'll see how to:\n",
|
||||
"\n",
|
||||
"1. Create a standard document Loader by sub-classing from `BaseLoader`.\n",
|
||||
"2. Create a parser using `BaseBlobParser` and use it in conjunction with `Blob` and `BlobLoaders`. This is useful primarily when working with files."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "20dc4c18-accc-4009-805c-961f3e8dc50a",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Standard Document Loader\n",
|
||||
"\n",
|
||||
"A document loader can be implemented by sub-classing from a `BaseLoader` which provides a standard interface for loading documents.\n",
|
||||
"\n",
|
||||
"### Interface \n",
|
||||
"\n",
|
||||
"| Method Name | Explanation |\n",
|
||||
"|-------------|-------------|\n",
|
||||
"| lazy_load | Used to load documents one by one **lazily**. Use for production code. |\n",
|
||||
"| alazy_load | Async variant of `lazy_load` |\n",
|
||||
"| load | Used to load all the documents into memory **eagerly**. Use for prototyping or interactive work. |\n",
|
||||
"| aload | Used to load all the documents into memory **eagerly**. Use for prototyping or interactive work. **Added in 2024-04 to LangChain.** |\n",
|
||||
"\n",
|
||||
"* The `load` methods is a convenience method meant solely for prototyping work -- it just invokes `list(self.lazy_load())`.\n",
|
||||
"* The `alazy_load` has a default implementation that will delegate to `lazy_load`. If you're using async, we recommend overriding the default implementation and providing a native async implementation.\n",
|
||||
"\n",
|
||||
"::: {.callout-important}\n",
|
||||
"When implementing a document loader do **NOT** provide parameters via the `lazy_load` or `alazy_load` methods.\n",
|
||||
"\n",
|
||||
"All configuration is expected to be passed through the initializer (__init__). This was a design choice made by LangChain to make sure that once a document loader has been instantiated it has all the information needed to load documents.\n",
|
||||
":::\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"### Implementation\n",
|
||||
"\n",
|
||||
"Let's create an example of a standard document loader that loads a file and creates a document from each line in the file."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "20f128c1-1a2c-43b9-9e7b-cf9b3a86d1db",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import AsyncIterator, Iterator\n",
|
||||
"\n",
|
||||
"from langchain_core.document_loaders import BaseLoader\n",
|
||||
"from langchain_core.documents import Document\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class CustomDocumentLoader(BaseLoader):\n",
|
||||
" \"\"\"An example document loader that reads a file line by line.\"\"\"\n",
|
||||
"\n",
|
||||
" def __init__(self, file_path: str) -> None:\n",
|
||||
" \"\"\"Initialize the loader with a file path.\n",
|
||||
"\n",
|
||||
" Args:\n",
|
||||
" file_path: The path to the file to load.\n",
|
||||
" \"\"\"\n",
|
||||
" self.file_path = file_path\n",
|
||||
"\n",
|
||||
" def lazy_load(self) -> Iterator[Document]: # <-- Does not take any arguments\n",
|
||||
" \"\"\"A lazy loader that reads a file line by line.\n",
|
||||
"\n",
|
||||
" When you're implementing lazy load methods, you should use a generator\n",
|
||||
" to yield documents one by one.\n",
|
||||
" \"\"\"\n",
|
||||
" with open(self.file_path, encoding=\"utf-8\") as f:\n",
|
||||
" line_number = 0\n",
|
||||
" for line in f:\n",
|
||||
" yield Document(\n",
|
||||
" page_content=line,\n",
|
||||
" metadata={\"line_number\": line_number, \"source\": self.file_path},\n",
|
||||
" )\n",
|
||||
" line_number += 1\n",
|
||||
"\n",
|
||||
" # alazy_load is OPTIONAL.\n",
|
||||
" # If you leave out the implementation, a default implementation which delegates to lazy_load will be used!\n",
|
||||
" async def alazy_load(\n",
|
||||
" self,\n",
|
||||
" ) -> AsyncIterator[Document]: # <-- Does not take any arguments\n",
|
||||
" \"\"\"An async lazy loader that reads a file line by line.\"\"\"\n",
|
||||
" # Requires aiofiles\n",
|
||||
" # Install with `pip install aiofiles`\n",
|
||||
" # https://github.com/Tinche/aiofiles\n",
|
||||
" import aiofiles\n",
|
||||
"\n",
|
||||
" async with aiofiles.open(self.file_path, encoding=\"utf-8\") as f:\n",
|
||||
" line_number = 0\n",
|
||||
" async for line in f:\n",
|
||||
" yield Document(\n",
|
||||
" page_content=line,\n",
|
||||
" metadata={\"line_number\": line_number, \"source\": self.file_path},\n",
|
||||
" )\n",
|
||||
" line_number += 1"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "eb845512-3d46-44fa-a4c6-ff723533abbe",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"source": [
|
||||
"### Test 🧪\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"To test out the document loader, we need a file with some quality content."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "b1751198-c6dd-4149-95bd-6370ce8fa06f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"with open(\"./meow.txt\", \"w\", encoding=\"utf-8\") as f:\n",
|
||||
" quality_content = \"meow meow🐱 \\n meow meow🐱 \\n meow😻😻\"\n",
|
||||
" f.write(quality_content)\n",
|
||||
"\n",
|
||||
"loader = CustomDocumentLoader(\"./meow.txt\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "71ef1482-f9de-4852-b5a4-0938f350612e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"<class 'langchain_core.documents.base.Document'>\n",
|
||||
"page_content='meow meow🐱 \\n' metadata={'line_number': 0, 'source': './meow.txt'}\n",
|
||||
"\n",
|
||||
"<class 'langchain_core.documents.base.Document'>\n",
|
||||
"page_content=' meow meow🐱 \\n' metadata={'line_number': 1, 'source': './meow.txt'}\n",
|
||||
"\n",
|
||||
"<class 'langchain_core.documents.base.Document'>\n",
|
||||
"page_content=' meow😻😻' metadata={'line_number': 2, 'source': './meow.txt'}\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"## Test out the lazy load interface\n",
|
||||
"for doc in loader.lazy_load():\n",
|
||||
" print()\n",
|
||||
" print(type(doc))\n",
|
||||
" print(doc)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "1588e78c-e81a-4d40-b36c-634242c84a6a",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"<class 'langchain_core.documents.base.Document'>\n",
|
||||
"page_content='meow meow🐱 \\n' metadata={'line_number': 0, 'source': './meow.txt'}\n",
|
||||
"\n",
|
||||
"<class 'langchain_core.documents.base.Document'>\n",
|
||||
"page_content=' meow meow🐱 \\n' metadata={'line_number': 1, 'source': './meow.txt'}\n",
|
||||
"\n",
|
||||
"<class 'langchain_core.documents.base.Document'>\n",
|
||||
"page_content=' meow😻😻' metadata={'line_number': 2, 'source': './meow.txt'}\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"## Test out the async implementation\n",
|
||||
"async for doc in loader.alazy_load():\n",
|
||||
" print()\n",
|
||||
" print(type(doc))\n",
|
||||
" print(doc)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "56cb443e-f987-4386-b4ec-975ee129adb2",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"::: {.callout-tip}\n",
|
||||
"\n",
|
||||
"`load()` can be helpful in an interactive environment such as a jupyter notebook.\n",
|
||||
"\n",
|
||||
"Avoid using it for production code since eager loading assumes that all the content\n",
|
||||
"can fit into memory, which is not always the case, especially for enterprise data.\n",
|
||||
":::"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 6,
|
||||
"id": "df5ad46a-9e00-4073-8505-489fc4f3799e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[Document(page_content='meow meow🐱 \\n', metadata={'line_number': 0, 'source': './meow.txt'}),\n",
|
||||
" Document(page_content=' meow meow🐱 \\n', metadata={'line_number': 1, 'source': './meow.txt'}),\n",
|
||||
" Document(page_content=' meow😻😻', metadata={'line_number': 2, 'source': './meow.txt'})]"
|
||||
]
|
||||
},
|
||||
"execution_count": 6,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"loader.load()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "639fe87c-b65f-4bef-8fe2-d10be85589f4",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Working with Files\n",
|
||||
"\n",
|
||||
"Many document loaders invovle parsing files. The difference between such loaders usually stems from how the file is parsed rather than how the file is loaded. For example, you can use `open` to read the binary content of either a PDF or a markdown file, but you need different parsing logic to convert that binary data into text.\n",
|
||||
"\n",
|
||||
"As a result, it can be helpful to decouple the parsing logic from the loading logic, which makes it easier to re-use a given parser regardless of how the data was loaded.\n",
|
||||
"\n",
|
||||
"### BaseBlobParser\n",
|
||||
"\n",
|
||||
"A `BaseBlobParser` is an interface that accepts a `blob` and outputs a list of `Document` objects. A `blob` is a representation of data that lives either in memory or in a file. LangChain python has a `Blob` primitive which is inspired by the [Blob WebAPI spec](https://developer.mozilla.org/en-US/docs/Web/API/Blob)."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 7,
|
||||
"id": "209f6a91-2f15-4cb2-9237-f79fc9493b82",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain_core.document_loaders import BaseBlobParser, Blob\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class MyParser(BaseBlobParser):\n",
|
||||
" \"\"\"A simple parser that creates a document from each line.\"\"\"\n",
|
||||
"\n",
|
||||
" def lazy_parse(self, blob: Blob) -> Iterator[Document]:\n",
|
||||
" \"\"\"Parse a blob into a document line by line.\"\"\"\n",
|
||||
" line_number = 0\n",
|
||||
" with blob.as_bytes_io() as f:\n",
|
||||
" for line in f:\n",
|
||||
" line_number += 1\n",
|
||||
" yield Document(\n",
|
||||
" page_content=line,\n",
|
||||
" metadata={\"line_number\": line_number, \"source\": blob.source},\n",
|
||||
" )"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "b1275c59-06d4-458f-abd2-fcbad0bde442",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"blob = Blob.from_path(\"./meow.txt\")\n",
|
||||
"parser = MyParser()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "56a3d707-2086-413b-ae82-50e92ddb27f6",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[Document(page_content='meow meow🐱 \\n', metadata={'line_number': 1, 'source': './meow.txt'}),\n",
|
||||
" Document(page_content=' meow meow🐱 \\n', metadata={'line_number': 2, 'source': './meow.txt'}),\n",
|
||||
" Document(page_content=' meow😻😻', metadata={'line_number': 3, 'source': './meow.txt'})]"
|
||||
]
|
||||
},
|
||||
"execution_count": 8,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"list(parser.lazy_parse(blob))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "433bfb7c-7767-43bc-b71e-42413d7494a8",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Using the **blob** API also allows one to load content direclty from memory without having to read it from a file!"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 9,
|
||||
"id": "20d03092-ba35-47d7-b612-9d1631c261cd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"[Document(page_content='some data from memory\\n', metadata={'line_number': 1, 'source': None}),\n",
|
||||
" Document(page_content='meow', metadata={'line_number': 2, 'source': None})]"
|
||||
]
|
||||
},
|
||||
"execution_count": 9,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"blob = Blob(data=b\"some data from memory\\nmeow\")\n",
|
||||
"list(parser.lazy_parse(blob))"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "d401c5e9-32cc-41e2-973f-c70d1cd3ba76",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Blob\n",
|
||||
"\n",
|
||||
"Let's take a quick look through some of the Blob API."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 10,
|
||||
"id": "a9e92e0e-c8da-401c-b8c6-f0676004cf58",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"blob = Blob.from_path(\"./meow.txt\", metadata={\"foo\": \"bar\"})"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "6b559d30-8b0c-4e45-86b1-e4602d9aaa7e",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'utf-8'"
|
||||
]
|
||||
},
|
||||
"execution_count": 11,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"blob.encoding"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 12,
|
||||
"id": "2f7b145a-9c6f-47f9-9487-1f4b25aff46f",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"b'meow meow\\xf0\\x9f\\x90\\xb1 \\n meow meow\\xf0\\x9f\\x90\\xb1 \\n meow\\xf0\\x9f\\x98\\xbb\\xf0\\x9f\\x98\\xbb'"
|
||||
]
|
||||
},
|
||||
"execution_count": 12,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"blob.as_bytes()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 13,
|
||||
"id": "9b9482fa-c49c-42cd-a2ef-80bc93214631",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'meow meow🐱 \\n meow meow🐱 \\n meow😻😻'"
|
||||
]
|
||||
},
|
||||
"execution_count": 13,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"blob.as_string()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 14,
|
||||
"id": "04cc7a81-290e-4ef8-b7e1-d885fcc59ece",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"<contextlib._GeneratorContextManager at 0x743f34324450>"
|
||||
]
|
||||
},
|
||||
"execution_count": 14,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"blob.as_bytes_io()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 15,
|
||||
"id": "ec8de0ab-51d7-4e41-82c9-3ce0a6fdc2cd",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"{'foo': 'bar'}"
|
||||
]
|
||||
},
|
||||
"execution_count": 15,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"blob.metadata"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 16,
|
||||
"id": "19eae991-ae48-43c2-8952-7347cdb76a34",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": [
|
||||
"'./meow.txt'"
|
||||
]
|
||||
},
|
||||
"execution_count": 16,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"blob.source"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3ea67645-a367-48ce-b164-0d9f00c17370",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Blob Loaders\n",
|
||||
"\n",
|
||||
"While a parser encapsulates the logic needed to parse binary data into documents, *blob loaders* encapsulate the logic that's necessary to load blobs from a given storage location.\n",
|
||||
"\n",
|
||||
"A the moment, `LangChain` only supports `FileSystemBlobLoader`.\n",
|
||||
"\n",
|
||||
"You can use the `FileSystemBlobLoader` to load blobs and then use the parser to parse them."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 17,
|
||||
"id": "c093becb-2e84-4329-89e3-956a3bd765e5",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain_community.document_loaders.blob_loaders import FileSystemBlobLoader\n",
|
||||
"\n",
|
||||
"blob_loader = FileSystemBlobLoader(path=\".\", glob=\"*.mdx\", show_progress=True)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 18,
|
||||
"id": "77739dab-2a1e-4b64-8daa-fee8aa029972",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "45e85d3f63224bb59db02a40ae2e3268",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
" 0%| | 0/8 [00:00<?, ?it/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"page_content='# Microsoft Office\\n' metadata={'line_number': 1, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='# Markdown\\n' metadata={'line_number': 1, 'source': 'markdown.mdx'}\n",
|
||||
"page_content='# JSON\\n' metadata={'line_number': 1, 'source': 'json.mdx'}\n",
|
||||
"page_content='---\\n' metadata={'line_number': 1, 'source': 'pdf.mdx'}\n",
|
||||
"page_content='---\\n' metadata={'line_number': 1, 'source': 'index.mdx'}\n",
|
||||
"page_content='# File Directory\\n' metadata={'line_number': 1, 'source': 'file_directory.mdx'}\n",
|
||||
"page_content='# CSV\\n' metadata={'line_number': 1, 'source': 'csv.mdx'}\n",
|
||||
"page_content='# HTML\\n' metadata={'line_number': 1, 'source': 'html.mdx'}\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"parser = MyParser()\n",
|
||||
"for blob in blob_loader.yield_blobs():\n",
|
||||
" for doc in parser.lazy_parse(blob):\n",
|
||||
" print(doc)\n",
|
||||
" break"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "f016390c-d38b-4261-946d-34eefe546df7",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"### Generic Loader\n",
|
||||
"\n",
|
||||
"LangChain has a `GenericLoader` abstraction which composes a `BlobLoader` with a `BaseBlobParser`.\n",
|
||||
"\n",
|
||||
"`GenericLoader` is meant to provide standardized classmethods that make it easy to use existing `BlobLoader` implementations. At the moment, only the `FileSystemBlobLoader` is supported."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 19,
|
||||
"id": "1de74daf-70ee-4616-9089-d28e26b16851",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "5f1f6810a71a4909ac9fe1e8f8cb9e0a",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
" 0%| | 0/8 [00:00<?, ?it/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"page_content='# Microsoft Office\\n' metadata={'line_number': 1, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='\\n' metadata={'line_number': 2, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='>[The Microsoft Office](https://www.office.com/) suite of productivity software includes Microsoft Word, Microsoft Excel, Microsoft PowerPoint, Microsoft Outlook, and Microsoft OneNote. It is available for Microsoft Windows and macOS operating systems. It is also available on Android and iOS.\\n' metadata={'line_number': 3, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='\\n' metadata={'line_number': 4, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='This covers how to load commonly used file formats including `DOCX`, `XLSX` and `PPTX` documents into a document format that we can use downstream.\\n' metadata={'line_number': 5, 'source': 'office_file.mdx'}\n",
|
||||
"... output truncated for demo purposes\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from langchain_community.document_loaders.generic import GenericLoader\n",
|
||||
"\n",
|
||||
"loader = GenericLoader.from_filesystem(\n",
|
||||
" path=\".\", glob=\"*.mdx\", show_progress=True, parser=MyParser()\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"for idx, doc in enumerate(loader.lazy_load()):\n",
|
||||
" if idx < 5:\n",
|
||||
" print(doc)\n",
|
||||
"\n",
|
||||
"print(\"... output truncated for demo purposes\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "902048b7-ff04-46c0-97b5-935b40ff8511",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"#### Custom Generic Loader\n",
|
||||
"\n",
|
||||
"If you really like creating classes, you can sub-class and create a class to encapsulate the logic together.\n",
|
||||
"\n",
|
||||
"You can sub-class from this class to load content using an existing loader."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 20,
|
||||
"id": "23633102-dc44-4fed-a4e1-8159489101c8",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from typing import Any\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"class MyCustomLoader(GenericLoader):\n",
|
||||
" @staticmethod\n",
|
||||
" def get_parser(**kwargs: Any) -> BaseBlobParser:\n",
|
||||
" \"\"\"Override this method to associate a default parser with the class.\"\"\"\n",
|
||||
" return MyParser()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 21,
|
||||
"id": "dc95be85-4a29-4c6f-a260-08afa3c95538",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"application/vnd.jupyter.widget-view+json": {
|
||||
"model_id": "4320598ea3b44a52b1873e1c801db312",
|
||||
"version_major": 2,
|
||||
"version_minor": 0
|
||||
},
|
||||
"text/plain": [
|
||||
" 0%| | 0/8 [00:00<?, ?it/s]"
|
||||
]
|
||||
},
|
||||
"metadata": {},
|
||||
"output_type": "display_data"
|
||||
},
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"page_content='# Microsoft Office\\n' metadata={'line_number': 1, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='\\n' metadata={'line_number': 2, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='>[The Microsoft Office](https://www.office.com/) suite of productivity software includes Microsoft Word, Microsoft Excel, Microsoft PowerPoint, Microsoft Outlook, and Microsoft OneNote. It is available for Microsoft Windows and macOS operating systems. It is also available on Android and iOS.\\n' metadata={'line_number': 3, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='\\n' metadata={'line_number': 4, 'source': 'office_file.mdx'}\n",
|
||||
"page_content='This covers how to load commonly used file formats including `DOCX`, `XLSX` and `PPTX` documents into a document format that we can use downstream.\\n' metadata={'line_number': 5, 'source': 'office_file.mdx'}\n",
|
||||
"... output truncated for demo purposes\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"loader = MyCustomLoader.from_filesystem(path=\".\", glob=\"*.mdx\", show_progress=True)\n",
|
||||
"\n",
|
||||
"for idx, doc in enumerate(loader.lazy_load()):\n",
|
||||
" if idx < 5:\n",
|
||||
" print(doc)\n",
|
||||
"\n",
|
||||
"print(\"... output truncated for demo purposes\")"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.4"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Loading…
Reference in New Issue