{ "cells": [ { "cell_type": "markdown", "id": "79f24a6b", "metadata": {}, "source": [ "# File Directory\n", "\n", "This covers how to use the `DirectoryLoader` to load all documents in a directory. Under the hood, by default this uses the [UnstructuredLoader](./unstructured_file.ipynb)" ] }, { "cell_type": "code", "execution_count": 1, "id": "019d8520", "metadata": {}, "outputs": [], "source": [ "from langchain.document_loaders import DirectoryLoader" ] }, { "cell_type": "markdown", "id": "0c76cdc5", "metadata": {}, "source": [ "We can use the `glob` parameter to control which files to load. Note that here it doesn't load the `.rst` file or the `.ipynb` files." ] }, { "cell_type": "code", "execution_count": 2, "id": "891fe56f", "metadata": {}, "outputs": [], "source": [ "loader = DirectoryLoader('../', glob=\"**/*.md\")" ] }, { "cell_type": "code", "execution_count": 3, "id": "addfe9cf", "metadata": {}, "outputs": [], "source": [ "docs = loader.load()" ] }, { "cell_type": "code", "execution_count": 4, "id": "b042086d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(docs)" ] }, { "cell_type": "markdown", "id": "e633d62f", "metadata": {}, "source": [ "## Show a progress bar" ] }, { "cell_type": "markdown", "id": "43911860", "metadata": {}, "source": [ "By default a progress bar will not be shown. To show a progress bar, install the `tqdm` library (e.g. `pip install tqdm`), and set the `show_progress` parameter to `True`." ] }, { "cell_type": "code", "execution_count": 10, "id": "bb93daac", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Requirement already satisfied: tqdm in /Users/jon/.pyenv/versions/3.9.16/envs/microbiome-app/lib/python3.9/site-packages (4.65.0)\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "0it [00:00, ?it/s]\n" ] } ], "source": [ "%pip install tqdm\n", "loader = DirectoryLoader('../', glob=\"**/*.md\", show_progress=True)\n", "docs = loader.load()" ] }, { "cell_type": "markdown", "id": "c16ed46a", "metadata": {}, "source": [ "## Use multithreading" ] }, { "attachments": {}, "cell_type": "markdown", "id": "5752e23e", "metadata": {}, "source": [ "By default the loading happens in one thread. In order to utilize several threads set the `use_multithreading` flag to true." ] }, { "cell_type": "code", "execution_count": null, "id": "f8d84f52", "metadata": {}, "outputs": [], "source": [ "loader = DirectoryLoader('../', glob=\"**/*.md\", use_multithreading=True)\n", "docs = loader.load()" ] }, { "cell_type": "markdown", "id": "c5652850", "metadata": {}, "source": [ "## Change loader class\n", "By default this uses the `UnstructuredLoader` class. However, you can change up the type of loader pretty easily." ] }, { "cell_type": "code", "execution_count": 15, "id": "81c92da3", "metadata": {}, "outputs": [], "source": [ "from langchain.document_loaders import TextLoader" ] }, { "cell_type": "code", "execution_count": 6, "id": "ab38ee36", "metadata": {}, "outputs": [], "source": [ "loader = DirectoryLoader('../', glob=\"**/*.md\", loader_cls=TextLoader)" ] }, { "cell_type": "code", "execution_count": 7, "id": "25c8740f", "metadata": {}, "outputs": [], "source": [ "docs = loader.load()" ] }, { "cell_type": "code", "execution_count": 8, "id": "38337763", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "1" ] }, "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(docs)" ] }, { "cell_type": "markdown", "id": "598a2805", "metadata": {}, "source": [ "If you need to load Python source code files, use the `PythonLoader`." ] }, { "cell_type": "code", "execution_count": 14, "id": "c558bd73", "metadata": {}, "outputs": [], "source": [ "from langchain.document_loaders import PythonLoader" ] }, { "cell_type": "code", "execution_count": 13, "id": "a3cfaba7", "metadata": {}, "outputs": [], "source": [ "loader = DirectoryLoader('../../../../../', glob=\"**/*.py\", loader_cls=PythonLoader)" ] }, { "cell_type": "code", "execution_count": 14, "id": "e2e1e26a", "metadata": {}, "outputs": [], "source": [ "docs = loader.load()" ] }, { "cell_type": "code", "execution_count": 15, "id": "ffb8ff36", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "691" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(docs)" ] }, { "cell_type": "markdown", "id": "6411a0cb", "metadata": {}, "source": [ "## Auto detect file encodings with TextLoader\n", "\n", "In this example we will see some strategies that can be useful when loading a big list of arbitrary files from a directory using the `TextLoader` class.\n", "\n", "First to illustrate the problem, let's try to load multiple text with arbitrary encodings." ] }, { "cell_type": "code", "execution_count": 16, "id": "2c787a69", "metadata": {}, "outputs": [], "source": [ "path = '../../../../../tests/integration_tests/examples'\n", "loader = DirectoryLoader(path, glob=\"**/*.txt\", loader_cls=TextLoader)" ] }, { "cell_type": "markdown", "id": "e9001e12", "metadata": {}, "source": [ "### A. Default Behavior" ] }, { "cell_type": "code", "execution_count": 19, "id": "b1e88c31", "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/html": [ "
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮\n",
       " /data/source/langchain/langchain/document_loaders/text.py:29 in load                             \n",
       "                                                                                                  \n",
       "   26 │   │   text = \"\"                                                                           \n",
       "   27 │   │   with open(self.file_path, encoding=self.encoding) as f:                             \n",
       "   28 │   │   │   try:                                                                            \n",
       " 29 │   │   │   │   text = f.read()                                                             \n",
       "   30 │   │   │   except UnicodeDecodeError as e:                                                 \n",
       "   31 │   │   │   │   if self.autodetect_encoding:                                                \n",
       "   32 │   │   │   │   │   detected_encodings = self.detect_file_encodings()                       \n",
       "                                                                                                  \n",
       " /home/spike/.pyenv/versions/3.9.11/lib/python3.9/codecs.py:322 in decode                         \n",
       "                                                                                                  \n",
       "    319 def decode(self, input, final=False):                                                 \n",
       "    320 │   │   # decode input (taking the buffer into account)                                   \n",
       "    321 │   │   data = self.buffer + input                                                        \n",
       "  322 │   │   (result, consumed) = self._buffer_decode(data, self.errors, final)                \n",
       "    323 │   │   # keep undecoded input until the next call                                        \n",
       "    324 │   │   self.buffer = data[consumed:]                                                     \n",
       "    325 │   │   return result                                                                     \n",
       "╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
       "UnicodeDecodeError: 'utf-8' codec can't decode byte 0xca in position 0: invalid continuation byte\n",
       "\n",
       "The above exception was the direct cause of the following exception:\n",
       "\n",
       "╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮\n",
       " in <module>:1                                                                                    \n",
       "                                                                                                  \n",
       " 1 loader.load()                                                                                \n",
       "   2                                                                                              \n",
       "                                                                                                  \n",
       " /data/source/langchain/langchain/document_loaders/directory.py:84 in load                        \n",
       "                                                                                                  \n",
       "   81 │   │   │   │   │   │   if self.silent_errors:                                              \n",
       "   82 │   │   │   │   │   │   │   logger.warning(e)                                               \n",
       "   83 │   │   │   │   │   │   else:                                                               \n",
       " 84 │   │   │   │   │   │   │   raise e                                                         \n",
       "   85 │   │   │   │   │   finally:                                                                \n",
       "   86 │   │   │   │   │   │   if pbar:                                                            \n",
       "   87 │   │   │   │   │   │   │   pbar.update(1)                                                  \n",
       "                                                                                                  \n",
       " /data/source/langchain/langchain/document_loaders/directory.py:78 in load                        \n",
       "                                                                                                  \n",
       "   75 │   │   │   if i.is_file():                                                                 \n",
       "   76 │   │   │   │   if _is_visible(i.relative_to(p)) or self.load_hidden:                       \n",
       "   77 │   │   │   │   │   try:                                                                    \n",
       " 78 │   │   │   │   │   │   sub_docs = self.loader_cls(str(i), **self.loader_kwargs).load()     \n",
       "   79 │   │   │   │   │   │   docs.extend(sub_docs)                                               \n",
       "   80 │   │   │   │   │   except Exception as e:                                                  \n",
       "   81 │   │   │   │   │   │   if self.silent_errors:                                              \n",
       "                                                                                                  \n",
       " /data/source/langchain/langchain/document_loaders/text.py:44 in load                             \n",
       "                                                                                                  \n",
       "   41 │   │   │   │   │   │   except UnicodeDecodeError:                                          \n",
       "   42 │   │   │   │   │   │   │   continue                                                        \n",
       "   43 │   │   │   │   else:                                                                       \n",
       " 44 │   │   │   │   │   raise RuntimeError(f\"Error loading {self.file_path}\") from e            \n",
       "   45 │   │   │   except Exception as e:                                                          \n",
       "   46 │   │   │   │   raise RuntimeError(f\"Error loading {self.file_path}\") from e                \n",
       "   47                                                                                             \n",
       "╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\n",
       "RuntimeError: Error loading ../../../../../tests/integration_tests/examples/example-non-utf8.txt\n",
       "
\n" ], "text/plain": [ "\u001b[31m╭─\u001b[0m\u001b[31m──────────────────────────────\u001b[0m\u001b[31m \u001b[0m\u001b[1;31mTraceback \u001b[0m\u001b[1;2;31m(most recent call last)\u001b[0m\u001b[31m \u001b[0m\u001b[31m───────────────────────────────\u001b[0m\u001b[31m─╮\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2;33m/data/source/langchain/langchain/document_loaders/\u001b[0m\u001b[1;33mtext.py\u001b[0m:\u001b[94m29\u001b[0m in \u001b[92mload\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m26 \u001b[0m\u001b[2m│ │ \u001b[0mtext = \u001b[33m\"\u001b[0m\u001b[33m\"\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m27 \u001b[0m\u001b[2m│ │ \u001b[0m\u001b[94mwith\u001b[0m \u001b[96mopen\u001b[0m(\u001b[96mself\u001b[0m.file_path, encoding=\u001b[96mself\u001b[0m.encoding) \u001b[94mas\u001b[0m f: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m28 \u001b[0m\u001b[2m│ │ │ \u001b[0m\u001b[94mtry\u001b[0m: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m29 \u001b[2m│ │ │ │ \u001b[0mtext = f.read() \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m30 \u001b[0m\u001b[2m│ │ │ \u001b[0m\u001b[94mexcept\u001b[0m \u001b[96mUnicodeDecodeError\u001b[0m \u001b[94mas\u001b[0m e: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m31 \u001b[0m\u001b[2m│ │ │ │ \u001b[0m\u001b[94mif\u001b[0m \u001b[96mself\u001b[0m.autodetect_encoding: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m32 \u001b[0m\u001b[2m│ │ │ │ │ \u001b[0mdetected_encodings = \u001b[96mself\u001b[0m.detect_file_encodings() \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2;33m/home/spike/.pyenv/versions/3.9.11/lib/python3.9/\u001b[0m\u001b[1;33mcodecs.py\u001b[0m:\u001b[94m322\u001b[0m in \u001b[92mdecode\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m 319 \u001b[0m\u001b[2m│ \u001b[0m\u001b[94mdef\u001b[0m \u001b[92mdecode\u001b[0m(\u001b[96mself\u001b[0m, \u001b[96minput\u001b[0m, final=\u001b[94mFalse\u001b[0m): \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m 320 \u001b[0m\u001b[2m│ │ \u001b[0m\u001b[2m# decode input (taking the buffer into account)\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m 321 \u001b[0m\u001b[2m│ │ \u001b[0mdata = \u001b[96mself\u001b[0m.buffer + \u001b[96minput\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m 322 \u001b[2m│ │ \u001b[0m(result, consumed) = \u001b[96mself\u001b[0m._buffer_decode(data, \u001b[96mself\u001b[0m.errors, final) \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m 323 \u001b[0m\u001b[2m│ │ \u001b[0m\u001b[2m# keep undecoded input until the next call\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m 324 \u001b[0m\u001b[2m│ │ \u001b[0m\u001b[96mself\u001b[0m.buffer = data[consumed:] \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m 325 \u001b[0m\u001b[2m│ │ \u001b[0m\u001b[94mreturn\u001b[0m result \u001b[31m│\u001b[0m\n", "\u001b[31m╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n", "\u001b[1;91mUnicodeDecodeError: \u001b[0m\u001b[32m'utf-8'\u001b[0m codec can't decode byte \u001b[1;36m0xca\u001b[0m in position \u001b[1;36m0\u001b[0m: invalid continuation byte\n", "\n", "\u001b[3mThe above exception was the direct cause of the following exception:\u001b[0m\n", "\n", "\u001b[31m╭─\u001b[0m\u001b[31m──────────────────────────────\u001b[0m\u001b[31m \u001b[0m\u001b[1;31mTraceback \u001b[0m\u001b[1;2;31m(most recent call last)\u001b[0m\u001b[31m \u001b[0m\u001b[31m───────────────────────────────\u001b[0m\u001b[31m─╮\u001b[0m\n", "\u001b[31m│\u001b[0m in \u001b[92m\u001b[0m:\u001b[94m1\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m1 loader.load() \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m2 \u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2;33m/data/source/langchain/langchain/document_loaders/\u001b[0m\u001b[1;33mdirectory.py\u001b[0m:\u001b[94m84\u001b[0m in \u001b[92mload\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m81 \u001b[0m\u001b[2m│ │ │ │ │ │ \u001b[0m\u001b[94mif\u001b[0m \u001b[96mself\u001b[0m.silent_errors: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m82 \u001b[0m\u001b[2m│ │ │ │ │ │ │ \u001b[0mlogger.warning(e) \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m83 \u001b[0m\u001b[2m│ │ │ │ │ │ \u001b[0m\u001b[94melse\u001b[0m: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m84 \u001b[2m│ │ │ │ │ │ │ \u001b[0m\u001b[94mraise\u001b[0m e \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m85 \u001b[0m\u001b[2m│ │ │ │ │ \u001b[0m\u001b[94mfinally\u001b[0m: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m86 \u001b[0m\u001b[2m│ │ │ │ │ │ \u001b[0m\u001b[94mif\u001b[0m pbar: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m87 \u001b[0m\u001b[2m│ │ │ │ │ │ │ \u001b[0mpbar.update(\u001b[94m1\u001b[0m) \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2;33m/data/source/langchain/langchain/document_loaders/\u001b[0m\u001b[1;33mdirectory.py\u001b[0m:\u001b[94m78\u001b[0m in \u001b[92mload\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m75 \u001b[0m\u001b[2m│ │ │ \u001b[0m\u001b[94mif\u001b[0m i.is_file(): \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m76 \u001b[0m\u001b[2m│ │ │ │ \u001b[0m\u001b[94mif\u001b[0m _is_visible(i.relative_to(p)) \u001b[95mor\u001b[0m \u001b[96mself\u001b[0m.load_hidden: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m77 \u001b[0m\u001b[2m│ │ │ │ │ \u001b[0m\u001b[94mtry\u001b[0m: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m78 \u001b[2m│ │ │ │ │ │ \u001b[0msub_docs = \u001b[96mself\u001b[0m.loader_cls(\u001b[96mstr\u001b[0m(i), **\u001b[96mself\u001b[0m.loader_kwargs).load() \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m79 \u001b[0m\u001b[2m│ │ │ │ │ │ \u001b[0mdocs.extend(sub_docs) \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m80 \u001b[0m\u001b[2m│ │ │ │ │ \u001b[0m\u001b[94mexcept\u001b[0m \u001b[96mException\u001b[0m \u001b[94mas\u001b[0m e: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m81 \u001b[0m\u001b[2m│ │ │ │ │ │ \u001b[0m\u001b[94mif\u001b[0m \u001b[96mself\u001b[0m.silent_errors: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2;33m/data/source/langchain/langchain/document_loaders/\u001b[0m\u001b[1;33mtext.py\u001b[0m:\u001b[94m44\u001b[0m in \u001b[92mload\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m41 \u001b[0m\u001b[2m│ │ │ │ │ │ \u001b[0m\u001b[94mexcept\u001b[0m \u001b[96mUnicodeDecodeError\u001b[0m: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m42 \u001b[0m\u001b[2m│ │ │ │ │ │ │ \u001b[0m\u001b[94mcontinue\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m43 \u001b[0m\u001b[2m│ │ │ │ \u001b[0m\u001b[94melse\u001b[0m: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[31m❱ \u001b[0m44 \u001b[2m│ │ │ │ │ \u001b[0m\u001b[94mraise\u001b[0m \u001b[96mRuntimeError\u001b[0m(\u001b[33mf\u001b[0m\u001b[33m\"\u001b[0m\u001b[33mError loading \u001b[0m\u001b[33m{\u001b[0m\u001b[96mself\u001b[0m.file_path\u001b[33m}\u001b[0m\u001b[33m\"\u001b[0m) \u001b[94mfrom\u001b[0m \u001b[4;96me\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m45 \u001b[0m\u001b[2m│ │ │ \u001b[0m\u001b[94mexcept\u001b[0m \u001b[96mException\u001b[0m \u001b[94mas\u001b[0m e: \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m46 \u001b[0m\u001b[2m│ │ │ │ \u001b[0m\u001b[94mraise\u001b[0m \u001b[96mRuntimeError\u001b[0m(\u001b[33mf\u001b[0m\u001b[33m\"\u001b[0m\u001b[33mError loading \u001b[0m\u001b[33m{\u001b[0m\u001b[96mself\u001b[0m.file_path\u001b[33m}\u001b[0m\u001b[33m\"\u001b[0m) \u001b[94mfrom\u001b[0m \u001b[4;96me\u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m│\u001b[0m \u001b[2m47 \u001b[0m \u001b[31m│\u001b[0m\n", "\u001b[31m╰──────────────────────────────────────────────────────────────────────────────────────────────────╯\u001b[0m\n", "\u001b[1;91mRuntimeError: \u001b[0mError loading ..\u001b[35m/../../../../tests/integration_tests/examples/\u001b[0m\u001b[95mexample-non-utf8.txt\u001b[0m\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "loader.load()" ] }, { "cell_type": "markdown", "id": "da554f9a", "metadata": {}, "source": [ "The file `example-non-utf8.txt` uses a different encoding the `load()` function fails with a helpful message indicating which file failed decoding. \n", "\n", "With the default behavior of `TextLoader` any failure to load any of the documents will fail the whole loading process and no documents are loaded. " ] }, { "cell_type": "markdown", "id": "eb844b7e", "metadata": {}, "source": [ "### B. Silent fail\n", "\n", "We can pass the parameter `silent_errors` to the `DirectoryLoader` to skip the files which could not be loaded and continue the load process." ] }, { "cell_type": "code", "execution_count": 30, "id": "5314ec39", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ "Error loading ../../../../../tests/integration_tests/examples/example-non-utf8.txt\n" ] } ], "source": [ "loader = DirectoryLoader(path, glob=\"**/*.txt\", loader_cls=TextLoader, silent_errors=True)\n", "docs = loader.load()" ] }, { "cell_type": "code", "execution_count": 35, "id": "216337e5", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['../../../../../tests/integration_tests/examples/whatsapp_chat.txt',\n", " '../../../../../tests/integration_tests/examples/example-utf8.txt']" ] }, "execution_count": 35, "metadata": {}, "output_type": "execute_result" } ], "source": [ "doc_sources = [doc.metadata['source'] for doc in docs]\n", "doc_sources" ] }, { "cell_type": "markdown", "id": "4cba0e53", "metadata": {}, "source": [ "### C. Auto detect encodings\n", "\n", "We can also ask `TextLoader` to auto detect the file encoding before failing, by passing the `autodetect_encoding` to the loader class." ] }, { "cell_type": "code", "execution_count": 37, "id": "96d527d2", "metadata": {}, "outputs": [], "source": [ "text_loader_kwargs={'autodetect_encoding': True}\n", "loader = DirectoryLoader(path, glob=\"**/*.txt\", loader_cls=TextLoader, loader_kwargs=text_loader_kwargs)\n", "docs = loader.load()\n" ] }, { "cell_type": "code", "execution_count": 38, "id": "f1a136a5", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "['../../../../../tests/integration_tests/examples/example-non-utf8.txt',\n", " '../../../../../tests/integration_tests/examples/whatsapp_chat.txt',\n", " '../../../../../tests/integration_tests/examples/example-utf8.txt']" ] }, "execution_count": 38, "metadata": {}, "output_type": "execute_result" } ], "source": [ "doc_sources = [doc.metadata['source'] for doc in docs]\n", "doc_sources" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.11" } }, "nbformat": 4, "nbformat_minor": 5 }