mirror of
https://github.com/hwchase17/langchain
synced 2024-10-29 17:07:25 +00:00
280 lines
9.6 KiB
Plaintext
280 lines
9.6 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b0ed136e-6983-4893-ae1b-b75753af05f8",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Google Drive Retriever\n",
|
|
"This notebook covers how to retrieve documents from Google Drive.\n",
|
|
"\n",
|
|
"## Prerequisites\n",
|
|
"\n",
|
|
"1. Create a Google Cloud project or use an existing project\n",
|
|
"1. Enable the [Google Drive API](https://console.cloud.google.com/flows/enableapi?apiid=drive.googleapis.com)\n",
|
|
"1. [Authorize credentials for desktop app](https://developers.google.com/drive/api/quickstart/python#authorize_credentials_for_a_desktop_application)\n",
|
|
"1. `pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib`\n",
|
|
"\n",
|
|
"## Instructions for retrieving your Google Docs data\n",
|
|
"By default, the `GoogleDriveRetriever` expects the `credentials.json` file to be `~/.credentials/credentials.json`, but this is configurable using the `GOOGLE_ACCOUNT_FILE` environment variable. \n",
|
|
"The location of `token.json` use the same directory (or use the parameter `token_path`). Note that `token.json` will be created automatically the first time you use the retriever.\n",
|
|
"\n",
|
|
"`GoogleDriveRetriever` can retrieve a selection of files with some requests. \n",
|
|
"\n",
|
|
"By default, If you use a `folder_id`, all the files inside this folder can be retrieved to `Document`.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "35b94a93-97de-4af8-9cca-de9ffb7930c3",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can obtain your folder and document id from the URL:\n",
|
|
"* Folder: https://drive.google.com/drive/u/0/folders/1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5 -> folder id is `\"1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5\"`\n",
|
|
"* Document: https://docs.google.com/document/d/1bfaMQ18_i56204VaQDVeAFpqEijJTgvurupdEDiaUQw/edit -> document id is `\"1bfaMQ18_i56204VaQDVeAFpqEijJTgvurupdEDiaUQw\"`\n",
|
|
"\n",
|
|
"The special value `root` is for your personal home."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9c9665c9-a023-4078-9d95-e43021cecb6f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#!pip install --upgrade google-api-python-client google-auth-httplib2 google-auth-oauthlib"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "878928a6-a5ae-4f74-b351-64e3b01733fe",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-05-09T10:45:59.438650905Z",
|
|
"start_time": "2023-05-09T10:45:57.955900302Z"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain_googledrive.retrievers import GoogleDriveRetriever"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "755907c2-145d-4f0f-9b15-07a628a2d2d2",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-05-09T10:45:59.442890834Z",
|
|
"start_time": "2023-05-09T10:45:59.440941528Z"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"folder_id=\"root\"\n",
|
|
"#folder_id='1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2216c83f-68e4-4d2f-8ea2-5878fb18bbe7",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-05-09T10:45:59.795842403Z",
|
|
"start_time": "2023-05-09T10:45:59.445262457Z"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"retriever = GoogleDriveRetriever(\n",
|
|
" num_results=2,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fa339ca0-f478-440c-ba80-0e5f41a19ce1",
|
|
"metadata": {},
|
|
"source": [
|
|
"By default, all files with these mime-type can be converted to `Document`.\n",
|
|
"- text/text\n",
|
|
"- text/plain\n",
|
|
"- text/html\n",
|
|
"- text/csv\n",
|
|
"- text/markdown\n",
|
|
"- image/png\n",
|
|
"- image/jpeg\n",
|
|
"- application/epub+zip\n",
|
|
"- application/pdf\n",
|
|
"- application/rtf\n",
|
|
"- application/vnd.google-apps.document (GDoc)\n",
|
|
"- application/vnd.google-apps.presentation (GSlide)\n",
|
|
"- application/vnd.google-apps.spreadsheet (GSheet)\n",
|
|
"- application/vnd.google.colaboratory (Notebook colab)\n",
|
|
"- application/vnd.openxmlformats-officedocument.presentationml.presentation (PPTX)\n",
|
|
"- application/vnd.openxmlformats-officedocument.wordprocessingml.document (DOCX)\n",
|
|
"\n",
|
|
"It's possible to update or customize this. See the documentation of `GDriveRetriever`.\n",
|
|
"\n",
|
|
"But, the corresponding packages must be installed."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9dadec48",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#!pip install unstructured"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8f3b6aa0-b45d-4e37-8c50-5bebe70fdb9d",
|
|
"metadata": {
|
|
"ExecuteTime": {
|
|
"end_time": "2023-05-09T10:46:00.990310466Z",
|
|
"start_time": "2023-05-09T10:45:59.798774595Z"
|
|
},
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"retriever.get_relevant_documents(\"machine learning\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8ff33817-8619-4897-8742-2216b9934d2a",
|
|
"metadata": {},
|
|
"source": [
|
|
"You can customize the criteria to select the files. A set of predefined filter are proposed:\n",
|
|
"| template | description |\n",
|
|
"| -------------------------------------- | --------------------------------------------------------------------- |\n",
|
|
"| gdrive-all-in-folder | Return all compatible files from a `folder_id` |\n",
|
|
"| gdrive-query | Search `query` in all drives |\n",
|
|
"| gdrive-by-name | Search file with name `query`) |\n",
|
|
"| gdrive-query-in-folder | Search `query` in `folder_id` (and sub-folders in `_recursive=true`) |\n",
|
|
"| gdrive-mime-type | Search a specific `mime_type` |\n",
|
|
"| gdrive-mime-type-in-folder | Search a specific `mime_type` in `folder_id` |\n",
|
|
"| gdrive-query-with-mime-type | Search `query` with a specific `mime_type` |\n",
|
|
"| gdrive-query-with-mime-type-and-folder | Search `query` with a specific `mime_type` and in `folder_id` |"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9977c712-9659-4959-b508-f59cc7d49d44",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"retriever = GoogleDriveRetriever(\n",
|
|
" template=\"gdrive-query\", # Search everywhere\n",
|
|
" num_results=2, # But take only 2 documents\n",
|
|
")\n",
|
|
"for doc in retriever.get_relevant_documents(\"machine learning\"):\n",
|
|
" print(\"---\")\n",
|
|
" print(doc.page_content.strip()[:60]+\"...\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a5a0f3ef-26fb-4a5c-85f0-5aba90b682b1",
|
|
"metadata": {},
|
|
"source": [
|
|
"Else, you can customize the prompt with a specialized `PromptTemplate`"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b0bbebde-0487-4d20-9d77-8070e4f0e0d6",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.prompts import PromptTemplate\n",
|
|
"retriever = GoogleDriveRetriever(\n",
|
|
" template=PromptTemplate(input_variables=['query'],\n",
|
|
" # See https://developers.google.com/drive/api/guides/search-files\n",
|
|
" template=\"(fullText contains '{query}') \"\n",
|
|
" \"and mimeType='application/vnd.google-apps.document' \"\n",
|
|
" \"and modifiedTime > '2000-01-01T00:00:00' \"\n",
|
|
" \"and trashed=false\"),\n",
|
|
" num_results=2,\n",
|
|
" # See https://developers.google.com/drive/api/v3/reference/files/list\n",
|
|
" includeItemsFromAllDrives=False,\n",
|
|
" supportsAllDrives=False,\n",
|
|
")\n",
|
|
"for doc in retriever.get_relevant_documents(\"machine learning\"):\n",
|
|
" print(f\"{doc.metadata['name']}:\")\n",
|
|
" print(\"---\")\n",
|
|
" print(doc.page_content.strip()[:60]+\"...\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9b6fed29-1666-452e-b677-401613270388",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Use GDrive 'description' metadata\n",
|
|
"Each Google Drive has a `description` field in metadata (see the *details of a file*).\n",
|
|
"Use the `snippets` mode to return the description of selected files.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "342dbe12-ed83-40f4-8957-0cc8c4609542",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"retriever = GoogleDriveRetriever(\n",
|
|
" template='gdrive-mime-type-in-folder',\n",
|
|
" folder_id=folder_id,\n",
|
|
" mime_type='application/vnd.google-apps.document', # Only Google Docs\n",
|
|
" num_results=2,\n",
|
|
" mode='snippets',\n",
|
|
" includeItemsFromAllDrives=False,\n",
|
|
" supportsAllDrives=False,\n",
|
|
")\n",
|
|
"retriever.get_relevant_documents(\"machine learning\")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|