Updated the MongoDB Atlas Vector Search docs to indicate the service is
Generally Available, updated the example to use the new index
definition, and added an example that uses metadata pre-filtering for
semantic search
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
">* This feature is in Public Preview and available for evaluation purposes, to validate functionality, and to gather feedback from public preview users. It is not recommended for production deployments as we may introduce breaking changes.\n",
">* This feature is Generally Available and ready for production deployments.\n",
">* The langchain version 0.0.305 ([release notes](https://github.com/langchain-ai/langchain/releases/tag/v0.0.305)) introduces the support for $vectorSearch MQL stage, which is available with MongoDB Atlas 6.0.11 and 7.0.2. Users utilizing earlier versions of MongoDB Atlas need to pin their LangChain version to <=0.0.304\n",
"> \n",
"> "
@ -34,7 +34,7 @@
"id": "1b5ce18d",
"metadata": {},
"source": [
"In the notebook we will demonstrate how to perform `Retrieval Augmented Generation` (RAG) using MongoDB Atlas, OpenAI and Langchain. We will be performing Similarity Search and Question Answering over the PDF document for [GPT 4 technical report](https://arxiv.org/pdf/2303.08774.pdf) that came out in March 2023 and hence is not part of the OpenAI's Large Language Model(LLM)'s parametric memory, which had a knowledge cutoff of September 2021."
"In the notebook we will demonstrate how to perform `Retrieval Augmented Generation` (RAG) using MongoDB Atlas, OpenAI and Langchain. We will be performing Similarity Search, Similarity Search with Metadata Pre-Filtering, and Question Answering over the PDF document for [GPT 4 technical report](https://arxiv.org/pdf/2303.08774.pdf) that came out in March 2023 and hence is not part of the OpenAI's Large Language Model(LLM)'s parametric memory, which had a knowledge cutoff of September 2021."
"Now, let's create a vector search index on your cluster. In the below example, `embedding` is the name of the field that contains the embedding vector. Please refer to the [documentation](https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector) to get more details on how to define an Atlas Vector Search index.\n",
"Now, let's create a vector search index on your cluster. In the below example, `embedding` is the name of the field that contains the embedding vector. Please refer to the [documentation](https://www.mongodb.com/docs/atlas/atlas-vector-search/create-index/) to get more details on how to define an Atlas Vector Search index.\n",
"You can name the index `{ATLAS_VECTOR_SEARCH_INDEX_NAME}` and create the index on the namespace `{DB_NAME}.{COLLECTION_NAME}`. Finally, write the following definition in the JSON editor on MongoDB Atlas:\n",
"\n",
"```json\n",
"{\n",
" \"mappings\": {\n",
" \"dynamic\": true,\n",
" \"fields\": {\n",
" \"embedding\": {\n",
" \"dimensions\": 1536,\n",
" \"similarity\": \"cosine\",\n",
" \"type\": \"knnVector\"\n",
" }\n",
" \"name\": \"index_name\",\n",
" \"type\": \"vectorSearch\",\n",
" \"fields\":[\n",
" {\n",
" \"type\": \"vector\",\n",
" \"path\": \"embedding\",\n",
" \"numDimensions\": 1536,\n",
" \"similarity\": \"cosine\"\n",
" }\n",
" }\n",
" ]\n",
"}\n",
"```"
]
@ -255,6 +253,65 @@
")"
]
},
{
"cell_type": "markdown",
"id": "02aef29c-5da0-41b8-b4fc-98fd71b94abf",
"metadata": {},
"source": [
"## Pre-filtering with Similarity Search"
]
},
{
"cell_type": "markdown",
"id": "f3b2d36d-d47a-482f-999d-85c23eb67eed",
"metadata": {},
"source": [
"Atlas Vector Search supports pre-filtering using MQL Operators for filtering. Below is an example index and query on the same data loaded above that allows you do metadata filtering on the \"page\" field. You can update your existing index with the filter defined and do pre-filtering with vector search."
]
},
{
"cell_type": "markdown",
"id": "2b385a46-1e54-471f-95b2-202813d90bb2",
"metadata": {},
"source": [
"```json\n",
"{\n",
" \"name\": \"index_name\",\n",
" \"type\": \"vectorSearch\",\n",
" \"fields\":[\n",
" {\n",
" \"type\": \"vector\",\n",
" \"path\": \"embedding\",\n",
" \"numDimensions\": 1536,\n",
" \"similarity\": \"cosine\"\n",
" },\n",
" {\n",
" \"type\": \"filter\",\n",
" \"path\": \"page\"\n",
" }\n",
" ]\n",
"}\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "dfc8487d-14ec-42c9-9670-80fe02816196",
"metadata": {},
"outputs": [],
"source": [
"query = \"What were the compute requirements for training GPT 4\"\n",
Ben Flast
10 months ago
committed by
GitHub