mirror of
https://github.com/hwchase17/langchain
synced 2024-10-31 15:20:26 +00:00
03b16ed2b1
Fixed several inconsistencies: - file names and notebook titles should be similar otherwise ToC on the [retrievers page](https://python.langchain.com/en/latest/modules/indexes/retrievers.html) and on the left ToC tab are different. For example, now, `Self-querying with Chroma` is not correctly alphabetically sorted because its file named `chroma_self_query.ipynb` - `Stringing compressors and document transformers...` demoted from `#` to `##`. Otherwise, it appears in Toc. - several formatting problems #### Who can review? @hwchase17 @dev2049 Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
119 lines
4.4 KiB
Plaintext
119 lines
4.4 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "fc0db1bc",
|
|
"metadata": {},
|
|
"source": [
|
|
"# LOTR (Merger Retriever)\n",
|
|
"\n",
|
|
"`Lord of the Retrievers`, also known as `MergerRetriever`, takes a list of retrievers as input and merges the results of their get_relevant_documents() methods into a single list. The merged results will be a list of documents that are relevant to the query and that have been ranked by the different retrievers.\n",
|
|
"\n",
|
|
"The `MergerRetriever` class can be used to improve the accuracy of document retrieval in a number of ways. First, it can combine the results of multiple retrievers, which can help to reduce the risk of bias in the results. Second, it can rank the results of the different retrievers, which can help to ensure that the most relevant documents are returned first."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9fbcc58f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"import chromadb\n",
|
|
"from langchain.retrievers.merger_retriever import MergerRetriever\n",
|
|
"from langchain.vectorstores import Chroma\n",
|
|
"from langchain.embeddings import HuggingFaceEmbeddings\n",
|
|
"from langchain.embeddings import OpenAIEmbeddings\n",
|
|
"from langchain.document_transformers import EmbeddingsRedundantFilter\n",
|
|
"from langchain.retrievers.document_compressors import DocumentCompressorPipeline\n",
|
|
"from langchain.retrievers import ContextualCompressionRetriever\n",
|
|
"\n",
|
|
"# Get 3 diff embeddings.\n",
|
|
"all_mini = HuggingFaceEmbeddings(model_name=\"all-MiniLM-L6-v2\")\n",
|
|
"multi_qa_mini = HuggingFaceEmbeddings(model_name=\"multi-qa-MiniLM-L6-dot-v1\")\n",
|
|
"filter_embeddings = OpenAIEmbeddings()\n",
|
|
"\n",
|
|
"ABS_PATH = os.path.dirname(os.path.abspath(__file__))\n",
|
|
"DB_DIR = os.path.join(ABS_PATH, \"db\")\n",
|
|
"\n",
|
|
"# Instantiate 2 diff cromadb indexs, each one with a diff embedding.\n",
|
|
"client_settings = chromadb.config.Settings(\n",
|
|
" chroma_db_impl=\"duckdb+parquet\",\n",
|
|
" persist_directory=DB_DIR,\n",
|
|
" anonymized_telemetry=False,\n",
|
|
")\n",
|
|
"db_all = Chroma(\n",
|
|
" collection_name=\"project_store_all\",\n",
|
|
" persist_directory=DB_DIR,\n",
|
|
" client_settings=client_settings,\n",
|
|
" embedding_function=all_mini,\n",
|
|
")\n",
|
|
"db_multi_qa = Chroma(\n",
|
|
" collection_name=\"project_store_multi\",\n",
|
|
" persist_directory=DB_DIR,\n",
|
|
" client_settings=client_settings,\n",
|
|
" embedding_function=multi_qa_mini,\n",
|
|
")\n",
|
|
"\n",
|
|
"# Define 2 diff retrievers with 2 diff embeddings and diff search type.\n",
|
|
"retriever_all = db_all.as_retriever(\n",
|
|
" search_type=\"similarity\", search_kwargs={\"k\": 5, \"include_metadata\": True}\n",
|
|
")\n",
|
|
"retriever_multi_qa = db_multi_qa.as_retriever(\n",
|
|
" search_type=\"mmr\", search_kwargs={\"k\": 5, \"include_metadata\": True}\n",
|
|
")\n",
|
|
"\n",
|
|
"# The Lord of the Retrievers will hold the ouput of boths retrievers and can be used as any other\n",
|
|
"# retriever on different types of chains.\n",
|
|
"lotr = MergerRetriever(retrievers=[retriever_all, retriever_multi_qa])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "c152339d",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Remove redundant results from the merged retrievers."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "039faea6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# We can remove redundant results from both retrievers using yet another embedding.\n",
|
|
"# Using multiples embeddings in diff steps could help reduce biases.\n",
|
|
"filter = EmbeddingsRedundantFilter(embeddings=filter_embeddings)\n",
|
|
"pipeline = DocumentCompressorPipeline(transformers=[filter])\n",
|
|
"compression_retriever = ContextualCompressionRetriever(\n",
|
|
" base_compressor=pipeline, base_retriever=lotr\n",
|
|
")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.6"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|