forked from Archives/langchain
docs `retriever` improvements (#4430)
# Docs: improvements in the `retrievers/examples/` notebooks Its primary purpose is to make the Jupyter notebook examples **consistent** and more suitable for first-time viewers. - add links to the integration source (if applicable) with a short description of this source; - removed `_retriever` suffix from the file names (where it existed) for consistency; - removed ` retriever` from the notebook title (where it existed) for consistency; - added code to install necessary Python package(s); - added code to set up the necessary API Key. - very small fixes in notebooks from other folders (for consistency): - docs/modules/indexes/vectorstores/examples/elasticsearch.ipynb - docs/modules/indexes/vectorstores/examples/pinecone.ipynb - docs/modules/models/llms/integrations/cohere.ipynb - fixed misspelling in langchain/retrievers/time_weighted_retriever.py comment (sorry, about this change in a .py file ) ## Who can review @dev2049docker
Leonid Ganeline
1 year ago
committed by
GitHub
parent
0147f845f1
commit
b96ab4b763
@ -0,0 +1,138 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ce0f17b9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Vespa\n",
|
||||
"\n",
|
||||
">[Vespa](https://vespa.ai/) is a fully featured search engine and vector database. It supports vector search (ANN), lexical search, and search in structured data, all in the same query.\n",
|
||||
"\n",
|
||||
"This notebook shows how to use `Vespa.ai` as a LangChain retriever.\n",
|
||||
"\n",
|
||||
"In order to create a retriever, we use [pyvespa](https://pyvespa.readthedocs.io/en/latest/index.html) to\n",
|
||||
"create a connection a `Vespa` service."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7e6a11ab-38bd-4920-ba11-60cb2f075754",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"#!pip install pyvespa"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "c10dd962",
|
||||
"metadata": {
|
||||
"tags": []
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from vespa.application import Vespa\n",
|
||||
"\n",
|
||||
"vespa_app = Vespa(url=\"https://doc-search.vespa.oath.cloud\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3df4ce53",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"This creates a connection to a `Vespa` service, here the Vespa documentation search service.\n",
|
||||
"Using `pyvespa` package, you can also connect to a\n",
|
||||
"[Vespa Cloud instance](https://pyvespa.readthedocs.io/en/latest/deploy-vespa-cloud.html)\n",
|
||||
"or a local\n",
|
||||
"[Docker instance](https://pyvespa.readthedocs.io/en/latest/deploy-docker.html).\n",
|
||||
"\n",
|
||||
"\n",
|
||||
"After connecting to the service, you can set up the retriever:"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "7ccca1f4",
|
||||
"metadata": {
|
||||
"pycharm": {
|
||||
"name": "#%%\n"
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.retrievers.vespa_retriever import VespaRetriever\n",
|
||||
"\n",
|
||||
"vespa_query_body = {\n",
|
||||
" \"yql\": \"select content from paragraph where userQuery()\",\n",
|
||||
" \"hits\": 5,\n",
|
||||
" \"ranking\": \"documentation\",\n",
|
||||
" \"locale\": \"en-us\"\n",
|
||||
"}\n",
|
||||
"vespa_content_field = \"content\"\n",
|
||||
"retriever = VespaRetriever(vespa_app, vespa_query_body, vespa_content_field)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "1e7e34e1",
|
||||
"metadata": {
|
||||
"pycharm": {
|
||||
"name": "#%% md\n"
|
||||
}
|
||||
},
|
||||
"source": [
|
||||
"This sets up a LangChain retriever that fetches documents from the Vespa application.\n",
|
||||
"Here, up to 5 results are retrieved from the `content` field in the `paragraph` document type,\n",
|
||||
"using `doumentation` as the ranking method. The `userQuery()` is replaced with the actual query\n",
|
||||
"passed from LangChain.\n",
|
||||
"\n",
|
||||
"Please refer to the [pyvespa documentation](https://pyvespa.readthedocs.io/en/latest/getting-started-pyvespa.html#Query)\n",
|
||||
"for more information.\n",
|
||||
"\n",
|
||||
"Now you can return the results and continue using the results in LangChain."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "f47a2bfe",
|
||||
"metadata": {
|
||||
"pycharm": {
|
||||
"name": "#%%\n"
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"retriever.get_relevant_documents(\"what is vespa?\")"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.6"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@ -1,226 +0,0 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "ce0f17b9",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Vespa retriever\n",
|
||||
"\n",
|
||||
"This notebook shows how to use Vespa.ai as a LangChain retriever.\n",
|
||||
"Vespa.ai is a platform for highly efficient structured text and vector search.\n",
|
||||
"Please refer to [Vespa.ai](https://vespa.ai) for more information.\n",
|
||||
"\n",
|
||||
"In this example we'll work with the public [cord-19-search](https://github.com/vespa-cloud/cord-19-search) app which serves an index for the [CORD-19](https://allenai.org/data/cord-19) dataset containing Covid-19 research papers.\n",
|
||||
"\n",
|
||||
"In order to create a retriever, we use [pyvespa](https://pyvespa.readthedocs.io/en/latest/index.html) to\n",
|
||||
"create a connection a Vespa service."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 1,
|
||||
"id": "101c8eb3",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"# Uncomment below if you haven't install pyvespa\n",
|
||||
"\n",
|
||||
"# !pip install pyvespa"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "9f0406d2",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"def _pretty_print(docs):\n",
|
||||
" for doc in docs:\n",
|
||||
" print(\"-\" * 80)\n",
|
||||
" print(\"CONTENT: \" + doc.page_content + \"\\n\")\n",
|
||||
" print(\"METADATA: \" + str(doc.metadata))\n",
|
||||
" print(\"-\" * 80)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "3db3bfea",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Retrieving documents"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "d83331fa",
|
||||
"metadata": {
|
||||
"pycharm": {
|
||||
"name": "#%%\n"
|
||||
}
|
||||
},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.retrievers import VespaRetriever\n",
|
||||
"\n",
|
||||
"# Retrieve the abstracts of the top 2 papers that best match the user query.\n",
|
||||
"retriever = VespaRetriever.from_params(\n",
|
||||
" 'https://api.cord19.vespa.ai', \n",
|
||||
" \"abstract\",\n",
|
||||
" k=2,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "f47a2bfe",
|
||||
"metadata": {
|
||||
"pycharm": {
|
||||
"name": "#%%\n"
|
||||
}
|
||||
},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"CONTENT: <sep />and peak hospitalizations by 4-96x, without contact tracing. Although contact tracing was highly <hi>effective</hi> at reducing spread, it was insufficient to stop outbreaks caused by <hi>travellers</hi> in even the best-case scenario, and the likelihood of exceeding contact tracing capacity was a concern in most scenarios. Quarantine compliance had only a small impact on <hi>COVID</hi> spread; <hi>travel</hi> volume and infection rate drove spread. Interpretation: NL's <hi>travel</hi> <hi>ban</hi> was likely a critically important intervention to prevent <hi>COVID</hi> spread. Even a small number<sep />\n",
|
||||
"\n",
|
||||
"METADATA: {'id': 'index:content/1/544bbfee3466d2c126719d5f'}\n",
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"CONTENT: How <hi>effective</hi> are restrictions on mobility in limiting <hi>COVID</hi>-19 spread? Using zip code data across five U.S. cities, we estimate that total cases per capita decrease by 20% for every ten percentage point fall in mobility. Addressing endogeneity concerns, we instrument for <hi>travel</hi> by residential teleworkable and essential shares and find a 27% decline in cases per capita. Using panel data for NYC with week and zip code fixed effects, we estimate a decline of 17%. We find substantial spatial and temporal heterogeneity;east coast cities have stronger effects, with the largest for NYC<sep />\n",
|
||||
"\n",
|
||||
"METADATA: {'id': 'index:content/0/911dfc6986f1c8bc15fc3a26'}\n",
|
||||
"--------------------------------------------------------------------------------\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"docs = retriever.get_relevant_documents(\"How effective are covid travel bans?\")\n",
|
||||
"_pretty_print(docs)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "4a158b8e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Configuring the retriever\n",
|
||||
"We can further configure our results by specifying metadata fields to retrieve, specifying sources to pull from, adding filters and adding index-specific parameters."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 8,
|
||||
"id": "dc6be773",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"CONTENT: ...and peak hospitalizations by 4-96x, without contact tracing. Although contact tracing was highly effective at reducing spread, it was insufficient to stop outbreaks caused by travellers in even the best-case scenario, and the likelihood of exceeding contact tracing capacity was a concern in most scenarios. Quarantine compliance had only a small impact on COVID spread; travel volume and infection rate drove spread. Interpretation: NL's travel ban was likely a critically important intervention to prevent COVID spread. Even a small number...\n",
|
||||
"\n",
|
||||
"METADATA: {'matchfeatures': {'bm25': 35.5404665009022, 'colbert_maxsim': 78.48671418428421}, 'sddocname': 'doc', 'title': \"How effective was Newfoundland & Labrador's travel ban to prevent the spread of COVID-19? An agent-based analysis\", 'id': 'index:content/1/544bbfee3466d2c126719d5f', 'timestamp': 1612738800, 'license': 'medrxiv', 'doi': 'https://doi.org/10.1101/2021.02.05.21251157', 'authors': [{'first': ' D. M.', 'name': ' D. M. Aleman', 'last': 'Aleman'}, {'first': ' B. Z.', 'name': ' B. Z. Tham', 'last': ' Tham'}, {'first': ' S. J.', 'name': ' S. J. Wagner', 'last': ' Wagner'}, {'first': ' J.', 'name': ' J. Semelhago', 'last': ' Semelhago'}, {'first': ' A.', 'name': ' A. Mohammadi', 'last': ' Mohammadi'}, {'first': ' P.', 'name': ' P. Price', 'last': ' Price'}, {'first': ' R.', 'name': ' R. Giffen', 'last': ' Giffen'}, {'first': ' P.', 'name': ' P. Rahman', 'last': ' Rahman'}], 'source': 'MedRxiv; WHO', 'cord_uid': '9b9kt4sp'}\n",
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"CONTENT: ...reduction in COVID-19 importation and a delay of the COVID-19 outbreak in Australia by approximately one month. Further projection of COVID-19 to May 2020 showed spread patterns depending on the basic reproduction number. CONCLUSION: Imposing the travel ban was effective in delaying widespread transmission of COVID-19. However, strengthening of the domestic control measures is needed to prevent Australia from becoming another epicentre. Implications for public health: This report has shown the importance of border closure to pandemic control.\n",
|
||||
"\n",
|
||||
"METADATA: {'matchfeatures': {'bm25': 32.398379319326295, 'colbert_maxsim': 73.91238763928413}, 'sddocname': 'doc', 'title': 'Delaying the COVID-19 epidemic in Australia: evaluating the effectiveness of international travel bans', 'id': 'index:content/1/decd6a8642418607b0d7dff9', 'timestamp': 0, 'license': 'unk', 'authors': [{'first': ' Adeshina', 'name': ' Adeshina Adekunle', 'last': 'Adekunle'}, {'first': ' Michael', 'name': ' Michael Meehan', 'last': ' Meehan'}, {'first': ' Diana', 'name': ' Diana Rojas-Alvarez', 'last': ' Rojas-Alvarez'}, {'first': ' James', 'name': ' James Trauer', 'last': ' Trauer'}, {'first': ' Emma', 'name': ' Emma McBryde', 'last': ' McBryde'}], 'source': 'WHO', 'cord_uid': 'jdh33itm', 'journal': 'Aust N Z J Public Health'}\n",
|
||||
"--------------------------------------------------------------------------------\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"retriever = VespaRetriever.from_params(\n",
|
||||
" 'https://api.cord19.vespa.ai', \n",
|
||||
" \"abstract\",\n",
|
||||
" k=2,\n",
|
||||
" metadata_fields=\"*\", # return all data fields and store as metadata\n",
|
||||
" ranking=\"hybrid-colbert\", # other valid values: colbert, bm25\n",
|
||||
" bolding=False,\n",
|
||||
")\n",
|
||||
"docs = retriever.get_relevant_documents(\"How effective are covid travel bans?\")\n",
|
||||
"_pretty_print(docs)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "11242e84",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Querying with filtering conditions\n",
|
||||
"\n",
|
||||
"Vespa has powerful querying abilities, and lets you specify many different conditions in YQL. You can add these filtering conditions using the `get_relevant_documents_with_filter` function.\n",
|
||||
"\n",
|
||||
"Read more on the Vespa query language here: https://docs.vespa.ai/en/query-language.html"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 11,
|
||||
"id": "223aeaa9",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"CONTENT: Importance: As countermeasures against the economic downturn caused by the coronavirus 2019 (COVID-19) pandemic, many countries have introduced or considering financial incentives for people to engage in economic activities such as travel and use restaurants. Japan has implemented a large-scale, nationwide government-funded program that subsidizes up to 50% of all travel expenses since July 2020 with the aim of reviving the travel industry. However, it remains unknown as to how such provision of government subsidies for travel impacted the COVID-19 pandemic...\n",
|
||||
"\n",
|
||||
"METADATA: {'matchfeatures': {'bm25': 22.54935242101209, 'colbert_maxsim': 55.04242363572121}, 'sddocname': 'doc', 'title': 'Association between Participation in Government Subsidy Program for Domestic Travel and Symptoms Indicative of COVID-19 Infection', 'journal': 'medRxiv : the preprint server for health sciences', 'id': 'index:content/0/d88422d1d176ab0a854caccc', 'timestamp': 1607036400, 'license': 'medrxiv', 'doi': 'https://doi.org/10.1101/2020.12.03.20243352', 'authors': [{'first': ' A.', 'name': ' A. Miyawaki', 'last': 'Miyawaki'}, {'first': ' T.', 'name': ' T. Tabuchi', 'last': ' Tabuchi'}, {'first': ' Y.', 'name': ' Y. Tomata', 'last': ' Tomata'}, {'first': ' Y.', 'name': ' Y. Tsugawa', 'last': ' Tsugawa'}], 'source': 'MedRxiv; Medline; WHO', 'cord_uid': '0isi7yd4'}\n",
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"--------------------------------------------------------------------------------\n",
|
||||
"CONTENT: The Japanese government has declared a national emergency and travel entry ban since the coronavirus disease 2019 (COVID-19) pandemic began. As of June 19, 2020, there have been no confirmed cases of COVID-19 in Iwate, a prefecture of Japan. Here, we analyzed the excess deaths as well as the number of patients and medical earnings due to the pandemic from prefectural ...\n",
|
||||
"\n",
|
||||
"METADATA: {'matchfeatures': {'bm25': 19.348708049098548, 'colbert_maxsim': 58.35367426276207}, 'sddocname': 'doc', 'title': 'Affected medical services in Iwate prefecture in the absence of a COVID-19 outbreak', 'id': 'index:content/1/9f27176791532b37ef8e4a24', 'timestamp': 1592604000, 'license': 'medrxiv', 'doi': 'https://doi.org/10.1101/2020.06.19.20135269', 'authors': [{'first': ' N.', 'name': ' N. Sasaki', 'last': 'Sasaki'}, {'first': ' S. S.', 'name': ' S. S. Nishizuka', 'last': ' Nishizuka'}], 'source': 'MedRxiv; WHO', 'cord_uid': '7egroqb1'}\n",
|
||||
"--------------------------------------------------------------------------------\n"
|
||||
]
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"docs = retriever.get_relevant_documents_with_filter(\n",
|
||||
" \"How effective are covid travel bans?\", \n",
|
||||
" _filter='abstract contains \"Japan\" and license matches \"medrxiv\"'\n",
|
||||
")\n",
|
||||
"_pretty_print(docs)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"id": "13039caf",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
Loading…
Reference in New Issue