forked from Archives/langchain
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
227 lines
12 KiB
Plaintext
227 lines
12 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ce0f17b9",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Vespa retriever\n",
|
|
"\n",
|
|
"This notebook shows how to use Vespa.ai as a LangChain retriever.\n",
|
|
"Vespa.ai is a platform for highly efficient structured text and vector search.\n",
|
|
"Please refer to [Vespa.ai](https://vespa.ai) for more information.\n",
|
|
"\n",
|
|
"In this example we'll work with the public [cord-19-search](https://github.com/vespa-cloud/cord-19-search) app which serves an index for the [CORD-19](https://allenai.org/data/cord-19) dataset containing Covid-19 research papers.\n",
|
|
"\n",
|
|
"In order to create a retriever, we use [pyvespa](https://pyvespa.readthedocs.io/en/latest/index.html) to\n",
|
|
"create a connection a Vespa service."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "101c8eb3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Uncomment below if you haven't install pyvespa\n",
|
|
"\n",
|
|
"# !pip install pyvespa"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "9f0406d2",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def _pretty_print(docs):\n",
|
|
" for doc in docs:\n",
|
|
" print(\"-\" * 80)\n",
|
|
" print(\"CONTENT: \" + doc.page_content + \"\\n\")\n",
|
|
" print(\"METADATA: \" + str(doc.metadata))\n",
|
|
" print(\"-\" * 80)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3db3bfea",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Retrieving documents"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "d83331fa",
|
|
"metadata": {
|
|
"pycharm": {
|
|
"name": "#%%\n"
|
|
}
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.retrievers import VespaRetriever\n",
|
|
"\n",
|
|
"# Retrieve the abstracts of the top 2 papers that best match the user query.\n",
|
|
"retriever = VespaRetriever.from_params(\n",
|
|
" 'https://api.cord19.vespa.ai', \n",
|
|
" \"abstract\",\n",
|
|
" k=2,\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "f47a2bfe",
|
|
"metadata": {
|
|
"pycharm": {
|
|
"name": "#%%\n"
|
|
}
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"--------------------------------------------------------------------------------\n",
|
|
"CONTENT: <sep />and peak hospitalizations by 4-96x, without contact tracing. Although contact tracing was highly <hi>effective</hi> at reducing spread, it was insufficient to stop outbreaks caused by <hi>travellers</hi> in even the best-case scenario, and the likelihood of exceeding contact tracing capacity was a concern in most scenarios. Quarantine compliance had only a small impact on <hi>COVID</hi> spread; <hi>travel</hi> volume and infection rate drove spread. Interpretation: NL's <hi>travel</hi> <hi>ban</hi> was likely a critically important intervention to prevent <hi>COVID</hi> spread. Even a small number<sep />\n",
|
|
"\n",
|
|
"METADATA: {'id': 'index:content/1/544bbfee3466d2c126719d5f'}\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"CONTENT: How <hi>effective</hi> are restrictions on mobility in limiting <hi>COVID</hi>-19 spread? Using zip code data across five U.S. cities, we estimate that total cases per capita decrease by 20% for every ten percentage point fall in mobility. Addressing endogeneity concerns, we instrument for <hi>travel</hi> by residential teleworkable and essential shares and find a 27% decline in cases per capita. Using panel data for NYC with week and zip code fixed effects, we estimate a decline of 17%. We find substantial spatial and temporal heterogeneity;east coast cities have stronger effects, with the largest for NYC<sep />\n",
|
|
"\n",
|
|
"METADATA: {'id': 'index:content/0/911dfc6986f1c8bc15fc3a26'}\n",
|
|
"--------------------------------------------------------------------------------\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"docs = retriever.get_relevant_documents(\"How effective are covid travel bans?\")\n",
|
|
"_pretty_print(docs)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4a158b8e",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Configuring the retriever\n",
|
|
"We can further configure our results by specifying metadata fields to retrieve, specifying sources to pull from, adding filters and adding index-specific parameters."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "dc6be773",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"--------------------------------------------------------------------------------\n",
|
|
"CONTENT: ...and peak hospitalizations by 4-96x, without contact tracing. Although contact tracing was highly effective at reducing spread, it was insufficient to stop outbreaks caused by travellers in even the best-case scenario, and the likelihood of exceeding contact tracing capacity was a concern in most scenarios. Quarantine compliance had only a small impact on COVID spread; travel volume and infection rate drove spread. Interpretation: NL's travel ban was likely a critically important intervention to prevent COVID spread. Even a small number...\n",
|
|
"\n",
|
|
"METADATA: {'matchfeatures': {'bm25': 35.5404665009022, 'colbert_maxsim': 78.48671418428421}, 'sddocname': 'doc', 'title': \"How effective was Newfoundland & Labrador's travel ban to prevent the spread of COVID-19? An agent-based analysis\", 'id': 'index:content/1/544bbfee3466d2c126719d5f', 'timestamp': 1612738800, 'license': 'medrxiv', 'doi': 'https://doi.org/10.1101/2021.02.05.21251157', 'authors': [{'first': ' D. M.', 'name': ' D. M. Aleman', 'last': 'Aleman'}, {'first': ' B. Z.', 'name': ' B. Z. Tham', 'last': ' Tham'}, {'first': ' S. J.', 'name': ' S. J. Wagner', 'last': ' Wagner'}, {'first': ' J.', 'name': ' J. Semelhago', 'last': ' Semelhago'}, {'first': ' A.', 'name': ' A. Mohammadi', 'last': ' Mohammadi'}, {'first': ' P.', 'name': ' P. Price', 'last': ' Price'}, {'first': ' R.', 'name': ' R. Giffen', 'last': ' Giffen'}, {'first': ' P.', 'name': ' P. Rahman', 'last': ' Rahman'}], 'source': 'MedRxiv; WHO', 'cord_uid': '9b9kt4sp'}\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"CONTENT: ...reduction in COVID-19 importation and a delay of the COVID-19 outbreak in Australia by approximately one month. Further projection of COVID-19 to May 2020 showed spread patterns depending on the basic reproduction number. CONCLUSION: Imposing the travel ban was effective in delaying widespread transmission of COVID-19. However, strengthening of the domestic control measures is needed to prevent Australia from becoming another epicentre. Implications for public health: This report has shown the importance of border closure to pandemic control.\n",
|
|
"\n",
|
|
"METADATA: {'matchfeatures': {'bm25': 32.398379319326295, 'colbert_maxsim': 73.91238763928413}, 'sddocname': 'doc', 'title': 'Delaying the COVID-19 epidemic in Australia: evaluating the effectiveness of international travel bans', 'id': 'index:content/1/decd6a8642418607b0d7dff9', 'timestamp': 0, 'license': 'unk', 'authors': [{'first': ' Adeshina', 'name': ' Adeshina Adekunle', 'last': 'Adekunle'}, {'first': ' Michael', 'name': ' Michael Meehan', 'last': ' Meehan'}, {'first': ' Diana', 'name': ' Diana Rojas-Alvarez', 'last': ' Rojas-Alvarez'}, {'first': ' James', 'name': ' James Trauer', 'last': ' Trauer'}, {'first': ' Emma', 'name': ' Emma McBryde', 'last': ' McBryde'}], 'source': 'WHO', 'cord_uid': 'jdh33itm', 'journal': 'Aust N Z J Public Health'}\n",
|
|
"--------------------------------------------------------------------------------\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"retriever = VespaRetriever.from_params(\n",
|
|
" 'https://api.cord19.vespa.ai', \n",
|
|
" \"abstract\",\n",
|
|
" k=2,\n",
|
|
" metadata_fields=\"*\", # return all data fields and store as metadata\n",
|
|
" ranking=\"hybrid-colbert\", # other valid values: colbert, bm25\n",
|
|
" bolding=False,\n",
|
|
")\n",
|
|
"docs = retriever.get_relevant_documents(\"How effective are covid travel bans?\")\n",
|
|
"_pretty_print(docs)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "11242e84",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Querying with filtering conditions\n",
|
|
"\n",
|
|
"Vespa has powerful querying abilities, and lets you specify many different conditions in YQL. You can add these filtering conditions using the `get_relevant_documents_with_filter` function.\n",
|
|
"\n",
|
|
"Read more on the Vespa query language here: https://docs.vespa.ai/en/query-language.html"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "223aeaa9",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"--------------------------------------------------------------------------------\n",
|
|
"CONTENT: Importance: As countermeasures against the economic downturn caused by the coronavirus 2019 (COVID-19) pandemic, many countries have introduced or considering financial incentives for people to engage in economic activities such as travel and use restaurants. Japan has implemented a large-scale, nationwide government-funded program that subsidizes up to 50% of all travel expenses since July 2020 with the aim of reviving the travel industry. However, it remains unknown as to how such provision of government subsidies for travel impacted the COVID-19 pandemic...\n",
|
|
"\n",
|
|
"METADATA: {'matchfeatures': {'bm25': 22.54935242101209, 'colbert_maxsim': 55.04242363572121}, 'sddocname': 'doc', 'title': 'Association between Participation in Government Subsidy Program for Domestic Travel and Symptoms Indicative of COVID-19 Infection', 'journal': 'medRxiv : the preprint server for health sciences', 'id': 'index:content/0/d88422d1d176ab0a854caccc', 'timestamp': 1607036400, 'license': 'medrxiv', 'doi': 'https://doi.org/10.1101/2020.12.03.20243352', 'authors': [{'first': ' A.', 'name': ' A. Miyawaki', 'last': 'Miyawaki'}, {'first': ' T.', 'name': ' T. Tabuchi', 'last': ' Tabuchi'}, {'first': ' Y.', 'name': ' Y. Tomata', 'last': ' Tomata'}, {'first': ' Y.', 'name': ' Y. Tsugawa', 'last': ' Tsugawa'}], 'source': 'MedRxiv; Medline; WHO', 'cord_uid': '0isi7yd4'}\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"--------------------------------------------------------------------------------\n",
|
|
"CONTENT: The Japanese government has declared a national emergency and travel entry ban since the coronavirus disease 2019 (COVID-19) pandemic began. As of June 19, 2020, there have been no confirmed cases of COVID-19 in Iwate, a prefecture of Japan. Here, we analyzed the excess deaths as well as the number of patients and medical earnings due to the pandemic from prefectural ...\n",
|
|
"\n",
|
|
"METADATA: {'matchfeatures': {'bm25': 19.348708049098548, 'colbert_maxsim': 58.35367426276207}, 'sddocname': 'doc', 'title': 'Affected medical services in Iwate prefecture in the absence of a COVID-19 outbreak', 'id': 'index:content/1/9f27176791532b37ef8e4a24', 'timestamp': 1592604000, 'license': 'medrxiv', 'doi': 'https://doi.org/10.1101/2020.06.19.20135269', 'authors': [{'first': ' N.', 'name': ' N. Sasaki', 'last': 'Sasaki'}, {'first': ' S. S.', 'name': ' S. S. Nishizuka', 'last': ' Nishizuka'}], 'source': 'MedRxiv; WHO', 'cord_uid': '7egroqb1'}\n",
|
|
"--------------------------------------------------------------------------------\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"docs = retriever.get_relevant_documents_with_filter(\n",
|
|
" \"How effective are covid travel bans?\", \n",
|
|
" _filter='abstract contains \"Japan\" and license matches \"medrxiv\"'\n",
|
|
")\n",
|
|
"_pretty_print(docs)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "13039caf",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|