- Improved the search wrapper API to mirror the usage of the google
search one.
- Register searx-search as loadable tool
- Added documentation and example notebook
This page covers how to use the SearxNG search API within LangChain.
It is broken into two parts: installation and setup, and then references to the specific SearxNG API wrapper.
## Installation and Setup
- You can find a list of public SearxNG instances [here](https://searx.space/).
- It recommended to use a self-hosted instance to avoid abuse on the public instances. Also note that public instances often have a limit on the number of requests.
- To run a self-hosted instance see [this page](https://searxng.github.io/searxng/admin/installation.html) for more information.
- To use the tool you need to provide the searx host url by:
1. passing the named parameter `searx_host` when creating the instance.
2. exporting the environment variable `SEARXNG_HOST`.
## Wrappers
### Utility
You can use the wrapper to get results from a SearxNG instance.
```python
from langchain.utilities import SearxSearchWrapper
```
### Tool
You can also easily load this wrapper as a Tool (to use with an Agent).
"'In all, 45 individuals have served 46 presidencies spanning 58 full four-year terms. Joe Biden is the 46th and current president of the United States, having assumed office on January 20, 2021.'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"search.run(\"Who is the current president of the united states of america?\")"
]
},
{
"cell_type": "markdown",
"id": "bf728728",
"metadata": {},
"source": [
"For some engines, if a direct `answer` is available the warpper will print the answer instead of the full search results. You can use the `results` method of the wrapper if you want to obtain all the results."
]
},
{
"cell_type": "markdown",
"id": "cbac93d4",
"metadata": {},
"source": [
"\n",
"# Custom Parameters\n",
"\n",
"SearxNG supports up to [139 search engines](https://docs.searxng.org/admin/engines/configured_engines.html#configured-engines). You can also customize the Searx wrapper with arbitrary named parameters that will be passed to the Searx search API . In the below example we will making a more interesting use of custom search parameters from searx search api."
]
},
{
"cell_type": "markdown",
"id": "7844deaa",
"metadata": {},
"source": [
"In this example we will be using the `engines` parameters to query wikipedia"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "1517e24b",
"metadata": {},
"outputs": [],
"source": [
"search = SearxSearchWrapper(searx_host=\"http://127.0.0.1:8888\", k=5) # k is for max number of items"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "4ded48b0",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Large language models (LLMs) represent a major advancement in AI, with the promise of transforming domains through learned knowledge. LLM sizes have been increasing 10X every year for the last few years, and as these models grow in complexity and size, so do their capabilities.\\n\\nGPT-3 can translate language, write essays, generate computer code, and more — all with limited to no supervision. In July 2020, OpenAI unveiled GPT-3, a language model that was easily the largest known at the time. Put simply, GPT-3 is trained to predict the next word in a sentence, much like how a text message autocomplete feature works.\\n\\nAll of today’s well-known language models—e.g., GPT-3 from OpenAI, PaLM or LaMDA from Google, Galactica or OPT from Meta, Megatron-Turing from Nvidia/Microsoft, Jurassic-1 from AI21 Labs—are...\\n\\nLarge language models are computer programs that open new possibilities of text understanding and generation in software systems. Consider this: ...\\n\\nLarge language models (LLMs) such as GPT-3are increasingly being used to generate text. These tools should be used with care, since they can generate content that is biased, non-verifiable, constitutes original research, or violates copyrights.'"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"search.run(\"large language model \", engines='wiki')"
]
},
{
"cell_type": "markdown",
"id": "259f5a5b",
"metadata": {},
"source": [
"## Obtaining results with metadata"
]
},
{
"cell_type": "markdown",
"id": "3c4cf1db",
"metadata": {},
"source": [
"In this example we will be looking for scientific paper using the `categories` parameter and limiting the results to a `time_range` (not all engines support the time range option).\n",
"\n",
"We also would like to obtain the results in a structured way including metadata. For this we will be using the `results` method of the wrapper."
"[{'snippet': '… on natural language instructions, large language models (… the prompt used to steer the model, and most effective prompts … to prompt engineering, we propose Automatic Prompt …',\n",
" 'title': 'Large language models are human-level prompt engineers',\n",
" {'snippet': '… Large language models (LLMs) have introduced new possibilities for prototyping with AI [18]. Pre-trained on a large amount of text data, models … language instructions called prompts. …',\n",
" 'title': 'Promptchainer: Chaining large language model prompts through visual programming',\n",
" {'snippet': '… can introspect the large prompt model. We derive the view ϕ0(X) and the model h0 from T01. However, instead of fully fine-tuning T0 during co-training, we focus on soft prompt tuning, …',\n",
" 'title': 'Co-training improves prompt-based learning for large language models',\n",
" {'snippet': '… With the success of large language models (LLMs) of code and their use as … prompt design process become important. In this work, we propose a framework called Repo-Level Prompt …',\n",
" 'title': 'Repository-level prompt generation for large language models of code',\n",
" {'snippet': '… Figure 2 | The benefits of different components of a prompt for the largest language model (Gopher), as estimated from hierarchical logistic regression. Each point estimates the unique …',\n",
" 'title': 'Can language models learn from explanations in context?',\n",
" 'link': 'https://arxiv.org/abs/2204.02329'}]"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"search.results(\"Large Language Model prompt\", num_results=5, categories='science', time_range='year')"
blob42
1 year ago