forked from Archives/langchain
Google Search API integration with serper.dev (wrapper, tests, docs, … (#909)
Adds Google Search integration with [Serper](https://serper.dev) a low-cost alternative to SerpAPI (10x cheaper + generous free tier). Includes documentation, tests and examples. Hopefully I am not missing anything. Developers can sign up for a free account at [serper.dev](https://serper.dev) and obtain an api key. ## Usage ```python from langchain.utilities import GoogleSerperAPIWrapper from langchain.llms.openai import OpenAI from langchain.agents import initialize_agent, Tool import os os.environ["SERPER_API_KEY"] = "" os.environ['OPENAI_API_KEY'] = "" llm = OpenAI(temperature=0) search = GoogleSerperAPIWrapper() tools = [ Tool( name="Intermediate Answer", func=search.run ) ] self_ask_with_search = initialize_agent(tools, llm, agent="self-ask-with-search", verbose=True) self_ask_with_search.run("What is the hometown of the reigning men's U.S. Open champion?") ``` ### Output ``` Entering new AgentExecutor chain... Yes. Follow up: Who is the reigning men's U.S. Open champion? Intermediate answer: Current champions Carlos Alcaraz, 2022 men's singles champion. Follow up: Where is Carlos Alcaraz from? Intermediate answer: El Palmar, Spain So the final answer is: El Palmar, Spain > Finished chain. 'El Palmar, Spain' ```searx-api
rogerserper
1 year ago
committed by
GitHub
parent
52753066ef
commit
e46cd3b7db
@ -0,0 +1,70 @@
|
|||||||
|
# Google Serper Wrapper
|
||||||
|
|
||||||
|
This page covers how to use the [Serper](https://serper.dev) Google Search API within LangChain. Serper is a low-cost Google Search API that can be used to add answer box, knowledge graph, and organic results data from Google Search.
|
||||||
|
It is broken into two parts: setup, and then references to the specific Google Serper wrapper.
|
||||||
|
|
||||||
|
## Setup
|
||||||
|
- Go to [serper.dev](https://serper.dev) to sign up for a free account
|
||||||
|
- Get the api key and set it as an environment variable (`SERPER_API_KEY`)
|
||||||
|
|
||||||
|
## Wrappers
|
||||||
|
|
||||||
|
### Utility
|
||||||
|
|
||||||
|
There exists a GoogleSerperAPIWrapper utility which wraps this API. To import this utility:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from langchain.utilities import GoogleSerperAPIWrapper
|
||||||
|
```
|
||||||
|
|
||||||
|
You can use it as part of a Self Ask chain:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from langchain.utilities import GoogleSerperAPIWrapper
|
||||||
|
from langchain.llms.openai import OpenAI
|
||||||
|
from langchain.agents import initialize_agent, Tool
|
||||||
|
|
||||||
|
import os
|
||||||
|
os.environ["SERPER_API_KEY"] = ""
|
||||||
|
os.environ['OPENAI_API_KEY'] = ""
|
||||||
|
|
||||||
|
llm = OpenAI(temperature=0)
|
||||||
|
search = GoogleSerperAPIWrapper()
|
||||||
|
tools = [
|
||||||
|
Tool(
|
||||||
|
name="Intermediate Answer",
|
||||||
|
func=search.run
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
self_ask_with_search = initialize_agent(tools, llm, agent="self-ask-with-search", verbose=True)
|
||||||
|
self_ask_with_search.run("What is the hometown of the reigning men's U.S. Open champion?")
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Output
|
||||||
|
```
|
||||||
|
Entering new AgentExecutor chain...
|
||||||
|
Yes.
|
||||||
|
Follow up: Who is the reigning men's U.S. Open champion?
|
||||||
|
Intermediate answer: Current champions Carlos Alcaraz, 2022 men's singles champion.
|
||||||
|
Follow up: Where is Carlos Alcaraz from?
|
||||||
|
Intermediate answer: El Palmar, Spain
|
||||||
|
So the final answer is: El Palmar, Spain
|
||||||
|
|
||||||
|
> Finished chain.
|
||||||
|
|
||||||
|
'El Palmar, Spain'
|
||||||
|
```
|
||||||
|
|
||||||
|
For a more detailed walkthrough of this wrapper, see [this notebook](../modules/utils/examples/google_serper.ipynb).
|
||||||
|
|
||||||
|
### Tool
|
||||||
|
|
||||||
|
You can also easily load this wrapper as a Tool (to use with an Agent).
|
||||||
|
You can do this with:
|
||||||
|
```python
|
||||||
|
from langchain.agents import load_tools
|
||||||
|
tools = load_tools(["google-serper"])
|
||||||
|
```
|
||||||
|
|
||||||
|
For more information on this, see [this page](../modules/agents/tools.md)
|
||||||
@ -0,0 +1,157 @@
|
|||||||
|
{
|
||||||
|
"cells": [
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"id": "dc23c48e",
|
||||||
|
"metadata": {},
|
||||||
|
"source": [
|
||||||
|
"# Google Serper API\n",
|
||||||
|
"\n",
|
||||||
|
"This notebook goes over how to use the Google Serper component to search the web. First you need to sign up for a free account at [serper.dev](https://serper.dev) and get your api key."
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"import os\n",
|
||||||
|
"os.environ[\"SERPER_API_KEY\"] = \"\""
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"collapsed": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 2,
|
||||||
|
"id": "54bf5afd",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"from langchain.utilities import GoogleSerperAPIWrapper"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 3,
|
||||||
|
"id": "31f8f382",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"search = GoogleSerperAPIWrapper()"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 4,
|
||||||
|
"id": "25ce0225",
|
||||||
|
"metadata": {},
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": "'Barack Hussein Obama II'"
|
||||||
|
},
|
||||||
|
"execution_count": 4,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"search.run(\"Obama's first name?\")"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "markdown",
|
||||||
|
"source": [
|
||||||
|
"## As part of a Self Ask With Search Chain"
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"collapsed": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": null,
|
||||||
|
"outputs": [],
|
||||||
|
"source": [
|
||||||
|
"os.environ['OPENAI_API_KEY'] = \"\""
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"collapsed": false
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"cell_type": "code",
|
||||||
|
"execution_count": 5,
|
||||||
|
"outputs": [
|
||||||
|
{
|
||||||
|
"name": "stdout",
|
||||||
|
"output_type": "stream",
|
||||||
|
"text": [
|
||||||
|
"\n",
|
||||||
|
"\n",
|
||||||
|
"\u001B[1m> Entering new AgentExecutor chain...\u001B[0m\n",
|
||||||
|
"\u001B[32;1m\u001B[1;3m Yes.\n",
|
||||||
|
"Follow up: Who is the reigning men's U.S. Open champion?\u001B[0m\n",
|
||||||
|
"Intermediate answer: \u001B[36;1m\u001B[1;3mCurrent champions Carlos Alcaraz, 2022 men's singles champion.\u001B[0m\n",
|
||||||
|
"\u001B[32;1m\u001B[1;3mFollow up: Where is Carlos Alcaraz from?\u001B[0m\n",
|
||||||
|
"Intermediate answer: \u001B[36;1m\u001B[1;3mEl Palmar, Spain\u001B[0m\n",
|
||||||
|
"\u001B[32;1m\u001B[1;3mSo the final answer is: El Palmar, Spain\u001B[0m\n",
|
||||||
|
"\n",
|
||||||
|
"\u001B[1m> Finished chain.\u001B[0m\n"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"data": {
|
||||||
|
"text/plain": "'El Palmar, Spain'"
|
||||||
|
},
|
||||||
|
"execution_count": 5,
|
||||||
|
"metadata": {},
|
||||||
|
"output_type": "execute_result"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"source": [
|
||||||
|
"from langchain.utilities import GoogleSerperAPIWrapper\n",
|
||||||
|
"from langchain.llms.openai import OpenAI\n",
|
||||||
|
"from langchain.agents import initialize_agent, Tool\n",
|
||||||
|
"\n",
|
||||||
|
"llm = OpenAI(temperature=0)\n",
|
||||||
|
"search = GoogleSerperAPIWrapper()\n",
|
||||||
|
"tools = [\n",
|
||||||
|
" Tool(\n",
|
||||||
|
" name=\"Intermediate Answer\",\n",
|
||||||
|
" func=search.run\n",
|
||||||
|
" )\n",
|
||||||
|
"]\n",
|
||||||
|
"\n",
|
||||||
|
"self_ask_with_search = initialize_agent(tools, llm, agent=\"self-ask-with-search\", verbose=True)\n",
|
||||||
|
"self_ask_with_search.run(\"What is the hometown of the reigning men's U.S. Open champion?\")"
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"collapsed": false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"metadata": {
|
||||||
|
"kernelspec": {
|
||||||
|
"display_name": "Python 3 (ipykernel)",
|
||||||
|
"language": "python",
|
||||||
|
"name": "python3"
|
||||||
|
},
|
||||||
|
"language_info": {
|
||||||
|
"codemirror_mode": {
|
||||||
|
"name": "ipython",
|
||||||
|
"version": 3
|
||||||
|
},
|
||||||
|
"file_extension": ".py",
|
||||||
|
"mimetype": "text/x-python",
|
||||||
|
"name": "python",
|
||||||
|
"nbconvert_exporter": "python",
|
||||||
|
"pygments_lexer": "ipython3",
|
||||||
|
"version": "3.10.9"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"nbformat": 4,
|
||||||
|
"nbformat_minor": 5
|
||||||
|
}
|
||||||
@ -0,0 +1,94 @@
|
|||||||
|
"""Util that calls Google Search using the Serper.dev API."""
|
||||||
|
from typing import Dict, Optional
|
||||||
|
|
||||||
|
import requests
|
||||||
|
from pydantic.class_validators import root_validator
|
||||||
|
from pydantic.main import BaseModel
|
||||||
|
|
||||||
|
from langchain.utils import get_from_dict_or_env
|
||||||
|
|
||||||
|
|
||||||
|
class GoogleSerperAPIWrapper(BaseModel):
|
||||||
|
"""Wrapper around the Serper.dev Google Search API.
|
||||||
|
|
||||||
|
You can create a free API key at https://serper.dev.
|
||||||
|
|
||||||
|
To use, you should have the environment variable ``SERPER_API_KEY``
|
||||||
|
set with your API key, or pass `serper_api_key` as a named parameter
|
||||||
|
to the constructor.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
from langchain import GoogleSerperAPIWrapper
|
||||||
|
google_serper = GoogleSerperAPIWrapper()
|
||||||
|
"""
|
||||||
|
|
||||||
|
k: int = 10
|
||||||
|
gl: str = "us"
|
||||||
|
hl: str = "en"
|
||||||
|
serper_api_key: Optional[str] = None
|
||||||
|
|
||||||
|
@root_validator()
|
||||||
|
def validate_environment(cls, values: Dict) -> Dict:
|
||||||
|
"""Validate that api key exists in environment."""
|
||||||
|
serper_api_key = get_from_dict_or_env(
|
||||||
|
values, "serper_api_key", "SERPER_API_KEY"
|
||||||
|
)
|
||||||
|
values["serper_api_key"] = serper_api_key
|
||||||
|
|
||||||
|
return values
|
||||||
|
|
||||||
|
def run(self, query: str) -> str:
|
||||||
|
"""Run query through GoogleSearch and parse result."""
|
||||||
|
results = self._google_serper_search_results(query, gl=self.gl, hl=self.hl)
|
||||||
|
|
||||||
|
return self._parse_results(results)
|
||||||
|
|
||||||
|
def _parse_results(self, results: dict) -> str:
|
||||||
|
snippets = []
|
||||||
|
|
||||||
|
if results.get("answerBox"):
|
||||||
|
answer_box = results.get("answerBox", {})
|
||||||
|
if answer_box.get("answer"):
|
||||||
|
return answer_box.get("answer")
|
||||||
|
elif answer_box.get("snippet"):
|
||||||
|
return answer_box.get("snippet").replace("\n", " ")
|
||||||
|
elif answer_box.get("snippetHighlighted"):
|
||||||
|
return ", ".join(answer_box.get("snippetHighlighted"))
|
||||||
|
|
||||||
|
if results.get("knowledgeGraph"):
|
||||||
|
kg = results.get("knowledgeGraph", {})
|
||||||
|
title = kg.get("title")
|
||||||
|
entity_type = kg.get("type")
|
||||||
|
if entity_type:
|
||||||
|
snippets.append(f"{title}: {entity_type}.")
|
||||||
|
description = kg.get("description")
|
||||||
|
if description:
|
||||||
|
snippets.append(description)
|
||||||
|
for attribute, value in kg.get("attributes", {}).items():
|
||||||
|
snippets.append(f"{title} {attribute}: {value}.")
|
||||||
|
|
||||||
|
for result in results["organic"][: self.k]:
|
||||||
|
if "snippet" in result:
|
||||||
|
snippets.append(result["snippet"])
|
||||||
|
for attribute, value in result.get("attributes", {}).items():
|
||||||
|
snippets.append(f"{attribute}: {value}.")
|
||||||
|
|
||||||
|
if len(snippets) == 0:
|
||||||
|
return "No good Google Search Result was found"
|
||||||
|
|
||||||
|
return " ".join(snippets)
|
||||||
|
|
||||||
|
def _google_serper_search_results(self, search_term: str, gl: str, hl: str) -> dict:
|
||||||
|
headers = {
|
||||||
|
"X-API-KEY": self.serper_api_key or "",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
}
|
||||||
|
params = {"q": search_term, "gl": gl, "hl": hl}
|
||||||
|
response = requests.post(
|
||||||
|
"https://google.serper.dev/search", headers=headers, params=params
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
search_results = response.json()
|
||||||
|
return search_results
|
||||||
@ -0,0 +1,9 @@
|
|||||||
|
"""Integration test for Serper.dev's Google Search API Wrapper."""
|
||||||
|
from langchain.utilities.google_serper import GoogleSerperAPIWrapper
|
||||||
|
|
||||||
|
|
||||||
|
def test_call() -> None:
|
||||||
|
"""Test that call gives the correct answer."""
|
||||||
|
search = GoogleSerperAPIWrapper()
|
||||||
|
output = search.run("What was Obama's first name?")
|
||||||
|
assert "Barack Hussein Obama II" in output
|
||||||
Loading…
Reference in New Issue