forked from Archives/langchain
Google Search API integration with serper.dev (wrapper, tests, docs, … (#909)
Adds Google Search integration with [Serper](https://serper.dev) a low-cost alternative to SerpAPI (10x cheaper + generous free tier). Includes documentation, tests and examples. Hopefully I am not missing anything. Developers can sign up for a free account at [serper.dev](https://serper.dev) and obtain an api key. ## Usage ```python from langchain.utilities import GoogleSerperAPIWrapper from langchain.llms.openai import OpenAI from langchain.agents import initialize_agent, Tool import os os.environ["SERPER_API_KEY"] = "" os.environ['OPENAI_API_KEY'] = "" llm = OpenAI(temperature=0) search = GoogleSerperAPIWrapper() tools = [ Tool( name="Intermediate Answer", func=search.run ) ] self_ask_with_search = initialize_agent(tools, llm, agent="self-ask-with-search", verbose=True) self_ask_with_search.run("What is the hometown of the reigning men's U.S. Open champion?") ``` ### Output ``` Entering new AgentExecutor chain... Yes. Follow up: Who is the reigning men's U.S. Open champion? Intermediate answer: Current champions Carlos Alcaraz, 2022 men's singles champion. Follow up: Where is Carlos Alcaraz from? Intermediate answer: El Palmar, Spain So the final answer is: El Palmar, Spain > Finished chain. 'El Palmar, Spain' ```searx-api
rogerserper
1 year ago
committed by
GitHub
parent
52753066ef
commit
e46cd3b7db
@ -0,0 +1,70 @@
|
||||
# Google Serper Wrapper
|
||||
|
||||
This page covers how to use the [Serper](https://serper.dev) Google Search API within LangChain. Serper is a low-cost Google Search API that can be used to add answer box, knowledge graph, and organic results data from Google Search.
|
||||
It is broken into two parts: setup, and then references to the specific Google Serper wrapper.
|
||||
|
||||
## Setup
|
||||
- Go to [serper.dev](https://serper.dev) to sign up for a free account
|
||||
- Get the api key and set it as an environment variable (`SERPER_API_KEY`)
|
||||
|
||||
## Wrappers
|
||||
|
||||
### Utility
|
||||
|
||||
There exists a GoogleSerperAPIWrapper utility which wraps this API. To import this utility:
|
||||
|
||||
```python
|
||||
from langchain.utilities import GoogleSerperAPIWrapper
|
||||
```
|
||||
|
||||
You can use it as part of a Self Ask chain:
|
||||
|
||||
```python
|
||||
from langchain.utilities import GoogleSerperAPIWrapper
|
||||
from langchain.llms.openai import OpenAI
|
||||
from langchain.agents import initialize_agent, Tool
|
||||
|
||||
import os
|
||||
os.environ["SERPER_API_KEY"] = ""
|
||||
os.environ['OPENAI_API_KEY'] = ""
|
||||
|
||||
llm = OpenAI(temperature=0)
|
||||
search = GoogleSerperAPIWrapper()
|
||||
tools = [
|
||||
Tool(
|
||||
name="Intermediate Answer",
|
||||
func=search.run
|
||||
)
|
||||
]
|
||||
|
||||
self_ask_with_search = initialize_agent(tools, llm, agent="self-ask-with-search", verbose=True)
|
||||
self_ask_with_search.run("What is the hometown of the reigning men's U.S. Open champion?")
|
||||
```
|
||||
|
||||
#### Output
|
||||
```
|
||||
Entering new AgentExecutor chain...
|
||||
Yes.
|
||||
Follow up: Who is the reigning men's U.S. Open champion?
|
||||
Intermediate answer: Current champions Carlos Alcaraz, 2022 men's singles champion.
|
||||
Follow up: Where is Carlos Alcaraz from?
|
||||
Intermediate answer: El Palmar, Spain
|
||||
So the final answer is: El Palmar, Spain
|
||||
|
||||
> Finished chain.
|
||||
|
||||
'El Palmar, Spain'
|
||||
```
|
||||
|
||||
For a more detailed walkthrough of this wrapper, see [this notebook](../modules/utils/examples/google_serper.ipynb).
|
||||
|
||||
### Tool
|
||||
|
||||
You can also easily load this wrapper as a Tool (to use with an Agent).
|
||||
You can do this with:
|
||||
```python
|
||||
from langchain.agents import load_tools
|
||||
tools = load_tools(["google-serper"])
|
||||
```
|
||||
|
||||
For more information on this, see [this page](../modules/agents/tools.md)
|
||||
@ -0,0 +1,157 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"id": "dc23c48e",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Google Serper API\n",
|
||||
"\n",
|
||||
"This notebook goes over how to use the Google Serper component to search the web. First you need to sign up for a free account at [serper.dev](https://serper.dev) and get your api key."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"os.environ[\"SERPER_API_KEY\"] = \"\""
|
||||
],
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 2,
|
||||
"id": "54bf5afd",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"from langchain.utilities import GoogleSerperAPIWrapper"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 3,
|
||||
"id": "31f8f382",
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"search = GoogleSerperAPIWrapper()"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 4,
|
||||
"id": "25ce0225",
|
||||
"metadata": {},
|
||||
"outputs": [
|
||||
{
|
||||
"data": {
|
||||
"text/plain": "'Barack Hussein Obama II'"
|
||||
},
|
||||
"execution_count": 4,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"search.run(\"Obama's first name?\")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"source": [
|
||||
"## As part of a Self Ask With Search Chain"
|
||||
],
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"os.environ['OPENAI_API_KEY'] = \"\""
|
||||
],
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
}
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": 5,
|
||||
"outputs": [
|
||||
{
|
||||
"name": "stdout",
|
||||
"output_type": "stream",
|
||||
"text": [
|
||||
"\n",
|
||||
"\n",
|
||||
"\u001B[1m> Entering new AgentExecutor chain...\u001B[0m\n",
|
||||
"\u001B[32;1m\u001B[1;3m Yes.\n",
|
||||
"Follow up: Who is the reigning men's U.S. Open champion?\u001B[0m\n",
|
||||
"Intermediate answer: \u001B[36;1m\u001B[1;3mCurrent champions Carlos Alcaraz, 2022 men's singles champion.\u001B[0m\n",
|
||||
"\u001B[32;1m\u001B[1;3mFollow up: Where is Carlos Alcaraz from?\u001B[0m\n",
|
||||
"Intermediate answer: \u001B[36;1m\u001B[1;3mEl Palmar, Spain\u001B[0m\n",
|
||||
"\u001B[32;1m\u001B[1;3mSo the final answer is: El Palmar, Spain\u001B[0m\n",
|
||||
"\n",
|
||||
"\u001B[1m> Finished chain.\u001B[0m\n"
|
||||
]
|
||||
},
|
||||
{
|
||||
"data": {
|
||||
"text/plain": "'El Palmar, Spain'"
|
||||
},
|
||||
"execution_count": 5,
|
||||
"metadata": {},
|
||||
"output_type": "execute_result"
|
||||
}
|
||||
],
|
||||
"source": [
|
||||
"from langchain.utilities import GoogleSerperAPIWrapper\n",
|
||||
"from langchain.llms.openai import OpenAI\n",
|
||||
"from langchain.agents import initialize_agent, Tool\n",
|
||||
"\n",
|
||||
"llm = OpenAI(temperature=0)\n",
|
||||
"search = GoogleSerperAPIWrapper()\n",
|
||||
"tools = [\n",
|
||||
" Tool(\n",
|
||||
" name=\"Intermediate Answer\",\n",
|
||||
" func=search.run\n",
|
||||
" )\n",
|
||||
"]\n",
|
||||
"\n",
|
||||
"self_ask_with_search = initialize_agent(tools, llm, agent=\"self-ask-with-search\", verbose=True)\n",
|
||||
"self_ask_with_search.run(\"What is the hometown of the reigning men's U.S. Open champion?\")"
|
||||
],
|
||||
"metadata": {
|
||||
"collapsed": false
|
||||
}
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.10.9"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 5
|
||||
}
|
||||
@ -0,0 +1,94 @@
|
||||
"""Util that calls Google Search using the Serper.dev API."""
|
||||
from typing import Dict, Optional
|
||||
|
||||
import requests
|
||||
from pydantic.class_validators import root_validator
|
||||
from pydantic.main import BaseModel
|
||||
|
||||
from langchain.utils import get_from_dict_or_env
|
||||
|
||||
|
||||
class GoogleSerperAPIWrapper(BaseModel):
|
||||
"""Wrapper around the Serper.dev Google Search API.
|
||||
|
||||
You can create a free API key at https://serper.dev.
|
||||
|
||||
To use, you should have the environment variable ``SERPER_API_KEY``
|
||||
set with your API key, or pass `serper_api_key` as a named parameter
|
||||
to the constructor.
|
||||
|
||||
Example:
|
||||
.. code-block:: python
|
||||
|
||||
from langchain import GoogleSerperAPIWrapper
|
||||
google_serper = GoogleSerperAPIWrapper()
|
||||
"""
|
||||
|
||||
k: int = 10
|
||||
gl: str = "us"
|
||||
hl: str = "en"
|
||||
serper_api_key: Optional[str] = None
|
||||
|
||||
@root_validator()
|
||||
def validate_environment(cls, values: Dict) -> Dict:
|
||||
"""Validate that api key exists in environment."""
|
||||
serper_api_key = get_from_dict_or_env(
|
||||
values, "serper_api_key", "SERPER_API_KEY"
|
||||
)
|
||||
values["serper_api_key"] = serper_api_key
|
||||
|
||||
return values
|
||||
|
||||
def run(self, query: str) -> str:
|
||||
"""Run query through GoogleSearch and parse result."""
|
||||
results = self._google_serper_search_results(query, gl=self.gl, hl=self.hl)
|
||||
|
||||
return self._parse_results(results)
|
||||
|
||||
def _parse_results(self, results: dict) -> str:
|
||||
snippets = []
|
||||
|
||||
if results.get("answerBox"):
|
||||
answer_box = results.get("answerBox", {})
|
||||
if answer_box.get("answer"):
|
||||
return answer_box.get("answer")
|
||||
elif answer_box.get("snippet"):
|
||||
return answer_box.get("snippet").replace("\n", " ")
|
||||
elif answer_box.get("snippetHighlighted"):
|
||||
return ", ".join(answer_box.get("snippetHighlighted"))
|
||||
|
||||
if results.get("knowledgeGraph"):
|
||||
kg = results.get("knowledgeGraph", {})
|
||||
title = kg.get("title")
|
||||
entity_type = kg.get("type")
|
||||
if entity_type:
|
||||
snippets.append(f"{title}: {entity_type}.")
|
||||
description = kg.get("description")
|
||||
if description:
|
||||
snippets.append(description)
|
||||
for attribute, value in kg.get("attributes", {}).items():
|
||||
snippets.append(f"{title} {attribute}: {value}.")
|
||||
|
||||
for result in results["organic"][: self.k]:
|
||||
if "snippet" in result:
|
||||
snippets.append(result["snippet"])
|
||||
for attribute, value in result.get("attributes", {}).items():
|
||||
snippets.append(f"{attribute}: {value}.")
|
||||
|
||||
if len(snippets) == 0:
|
||||
return "No good Google Search Result was found"
|
||||
|
||||
return " ".join(snippets)
|
||||
|
||||
def _google_serper_search_results(self, search_term: str, gl: str, hl: str) -> dict:
|
||||
headers = {
|
||||
"X-API-KEY": self.serper_api_key or "",
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
params = {"q": search_term, "gl": gl, "hl": hl}
|
||||
response = requests.post(
|
||||
"https://google.serper.dev/search", headers=headers, params=params
|
||||
)
|
||||
response.raise_for_status()
|
||||
search_results = response.json()
|
||||
return search_results
|
||||
@ -0,0 +1,9 @@
|
||||
"""Integration test for Serper.dev's Google Search API Wrapper."""
|
||||
from langchain.utilities.google_serper import GoogleSerperAPIWrapper
|
||||
|
||||
|
||||
def test_call() -> None:
|
||||
"""Test that call gives the correct answer."""
|
||||
search = GoogleSerperAPIWrapper()
|
||||
output = search.run("What was Obama's first name?")
|
||||
assert "Barack Hussein Obama II" in output
|
||||
Loading…
Reference in New Issue