blob42/langchain
1
0
Fork 0
forked from Archives/langchain
Code Issues Pull Requests Packages Projects Releases Wiki Activity

hierarchical planning agent for multi-step queries against larger openapi specs (#2170)

Browse Source 
The specs used in chat-gpt plugins have only a few endpoints and have
unrealistically small specifications. By contrast, a spec like spotify's
has 60+ endpoints and is comprised 100k+ tokens.

Here are some impressive traces from gpt-4 that string together
non-trivial sequences of API calls. As noted in `planner.py`, gpt-3 is
not as robust but can be improved with i) better retry, self-reflect,
etc. logic and ii) better few-shots iii) etc. This PR's just a first
attempt probing a few different directions that eventually can be made
more core.
 

`make me a playlist with songs from kind of blue. call it machine
blues.`

```
> Entering new AgentExecutor chain...
Action: api_planner
Action Input: I need to find the right API calls to create a playlist with songs from Kind of Blue and name it Machine Blues
Observation: 1. GET /search to find the album ID for "Kind of Blue".
2. GET /albums/{id}/tracks to get the tracks from the "Kind of Blue" album.
3. GET /me to get the current user's ID.
4. POST /users/{user_id}/playlists to create a new playlist named "Machine Blues" for the current user.
5. POST /playlists/{playlist_id}/tracks to add the tracks from "Kind of Blue" to the newly created "Machine Blues" playlist.
Thought:I have a plan to create the playlist. Now, I will execute the API calls.
Action: api_controller
Action Input: 1. GET /search to find the album ID for "Kind of Blue".
2. GET /albums/{id}/tracks to get the tracks from the "Kind of Blue" album.
3. GET /me to get the current user's ID.
4. POST /users/{user_id}/playlists to create a new playlist named "Machine Blues" for the current user.
5. POST /playlists/{playlist_id}/tracks to add the tracks from "Kind of Blue" to the newly created "Machine Blues" playlist.

> Entering new AgentExecutor chain...
Action: requests_get
Action Input: {"url": "https://api.spotify.com/v1/search?q=Kind%20of%20Blue&type=album", "output_instructions": "Extract the id of the first album in the search results"}
Observation: 1weenld61qoidwYuZ1GESA
Thought:Action: requests_get
Action Input: {"url": "https://api.spotify.com/v1/albums/1weenld61qoidwYuZ1GESA/tracks", "output_instructions": "Extract the ids of all the tracks in the album"}
Observation: ["7q3kkfAVpmcZ8g6JUThi3o"]
Thought:Action: requests_get
Action Input: {"url": "https://api.spotify.com/v1/me", "output_instructions": "Extract the id of the current user"}
Observation: 22rhrz4m4kvpxlsb5hezokzwi
Thought:Action: requests_post
Action Input: {"url": "https://api.spotify.com/v1/users/22rhrz4m4kvpxlsb5hezokzwi/playlists", "data": {"name": "Machine Blues"}, "output_instructions": "Extract the id of the newly created playlist"}
Observation: 48YP9TMcEtFu9aGN8n10lg
Thought:Action: requests_post
Action Input: {"url": "https://api.spotify.com/v1/playlists/48YP9TMcEtFu9aGN8n10lg/tracks", "data": {"uris": ["spotify:track:7q3kkfAVpmcZ8g6JUThi3o"]}, "output_instructions": "Confirm that the tracks were added to the playlist"}
Observation: The tracks were added to the playlist. The snapshot_id is "Miw4NTdmMWUxOGU5YWMxMzVmYmE3ZWE5MWZlYWNkMTc2NGVmNTI1ZjY5".
Thought:I am finished executing the plan.
Final Answer: The tracks from the "Kind of Blue" album have been added to the newly created "Machine Blues" playlist. The playlist ID is 48YP9TMcEtFu9aGN8n10lg.

> Finished chain.

Observation: The tracks from the "Kind of Blue" album have been added to the newly created "Machine Blues" playlist. The playlist ID is 48YP9TMcEtFu9aGN8n10lg.
Thought:I am finished executing the plan and have created the playlist with songs from Kind of Blue, named Machine Blues.
Final Answer: I have created a playlist called "Machine Blues" with songs from the "Kind of Blue" album. The playlist ID is 48YP9TMcEtFu9aGN8n10lg.

> Finished chain.
```

or

`give me a song in the style of tobe nwige`

```
> Entering new AgentExecutor chain...
Action: api_planner
Action Input: I need to find the right API calls to get a song in the style of Tobe Nwigwe

Observation: 1. GET /search to find the artist ID for Tobe Nwigwe.
2. GET /artists/{id}/related-artists to find similar artists to Tobe Nwigwe.
3. Pick one of the related artists and use their artist ID in the next step.
4. GET /artists/{id}/top-tracks to get the top tracks of the chosen related artist.
Thought:


I'm ready to execute the API calls.
Action: api_controller
Action Input: 1. GET /search to find the artist ID for Tobe Nwigwe.
2. GET /artists/{id}/related-artists to find similar artists to Tobe Nwigwe.
3. Pick one of the related artists and use their artist ID in the next step.
4. GET /artists/{id}/top-tracks to get the top tracks of the chosen related artist.

> Entering new AgentExecutor chain...
Action: requests_get
Action Input: {"url": "https://api.spotify.com/v1/search?q=Tobe%20Nwigwe&type=artist", "output_instructions": "Extract the artist id for Tobe Nwigwe"}
Observation: 3Qh89pgJeZq6d8uM1bTot3
Thought:Action: requests_get
Action Input: {"url": "https://api.spotify.com/v1/artists/3Qh89pgJeZq6d8uM1bTot3/related-artists", "output_instructions": "Extract the ids and names of the related artists"}
Observation: [
  {
    "id": "75WcpJKWXBV3o3cfluWapK",
    "name": "Lute"
  },
  {
    "id": "5REHfa3YDopGOzrxwTsPvH",
    "name": "Deante' Hitchcock"
  },
  {
    "id": "6NL31G53xThQXkFs7lDpL5",
    "name": "Rapsody"
  },
  {
    "id": "5MbNzCW3qokGyoo9giHA3V",
    "name": "EARTHGANG"
  },
  {
    "id": "7Hjbimq43OgxaBRpFXic4x",
    "name": "Saba"
  },
  {
    "id": "1ewyVtTZBqFYWIcepopRhp",
    "name": "Mick Jenkins"
  }
]
Thought:Action: requests_get
Action Input: {"url": "https://api.spotify.com/v1/artists/75WcpJKWXBV3o3cfluWapK/top-tracks?country=US", "output_instructions": "Extract the ids and names of the top tracks"}
Observation: [
  {
    "id": "6MF4tRr5lU8qok8IKaFOBE",
    "name": "Under The Sun (with J. Cole & Lute feat. DaBaby)"
  }
]
Thought:I am finished executing the plan.

Final Answer: The top track of the related artist Lute is "Under The Sun (with J. Cole & Lute feat. DaBaby)" with the track ID "6MF4tRr5lU8qok8IKaFOBE".

> Finished chain.

Observation: The top track of the related artist Lute is "Under The Sun (with J. Cole & Lute feat. DaBaby)" with the track ID "6MF4tRr5lU8qok8IKaFOBE".
Thought:I am finished executing the plan and have the information the user asked for.
Final Answer: The song "Under The Sun (with J. Cole & Lute feat. DaBaby)" by Lute is in the style of Tobe Nwigwe.

> Finished chain.
```

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
This commit is contained in:
jerwelborn 2023-04-04 19:49:42 -07:00 committed by GitHub
parent d6d6f322a9
commit b026a62bc4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
6 changed files with 1057 additions and 47 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

615
docs/modules/agents/toolkits/examples/openapi.ipynb
View File

@ -5,33 +5,576 @@
"id": "85fb2c03-ab88-4c8c-97e3-a7f2954555ab",
"metadata": {},
"source": [
"# OpenAPI Agent\n",
"# OpenAPI agents\n",
"\n",
"This notebook showcases an agent designed to interact with an OpenAPI spec and make a correct API request based on the information it has gathered from the spec.\n",
"\n",
"In the below example, we are using the OpenAPI spec for the OpenAI API, which you can find [here](https://github.com/openai/openai-openapi/blob/master/openapi.yaml)."
"We can construct agents to consume arbitrary APIs, here APIs conformant to the OpenAPI/Swagger specification."
]
},
{
"cell_type": "markdown",
"id": "893f90fd-f8f6-470a-a76d-1f200ba02e2f",
"id": "a389367b",
"metadata": {},
"source": [
"## Initialization"
"# 1st example: hierarchical planning agent\n",
"\n",
"In this example, we'll consider an approach called hierarchical planning, common in robotics and appearing in recent works for LLMs X robotics. We'll see it's a viable approach to start working with a massive API spec AND to assist with user queries that require multiple steps against the API.\n",
"\n",
"The idea is simple: to get coherent agent behavior over long sequences behavior & to save on tokens, we'll separate concerns: a \"planner\" will be responsible for what endpoints to call and a \"controller\" will be responsible for how to call them.\n",
"\n",
"In the initial implementation, the planner is an LLM chain that has the name and a short description for each endpoint in context. The controller is an LLM agent that is instantiated with documentation for only the endpoints for a particular plan. There's a lot left to get this working very robustly :)\n",
"\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "4b6ecf6e",
"metadata": {},
"source": [
"## To start, let's collect some OpenAPI specs."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "ff988466-c389-4ec6-b6ac-14364a537fd5",
"metadata": {
"tags": []
},
"id": "0adf3537",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"import yaml\n",
"import os, yaml"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "eb15cea0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"--2023-03-31 15:45:56-- https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 122995 (120K) [text/plain]\n",
"Saving to: ‘openapi.yaml’\n",
"\n",
"openapi.yaml 100%[===================>] 120.11K --.-KB/s in 0.01s \n",
"\n",
"2023-03-31 15:45:56 (10.4 MB/s) - ‘openapi.yaml’ saved [122995/122995]\n",
"\n",
"--2023-03-31 15:45:57-- https://www.klarna.com/us/shopping/public/openai/v0/api-docs\n",
"Resolving www.klarna.com (www.klarna.com)... 52.84.150.34, 52.84.150.46, 52.84.150.61, ...\n",
"Connecting to www.klarna.com (www.klarna.com)|52.84.150.34|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: unspecified [application/json]\n",
"Saving to: ‘api-docs’\n",
"\n",
"api-docs [ <=> ] 1.87K --.-KB/s in 0s \n",
"\n",
"2023-03-31 15:45:57 (261 MB/s) - ‘api-docs’ saved [1916]\n",
"\n",
"--2023-03-31 15:45:57-- https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/spotify.com/1.0.0/openapi.yaml\n",
"Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...\n",
"Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n",
"HTTP request sent, awaiting response... 200 OK\n",
"Length: 286747 (280K) [text/plain]\n",
"Saving to: ‘openapi.yaml’\n",
"\n",
"openapi.yaml 100%[===================>] 280.03K --.-KB/s in 0.02s \n",
"\n",
"2023-03-31 15:45:58 (13.3 MB/s) - ‘openapi.yaml’ saved [286747/286747]\n",
"\n"
]
}
],
"source": [
"!wget https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml\n",
"!mv openapi.yaml openai_openapi.yaml\n",
"!wget https://www.klarna.com/us/shopping/public/openai/v0/api-docs\n",
"!mv api-docs klarna_openapi.yaml\n",
"!wget https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/spotify.com/1.0.0/openapi.yaml\n",
"!mv openapi.yaml spotify_openapi.yaml"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "690a35bf",
"metadata": {},
"outputs": [],
"source": [
"from langchain.agents.agent_toolkits.openapi.spec import reduce_openapi_spec"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "69a8e1b9",
"metadata": {},
"outputs": [],
"source": [
"with open(\"openai_openapi.yaml\") as f:\n",
" raw_openai_api_spec = yaml.load(f, Loader=yaml.Loader)\n",
"openai_api_spec = reduce_openapi_spec(raw_openai_api_spec)\n",
" \n",
"with open(\"klarna_openapi.yaml\") as f:\n",
" raw_klarna_api_spec = yaml.load(f, Loader=yaml.Loader)\n",
"klarna_api_spec = reduce_openapi_spec(raw_klarna_api_spec)\n",
"\n",
"with open(\"spotify_openapi.yaml\") as f:\n",
" raw_spotify_api_spec = yaml.load(f, Loader=yaml.Loader)\n",
"spotify_api_spec = reduce_openapi_spec(raw_spotify_api_spec)"
]
},
{
"cell_type": "markdown",
"id": "ba833d49",
"metadata": {},
"source": [
"---\n",
"\n",
"We'll work with the Spotify API as one of the examples of a somewhat complex API. There's a bit of auth-related setup to do if you want to replicate this.\n",
"\n",
"- You'll have to set up an application in the Spotify developer console, documented [here](https://developer.spotify.com/documentation/general/guides/authorization/), to get credentials: `CLIENT_ID`, `CLIENT_SECRET`, and `REDIRECT_URI`.\n",
"- To get an access tokens (and keep them fresh), you can implement the oauth flows, or you can use `spotipy`. If you've set your Spotify creedentials as environment variables `SPOTIPY_CLIENT_ID`, `SPOTIPY_CLIENT_SECRET`, and `SPOTIPY_REDIRECT_URI`, you can use the helper functions below:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "a82c2cfa",
"metadata": {},
"outputs": [],
"source": [
"import spotipy.util as util\n",
"from langchain.requests import RequestsWrapper\n",
"\n",
"def construct_spotify_auth_headers(raw_spec: dict):\n",
" scopes = list(raw_spec['components']['securitySchemes']['oauth_2_0']['flows']['authorizationCode']['scopes'].keys())\n",
" access_token = util.prompt_for_user_token(scope=','.join(scopes))\n",
" return {\n",
" 'Authorization': f'Bearer {access_token}'\n",
" }\n",
"\n",
"# Get API credentials.\n",
"headers = construct_spotify_auth_headers(raw_spotify_api_spec)\n",
"requests_wrapper = RequestsWrapper(headers=headers)"
]
},
{
"cell_type": "markdown",
"id": "76349780",
"metadata": {},
"source": [
"## How big is this spec?"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "2a93271e",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"63"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"endpoints = [\n",
" (route, operation)\n",
" for route, operations in raw_spotify_api_spec[\"paths\"].items()\n",
" for operation in operations\n",
" if operation in [\"get\", \"post\"]\n",
"]\n",
"len(endpoints)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "eb829190",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"80326"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"import tiktoken\n",
"enc = tiktoken.encoding_for_model('text-davinci-003')\n",
"def count_tokens(s): return len(enc.encode(s))\n",
"\n",
"count_tokens(yaml.dump(raw_spotify_api_spec))"
]
},
{
"cell_type": "markdown",
"id": "cbc4964e",
"metadata": {},
"source": [
"## Let's see some examples!\n",
"\n",
"Starting with GPT-4. (Some robustness iterations under way for GPT-3 family.)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "7f42ee84",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/jeremywelborn/src/langchain/langchain/llms/openai.py:169: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`\n",
" warnings.warn(\n",
"/Users/jeremywelborn/src/langchain/langchain/llms/openai.py:608: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`\n",
" warnings.warn(\n"
]
}
],
"source": [
"from langchain.llms.openai import OpenAI\n",
"from langchain.agents.agent_toolkits.openapi import planner\n",
"llm = OpenAI(model_name=\"gpt-4\", temperature=0.0)"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "38762cc0",
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mAction: api_planner\n",
"Action Input: I need to find the right API calls to create a playlist with the first song from Kind of Blue and name it Machine Blues\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m1. GET /search to search for the album \"Kind of Blue\"\n",
"2. GET /albums/{id}/tracks to get the tracks from the \"Kind of Blue\" album\n",
"3. GET /me to get the current user's information\n",
"4. POST /users/{user_id}/playlists to create a new playlist named \"Machine Blues\" for the current user\n",
"5. POST /playlists/{playlist_id}/tracks to add the first song from \"Kind of Blue\" to the \"Machine Blues\" playlist\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI have the plan, now I need to execute the API calls.\n",
"Action: api_controller\n",
"Action Input: 1. GET /search to search for the album \"Kind of Blue\"\n",
"2. GET /albums/{id}/tracks to get the tracks from the \"Kind of Blue\" album\n",
"3. GET /me to get the current user's information\n",
"4. POST /users/{user_id}/playlists to create a new playlist named \"Machine Blues\" for the current user\n",
"5. POST /playlists/{playlist_id}/tracks to add the first song from \"Kind of Blue\" to the \"Machine Blues\" playlist\u001b[0m\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.spotify.com/v1/search?q=Kind%20of%20Blue&type=album\", \"output_instructions\": \"Extract the id of the first album in the search results\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m1weenld61qoidwYuZ1GESA\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.spotify.com/v1/albums/1weenld61qoidwYuZ1GESA/tracks\", \"output_instructions\": \"Extract the id of the first track in the album\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m7q3kkfAVpmcZ8g6JUThi3o\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.spotify.com/v1/me\", \"output_instructions\": \"Extract the id of the current user\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m22rhrz4m4kvpxlsb5hezokzwi\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mAction: requests_post\n",
"Action Input: {\"url\": \"https://api.spotify.com/v1/users/22rhrz4m4kvpxlsb5hezokzwi/playlists\", \"data\": {\"name\": \"Machine Blues\"}, \"output_instructions\": \"Extract the id of the created playlist\"}\u001b[0m\n",
"Observation: \u001b[33;1m\u001b[1;3m7lzoEi44WOISnFYlrAIqyX\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mAction: requests_post\n",
"Action Input: {\"url\": \"https://api.spotify.com/v1/playlists/7lzoEi44WOISnFYlrAIqyX/tracks\", \"data\": {\"uris\": [\"spotify:track:7q3kkfAVpmcZ8g6JUThi3o\"]}, \"output_instructions\": \"Confirm that the track was added to the playlist\"}\n",
"\u001b[0m\n",
"Observation: \u001b[33;1m\u001b[1;3mThe track was added to the playlist, confirmed by the snapshot_id: MiwxODMxNTMxZTFlNzg3ZWFlZmMxYTlmYWQyMDFiYzUwNDEwMTAwZmE1.\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan.\n",
"Final Answer: The first song from the \"Kind of Blue\" album has been added to the \"Machine Blues\" playlist.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"\n",
"Observation: \u001b[33;1m\u001b[1;3mThe first song from the \"Kind of Blue\" album has been added to the \"Machine Blues\" playlist.\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan and have created the playlist with the first song from Kind of Blue.\n",
"Final Answer: I have created a playlist called \"Machine Blues\" with the first song from the \"Kind of Blue\" album.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
},
{
"data": {
"text/plain": [
"'I have created a playlist called \"Machine Blues\" with the first song from the \"Kind of Blue\" album.'"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"spotify_agent = planner.create_openapi_agent(spotify_api_spec, requests_wrapper, llm)\n",
"user_query = \"make me a playlist with the first song from kind of blue. call it machine blues.\"\n",
"spotify_agent.run(user_query)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "96184181",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mAction: api_planner\n",
"Action Input: I need to find the right API calls to get a blues song recommendation for the user\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m1. GET /me to get the current user's information\n",
"2. GET /recommendations/available-genre-seeds to retrieve a list of available genres\n",
"3. GET /recommendations with the seed_genre parameter set to \"blues\" to get a blues song recommendation for the user\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI have the plan, now I need to execute the API calls.\n",
"Action: api_controller\n",
"Action Input: 1. GET /me to get the current user's information\n",
"2. GET /recommendations/available-genre-seeds to retrieve a list of available genres\n",
"3. GET /recommendations with the seed_genre parameter set to \"blues\" to get a blues song recommendation for the user\u001b[0m\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.spotify.com/v1/me\", \"output_instructions\": \"Extract the user's id and username\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3mID: 22rhrz4m4kvpxlsb5hezokzwi, Username: Jeremy Welborn\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.spotify.com/v1/recommendations/available-genre-seeds\", \"output_instructions\": \"Extract the list of available genres\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3macoustic, afrobeat, alt-rock, alternative, ambient, anime, black-metal, bluegrass, blues, bossanova, brazil, breakbeat, british, cantopop, chicago-house, children, chill, classical, club, comedy, country, dance, dancehall, death-metal, deep-house, detroit-techno, disco, disney, drum-and-bass, dub, dubstep, edm, electro, electronic, emo, folk, forro, french, funk, garage, german, gospel, goth, grindcore, groove, grunge, guitar, happy, hard-rock, hardcore, hardstyle, heavy-metal, hip-hop, holidays, honky-tonk, house, idm, indian, indie, indie-pop, industrial, iranian, j-dance, j-idol, j-pop, j-rock, jazz, k-pop, kids, latin, latino, malay, mandopop, metal, metal-misc, metalcore, minimal-techno, movies, mpb, new-age, new-release, opera, pagode, party, philippines-\u001b[0m\n",
"Thought:"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"Retrying langchain.llms.openai.completion_with_retry.<locals>._completion_with_retry in 4.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 2167437a0072228238f3c0c5b3882764 in your message.).\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.spotify.com/v1/recommendations?seed_genres=blues\", \"output_instructions\": \"Extract the list of recommended tracks with their ids and names\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m[\n",
" {\n",
" id: '03lXHmokj9qsXspNsPoirR',\n",
" name: 'Get Away Jordan'\n",
" }\n",
"]\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan.\n",
"Final Answer: The recommended blues song for user Jeremy Welborn (ID: 22rhrz4m4kvpxlsb5hezokzwi) is \"Get Away Jordan\" with the track ID: 03lXHmokj9qsXspNsPoirR.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"\n",
"Observation: \u001b[33;1m\u001b[1;3mThe recommended blues song for user Jeremy Welborn (ID: 22rhrz4m4kvpxlsb5hezokzwi) is \"Get Away Jordan\" with the track ID: 03lXHmokj9qsXspNsPoirR.\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan and have the information the user asked for.\n",
"Final Answer: The recommended blues song for you is \"Get Away Jordan\" with the track ID: 03lXHmokj9qsXspNsPoirR.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
},
{
"data": {
"text/plain": [
"'The recommended blues song for you is \"Get Away Jordan\" with the track ID: 03lXHmokj9qsXspNsPoirR.'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"user_query = \"give me a song I'd like, make it blues-ey\"\n",
"spotify_agent.run(user_query)"
]
},
{
"cell_type": "markdown",
"id": "d5317926",
"metadata": {},
"source": [
"#### Try another API.\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "06c3d6a8",
"metadata": {},
"outputs": [],
"source": [
"headers = {\n",
" \"Authorization\": f\"Bearer {os.getenv('OPENAI_API_KEY')}\"\n",
"}\n",
"openai_requests_wrapper=RequestsWrapper(headers=headers)"
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "3a9cc939",
"metadata": {
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mAction: api_planner\n",
"Action Input: I need to find the right API calls to generate a short piece of advice\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m1. GET /engines to retrieve the list of available engines\n",
"2. POST /completions with the selected engine and a prompt for generating a short piece of advice\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI have the plan, now I need to execute the API calls.\n",
"Action: api_controller\n",
"Action Input: 1. GET /engines to retrieve the list of available engines\n",
"2. POST /completions with the selected engine and a prompt for generating a short piece of advice\u001b[0m\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.openai.com/v1/engines\", \"output_instructions\": \"Extract the ids of the engines\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3mbabbage, davinci, text-davinci-edit-001, babbage-code-search-code, text-similarity-babbage-001, code-davinci-edit-001, text-davinci-001, ada, babbage-code-search-text, babbage-similarity, whisper-1, code-search-babbage-text-001, text-curie-001, code-search-babbage-code-001, text-ada-001, text-embedding-ada-002, text-similarity-ada-001, curie-instruct-beta, ada-code-search-code, ada-similarity, text-davinci-003, code-search-ada-text-001, text-search-ada-query-001, davinci-search-document, ada-code-search-text, text-search-ada-doc-001, davinci-instruct-beta, text-similarity-curie-001, code-search-ada-code-001\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI will use the \"davinci\" engine to generate a short piece of advice.\n",
"Action: requests_post\n",
"Action Input: {\"url\": \"https://api.openai.com/v1/completions\", \"data\": {\"engine\": \"davinci\", \"prompt\": \"Give me a short piece of advice on how to be more productive.\"}, \"output_instructions\": \"Extract the text from the first choice\"}\u001b[0m\n",
"Observation: \u001b[33;1m\u001b[1;3m\"you must provide a model parameter\"\u001b[0m\n",
"Thought:!! Could not _extract_tool_and_input from \"I cannot finish executing the plan without knowing how to provide the model parameter correctly.\" in _get_next_action\n",
"\u001b[32;1m\u001b[1;3mI cannot finish executing the plan without knowing how to provide the model parameter correctly.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"\n",
"Observation: \u001b[33;1m\u001b[1;3mI need more information on how to provide the model parameter correctly in the POST request to generate a short piece of advice.\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI need to adjust my plan to include the model parameter in the POST request.\n",
"Action: api_planner\n",
"Action Input: I need to find the right API calls to generate a short piece of advice, including the model parameter in the POST request\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m1. GET /models to retrieve the list of available models\n",
"2. Choose a suitable model from the list\n",
"3. POST /completions with the chosen model as a parameter to generate a short piece of advice\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI have an updated plan, now I need to execute the API calls.\n",
"Action: api_controller\n",
"Action Input: 1. GET /models to retrieve the list of available models\n",
"2. Choose a suitable model from the list\n",
"3. POST /completions with the chosen model as a parameter to generate a short piece of advice\u001b[0m\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.openai.com/v1/models\", \"output_instructions\": \"Extract the ids of the available models\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3mbabbage, davinci, text-davinci-edit-001, babbage-code-search-code, text-similarity-babbage-001, code-davinci-edit-001, text-davinci-edit-001, ada\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mAction: requests_post\n",
"Action Input: {\"url\": \"https://api.openai.com/v1/completions\", \"data\": {\"model\": \"davinci\", \"prompt\": \"Give me a short piece of advice on how to improve communication skills.\"}, \"output_instructions\": \"Extract the text from the first choice\"}\u001b[0m\n",
"Observation: \u001b[33;1m\u001b[1;3m\"I'd like to broaden my horizon.\\n\\nI was trying to\"\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI cannot finish executing the plan without knowing some other information.\n",
"\n",
"Final Answer: The generated text is not a piece of advice on improving communication skills. I would need to retry the API call with a different prompt or model to get a more relevant response.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"\n",
"Observation: \u001b[33;1m\u001b[1;3mThe generated text is not a piece of advice on improving communication skills. I would need to retry the API call with a different prompt or model to get a more relevant response.\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI need to adjust my plan to include a more specific prompt for generating a short piece of advice on improving communication skills.\n",
"Action: api_planner\n",
"Action Input: I need to find the right API calls to generate a short piece of advice on improving communication skills, including the model parameter in the POST request\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m1. GET /models to retrieve the list of available models\n",
"2. Choose a suitable model for generating text (e.g., text-davinci-002)\n",
"3. POST /completions with the chosen model and a prompt related to improving communication skills to generate a short piece of advice\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI have an updated plan, now I need to execute the API calls.\n",
"Action: api_controller\n",
"Action Input: 1. GET /models to retrieve the list of available models\n",
"2. Choose a suitable model for generating text (e.g., text-davinci-002)\n",
"3. POST /completions with the chosen model and a prompt related to improving communication skills to generate a short piece of advice\u001b[0m\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mAction: requests_get\n",
"Action Input: {\"url\": \"https://api.openai.com/v1/models\", \"output_instructions\": \"Extract the names of the models\"}\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3mbabbage, davinci, text-davinci-edit-001, babbage-code-search-code, text-similarity-babbage-001, code-davinci-edit-001, text-davinci-edit-001, ada\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mAction: requests_post\n",
"Action Input: {\"url\": \"https://api.openai.com/v1/completions\", \"data\": {\"model\": \"text-davinci-002\", \"prompt\": \"Give a short piece of advice on how to improve communication skills\"}, \"output_instructions\": \"Extract the text from the first choice\"}\u001b[0m\n",
"Observation: \u001b[33;1m\u001b[1;3m\"Some basic advice for improving communication skills would be to make sure to listen\"\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan.\n",
"\n",
"Final Answer: Some basic advice for improving communication skills would be to make sure to listen.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"\n",
"Observation: \u001b[33;1m\u001b[1;3mSome basic advice for improving communication skills would be to make sure to listen.\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan and have the information the user asked for.\n",
"Final Answer: A short piece of advice for improving communication skills is to make sure to listen.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
},
{
"data": {
"text/plain": [
"'A short piece of advice for improving communication skills is to make sure to listen.'"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Meta!\n",
"llm = OpenAI(model_name=\"gpt-4\", temperature=0.25)\n",
"openai_agent = planner.create_openapi_agent(openai_api_spec, openai_requests_wrapper, llm)\n",
"user_query = \"generate a short piece of advice\"\n",
"openai_agent.run(user_query)"
]
},
{
"cell_type": "markdown",
"id": "f32bc6ec",
"metadata": {},
"source": [
"Takes awhile to get there!"
]
},
{
"cell_type": "markdown",
"id": "461229e4",
"metadata": {},
"source": [
"## 2nd example: \"json explorer\" agent\n",
"\n",
"Here's an agent that's not particularly practical, but neat! The agent has access to 2 toolkits. One comprises tools to interact with json: one tool to list the keys of a json object and another tool to get the value for a given key. The other toolkit comprises `requests` wrappers to send GET and POST requests. This agent consumes a lot calls to the language model, but does a surprisingly decent job.\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"id": "f8dfa1d3",
"metadata": {},
"outputs": [],
"source": [
"from langchain.agents import create_openapi_agent\n",
"from langchain.agents.agent_toolkits import OpenAPIToolkit\n",
"from langchain.llms.openai import OpenAI\n",
@ -41,21 +584,19 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 32,
"id": "9ecd1ba0-3937-4359-a41e-68605f0596a1",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"with open(\"openai_openapi.yml\") as f:\n",
"with open(\"openai_openapi.yaml\") as f:\n",
" data = yaml.load(f, Loader=yaml.FullLoader)\n",
"json_spec=JsonSpec(dict_=data, max_value_length=4000)\n",
"headers = {\n",
" \"Authorization\": f\"Bearer {os.getenv('OPENAI_API_KEY')}\"\n",
"}\n",
"requests_wrapper=TextRequestsWrapper(headers=headers)\n",
"openapi_toolkit = OpenAPIToolkit.from_llm(OpenAI(temperature=0), json_spec, requests_wrapper, verbose=True)\n",
"\n",
"\n",
"openapi_toolkit = OpenAPIToolkit.from_llm(OpenAI(temperature=0), json_spec, openai_requests_wrapper, verbose=True)\n",
"openapi_agent_executor = create_openapi_agent(\n",
" llm=OpenAI(temperature=0),\n",
" toolkit=openapi_toolkit,\n",
@ -63,17 +604,9 @@
")"
]
},
{
"cell_type": "markdown",
"id": "f111879d-ae84-41f9-ad82-d3e6b72c41ba",
"metadata": {},
"source": [
"## Example: agent capable of analyzing OpenAPI spec and making requests"
]
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 33,
"id": "548db7f7-337b-4ba8-905c-e7fd58c01799",
"metadata": {
"tags": []
@ -118,13 +651,13 @@
"Thought:\u001b[32;1m\u001b[1;3m I should look at the paths key to see what endpoints exist\n",
"Action: json_spec_list_keys\n",
"Action Input: data[\"paths\"]\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m['/engines', '/engines/{engine_id}', '/completions', '/edits', '/images/generations', '/images/edits', '/images/variations', '/embeddings', '/engines/{engine_id}/search', '/files', '/files/{file_id}', '/files/{file_id}/content', '/answers', '/classifications', '/fine-tunes', '/fine-tunes/{fine_tune_id}', '/fine-tunes/{fine_tune_id}/cancel', '/fine-tunes/{fine_tune_id}/events', '/models', '/models/{model}', '/moderations']\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m['/engines', '/engines/{engine_id}', '/completions', '/chat/completions', '/edits', '/images/generations', '/images/edits', '/images/variations', '/embeddings', '/audio/transcriptions', '/audio/translations', '/engines/{engine_id}/search', '/files', '/files/{file_id}', '/files/{file_id}/content', '/answers', '/classifications', '/fine-tunes', '/fine-tunes/{fine_tune_id}', '/fine-tunes/{fine_tune_id}/cancel', '/fine-tunes/{fine_tune_id}/events', '/models', '/models/{model}', '/moderations']\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3m I now know the path for the /completions endpoint\n",
"Final Answer: data[\"paths\"][2]\u001b[0m\n",
"Final Answer: The path for the /completions endpoint is data[\"paths\"][2]\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
"\n",
"Observation: \u001b[33;1m\u001b[1;3mdata[\"paths\"][2]\u001b[0m\n",
"Observation: \u001b[33;1m\u001b[1;3mThe path for the /completions endpoint is data[\"paths\"][2]\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3m I should find the required parameters for the POST request.\n",
"Action: json_explorer\n",
"Action Input: What are the required parameters for a POST request to the /completions endpoint?\u001b[0m\n",
@ -136,7 +669,7 @@
"Thought:\u001b[32;1m\u001b[1;3m I should look at the paths key to see what endpoints exist\n",
"Action: json_spec_list_keys\n",
"Action Input: data[\"paths\"]\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m['/engines', '/engines/{engine_id}', '/completions', '/edits', '/images/generations', '/images/edits', '/images/variations', '/embeddings', '/engines/{engine_id}/search', '/files', '/files/{file_id}', '/files/{file_id}/content', '/answers', '/classifications', '/fine-tunes', '/fine-tunes/{fine_tune_id}', '/fine-tunes/{fine_tune_id}/cancel', '/fine-tunes/{fine_tune_id}/events', '/models', '/models/{model}', '/moderations']\u001b[0m\n",
"Observation: \u001b[36;1m\u001b[1;3m['/engines', '/engines/{engine_id}', '/completions', '/chat/completions', '/edits', '/images/generations', '/images/edits', '/images/variations', '/embeddings', '/audio/transcriptions', '/audio/translations', '/engines/{engine_id}/search', '/files', '/files/{file_id}', '/files/{file_id}/content', '/answers', '/classifications', '/fine-tunes', '/fine-tunes/{fine_tune_id}', '/fine-tunes/{fine_tune_id}/cancel', '/fine-tunes/{fine_tune_id}/events', '/models', '/models/{model}', '/moderations']\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3m I should look at the /completions endpoint to see what parameters are required\n",
"Action: json_spec_list_keys\n",
"Action Input: data[\"paths\"][\"/completions\"]\u001b[0m\n",
@ -186,10 +719,10 @@
"Thought:\u001b[32;1m\u001b[1;3m I now know the parameters needed to make the request.\n",
"Action: requests_post\n",
"Action Input: { \"url\": \"https://api.openai.com/v1/completions\", \"data\": { \"model\": \"davinci\", \"prompt\": \"tell me a joke\" } }\u001b[0m\n",
"Observation: \u001b[33;1m\u001b[1;3m{\"id\":\"cmpl-6oeEcNETfq8TOuIUQvAct6NrBXihs\",\"object\":\"text_completion\",\"created\":1677529082,\"model\":\"davinci\",\"choices\":[{\"text\":\"\\n\\n\\n\\nLove is a battlefield\\n\\n\\n\\nIt's me...And some\",\"index\":0,\"logprobs\":null,\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":16,\"total_tokens\":20}}\n",
"Observation: \u001b[33;1m\u001b[1;3m{\"id\":\"cmpl-70Ivzip3dazrIXU8DSVJGzFJj2rdv\",\"object\":\"text_completion\",\"created\":1680307139,\"model\":\"davinci\",\"choices\":[{\"text\":\" with mummy not there”\\n\\nYou dig deep and come up with,\",\"index\":0,\"logprobs\":null,\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":16,\"total_tokens\":20}}\n",
"\u001b[0m\n",
"Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n",
"Final Answer: Love is a battlefield. It's me...And some.\u001b[0m\n",
"Final Answer: The response of the POST request is {\"id\":\"cmpl-70Ivzip3dazrIXU8DSVJGzFJj2rdv\",\"object\":\"text_completion\",\"created\":1680307139,\"model\":\"davinci\",\"choices\":[{\"text\":\" with mummy not there”\\n\\nYou dig deep and come up with,\",\"index\":0,\"logprobs\":null,\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":16,\"total_tokens\":20}}\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
@ -197,10 +730,10 @@
{
"data": {
"text/plain": [
"\"Love is a battlefield. It's me...And some.\""
"'The response of the POST request is {\"id\":\"cmpl-70Ivzip3dazrIXU8DSVJGzFJj2rdv\",\"object\":\"text_completion\",\"created\":1680307139,\"model\":\"davinci\",\"choices\":[{\"text\":\" with mummy not there”\\\\n\\\\nYou dig deep and come up with,\",\"index\":0,\"logprobs\":null,\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":16,\"total_tokens\":20}}'"
]
},
"execution_count": 3,
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
@ -208,14 +741,6 @@
"source": [
"openapi_agent_executor.run(\"Make a post request to openai /completions. The prompt should be 'tell me a joke.'\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6ec9582b",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
@ -234,7 +759,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.9"
"version": "3.9.0"
}
},
"nbformat": 4,

6
langchain/agents/agent_toolkits/openapi/base.py
View File

@ -23,6 +23,7 @@ def create_openapi_agent(
format_instructions: str = FORMAT_INSTRUCTIONS,
input_variables: Optional[List[str]] = None,
verbose: bool = False,
return_intermediate_steps: bool = False,
**kwargs: Any,
) -> AgentExecutor:
"""Construct a json agent from an LLM and tools."""
@ -42,5 +43,8 @@ def create_openapi_agent(
tool_names = [tool.name for tool in tools]
agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs)
return AgentExecutor.from_agent_and_tools(
agent=agent, tools=toolkit.get_tools(), verbose=verbose
agent=agent,
tools=toolkit.get_tools(),
verbose=verbose,
return_intermediate_steps=return_intermediate_steps,
)

215
langchain/agents/agent_toolkits/openapi/planner.py Normal file
View File

@ -0,0 +1,215 @@
"""Agent that interacts with OpenAPI APIs via a hierarchical planning approach."""
import json
import re
from typing import List, Optional
import yaml
from langchain.agents.agent import AgentExecutor
from langchain.agents.agent_toolkits.openapi.planner_prompt import (
API_CONTROLLER_PROMPT,
API_CONTROLLER_TOOL_DESCRIPTION,
API_CONTROLLER_TOOL_NAME,
API_ORCHESTRATOR_PROMPT,
API_PLANNER_PROMPT,
API_PLANNER_TOOL_DESCRIPTION,
API_PLANNER_TOOL_NAME,
PARSING_GET_PROMPT,
PARSING_POST_PROMPT,
REQUESTS_GET_TOOL_DESCRIPTION,
REQUESTS_POST_TOOL_DESCRIPTION,
)
from langchain.agents.agent_toolkits.openapi.spec import (
ReducedOpenAPISpec,
)
from langchain.agents.mrkl.base import ZeroShotAgent
from langchain.agents.tools import Tool
from langchain.chains.llm import LLMChain
from langchain.llms.openai import OpenAI
from langchain.prompts import PromptTemplate
from langchain.requests import RequestsWrapper
from langchain.schema import BaseLanguageModel
from langchain.tools.base import BaseTool
from langchain.tools.requests.tool import BaseRequestsTool
#
# Requests tools with LLM-instructed extraction of truncated responses.
#
# Of course, truncating so bluntly may lose a lot of valuable
# information in the response.
# However, the goal for now is to have only a single inference step.
MAX_RESPONSE_LENGTH = 5000
class RequestsGetToolWithParsing(BaseRequestsTool, BaseTool):
name = "requests_get"
description = REQUESTS_GET_TOOL_DESCRIPTION
response_length: Optional[int] = MAX_RESPONSE_LENGTH
llm_chain = LLMChain(
llm=OpenAI(),
prompt=PARSING_GET_PROMPT,
)
def _run(self, text: str) -> str:
try:
data = json.loads(text)
except json.JSONDecodeError as e:
raise e
response = self.requests_wrapper.get(data["url"])
response = response[: self.response_length]
return self.llm_chain.predict(
response=response, instructions=data["output_instructions"]
).strip()
async def _arun(self, text: str) -> str:
raise NotImplementedError()
class RequestsPostToolWithParsing(BaseRequestsTool, BaseTool):
name = "requests_post"
description = REQUESTS_POST_TOOL_DESCRIPTION
response_length: Optional[int] = MAX_RESPONSE_LENGTH
llm_chain = LLMChain(
llm=OpenAI(),
prompt=PARSING_POST_PROMPT,
)
def _run(self, text: str) -> str:
try:
data = json.loads(text)
except json.JSONDecodeError as e:
raise e
response = self.requests_wrapper.post(data["url"], data["data"])
response = response[: self.response_length]
return self.llm_chain.predict(
response=response, instructions=data["output_instructions"]
).strip()
async def _arun(self, text: str) -> str:
raise NotImplementedError()
#
# Orchestrator, planner, controller.
#
def _create_api_planner_tool(
api_spec: ReducedOpenAPISpec, llm: BaseLanguageModel
) -> Tool:
endpoint_descriptions = [
f"{name} {description}" for name, description, _ in api_spec.endpoints
]
prompt = PromptTemplate(
template=API_PLANNER_PROMPT,
input_variables=["query"],
partial_variables={"endpoints": "- " + "- ".join(endpoint_descriptions)},
)
chain = LLMChain(llm=llm, prompt=prompt)
tool = Tool(
name=API_PLANNER_TOOL_NAME,
description=API_PLANNER_TOOL_DESCRIPTION,
func=chain.run,
)
return tool
def _create_api_controller_agent(
api_url: str,
api_docs: str,
requests_wrapper: RequestsWrapper,
llm: BaseLanguageModel,
) -> AgentExecutor:
tools: List[BaseTool] = [
RequestsGetToolWithParsing(requests_wrapper=requests_wrapper),
RequestsPostToolWithParsing(requests_wrapper=requests_wrapper),
]
prompt = PromptTemplate(
template=API_CONTROLLER_PROMPT,
input_variables=["input", "agent_scratchpad"],
partial_variables={
"api_url": api_url,
"api_docs": api_docs,
"tool_names": ", ".join([tool.name for tool in tools]),
"tool_descriptions": "\n".join(
[f"{tool.name}: {tool.description}" for tool in tools]
),
},
)
agent = ZeroShotAgent(
llm_chain=LLMChain(llm=llm, prompt=prompt),
allowed_tools=[tool.name for tool in tools],
)
return AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
def _create_api_controller_tool(
api_spec: ReducedOpenAPISpec,
requests_wrapper: RequestsWrapper,
llm: BaseLanguageModel,
) -> Tool:
"""Expose controller as a tool.
The tool is invoked with a plan from the planner, and dynamically
creates a controller agent with relevant documentation only to
constrain the context.
"""
base_url = api_spec.servers[0]["url"] # TODO: do better.
def _create_and_run_api_controller_agent(plan_str: str) -> str:
pattern = r"\b(GET|POST)\s+(/\S+)*"
matches = re.findall(pattern, plan_str)
endpoint_names = [
"{method} {route}".format(method=method, route=route.split("?")[0])
for method, route in matches
]
endpoint_docs_by_name = {name: docs for name, _, docs in api_spec.endpoints}
docs_str = ""
for endpoint_name in endpoint_names:
docs = endpoint_docs_by_name.get(endpoint_name)
if not docs:
raise ValueError(f"{endpoint_name} endpoint does not exist.")
docs_str += f"== Docs for {endpoint_name} == \n{yaml.dump(docs)}\n"
agent = _create_api_controller_agent(base_url, docs_str, requests_wrapper, llm)
return agent.run(plan_str)
return Tool(
name=API_CONTROLLER_TOOL_NAME,
func=_create_and_run_api_controller_agent,
description=API_CONTROLLER_TOOL_DESCRIPTION,
)
def create_openapi_agent(
api_spec: ReducedOpenAPISpec,
requests_wrapper: RequestsWrapper,
llm: BaseLanguageModel,
) -> AgentExecutor:
"""Instantiate API planner and controller for a given spec.
Inject credentials via requests_wrapper.
We use a top-level "orchestrator" agent to invoke the planner and controller,
rather than a top-level planner
that invokes a controller with its plan. This is to keep the planner simple.
"""
tools = [
_create_api_planner_tool(api_spec, llm),
_create_api_controller_tool(api_spec, requests_wrapper, llm),
]
prompt = PromptTemplate(
template=API_ORCHESTRATOR_PROMPT,
input_variables=["input", "agent_scratchpad"],
partial_variables={
"tool_names": ", ".join([tool.name for tool in tools]),
"tool_descriptions": "\n".join(
[f"{tool.name}: {tool.description}" for tool in tools]
),
},
)
agent = ZeroShotAgent(
llm_chain=LLMChain(llm=llm, prompt=prompt),
allowed_tools=[tool.name for tool in tools],
)
return AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)

156
langchain/agents/agent_toolkits/openapi/planner_prompt.py Normal file
View File

@ -0,0 +1,156 @@
# flake8: noqa
from langchain.prompts.prompt import PromptTemplate
API_PLANNER_PROMPT = """You are a planner that plans a sequence of API calls to assist with user queries against an API.
You should:
1) evaluate whether the user query can be solved by the API documentated below. If no, say why.
2) if yes, generate a plan of API calls and say what they are doing step by step.
You should only use API endpoints documented below ("Endpoints you can use:").
Some user queries can be resolved in a single API call, but some will require several API calls.
The plan will be passed to an API controller that can format it into web requests and return the responses.
----
Here are some examples:
Fake endpoints for examples:
GET /user to get information about the current user
GET /products/search search across products
POST /users/{{id}}/cart to add products to a user's cart
User query: tell me a joke
Plan: Sorry, this API's domain is shopping, not comedy.
Usery query: I want to buy a couch
Plan: 1. GET /products/search to search for couches
2. GET /user to find the user's id
3. POST /users/{{id}}/cart to add a couch to the user's cart
----
Here are endpoints you can use. Do not reference any of the endpoints above.
{endpoints}
----
User query: {query}
Plan:"""
API_PLANNER_TOOL_NAME = "api_planner"
API_PLANNER_TOOL_DESCRIPTION = f"Can be used to generate the right API calls to assist with a user query, like {API_PLANNER_TOOL_NAME}(query). Should always be called before trying to calling the API controller."
# Execution.
API_CONTROLLER_PROMPT = """You are an agent that gets a sequence of API calls and given their documentation, should execute them and return the final response.
If you cannot complete them and run into issues, you should explain the issue. If you're able to resolve an API call call, you can retry the API call. When interacting with API objects, you should extract ids for inputs to other API calls but ids and names for outputs returned to the User.
Here is documentation on the API:
Base url: {api_url}
Endpoints:
{api_docs}
Here are tools to execute requests against the API: {tool_descriptions}
Starting below, you should follow this format:
Plan: the plan of API calls to execute
Thought: you should always think about what to do
Action: the action to take, should be one of the tools [{tool_names}]
Action Input: the input to the action
Observation: the output of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I am finished executing the plan (or, I cannot finish executing the plan without knowing some other information.)
Final Answer: the final output from executing the plan or missing information I'd need to re-plan correctly.
Begin!
Plan: {input}
Thought:
{agent_scratchpad}
"""
API_CONTROLLER_TOOL_NAME = "api_controller"
API_CONTROLLER_TOOL_DESCRIPTION = f"Can be used to execute a plan of API calls, like {API_CONTROLLER_TOOL_NAME}(plan)."
# Orchestrate planning + execution.
# The goal is to have an agent at the top-level (e.g. so it can recover from errors and re-plan) while
# keeping planning (and specifically the planning prompt) simple.
API_ORCHESTRATOR_PROMPT = """You are an agent that assists with user queries against API, things like querying information or creating resources.
Some user queries can be resolved in a single API call though some require several API call.
You should always plan your API calls first, and then execute the plan second.
You should never return information without executing the api_controller tool.
Here are the tools to plan and execute API requests: {tool_descriptions}
Starting below, you should follow this format:
User query: the query a User wants help with related to the API
Thought: you should always think about what to do
Action: the action to take, should be one of the tools [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I am finished executing a plan and have the information the user asked for or the data the used asked to create
Final Answer: the final output from executing the plan
Example:
User query: can you add some trendy stuff to my shopping cart.
Thought: I should plan API calls first.
Action: api_planner
Action Input: I need to find the right API calls to add trendy items to the users shopping cart
Observation: 1) GET /items/trending to get trending item ids
2) GET /user to get user
3) POST /cart to post the trending items to the user's cart
Thought: I'm ready to execute the API calls.
Action: api_controller
Action Input: 1) GET /items/trending to get trending item ids
2) GET /user to get user
3) POST /cart to post the trending items to the user's cart
...
Begin!
User query: {input}
Thought: I should generate a plan to help with this query and then copy that plan exactly to the controller.
{agent_scratchpad}"""
REQUESTS_GET_TOOL_DESCRIPTION = """Use this to GET content from a website.
Input to the tool should be a json string with 2 keys: "url" and "output_instructions".
The value of "url" should be a string. The value of "output_instructions" should be instructions on what information to extract from the response, for example the id(s) for a resource(s) that the GET request fetches.
"""
PARSING_GET_PROMPT = PromptTemplate(
template="""Here is an API response:\n\n{response}\n\n====
Your task is to extract some information according to these instructions: {instructions}
When working with API objects, you should usually use ids over names.
If the response indicates an error, you should instead output a summary of the error.
Output:""",
input_variables=["response", "instructions"],
)
REQUESTS_POST_TOOL_DESCRIPTION = """Use this when you want to POST to a website.
Input to the tool should be a json string with 3 keys: "url", "data", and "output_instructions".
The value of "url" should be a string.
The value of "data" should be a dictionary of key-value pairs you want to POST to the url.
The value of "summary_instructions" should be instructions on what information to extract from the response, for example the id(s) for a resource(s) that the POST request creates.
Always use double quotes for strings in the json string."""
PARSING_POST_PROMPT = PromptTemplate(
template="""Here is an API response:\n\n{response}\n\n====
Your task is to extract some information according to these instructions: {instructions}
When working with API objects, you should usually use ids over names. Do not return any ids or names that are not in the response.
If the response indicates an error, you should instead output a summary of the error.
Output:""",
input_variables=["response", "instructions"],
)

2
langchain/agents/agent_toolkits/openapi/prompt.py
View File

@ -16,7 +16,7 @@ Fourth, make the requests needed to answer the question. Ensure that you are sen
Use the exact parameter names as listed in the spec, do not make up any names or abbreviate the names of parameters.
If you get a not found error, ensure that you are using a path that actually exists in the spec.
"""
OPENAPI_SUFFIX = """Begin!"
OPENAPI_SUFFIX = """Begin!
Question: {input}
Thought: I should explore the spec to find the base url for the API.

110
langchain/agents/agent_toolkits/openapi/spec.py Normal file
View File

@ -0,0 +1,110 @@
"""Quick and dirty representation for OpenAPI specs."""
from dataclasses import dataclass
from typing import Any, Dict, List, Tuple, Union
def dereference_refs(spec_obj: dict, full_spec: dict) -> Union[dict, list]:
"""Try to substitute $refs.
The goal is to get the complete docs for each endpoint in context for now.
In the few OpenAPI specs I studied, $refs referenced models
(or in OpenAPI terms, components) and could be nested. This code most
likely misses lots of cases.
"""
def _retrieve_ref_path(path: str, full_spec: dict) -> dict:
components = path.split("/")
if components[0] != "#":
raise RuntimeError(
"All $refs I've seen so far are uri fragments (start with hash)."
)
out = full_spec
for component in components[1:]:
out = out[component]
return out
def _dereference_refs(
obj: Union[dict, list], stop: bool = False
) -> Union[dict, list]:
if stop:
return obj
obj_out: Dict[str, Any] = {}
if isinstance(obj, dict):
for k, v in obj.items():
if k == "$ref":
# stop=True => don't dereference recursively.
return _dereference_refs(
_retrieve_ref_path(v, full_spec), stop=True
)
elif isinstance(v, list):
obj_out[k] = [_dereference_refs(el) for el in v]
elif isinstance(v, dict):
obj_out[k] = _dereference_refs(v)
else:
obj_out[k] = v
return obj_out
elif isinstance(obj, list):
return [_dereference_refs(el) for el in obj]
else:
return obj
return _dereference_refs(spec_obj)
@dataclass(frozen=True)
class ReducedOpenAPISpec:
servers: List[dict]
description: str
endpoints: List[Tuple[str, str, dict]]
def reduce_openapi_spec(spec: dict, dereference: bool = True) -> ReducedOpenAPISpec:
"""Simplify/distill/minify a spec somehow.
I want a smaller target for retrieval and (more importantly)
I want smaller results from retrieval.
I was hoping https://openapi.tools/ would have some useful bits
to this end, but doesn't seem so.
"""
# 1. Consider only get, post endpoints.
endpoints = [
(f"{operation_name.upper()} {route}", docs.get("description"), docs)
for route, operation in spec["paths"].items()
for operation_name, docs in operation.items()
if operation_name in ["get", "post"]
]
# 2. Replace any refs so that complete docs are retrieved.
# Note: probably want to do this post-retrieval, it blows up the size of the spec.
if dereference:
endpoints = [
(name, description, dereference_refs(docs, spec))
for name, description, docs in endpoints
]
# 3. Strip docs down to required request args + happy path response.
def reduce_endpoint_docs(docs: dict) -> dict:
out = {}
if docs.get("description"):
out["description"] = docs.get("description")
if docs.get("parameters"):
out["parameters"] = [
parameter
for parameter in docs.get("parameters", [])
if parameter.get("required")
]
if "200" in docs["responses"]:
out["responses"] = docs["responses"]["200"]
return out
endpoints = [
(name, description, reduce_endpoint_docs(docs))
for name, description, docs in endpoints
]
return ReducedOpenAPISpec(
servers=spec["servers"],
description=spec["info"].get("description", ""),
endpoints=endpoints,
)