diff --git a/docs/modules/agents/toolkits/examples/openapi.ipynb b/docs/modules/agents/toolkits/examples/openapi.ipynb index 9fe24136..0fee9948 100644 --- a/docs/modules/agents/toolkits/examples/openapi.ipynb +++ b/docs/modules/agents/toolkits/examples/openapi.ipynb @@ -5,33 +5,576 @@ "id": "85fb2c03-ab88-4c8c-97e3-a7f2954555ab", "metadata": {}, "source": [ - "# OpenAPI Agent\n", + "# OpenAPI agents\n", "\n", - "This notebook showcases an agent designed to interact with an OpenAPI spec and make a correct API request based on the information it has gathered from the spec.\n", - "\n", - "In the below example, we are using the OpenAPI spec for the OpenAI API, which you can find [here](https://github.com/openai/openai-openapi/blob/master/openapi.yaml)." + "We can construct agents to consume arbitrary APIs, here APIs conformant to the OpenAPI/Swagger specification." ] }, { "cell_type": "markdown", - "id": "893f90fd-f8f6-470a-a76d-1f200ba02e2f", + "id": "a389367b", "metadata": {}, "source": [ - "## Initialization" + "# 1st example: hierarchical planning agent\n", + "\n", + "In this example, we'll consider an approach called hierarchical planning, common in robotics and appearing in recent works for LLMs X robotics. We'll see it's a viable approach to start working with a massive API spec AND to assist with user queries that require multiple steps against the API.\n", + "\n", + "The idea is simple: to get coherent agent behavior over long sequences behavior & to save on tokens, we'll separate concerns: a \"planner\" will be responsible for what endpoints to call and a \"controller\" will be responsible for how to call them.\n", + "\n", + "In the initial implementation, the planner is an LLM chain that has the name and a short description for each endpoint in context. The controller is an LLM agent that is instantiated with documentation for only the endpoints for a particular plan. There's a lot left to get this working very robustly :)\n", + "\n", + "---" + ] + }, + { + "cell_type": "markdown", + "id": "4b6ecf6e", + "metadata": {}, + "source": [ + "## To start, let's collect some OpenAPI specs." ] }, { "cell_type": "code", "execution_count": 1, - "id": "ff988466-c389-4ec6-b6ac-14364a537fd5", - "metadata": { - "tags": [] - }, + "id": "0adf3537", + "metadata": {}, "outputs": [], "source": [ - "import os\n", - "import yaml\n", + "import os, yaml" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "eb15cea0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "--2023-03-31 15:45:56-- https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml\n", + "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...\n", + "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 122995 (120K) [text/plain]\n", + "Saving to: ‘openapi.yaml’\n", + "\n", + "openapi.yaml 100%[===================>] 120.11K --.-KB/s in 0.01s \n", + "\n", + "2023-03-31 15:45:56 (10.4 MB/s) - ‘openapi.yaml’ saved [122995/122995]\n", + "\n", + "--2023-03-31 15:45:57-- https://www.klarna.com/us/shopping/public/openai/v0/api-docs\n", + "Resolving www.klarna.com (www.klarna.com)... 52.84.150.34, 52.84.150.46, 52.84.150.61, ...\n", + "Connecting to www.klarna.com (www.klarna.com)|52.84.150.34|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: unspecified [application/json]\n", + "Saving to: ‘api-docs’\n", + "\n", + "api-docs [ <=> ] 1.87K --.-KB/s in 0s \n", + "\n", + "2023-03-31 15:45:57 (261 MB/s) - ‘api-docs’ saved [1916]\n", + "\n", + "--2023-03-31 15:45:57-- https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/spotify.com/1.0.0/openapi.yaml\n", + "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...\n", + "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.\n", + "HTTP request sent, awaiting response... 200 OK\n", + "Length: 286747 (280K) [text/plain]\n", + "Saving to: ‘openapi.yaml’\n", + "\n", + "openapi.yaml 100%[===================>] 280.03K --.-KB/s in 0.02s \n", + "\n", + "2023-03-31 15:45:58 (13.3 MB/s) - ‘openapi.yaml’ saved [286747/286747]\n", + "\n" + ] + } + ], + "source": [ + "!wget https://raw.githubusercontent.com/openai/openai-openapi/master/openapi.yaml\n", + "!mv openapi.yaml openai_openapi.yaml\n", + "!wget https://www.klarna.com/us/shopping/public/openai/v0/api-docs\n", + "!mv api-docs klarna_openapi.yaml\n", + "!wget https://raw.githubusercontent.com/APIs-guru/openapi-directory/main/APIs/spotify.com/1.0.0/openapi.yaml\n", + "!mv openapi.yaml spotify_openapi.yaml" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "690a35bf", + "metadata": {}, + "outputs": [], + "source": [ + "from langchain.agents.agent_toolkits.openapi.spec import reduce_openapi_spec" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "69a8e1b9", + "metadata": {}, + "outputs": [], + "source": [ + "with open(\"openai_openapi.yaml\") as f:\n", + " raw_openai_api_spec = yaml.load(f, Loader=yaml.Loader)\n", + "openai_api_spec = reduce_openapi_spec(raw_openai_api_spec)\n", + " \n", + "with open(\"klarna_openapi.yaml\") as f:\n", + " raw_klarna_api_spec = yaml.load(f, Loader=yaml.Loader)\n", + "klarna_api_spec = reduce_openapi_spec(raw_klarna_api_spec)\n", "\n", + "with open(\"spotify_openapi.yaml\") as f:\n", + " raw_spotify_api_spec = yaml.load(f, Loader=yaml.Loader)\n", + "spotify_api_spec = reduce_openapi_spec(raw_spotify_api_spec)" + ] + }, + { + "cell_type": "markdown", + "id": "ba833d49", + "metadata": {}, + "source": [ + "---\n", + "\n", + "We'll work with the Spotify API as one of the examples of a somewhat complex API. There's a bit of auth-related setup to do if you want to replicate this.\n", + "\n", + "- You'll have to set up an application in the Spotify developer console, documented [here](https://developer.spotify.com/documentation/general/guides/authorization/), to get credentials: `CLIENT_ID`, `CLIENT_SECRET`, and `REDIRECT_URI`.\n", + "- To get an access tokens (and keep them fresh), you can implement the oauth flows, or you can use `spotipy`. If you've set your Spotify creedentials as environment variables `SPOTIPY_CLIENT_ID`, `SPOTIPY_CLIENT_SECRET`, and `SPOTIPY_REDIRECT_URI`, you can use the helper functions below:" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "a82c2cfa", + "metadata": {}, + "outputs": [], + "source": [ + "import spotipy.util as util\n", + "from langchain.requests import RequestsWrapper\n", + "\n", + "def construct_spotify_auth_headers(raw_spec: dict):\n", + " scopes = list(raw_spec['components']['securitySchemes']['oauth_2_0']['flows']['authorizationCode']['scopes'].keys())\n", + " access_token = util.prompt_for_user_token(scope=','.join(scopes))\n", + " return {\n", + " 'Authorization': f'Bearer {access_token}'\n", + " }\n", + "\n", + "# Get API credentials.\n", + "headers = construct_spotify_auth_headers(raw_spotify_api_spec)\n", + "requests_wrapper = RequestsWrapper(headers=headers)" + ] + }, + { + "cell_type": "markdown", + "id": "76349780", + "metadata": {}, + "source": [ + "## How big is this spec?" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "2a93271e", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "63" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "endpoints = [\n", + " (route, operation)\n", + " for route, operations in raw_spotify_api_spec[\"paths\"].items()\n", + " for operation in operations\n", + " if operation in [\"get\", \"post\"]\n", + "]\n", + "len(endpoints)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "eb829190", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "80326" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "import tiktoken\n", + "enc = tiktoken.encoding_for_model('text-davinci-003')\n", + "def count_tokens(s): return len(enc.encode(s))\n", + "\n", + "count_tokens(yaml.dump(raw_spotify_api_spec))" + ] + }, + { + "cell_type": "markdown", + "id": "cbc4964e", + "metadata": {}, + "source": [ + "## Let's see some examples!\n", + "\n", + "Starting with GPT-4. (Some robustness iterations under way for GPT-3 family.)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "7f42ee84", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/jeremywelborn/src/langchain/langchain/llms/openai.py:169: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`\n", + " warnings.warn(\n", + "/Users/jeremywelborn/src/langchain/langchain/llms/openai.py:608: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`\n", + " warnings.warn(\n" + ] + } + ], + "source": [ + "from langchain.llms.openai import OpenAI\n", + "from langchain.agents.agent_toolkits.openapi import planner\n", + "llm = OpenAI(model_name=\"gpt-4\", temperature=0.0)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "38762cc0", + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\n", + "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", + "\u001b[32;1m\u001b[1;3mAction: api_planner\n", + "Action Input: I need to find the right API calls to create a playlist with the first song from Kind of Blue and name it Machine Blues\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m1. GET /search to search for the album \"Kind of Blue\"\n", + "2. GET /albums/{id}/tracks to get the tracks from the \"Kind of Blue\" album\n", + "3. GET /me to get the current user's information\n", + "4. POST /users/{user_id}/playlists to create a new playlist named \"Machine Blues\" for the current user\n", + "5. POST /playlists/{playlist_id}/tracks to add the first song from \"Kind of Blue\" to the \"Machine Blues\" playlist\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI have the plan, now I need to execute the API calls.\n", + "Action: api_controller\n", + "Action Input: 1. GET /search to search for the album \"Kind of Blue\"\n", + "2. GET /albums/{id}/tracks to get the tracks from the \"Kind of Blue\" album\n", + "3. GET /me to get the current user's information\n", + "4. POST /users/{user_id}/playlists to create a new playlist named \"Machine Blues\" for the current user\n", + "5. POST /playlists/{playlist_id}/tracks to add the first song from \"Kind of Blue\" to the \"Machine Blues\" playlist\u001b[0m\n", + "\n", + "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", + "\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.spotify.com/v1/search?q=Kind%20of%20Blue&type=album\", \"output_instructions\": \"Extract the id of the first album in the search results\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m1weenld61qoidwYuZ1GESA\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.spotify.com/v1/albums/1weenld61qoidwYuZ1GESA/tracks\", \"output_instructions\": \"Extract the id of the first track in the album\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m7q3kkfAVpmcZ8g6JUThi3o\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.spotify.com/v1/me\", \"output_instructions\": \"Extract the id of the current user\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m22rhrz4m4kvpxlsb5hezokzwi\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mAction: requests_post\n", + "Action Input: {\"url\": \"https://api.spotify.com/v1/users/22rhrz4m4kvpxlsb5hezokzwi/playlists\", \"data\": {\"name\": \"Machine Blues\"}, \"output_instructions\": \"Extract the id of the created playlist\"}\u001b[0m\n", + "Observation: \u001b[33;1m\u001b[1;3m7lzoEi44WOISnFYlrAIqyX\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mAction: requests_post\n", + "Action Input: {\"url\": \"https://api.spotify.com/v1/playlists/7lzoEi44WOISnFYlrAIqyX/tracks\", \"data\": {\"uris\": [\"spotify:track:7q3kkfAVpmcZ8g6JUThi3o\"]}, \"output_instructions\": \"Confirm that the track was added to the playlist\"}\n", + "\u001b[0m\n", + "Observation: \u001b[33;1m\u001b[1;3mThe track was added to the playlist, confirmed by the snapshot_id: MiwxODMxNTMxZTFlNzg3ZWFlZmMxYTlmYWQyMDFiYzUwNDEwMTAwZmE1.\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan.\n", + "Final Answer: The first song from the \"Kind of Blue\" album has been added to the \"Machine Blues\" playlist.\u001b[0m\n", + "\n", + "\u001b[1m> Finished chain.\u001b[0m\n", + "\n", + "Observation: \u001b[33;1m\u001b[1;3mThe first song from the \"Kind of Blue\" album has been added to the \"Machine Blues\" playlist.\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan and have created the playlist with the first song from Kind of Blue.\n", + "Final Answer: I have created a playlist called \"Machine Blues\" with the first song from the \"Kind of Blue\" album.\u001b[0m\n", + "\n", + "\u001b[1m> Finished chain.\u001b[0m\n" + ] + }, + { + "data": { + "text/plain": [ + "'I have created a playlist called \"Machine Blues\" with the first song from the \"Kind of Blue\" album.'" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "spotify_agent = planner.create_openapi_agent(spotify_api_spec, requests_wrapper, llm)\n", + "user_query = \"make me a playlist with the first song from kind of blue. call it machine blues.\"\n", + "spotify_agent.run(user_query)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "96184181", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\n", + "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", + "\u001b[32;1m\u001b[1;3mAction: api_planner\n", + "Action Input: I need to find the right API calls to get a blues song recommendation for the user\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m1. GET /me to get the current user's information\n", + "2. GET /recommendations/available-genre-seeds to retrieve a list of available genres\n", + "3. GET /recommendations with the seed_genre parameter set to \"blues\" to get a blues song recommendation for the user\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI have the plan, now I need to execute the API calls.\n", + "Action: api_controller\n", + "Action Input: 1. GET /me to get the current user's information\n", + "2. GET /recommendations/available-genre-seeds to retrieve a list of available genres\n", + "3. GET /recommendations with the seed_genre parameter set to \"blues\" to get a blues song recommendation for the user\u001b[0m\n", + "\n", + "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", + "\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.spotify.com/v1/me\", \"output_instructions\": \"Extract the user's id and username\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3mID: 22rhrz4m4kvpxlsb5hezokzwi, Username: Jeremy Welborn\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.spotify.com/v1/recommendations/available-genre-seeds\", \"output_instructions\": \"Extract the list of available genres\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3macoustic, afrobeat, alt-rock, alternative, ambient, anime, black-metal, bluegrass, blues, bossanova, brazil, breakbeat, british, cantopop, chicago-house, children, chill, classical, club, comedy, country, dance, dancehall, death-metal, deep-house, detroit-techno, disco, disney, drum-and-bass, dub, dubstep, edm, electro, electronic, emo, folk, forro, french, funk, garage, german, gospel, goth, grindcore, groove, grunge, guitar, happy, hard-rock, hardcore, hardstyle, heavy-metal, hip-hop, holidays, honky-tonk, house, idm, indian, indie, indie-pop, industrial, iranian, j-dance, j-idol, j-pop, j-rock, jazz, k-pop, kids, latin, latino, malay, mandopop, metal, metal-misc, metalcore, minimal-techno, movies, mpb, new-age, new-release, opera, pagode, party, philippines-\u001b[0m\n", + "Thought:" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Retrying langchain.llms.openai.completion_with_retry.._completion_with_retry in 4.0 seconds as it raised RateLimitError: That model is currently overloaded with other requests. You can retry your request, or contact us through our help center at help.openai.com if the error persists. (Please include the request ID 2167437a0072228238f3c0c5b3882764 in your message.).\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.spotify.com/v1/recommendations?seed_genres=blues\", \"output_instructions\": \"Extract the list of recommended tracks with their ids and names\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m[\n", + " {\n", + " id: '03lXHmokj9qsXspNsPoirR',\n", + " name: 'Get Away Jordan'\n", + " }\n", + "]\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan.\n", + "Final Answer: The recommended blues song for user Jeremy Welborn (ID: 22rhrz4m4kvpxlsb5hezokzwi) is \"Get Away Jordan\" with the track ID: 03lXHmokj9qsXspNsPoirR.\u001b[0m\n", + "\n", + "\u001b[1m> Finished chain.\u001b[0m\n", + "\n", + "Observation: \u001b[33;1m\u001b[1;3mThe recommended blues song for user Jeremy Welborn (ID: 22rhrz4m4kvpxlsb5hezokzwi) is \"Get Away Jordan\" with the track ID: 03lXHmokj9qsXspNsPoirR.\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan and have the information the user asked for.\n", + "Final Answer: The recommended blues song for you is \"Get Away Jordan\" with the track ID: 03lXHmokj9qsXspNsPoirR.\u001b[0m\n", + "\n", + "\u001b[1m> Finished chain.\u001b[0m\n" + ] + }, + { + "data": { + "text/plain": [ + "'The recommended blues song for you is \"Get Away Jordan\" with the track ID: 03lXHmokj9qsXspNsPoirR.'" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "user_query = \"give me a song I'd like, make it blues-ey\"\n", + "spotify_agent.run(user_query)" + ] + }, + { + "cell_type": "markdown", + "id": "d5317926", + "metadata": {}, + "source": [ + "#### Try another API.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "06c3d6a8", + "metadata": {}, + "outputs": [], + "source": [ + "headers = {\n", + " \"Authorization\": f\"Bearer {os.getenv('OPENAI_API_KEY')}\"\n", + "}\n", + "openai_requests_wrapper=RequestsWrapper(headers=headers)" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "3a9cc939", + "metadata": { + "scrolled": false + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "\n", + "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", + "\u001b[32;1m\u001b[1;3mAction: api_planner\n", + "Action Input: I need to find the right API calls to generate a short piece of advice\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m1. GET /engines to retrieve the list of available engines\n", + "2. POST /completions with the selected engine and a prompt for generating a short piece of advice\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI have the plan, now I need to execute the API calls.\n", + "Action: api_controller\n", + "Action Input: 1. GET /engines to retrieve the list of available engines\n", + "2. POST /completions with the selected engine and a prompt for generating a short piece of advice\u001b[0m\n", + "\n", + "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", + "\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.openai.com/v1/engines\", \"output_instructions\": \"Extract the ids of the engines\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3mbabbage, davinci, text-davinci-edit-001, babbage-code-search-code, text-similarity-babbage-001, code-davinci-edit-001, text-davinci-001, ada, babbage-code-search-text, babbage-similarity, whisper-1, code-search-babbage-text-001, text-curie-001, code-search-babbage-code-001, text-ada-001, text-embedding-ada-002, text-similarity-ada-001, curie-instruct-beta, ada-code-search-code, ada-similarity, text-davinci-003, code-search-ada-text-001, text-search-ada-query-001, davinci-search-document, ada-code-search-text, text-search-ada-doc-001, davinci-instruct-beta, text-similarity-curie-001, code-search-ada-code-001\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI will use the \"davinci\" engine to generate a short piece of advice.\n", + "Action: requests_post\n", + "Action Input: {\"url\": \"https://api.openai.com/v1/completions\", \"data\": {\"engine\": \"davinci\", \"prompt\": \"Give me a short piece of advice on how to be more productive.\"}, \"output_instructions\": \"Extract the text from the first choice\"}\u001b[0m\n", + "Observation: \u001b[33;1m\u001b[1;3m\"you must provide a model parameter\"\u001b[0m\n", + "Thought:!! Could not _extract_tool_and_input from \"I cannot finish executing the plan without knowing how to provide the model parameter correctly.\" in _get_next_action\n", + "\u001b[32;1m\u001b[1;3mI cannot finish executing the plan without knowing how to provide the model parameter correctly.\u001b[0m\n", + "\n", + "\u001b[1m> Finished chain.\u001b[0m\n", + "\n", + "Observation: \u001b[33;1m\u001b[1;3mI need more information on how to provide the model parameter correctly in the POST request to generate a short piece of advice.\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI need to adjust my plan to include the model parameter in the POST request.\n", + "Action: api_planner\n", + "Action Input: I need to find the right API calls to generate a short piece of advice, including the model parameter in the POST request\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m1. GET /models to retrieve the list of available models\n", + "2. Choose a suitable model from the list\n", + "3. POST /completions with the chosen model as a parameter to generate a short piece of advice\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI have an updated plan, now I need to execute the API calls.\n", + "Action: api_controller\n", + "Action Input: 1. GET /models to retrieve the list of available models\n", + "2. Choose a suitable model from the list\n", + "3. POST /completions with the chosen model as a parameter to generate a short piece of advice\u001b[0m\n", + "\n", + "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", + "\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.openai.com/v1/models\", \"output_instructions\": \"Extract the ids of the available models\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3mbabbage, davinci, text-davinci-edit-001, babbage-code-search-code, text-similarity-babbage-001, code-davinci-edit-001, text-davinci-edit-001, ada\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mAction: requests_post\n", + "Action Input: {\"url\": \"https://api.openai.com/v1/completions\", \"data\": {\"model\": \"davinci\", \"prompt\": \"Give me a short piece of advice on how to improve communication skills.\"}, \"output_instructions\": \"Extract the text from the first choice\"}\u001b[0m\n", + "Observation: \u001b[33;1m\u001b[1;3m\"I'd like to broaden my horizon.\\n\\nI was trying to\"\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI cannot finish executing the plan without knowing some other information.\n", + "\n", + "Final Answer: The generated text is not a piece of advice on improving communication skills. I would need to retry the API call with a different prompt or model to get a more relevant response.\u001b[0m\n", + "\n", + "\u001b[1m> Finished chain.\u001b[0m\n", + "\n", + "Observation: \u001b[33;1m\u001b[1;3mThe generated text is not a piece of advice on improving communication skills. I would need to retry the API call with a different prompt or model to get a more relevant response.\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI need to adjust my plan to include a more specific prompt for generating a short piece of advice on improving communication skills.\n", + "Action: api_planner\n", + "Action Input: I need to find the right API calls to generate a short piece of advice on improving communication skills, including the model parameter in the POST request\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m1. GET /models to retrieve the list of available models\n", + "2. Choose a suitable model for generating text (e.g., text-davinci-002)\n", + "3. POST /completions with the chosen model and a prompt related to improving communication skills to generate a short piece of advice\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI have an updated plan, now I need to execute the API calls.\n", + "Action: api_controller\n", + "Action Input: 1. GET /models to retrieve the list of available models\n", + "2. Choose a suitable model for generating text (e.g., text-davinci-002)\n", + "3. POST /completions with the chosen model and a prompt related to improving communication skills to generate a short piece of advice\u001b[0m\n", + "\n", + "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n", + "\u001b[32;1m\u001b[1;3mAction: requests_get\n", + "Action Input: {\"url\": \"https://api.openai.com/v1/models\", \"output_instructions\": \"Extract the names of the models\"}\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3mbabbage, davinci, text-davinci-edit-001, babbage-code-search-code, text-similarity-babbage-001, code-davinci-edit-001, text-davinci-edit-001, ada\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mAction: requests_post\n", + "Action Input: {\"url\": \"https://api.openai.com/v1/completions\", \"data\": {\"model\": \"text-davinci-002\", \"prompt\": \"Give a short piece of advice on how to improve communication skills\"}, \"output_instructions\": \"Extract the text from the first choice\"}\u001b[0m\n", + "Observation: \u001b[33;1m\u001b[1;3m\"Some basic advice for improving communication skills would be to make sure to listen\"\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan.\n", + "\n", + "Final Answer: Some basic advice for improving communication skills would be to make sure to listen.\u001b[0m\n", + "\n", + "\u001b[1m> Finished chain.\u001b[0m\n", + "\n", + "Observation: \u001b[33;1m\u001b[1;3mSome basic advice for improving communication skills would be to make sure to listen.\u001b[0m\n", + "Thought:\u001b[32;1m\u001b[1;3mI am finished executing the plan and have the information the user asked for.\n", + "Final Answer: A short piece of advice for improving communication skills is to make sure to listen.\u001b[0m\n", + "\n", + "\u001b[1m> Finished chain.\u001b[0m\n" + ] + }, + { + "data": { + "text/plain": [ + "'A short piece of advice for improving communication skills is to make sure to listen.'" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Meta!\n", + "llm = OpenAI(model_name=\"gpt-4\", temperature=0.25)\n", + "openai_agent = planner.create_openapi_agent(openai_api_spec, openai_requests_wrapper, llm)\n", + "user_query = \"generate a short piece of advice\"\n", + "openai_agent.run(user_query)" + ] + }, + { + "cell_type": "markdown", + "id": "f32bc6ec", + "metadata": {}, + "source": [ + "Takes awhile to get there!" + ] + }, + { + "cell_type": "markdown", + "id": "461229e4", + "metadata": {}, + "source": [ + "## 2nd example: \"json explorer\" agent\n", + "\n", + "Here's an agent that's not particularly practical, but neat! The agent has access to 2 toolkits. One comprises tools to interact with json: one tool to list the keys of a json object and another tool to get the value for a given key. The other toolkit comprises `requests` wrappers to send GET and POST requests. This agent consumes a lot calls to the language model, but does a surprisingly decent job.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "f8dfa1d3", + "metadata": {}, + "outputs": [], + "source": [ "from langchain.agents import create_openapi_agent\n", "from langchain.agents.agent_toolkits import OpenAPIToolkit\n", "from langchain.llms.openai import OpenAI\n", @@ -41,21 +584,19 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 32, "id": "9ecd1ba0-3937-4359-a41e-68605f0596a1", "metadata": { "tags": [] }, "outputs": [], "source": [ - "with open(\"openai_openapi.yml\") as f:\n", + "with open(\"openai_openapi.yaml\") as f:\n", " data = yaml.load(f, Loader=yaml.FullLoader)\n", "json_spec=JsonSpec(dict_=data, max_value_length=4000)\n", - "headers = {\n", - " \"Authorization\": f\"Bearer {os.getenv('OPENAI_API_KEY')}\"\n", - "}\n", - "requests_wrapper=TextRequestsWrapper(headers=headers)\n", - "openapi_toolkit = OpenAPIToolkit.from_llm(OpenAI(temperature=0), json_spec, requests_wrapper, verbose=True)\n", + "\n", + "\n", + "openapi_toolkit = OpenAPIToolkit.from_llm(OpenAI(temperature=0), json_spec, openai_requests_wrapper, verbose=True)\n", "openapi_agent_executor = create_openapi_agent(\n", " llm=OpenAI(temperature=0),\n", " toolkit=openapi_toolkit,\n", @@ -63,17 +604,9 @@ ")" ] }, - { - "cell_type": "markdown", - "id": "f111879d-ae84-41f9-ad82-d3e6b72c41ba", - "metadata": {}, - "source": [ - "## Example: agent capable of analyzing OpenAPI spec and making requests" - ] - }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 33, "id": "548db7f7-337b-4ba8-905c-e7fd58c01799", "metadata": { "tags": [] @@ -118,13 +651,13 @@ "Thought:\u001b[32;1m\u001b[1;3m I should look at the paths key to see what endpoints exist\n", "Action: json_spec_list_keys\n", "Action Input: data[\"paths\"]\u001b[0m\n", - "Observation: \u001b[36;1m\u001b[1;3m['/engines', '/engines/{engine_id}', '/completions', '/edits', '/images/generations', '/images/edits', '/images/variations', '/embeddings', '/engines/{engine_id}/search', '/files', '/files/{file_id}', '/files/{file_id}/content', '/answers', '/classifications', '/fine-tunes', '/fine-tunes/{fine_tune_id}', '/fine-tunes/{fine_tune_id}/cancel', '/fine-tunes/{fine_tune_id}/events', '/models', '/models/{model}', '/moderations']\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m['/engines', '/engines/{engine_id}', '/completions', '/chat/completions', '/edits', '/images/generations', '/images/edits', '/images/variations', '/embeddings', '/audio/transcriptions', '/audio/translations', '/engines/{engine_id}/search', '/files', '/files/{file_id}', '/files/{file_id}/content', '/answers', '/classifications', '/fine-tunes', '/fine-tunes/{fine_tune_id}', '/fine-tunes/{fine_tune_id}/cancel', '/fine-tunes/{fine_tune_id}/events', '/models', '/models/{model}', '/moderations']\u001b[0m\n", "Thought:\u001b[32;1m\u001b[1;3m I now know the path for the /completions endpoint\n", - "Final Answer: data[\"paths\"][2]\u001b[0m\n", + "Final Answer: The path for the /completions endpoint is data[\"paths\"][2]\u001b[0m\n", "\n", "\u001b[1m> Finished chain.\u001b[0m\n", "\n", - "Observation: \u001b[33;1m\u001b[1;3mdata[\"paths\"][2]\u001b[0m\n", + "Observation: \u001b[33;1m\u001b[1;3mThe path for the /completions endpoint is data[\"paths\"][2]\u001b[0m\n", "Thought:\u001b[32;1m\u001b[1;3m I should find the required parameters for the POST request.\n", "Action: json_explorer\n", "Action Input: What are the required parameters for a POST request to the /completions endpoint?\u001b[0m\n", @@ -136,7 +669,7 @@ "Thought:\u001b[32;1m\u001b[1;3m I should look at the paths key to see what endpoints exist\n", "Action: json_spec_list_keys\n", "Action Input: data[\"paths\"]\u001b[0m\n", - "Observation: \u001b[36;1m\u001b[1;3m['/engines', '/engines/{engine_id}', '/completions', '/edits', '/images/generations', '/images/edits', '/images/variations', '/embeddings', '/engines/{engine_id}/search', '/files', '/files/{file_id}', '/files/{file_id}/content', '/answers', '/classifications', '/fine-tunes', '/fine-tunes/{fine_tune_id}', '/fine-tunes/{fine_tune_id}/cancel', '/fine-tunes/{fine_tune_id}/events', '/models', '/models/{model}', '/moderations']\u001b[0m\n", + "Observation: \u001b[36;1m\u001b[1;3m['/engines', '/engines/{engine_id}', '/completions', '/chat/completions', '/edits', '/images/generations', '/images/edits', '/images/variations', '/embeddings', '/audio/transcriptions', '/audio/translations', '/engines/{engine_id}/search', '/files', '/files/{file_id}', '/files/{file_id}/content', '/answers', '/classifications', '/fine-tunes', '/fine-tunes/{fine_tune_id}', '/fine-tunes/{fine_tune_id}/cancel', '/fine-tunes/{fine_tune_id}/events', '/models', '/models/{model}', '/moderations']\u001b[0m\n", "Thought:\u001b[32;1m\u001b[1;3m I should look at the /completions endpoint to see what parameters are required\n", "Action: json_spec_list_keys\n", "Action Input: data[\"paths\"][\"/completions\"]\u001b[0m\n", @@ -186,10 +719,10 @@ "Thought:\u001b[32;1m\u001b[1;3m I now know the parameters needed to make the request.\n", "Action: requests_post\n", "Action Input: { \"url\": \"https://api.openai.com/v1/completions\", \"data\": { \"model\": \"davinci\", \"prompt\": \"tell me a joke\" } }\u001b[0m\n", - "Observation: \u001b[33;1m\u001b[1;3m{\"id\":\"cmpl-6oeEcNETfq8TOuIUQvAct6NrBXihs\",\"object\":\"text_completion\",\"created\":1677529082,\"model\":\"davinci\",\"choices\":[{\"text\":\"\\n\\n\\n\\nLove is a battlefield\\n\\n\\n\\nIt's me...And some\",\"index\":0,\"logprobs\":null,\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":16,\"total_tokens\":20}}\n", + "Observation: \u001b[33;1m\u001b[1;3m{\"id\":\"cmpl-70Ivzip3dazrIXU8DSVJGzFJj2rdv\",\"object\":\"text_completion\",\"created\":1680307139,\"model\":\"davinci\",\"choices\":[{\"text\":\" with mummy not there”\\n\\nYou dig deep and come up with,\",\"index\":0,\"logprobs\":null,\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":16,\"total_tokens\":20}}\n", "\u001b[0m\n", "Thought:\u001b[32;1m\u001b[1;3m I now know the final answer.\n", - "Final Answer: Love is a battlefield. It's me...And some.\u001b[0m\n", + "Final Answer: The response of the POST request is {\"id\":\"cmpl-70Ivzip3dazrIXU8DSVJGzFJj2rdv\",\"object\":\"text_completion\",\"created\":1680307139,\"model\":\"davinci\",\"choices\":[{\"text\":\" with mummy not there”\\n\\nYou dig deep and come up with,\",\"index\":0,\"logprobs\":null,\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":16,\"total_tokens\":20}}\u001b[0m\n", "\n", "\u001b[1m> Finished chain.\u001b[0m\n" ] @@ -197,10 +730,10 @@ { "data": { "text/plain": [ - "\"Love is a battlefield. It's me...And some.\"" + "'The response of the POST request is {\"id\":\"cmpl-70Ivzip3dazrIXU8DSVJGzFJj2rdv\",\"object\":\"text_completion\",\"created\":1680307139,\"model\":\"davinci\",\"choices\":[{\"text\":\" with mummy not there”\\\\n\\\\nYou dig deep and come up with,\",\"index\":0,\"logprobs\":null,\"finish_reason\":\"length\"}],\"usage\":{\"prompt_tokens\":4,\"completion_tokens\":16,\"total_tokens\":20}}'" ] }, - "execution_count": 3, + "execution_count": 33, "metadata": {}, "output_type": "execute_result" } @@ -208,14 +741,6 @@ "source": [ "openapi_agent_executor.run(\"Make a post request to openai /completions. The prompt should be 'tell me a joke.'\")" ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "6ec9582b", - "metadata": {}, - "outputs": [], - "source": [] } ], "metadata": { @@ -234,7 +759,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.10.9" + "version": "3.9.0" } }, "nbformat": 4, diff --git a/langchain/agents/agent_toolkits/openapi/base.py b/langchain/agents/agent_toolkits/openapi/base.py index 602ee87f..32232881 100644 --- a/langchain/agents/agent_toolkits/openapi/base.py +++ b/langchain/agents/agent_toolkits/openapi/base.py @@ -23,6 +23,7 @@ def create_openapi_agent( format_instructions: str = FORMAT_INSTRUCTIONS, input_variables: Optional[List[str]] = None, verbose: bool = False, + return_intermediate_steps: bool = False, **kwargs: Any, ) -> AgentExecutor: """Construct a json agent from an LLM and tools.""" @@ -42,5 +43,8 @@ def create_openapi_agent( tool_names = [tool.name for tool in tools] agent = ZeroShotAgent(llm_chain=llm_chain, allowed_tools=tool_names, **kwargs) return AgentExecutor.from_agent_and_tools( - agent=agent, tools=toolkit.get_tools(), verbose=verbose + agent=agent, + tools=toolkit.get_tools(), + verbose=verbose, + return_intermediate_steps=return_intermediate_steps, ) diff --git a/langchain/agents/agent_toolkits/openapi/planner.py b/langchain/agents/agent_toolkits/openapi/planner.py new file mode 100644 index 00000000..c08b3362 --- /dev/null +++ b/langchain/agents/agent_toolkits/openapi/planner.py @@ -0,0 +1,215 @@ +"""Agent that interacts with OpenAPI APIs via a hierarchical planning approach.""" +import json +import re +from typing import List, Optional + +import yaml + +from langchain.agents.agent import AgentExecutor +from langchain.agents.agent_toolkits.openapi.planner_prompt import ( + API_CONTROLLER_PROMPT, + API_CONTROLLER_TOOL_DESCRIPTION, + API_CONTROLLER_TOOL_NAME, + API_ORCHESTRATOR_PROMPT, + API_PLANNER_PROMPT, + API_PLANNER_TOOL_DESCRIPTION, + API_PLANNER_TOOL_NAME, + PARSING_GET_PROMPT, + PARSING_POST_PROMPT, + REQUESTS_GET_TOOL_DESCRIPTION, + REQUESTS_POST_TOOL_DESCRIPTION, +) +from langchain.agents.agent_toolkits.openapi.spec import ( + ReducedOpenAPISpec, +) +from langchain.agents.mrkl.base import ZeroShotAgent +from langchain.agents.tools import Tool +from langchain.chains.llm import LLMChain +from langchain.llms.openai import OpenAI +from langchain.prompts import PromptTemplate +from langchain.requests import RequestsWrapper +from langchain.schema import BaseLanguageModel +from langchain.tools.base import BaseTool +from langchain.tools.requests.tool import BaseRequestsTool + +# +# Requests tools with LLM-instructed extraction of truncated responses. +# +# Of course, truncating so bluntly may lose a lot of valuable +# information in the response. +# However, the goal for now is to have only a single inference step. +MAX_RESPONSE_LENGTH = 5000 + + +class RequestsGetToolWithParsing(BaseRequestsTool, BaseTool): + name = "requests_get" + description = REQUESTS_GET_TOOL_DESCRIPTION + response_length: Optional[int] = MAX_RESPONSE_LENGTH + llm_chain = LLMChain( + llm=OpenAI(), + prompt=PARSING_GET_PROMPT, + ) + + def _run(self, text: str) -> str: + try: + data = json.loads(text) + except json.JSONDecodeError as e: + raise e + response = self.requests_wrapper.get(data["url"]) + response = response[: self.response_length] + return self.llm_chain.predict( + response=response, instructions=data["output_instructions"] + ).strip() + + async def _arun(self, text: str) -> str: + raise NotImplementedError() + + +class RequestsPostToolWithParsing(BaseRequestsTool, BaseTool): + name = "requests_post" + description = REQUESTS_POST_TOOL_DESCRIPTION + + response_length: Optional[int] = MAX_RESPONSE_LENGTH + llm_chain = LLMChain( + llm=OpenAI(), + prompt=PARSING_POST_PROMPT, + ) + + def _run(self, text: str) -> str: + try: + data = json.loads(text) + except json.JSONDecodeError as e: + raise e + response = self.requests_wrapper.post(data["url"], data["data"]) + response = response[: self.response_length] + return self.llm_chain.predict( + response=response, instructions=data["output_instructions"] + ).strip() + + async def _arun(self, text: str) -> str: + raise NotImplementedError() + + +# +# Orchestrator, planner, controller. +# +def _create_api_planner_tool( + api_spec: ReducedOpenAPISpec, llm: BaseLanguageModel +) -> Tool: + endpoint_descriptions = [ + f"{name} {description}" for name, description, _ in api_spec.endpoints + ] + prompt = PromptTemplate( + template=API_PLANNER_PROMPT, + input_variables=["query"], + partial_variables={"endpoints": "- " + "- ".join(endpoint_descriptions)}, + ) + chain = LLMChain(llm=llm, prompt=prompt) + tool = Tool( + name=API_PLANNER_TOOL_NAME, + description=API_PLANNER_TOOL_DESCRIPTION, + func=chain.run, + ) + return tool + + +def _create_api_controller_agent( + api_url: str, + api_docs: str, + requests_wrapper: RequestsWrapper, + llm: BaseLanguageModel, +) -> AgentExecutor: + tools: List[BaseTool] = [ + RequestsGetToolWithParsing(requests_wrapper=requests_wrapper), + RequestsPostToolWithParsing(requests_wrapper=requests_wrapper), + ] + prompt = PromptTemplate( + template=API_CONTROLLER_PROMPT, + input_variables=["input", "agent_scratchpad"], + partial_variables={ + "api_url": api_url, + "api_docs": api_docs, + "tool_names": ", ".join([tool.name for tool in tools]), + "tool_descriptions": "\n".join( + [f"{tool.name}: {tool.description}" for tool in tools] + ), + }, + ) + agent = ZeroShotAgent( + llm_chain=LLMChain(llm=llm, prompt=prompt), + allowed_tools=[tool.name for tool in tools], + ) + return AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True) + + +def _create_api_controller_tool( + api_spec: ReducedOpenAPISpec, + requests_wrapper: RequestsWrapper, + llm: BaseLanguageModel, +) -> Tool: + """Expose controller as a tool. + + The tool is invoked with a plan from the planner, and dynamically + creates a controller agent with relevant documentation only to + constrain the context. + """ + + base_url = api_spec.servers[0]["url"] # TODO: do better. + + def _create_and_run_api_controller_agent(plan_str: str) -> str: + pattern = r"\b(GET|POST)\s+(/\S+)*" + matches = re.findall(pattern, plan_str) + endpoint_names = [ + "{method} {route}".format(method=method, route=route.split("?")[0]) + for method, route in matches + ] + endpoint_docs_by_name = {name: docs for name, _, docs in api_spec.endpoints} + docs_str = "" + for endpoint_name in endpoint_names: + docs = endpoint_docs_by_name.get(endpoint_name) + if not docs: + raise ValueError(f"{endpoint_name} endpoint does not exist.") + docs_str += f"== Docs for {endpoint_name} == \n{yaml.dump(docs)}\n" + + agent = _create_api_controller_agent(base_url, docs_str, requests_wrapper, llm) + return agent.run(plan_str) + + return Tool( + name=API_CONTROLLER_TOOL_NAME, + func=_create_and_run_api_controller_agent, + description=API_CONTROLLER_TOOL_DESCRIPTION, + ) + + +def create_openapi_agent( + api_spec: ReducedOpenAPISpec, + requests_wrapper: RequestsWrapper, + llm: BaseLanguageModel, +) -> AgentExecutor: + """Instantiate API planner and controller for a given spec. + + Inject credentials via requests_wrapper. + + We use a top-level "orchestrator" agent to invoke the planner and controller, + rather than a top-level planner + that invokes a controller with its plan. This is to keep the planner simple. + """ + tools = [ + _create_api_planner_tool(api_spec, llm), + _create_api_controller_tool(api_spec, requests_wrapper, llm), + ] + prompt = PromptTemplate( + template=API_ORCHESTRATOR_PROMPT, + input_variables=["input", "agent_scratchpad"], + partial_variables={ + "tool_names": ", ".join([tool.name for tool in tools]), + "tool_descriptions": "\n".join( + [f"{tool.name}: {tool.description}" for tool in tools] + ), + }, + ) + agent = ZeroShotAgent( + llm_chain=LLMChain(llm=llm, prompt=prompt), + allowed_tools=[tool.name for tool in tools], + ) + return AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True) diff --git a/langchain/agents/agent_toolkits/openapi/planner_prompt.py b/langchain/agents/agent_toolkits/openapi/planner_prompt.py new file mode 100644 index 00000000..0104a6db --- /dev/null +++ b/langchain/agents/agent_toolkits/openapi/planner_prompt.py @@ -0,0 +1,156 @@ +# flake8: noqa + +from langchain.prompts.prompt import PromptTemplate + + +API_PLANNER_PROMPT = """You are a planner that plans a sequence of API calls to assist with user queries against an API. + +You should: +1) evaluate whether the user query can be solved by the API documentated below. If no, say why. +2) if yes, generate a plan of API calls and say what they are doing step by step. + +You should only use API endpoints documented below ("Endpoints you can use:"). +Some user queries can be resolved in a single API call, but some will require several API calls. +The plan will be passed to an API controller that can format it into web requests and return the responses. + +---- + +Here are some examples: + +Fake endpoints for examples: +GET /user to get information about the current user +GET /products/search search across products +POST /users/{{id}}/cart to add products to a user's cart + +User query: tell me a joke +Plan: Sorry, this API's domain is shopping, not comedy. + +Usery query: I want to buy a couch +Plan: 1. GET /products/search to search for couches +2. GET /user to find the user's id +3. POST /users/{{id}}/cart to add a couch to the user's cart + +---- + +Here are endpoints you can use. Do not reference any of the endpoints above. + +{endpoints} + +---- + +User query: {query} +Plan:""" +API_PLANNER_TOOL_NAME = "api_planner" +API_PLANNER_TOOL_DESCRIPTION = f"Can be used to generate the right API calls to assist with a user query, like {API_PLANNER_TOOL_NAME}(query). Should always be called before trying to calling the API controller." + +# Execution. +API_CONTROLLER_PROMPT = """You are an agent that gets a sequence of API calls and given their documentation, should execute them and return the final response. +If you cannot complete them and run into issues, you should explain the issue. If you're able to resolve an API call call, you can retry the API call. When interacting with API objects, you should extract ids for inputs to other API calls but ids and names for outputs returned to the User. + + +Here is documentation on the API: +Base url: {api_url} +Endpoints: +{api_docs} + + +Here are tools to execute requests against the API: {tool_descriptions} + + +Starting below, you should follow this format: + +Plan: the plan of API calls to execute +Thought: you should always think about what to do +Action: the action to take, should be one of the tools [{tool_names}] +Action Input: the input to the action +Observation: the output of the action +... (this Thought/Action/Action Input/Observation can repeat N times) +Thought: I am finished executing the plan (or, I cannot finish executing the plan without knowing some other information.) +Final Answer: the final output from executing the plan or missing information I'd need to re-plan correctly. + + +Begin! + +Plan: {input} +Thought: +{agent_scratchpad} +""" +API_CONTROLLER_TOOL_NAME = "api_controller" +API_CONTROLLER_TOOL_DESCRIPTION = f"Can be used to execute a plan of API calls, like {API_CONTROLLER_TOOL_NAME}(plan)." + +# Orchestrate planning + execution. +# The goal is to have an agent at the top-level (e.g. so it can recover from errors and re-plan) while +# keeping planning (and specifically the planning prompt) simple. +API_ORCHESTRATOR_PROMPT = """You are an agent that assists with user queries against API, things like querying information or creating resources. +Some user queries can be resolved in a single API call though some require several API call. +You should always plan your API calls first, and then execute the plan second. +You should never return information without executing the api_controller tool. + + +Here are the tools to plan and execute API requests: {tool_descriptions} + + +Starting below, you should follow this format: + +User query: the query a User wants help with related to the API +Thought: you should always think about what to do +Action: the action to take, should be one of the tools [{tool_names}] +Action Input: the input to the action +Observation: the result of the action +... (this Thought/Action/Action Input/Observation can repeat N times) +Thought: I am finished executing a plan and have the information the user asked for or the data the used asked to create +Final Answer: the final output from executing the plan + + +Example: +User query: can you add some trendy stuff to my shopping cart. +Thought: I should plan API calls first. +Action: api_planner +Action Input: I need to find the right API calls to add trendy items to the users shopping cart +Observation: 1) GET /items/trending to get trending item ids +2) GET /user to get user +3) POST /cart to post the trending items to the user's cart +Thought: I'm ready to execute the API calls. +Action: api_controller +Action Input: 1) GET /items/trending to get trending item ids +2) GET /user to get user +3) POST /cart to post the trending items to the user's cart +... + +Begin! + +User query: {input} +Thought: I should generate a plan to help with this query and then copy that plan exactly to the controller. +{agent_scratchpad}""" + +REQUESTS_GET_TOOL_DESCRIPTION = """Use this to GET content from a website. +Input to the tool should be a json string with 2 keys: "url" and "output_instructions". +The value of "url" should be a string. The value of "output_instructions" should be instructions on what information to extract from the response, for example the id(s) for a resource(s) that the GET request fetches. +""" + +PARSING_GET_PROMPT = PromptTemplate( + template="""Here is an API response:\n\n{response}\n\n==== +Your task is to extract some information according to these instructions: {instructions} +When working with API objects, you should usually use ids over names. +If the response indicates an error, you should instead output a summary of the error. + +Output:""", + input_variables=["response", "instructions"], +) + +REQUESTS_POST_TOOL_DESCRIPTION = """Use this when you want to POST to a website. +Input to the tool should be a json string with 3 keys: "url", "data", and "output_instructions". +The value of "url" should be a string. +The value of "data" should be a dictionary of key-value pairs you want to POST to the url. +The value of "summary_instructions" should be instructions on what information to extract from the response, for example the id(s) for a resource(s) that the POST request creates. +Always use double quotes for strings in the json string.""" + +PARSING_POST_PROMPT = PromptTemplate( + template="""Here is an API response:\n\n{response}\n\n==== +Your task is to extract some information according to these instructions: {instructions} +When working with API objects, you should usually use ids over names. Do not return any ids or names that are not in the response. +If the response indicates an error, you should instead output a summary of the error. + +Output:""", + input_variables=["response", "instructions"], +) diff --git a/langchain/agents/agent_toolkits/openapi/prompt.py b/langchain/agents/agent_toolkits/openapi/prompt.py index 376c3d8c..0484f5bf 100644 --- a/langchain/agents/agent_toolkits/openapi/prompt.py +++ b/langchain/agents/agent_toolkits/openapi/prompt.py @@ -16,7 +16,7 @@ Fourth, make the requests needed to answer the question. Ensure that you are sen Use the exact parameter names as listed in the spec, do not make up any names or abbreviate the names of parameters. If you get a not found error, ensure that you are using a path that actually exists in the spec. """ -OPENAPI_SUFFIX = """Begin!" +OPENAPI_SUFFIX = """Begin! Question: {input} Thought: I should explore the spec to find the base url for the API. diff --git a/langchain/agents/agent_toolkits/openapi/spec.py b/langchain/agents/agent_toolkits/openapi/spec.py new file mode 100644 index 00000000..72f3ff9d --- /dev/null +++ b/langchain/agents/agent_toolkits/openapi/spec.py @@ -0,0 +1,110 @@ +"""Quick and dirty representation for OpenAPI specs.""" + +from dataclasses import dataclass +from typing import Any, Dict, List, Tuple, Union + + +def dereference_refs(spec_obj: dict, full_spec: dict) -> Union[dict, list]: + """Try to substitute $refs. + + The goal is to get the complete docs for each endpoint in context for now. + + In the few OpenAPI specs I studied, $refs referenced models + (or in OpenAPI terms, components) and could be nested. This code most + likely misses lots of cases. + """ + + def _retrieve_ref_path(path: str, full_spec: dict) -> dict: + components = path.split("/") + if components[0] != "#": + raise RuntimeError( + "All $refs I've seen so far are uri fragments (start with hash)." + ) + out = full_spec + for component in components[1:]: + out = out[component] + return out + + def _dereference_refs( + obj: Union[dict, list], stop: bool = False + ) -> Union[dict, list]: + if stop: + return obj + obj_out: Dict[str, Any] = {} + if isinstance(obj, dict): + for k, v in obj.items(): + if k == "$ref": + # stop=True => don't dereference recursively. + return _dereference_refs( + _retrieve_ref_path(v, full_spec), stop=True + ) + elif isinstance(v, list): + obj_out[k] = [_dereference_refs(el) for el in v] + elif isinstance(v, dict): + obj_out[k] = _dereference_refs(v) + else: + obj_out[k] = v + return obj_out + elif isinstance(obj, list): + return [_dereference_refs(el) for el in obj] + else: + return obj + + return _dereference_refs(spec_obj) + + +@dataclass(frozen=True) +class ReducedOpenAPISpec: + servers: List[dict] + description: str + endpoints: List[Tuple[str, str, dict]] + + +def reduce_openapi_spec(spec: dict, dereference: bool = True) -> ReducedOpenAPISpec: + """Simplify/distill/minify a spec somehow. + + I want a smaller target for retrieval and (more importantly) + I want smaller results from retrieval. + I was hoping https://openapi.tools/ would have some useful bits + to this end, but doesn't seem so. + """ + # 1. Consider only get, post endpoints. + endpoints = [ + (f"{operation_name.upper()} {route}", docs.get("description"), docs) + for route, operation in spec["paths"].items() + for operation_name, docs in operation.items() + if operation_name in ["get", "post"] + ] + + # 2. Replace any refs so that complete docs are retrieved. + # Note: probably want to do this post-retrieval, it blows up the size of the spec. + if dereference: + endpoints = [ + (name, description, dereference_refs(docs, spec)) + for name, description, docs in endpoints + ] + + # 3. Strip docs down to required request args + happy path response. + def reduce_endpoint_docs(docs: dict) -> dict: + out = {} + if docs.get("description"): + out["description"] = docs.get("description") + if docs.get("parameters"): + out["parameters"] = [ + parameter + for parameter in docs.get("parameters", []) + if parameter.get("required") + ] + if "200" in docs["responses"]: + out["responses"] = docs["responses"]["200"] + return out + + endpoints = [ + (name, description, reduce_endpoint_docs(docs)) + for name, description, docs in endpoints + ] + return ReducedOpenAPISpec( + servers=spec["servers"], + description=spec["info"].get("description", ""), + endpoints=endpoints, + )