Plan and execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the ["Plan-and-Solve" paper](https://arxiv.org/abs/2305.04091).
Plan-and-execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the ["Plan-and-Solve" paper](https://arxiv.org/abs/2305.04091).
Plan and execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the ["Plan-and-Solve" paper](https://arxiv.org/abs/2305.04091).
Plan-and-execute agents accomplish an objective by first planning what to do, then executing the sub tasks. This idea is largely inspired by [BabyAGI](https://github.com/yoheinakajima/babyagi) and then the ["Plan-and-Solve" paper](https://arxiv.org/abs/2305.04091).
"This notebook showcases using an agent that uses the OpenAI functions ability to respond to the prompts of the user using a Large Language Model\n",
"This notebook showcases using an agent that uses the OpenAI functions ability to respond to the prompts of the user using a Large Language Model.\n",
"\n",
"Install openai,google-search-results packages which are required as the langchain packages call them internally\n",
"Install `openai`, `google-search-results` packages which are required as the LangChain packages call them internally.\n",
"\n",
">pip install openai google-search-results\n"
"```bash\n",
"pip install openai google-search-results\n",
"```\n"
]
},
{
@ -34,8 +36,8 @@
"source": [
"The agent is given ability to perform search functionalities with the respective tool\n",
"\n",
"SerpAPIWrapper:\n",
">This initializes the SerpAPIWrapper for search functionality (search).\n"
"`SerpAPIWrapper`:\n",
">This initializes the `SerpAPIWrapper` for search functionality (search).\n"
]
},
{
@ -228,7 +230,7 @@
"source": [
"## Configuring max iteration behavior\n",
"\n",
"To make sure that our agent doesn't get stuck in excessively long loops, we can set max_iterations. We can also set an early stopping method, which will determine our agent's behavior once the number of max iterations is hit. By default, the early stopping uses method `force` which just returns that constant string. Alternatively, you could specify method `generate` which then does one FINAL pass through the LLM to generate an output."
"To make sure that our agent doesn't get stuck in excessively long loops, we can set `max_iterations`. We can also set an early stopping method, which will determine our agent's behavior once the number of max iterations is hit. By default, the early stopping uses method `force` which just returns that constant string. Alternatively, you could specify method `generate` which then does one FINAL pass through the LLM to generate an output."
]
},
{
@ -428,7 +430,7 @@
"id": "067a8d3e",
"metadata": {},
"source": [
"Notice that we never get around to looking up the weather the day before yesterday, due to hitting our max_iterations limit."
"Notice that we never get around to looking up the weather the day before yesterday, due to hitting our `max_iterations` limit."
"This notebook goes over how to add memory to OpenAI Functions agent."
"This notebook goes over how to add memory to an OpenAI Functions agent."
]
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": null,
"id": "ac594f26",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/harrisonchase/.pyenv/versions/3.9.1/envs/langchain/lib/python3.9/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.6.4) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.\n",
"This notebook covers how to combine agents and vectorstores. The use case for this is that you've ingested your data into a vectorstore and want to interact with it in an agentic manner.\n",
"This notebook covers how to combine agents and vectorstores. The use case for this is that you've ingested your data into a vectorstore and want to interact with it in an agentic manner.\n",
"\n",
"The recommended method for doing so is to create a RetrievalQA and then use that as a tool in the overall agent. Let's take a look at doing this below. You can do this with multiple different vectordbs, and use the agent as a way to route between them. There are two different ways of doing this - you can either let the agent use the vectorstores as normal tools, or you can set `return_direct=True` to really just use the agent as a router."
"The recommended method for doing so is to create a `RetrievalQA` and then use that as a tool in the overall agent. Let's take a look at doing this below. You can do this with multiple different vector DBs, and use the agent as a way to route between them. There are two different ways of doing this - you can either let the agent use the vectorstores as normal tools, or you can set `return_direct=True` to really just use the agent as a router."
]
},
{
@ -18,7 +18,7 @@
"id": "9b22020a",
"metadata": {},
"source": [
"## Create the Vectorstore"
"## Create the vector store"
]
},
{
@ -419,9 +419,9 @@
"id": "49a0cbbe",
"metadata": {},
"source": [
"## Multi-Hop vectorstore reasoning\n",
"## Multi-Hop vectorstore reasoning\n",
"\n",
"Because vectorstores are easily usable as tools in agents, it is easy to use answer multi-hop questions that depend on vectorstores using the existing agent framework"
"Because vectorstores are easily usable as tools in agents, it is easy to use answer multi-hop questions that depend on vectorstores using the existing agent framework."
"LangChain provides async support for Agents by leveraging the [asyncio](https://docs.python.org/3/library/asyncio.html) library.\n",
"\n",
"Async methods are currently supported for the following `Tools`: [`GoogleSerperAPIWrapper`](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/utilities/google_serper.py), [`SerpAPIWrapper`](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/utilities/serpapi.py), [`LLMMathChain`](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/chains/llm_math/base.py) and [`Qdrant`](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/vectorstores/qdrant.py). Async support for other agent tools are on the roadmap.\n",
"Async methods are currently supported for the following `Tool`s: [`GoogleSerperAPIWrapper`](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/utilities/google_serper.py), [`SerpAPIWrapper`](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/utilities/serpapi.py), [`LLMMathChain`](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/chains/llm_math/base.py) and [`Qdrant`](https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/vectorstores/qdrant.py). Async support for other agent tools are on the roadmap.\n",
"\n",
"For `Tool`s that have a `coroutine` implemented (the four mentioned above), the `AgentExecutor` will `await` them directly. Otherwise, the `AgentExecutor` will call the `Tool`'s `func` via `asyncio.get_event_loop().run_in_executor` to avoid blocking the main runloop.\n",
"\n",
@ -21,7 +21,7 @@
"id": "97800378-cc34-4283-9bd0-43f336bc914c",
"metadata": {},
"source": [
"## Serial vs. Concurrent Execution\n",
"## Serial vs. concurrent execution\n",
"\n",
"In this example, we kick off agents to answer some questions serially vs. concurrently. You can see that concurrent execution significantly speeds this up."
"The novel idea introduced in this notebook is the idea of using retrieval to select the set of tools to use to answer an agent query. This is useful when you have many many tools to select from. You cannot put the description of all the tools in the prompt (because of context length issues) so instead you dynamically select the N tools you do want to consider using at run time.\n",
"\n",
"In this notebook we will create a somewhat contrieved example. We will have one legitimate tool (search) and then 99 fake tools which are just nonsense. We will then add a step in the prompt template that takes the user input and retrieves tool relevant to the query."
"In this notebook we will create a somewhat contrived example. We will have one legitimate tool (search) and then 99 fake tools which are just nonsense. We will then add a step in the prompt template that takes the user input and retrieves tool relevant to the query."
]
},
{
@ -51,7 +51,7 @@
"source": [
"## Set up tools\n",
"\n",
"We will create one legitimate tool (search) and then 99 fake tools"
"We will create one legitimate tool (search) and then 99 fake tools."
]
},
{
@ -92,7 +92,7 @@
"source": [
"## Tool Retriever\n",
"\n",
"We will use a vectorstore to create embeddings for each tool description. Then, for an incoming query we can create embeddings for that query and do a similarity search for relevant tools."
"We will use a vectorstore to create embeddings for each tool description. Then, for an incoming query we can create embeddings for that query and do a similarity search for relevant tools."
]
},
{
@ -206,7 +206,7 @@
"id": "2e7a075c",
"metadata": {},
"source": [
"## Prompt Template\n",
"## Prompt template\n",
"\n",
"The prompt template is pretty standard, because we're not actually changing that much logic in the actual prompt template, but rather we are just changing how retrieval is done."
]
@ -245,7 +245,7 @@
"id": "1583acdc",
"metadata": {},
"source": [
"The custom prompt template now has the concept of a tools_getter, which we call on the input to select the tools to use"
"The custom prompt template now has the concept of a `tools_getter`, which we call on the input to select the tools to use."
]
},
{
@ -308,7 +308,7 @@
"id": "ef3a1af3",
"metadata": {},
"source": [
"## Output Parser\n",
"## Output parser\n",
"\n",
"The output parser is unchanged from the previous notebook, since we are not changing anything about the output format."
"This notebook goes through how to create your own custom MRKL agent.\n",
"\n",
"A MRKL agent consists of three parts:\n",
"\n",
"- Tools: The tools the agent has available to use.\n",
"- LLMChain: The LLMChain that produces the text that is parsed in a certain way to determine which action to take.\n",
"- The agent class itself: this parses the output of the LLMChain to determine which action to take.\n",
"\n",
"- Tools: The tools the agent has available to use.\n",
"- `LLMChain`: The `LLMChain` that produces the text that is parsed in a certain way to determine which action to take.\n",
"- The agent class itself: this parses the output of the `LLMChain` to determine which action to take.\n",
" \n",
" \n",
"In this notebook we walk through how to create a custom MRKL agent by creating a custom LLMChain."
"In this notebook we walk through how to create a custom MRKL agent by creating a custom `LLMChain`."
]
},
{
@ -26,16 +26,16 @@
"source": [
"### Custom LLMChain\n",
"\n",
"The first way to create a custom agent is to use an existing Agent class, but use a custom LLMChain. This is the simplest way to create a custom Agent. It is highly recommended that you work with the `ZeroShotAgent`, as at the moment that is by far the most generalizable one. \n",
"The first way to create a custom agent is to use an existing Agent class, but use a custom `LLMChain`. This is the simplest way to create a custom Agent. It is highly recommended that you work with the `ZeroShotAgent`, as at the moment that is by far the most generalizable one. \n",
"\n",
"Most of the work in creating the custom LLMChain comes down to the prompt. Because we are using an existing agent class to parse the output, it is very important that the prompt say to produce text in that format. Additionally, we currently require an `agent_scratchpad` input variable to put notes on previous actions and observations. This should almost always be the final part of the prompt. However, besides those instructions, you can customize the prompt as you wish.\n",
"Most of the work in creating the custom `LLMChain` comes down to the prompt. Because we are using an existing agent class to parse the output, it is very important that the prompt say to produce text in that format. Additionally, we currently require an `agent_scratchpad` input variable to put notes on previous actions and observations. This should almost always be the final part of the prompt. However, besides those instructions, you can customize the prompt as you wish.\n",
"\n",
"To ensure that the prompt contains the appropriate instructions, we will utilize a helper method on that class. The helper method for the `ZeroShotAgent` takes the following arguments:\n",
"\n",
"- tools: List of tools the agent will have access to, used to format the prompt.\n",
"- prefix: String to put before the list of tools.\n",
"- suffix: String to put after the list of tools.\n",
"- input_variables: List of input variables the final prompt will expect.\n",
"- `tools`: List of tools the agent will have access to, used to format the prompt.\n",
"- `prefix`: String to put before the list of tools.\n",
"- `suffix`: String to put after the list of tools.\n",
"- `input_variables`: List of input variables the final prompt will expect.\n",
"\n",
"For this exercise, we will give our agent access to Google Search, and we will customize it in that we will have it answer as a pirate."
"Occasionally the LLM cannot determine what step to take because it outputs format in incorrect form to be handled by the output parser. In this case, by default the agent errors. But you can easily control this functionality with `handle_parsing_errors`! Let's explore how."
"Occasionally the LLM cannot determine what step to take because its outputs are not correctly formatted to be handled by the output parser. In this case, by default the agent errors. But you can easily control this functionality with `handle_parsing_errors`! Let's explore how."
]
},
{
@ -130,7 +130,7 @@
"source": [
"## Default error handling\n",
"\n",
"Handle errors with `Invalid or incomplete response`"
"Handle errors with `Invalid or incomplete response`:"
]
},
{
@ -202,9 +202,9 @@
"id": "6613cc9c",
"metadata": {},
"source": [
"## Custom Error Message\n",
"## Custom error message\n",
"\n",
"You can easily customize the message to use when there are parsing errors"
"You can easily customize the message to use when there are parsing errors."
"First, let's do a run with a normal agent to show what would happen without this parameter. For this example, we will use a specifically crafter adversarial example that tries to trick it into continuing forever.\n",
"First, let's do a run with a normal agent to show what would happen without this parameter. For this example, we will use a specifically crafted adversarial example that tries to trick it into continuing forever.\n",
"\n",
"Try running the cell below and see what happens!"
]
@ -203,7 +203,7 @@
"id": "0f7a80fb",
"metadata": {},
"source": [
"By default, the early stopping uses method `force` which just returns that constant string. Alternatively, you could specify method `generate` which then does one FINAL pass through the LLM to generate an output."
"By default, the early stopping uses the `force` method which just returns that constant string. Alternatively, you could specify the `generate` method which then does one FINAL pass through the LLM to generate an output."
"First, let's do a run with a normal agent to show what would happen without this parameter. For this example, we will use a specifically crafter adversarial example that tries to trick it into continuing forever.\n",
"First, let's do a run with a normal agent to show what would happen without this parameter. For this example, we will use a specifically crafted adversarial example that tries to trick it into continuing forever.\n",
"\n",
"Try running the cell below and see what happens!"
]
@ -199,7 +199,7 @@
"id": "0f7a80fb",
"metadata": {},
"source": [
"By default, the early stopping uses method `force` which just returns that constant string. Alternatively, you could specify method `generate` which then does one FINAL pass through the LLM to generate an output."
"By default, the early stopping uses the `force` method which just returns that constant string. Alternatively, you could specify the `generate` method which then does one FINAL pass through the LLM to generate an output."
"This notebook goes over adding memory to **both** of an Agent and its tools. Before going through this notebook, please walk through the following notebooks, as this will build on top of both of them:\n",
"This notebook goes over adding memory to **both** an Agent and its tools. Before going through this notebook, please walk through the following notebooks, as this will build on top of both of them:\n",
"\n",
"- [Adding memory to an LLM Chain](/docs/modules/memory/integrations/adding_memory.html)\n",
"We are going to create a custom Agent. The agent has access to a conversation memory, search tool, and a summarization tool. And, the summarization tool also needs access to the conversation memory."
"We are going to create a custom Agent. The agent has access to a conversation memory, search tool, and a summarization tool. The summarization tool also needs access to the conversation memory."
]
},
{
@ -102,7 +102,7 @@
"id": "0021675b",
"metadata": {},
"source": [
"We can now construct the LLMChain, with the Memory object, and then create the agent."
"We can now construct the `LLMChain`, with the Memory object, and then create the agent."
"When constructing your own agent, you will need to provide it with a list of Tools that it can use. Besides the actual function that is called, the Tool consists of several components:\n",
"\n",
"- name (str), is required and must be unique within a set of tools provided to an agent\n",
"- description (str), is optional but recommended, as it is used by an agent to determine tool use\n",
"- return_direct (bool), defaults to False\n",
"- args_schema (Pydantic BaseModel), is optional but recommended, can be used to provide more information (e.g., few-shot examples) or validation for expected parameters.\n",
"- `name` (str), is required and must be unique within a set of tools provided to an agent\n",
"- `description` (str), is optional but recommended, as it is used by an agent to determine tool use\n",
"- `return_direct` (bool), defaults to False\n",
"- `args_schema` (Pydantic BaseModel), is optional but recommended, can be used to provide more information (e.g., few-shot examples) or validation for expected parameters.\n",
"\n",
"\n",
"There are two main ways to define a tool, we will cover both in the example below."
@ -116,7 +116,7 @@
"id": "e9b560f7",
"metadata": {},
"source": [
"You can also define a custom `args_schema`` to provide more information about inputs."
"You can also define a custom `args_schema` to provide more information about inputs."
]
},
{
@ -442,7 +442,7 @@
"id": "de34a6a3",
"metadata": {},
"source": [
"You can also provide `args_schema` to provide more information about the argument"
"You can also provide `args_schema` to provide more information about the argument."
]
},
{
@ -533,7 +533,7 @@
"source": [
"## Subclassing the BaseTool\n",
"\n",
"The BaseTool automatically infers the schema from the _run method's signature."
"The BaseTool automatically infers the schema from the `_run` method's signature."
]
},
{
@ -840,7 +840,7 @@
"metadata": {},
"source": [
"## Using tools to return directly\n",
"Often, it can be desirable to have a tool output returned directly to the user, if it’s called. You can do this easily with LangChain by setting the return_direct flag for a tool to be True."
"Often, it can be desirable to have a tool output returned directly to the user, if it’s called. You can do this easily with LangChain by setting the `return_direct` flag for a tool to be True."
"This walkthrough demonstrates how to add Human validation to any Tool. We'll do this using the `HumanApprovalCallbackhandler`.\n",
"This walkthrough demonstrates how to add human validation to any Tool. We'll do this using the `HumanApprovalCallbackhandler`.\n",
"\n",
"Let's suppose we need to make use of the ShellTool. Adding this tool to an automated flow poses obvious risks. Let's see how we could enforce manual human approval of inputs going into this tool.\n",
"Let's suppose we need to make use of the `ShellTool`. Adding this tool to an automated flow poses obvious risks. Let's see how we could enforce manual human approval of inputs going into this tool.\n",
"\n",
"**Note**: We generally recommend against using the ShellTool. There's a lot of ways to misuse it, and it's not required for most use cases. We employ it here only for demonstration purposes."
"**Note**: We generally recommend against using the `ShellTool`. There's a lot of ways to misuse it, and it's not required for most use cases. We employ it here only for demonstration purposes."
]
},
{
@ -60,7 +60,7 @@
"metadata": {},
"source": [
"## Adding Human Approval\n",
"Adding the default HumanApprovalCallbackHandler to the tool will make it so that a user has to manually approve every input to the tool before the command is actually executed."
"Adding the default `HumanApprovalCallbackHandler` to the tool will make it so that a user has to manually approve every input to the tool before the command is actually executed."
"An alternative to the structured tool would be to use the regular `Tool` class and accept a single string. The tool would then have to handle the parsing logic to extract the relavent values from the text, which tightly couples the tool representation to the agent prompt. This is still useful if the underlying language model can't reliabl generate structured schema. \n",
"An alternative to the structured tool would be to use the regular `Tool` class and accept a single string. The tool would then have to handle the parsing logic to extract the relavent values from the text, which tightly couples the tool representation to the agent prompt. This is still useful if the underlying language model can't reliably generate structured schema. \n",
"\n",
"Let's take the multiplication function as an example. In order to use this, we will tell the agent to generate the \"Action Input\" as a comma-separated list of length two. We will then write a thin wrapper that takes a string, splits it into two around a comma, and passes both parsed sides as integers to the multiplication function."
"If you are planning to use the async API, it is recommended to use `AsyncCallbackHandler` to avoid blocking the runloop. \n",
"\n",
"**Advanced** if you use a sync `CallbackHandler` while using an async method to run your llm/chain/tool/agent, it will still work. However, under the hood, it will be called with [`run_in_executor`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor) which can cause issues if your `CallbackHandler` is not thread-safe."
"**Advanced** if you use a sync `CallbackHandler` while using an async method to run your LLM / Chain / Tool / Agent, it will still work. However, under the hood, it will be called with [`run_in_executor`](https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor) which can cause issues if your `CallbackHandler` is not thread-safe."
You can add tags to your callbacks by passing a `tags` argument to the `call()`/`run()`/`apply()` methods. This is useful for filtering your logs, e.g. if you want to log all requests made to a specific LLMChain, you can add a tag, and then filter your logs by that tag. You can pass tags to both constructor and request callbacks, see the examples above for details. These tags are then passed to the `tags` argument of the "start" callback methods, ie. `on_llm_start`, `on_chat_model_start`, `on_chain_start`, `on_tool_start`.
You can add tags to your callbacks by passing a `tags` argument to the `call()`/`run()`/`apply()` methods. This is useful for filtering your logs, e.g. if you want to log all requests made to a specific `LLMChain`, you can add a tag, and then filter your logs by that tag. You can pass tags to both constructor and request callbacks, see the examples above for details. These tags are then passed to the `tags` argument of the "start" callback methods, ie. `on_llm_start`, `on_chat_model_start`, `on_chain_start`, `on_tool_start`.
LangChain provides a few built-in handlers that you can use to get started. These are available in the `langchain/callbacks` module. The most basic handler is the `StdOutCallbackHandler`, which simply logs all events to `stdout`.
**Note** when the `verbose` flag on the object is set to true, the `StdOutCallbackHandler` will be invoked even without being explicitly passed in.
**Note**: when the `verbose` flag on the object is set to true, the `StdOutCallbackHandler` will be invoked even without being explicitly passed in.
```python
from langchain.callbacks import StdOutCallbackHandler
@ -137,6 +137,6 @@ The `verbose` argument is available on most objects throughout the API (Chains,
### When do you want to use each of these?
- Constructor callbacks are most useful for use cases such as logging, monitoring, etc., which are _not specific to a single request_, but rather to the entire chain. For example, if you want to log all the requests made to an LLMChain, you would pass a handler to the constructor.
- Constructor callbacks are most useful for use cases such as logging, monitoring, etc., which are _not specific to a single request_, but rather to the entire chain. For example, if you want to log all the requests made to an `LLMChain`, you would pass a handler to the constructor.
- Request callbacks are most useful for use cases such as streaming, where you want to stream the output of a single request to a specific websocket connection, or other similar use cases. For example, if you want to stream the output of a single request to a websocket, you would pass a handler to the `call()` method
seamusp
10 months ago
committed by
GitHub