Archives
/
langchain
mirror of https://github.com/hwchase17/langchain
2
0
Fork 1
Code Issues Packages Projects Releases Wiki Activity

docs: show tool msg in tool call docs (#20358)

Browse Source
pull/20361/head
Bagatur 6 months ago committed by GitHub
parent bac9fb9a7c
commit f78564d75c
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 707 additions and 324 deletions
Show Stats Download Patch File Download Diff File

707
docs/docs/modules/model_io/chat/function_calling.ipynb
Unescape Escape View File

@ -0,0 +1,707 @@
{
"cells": [
{
"cell_type": "raw",
"id": "a413ade7-48f0-4d43-a1f3-d87f550a8018",
"metadata": {},
"source": [
"---\n",
"sidebar_position: 2\n",
"title: Tool/function calling\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "50d59b14-c434-4359-be8e-4a21378e762f",
"metadata": {},
"source": [
"# Tool calling\n",
"\n",
"```{=mdx}\n",
":::info\n",
"We use the term tool calling interchangeably with function calling. Although\n",
"function calling is sometimes meant to refer to invocations of a single function,\n",
"we treat all models as though they can return multiple tool or function calls in \n",
"each message.\n",
":::\n",
"```\n",
"\n",
"Tool calling allows a model to respond to a given prompt by generating output that \n",
"matches a user-defined schema. While the name implies that the model is performing \n",
"some action, this is actually not the case! The model is coming up with the \n",
"arguments to a tool, and actually running the tool (or not) is up to the user - \n",
"for example, if you want to [extract output matching some schema](/docs/use_cases/extraction/) \n",
"from unstructured text, you could give the model an \"extraction\" tool that takes \n",
"parameters matching the desired schema, then treat the generated output as your final \n",
"result.\n",
"\n",
"A tool call includes a name, arguments dict, and an optional identifier. The \n",
"arguments dict is structured `{argument_name: argument_value}`.\n",
"\n",
"Many LLM providers, including [Anthropic](https://www.anthropic.com/), \n",
"[Cohere](https://cohere.com/), [Google](https://cloud.google.com/vertex-ai), \n",
"[Mistral](https://mistral.ai/), [OpenAI](https://openai.com/), and others, \n",
"support variants of a tool calling feature. These features typically allow requests \n",
"to the LLM to include available tools and their schemas, and for responses to include \n",
"calls to these tools. For instance, given a search engine tool, an LLM might handle a \n",
"query by first issuing a call to the search engine. The system calling the LLM can \n",
"receive the tool call, execute it, and return the output to the LLM to inform its \n",
"response. LangChain includes a suite of [built-in tools](/docs/integrations/tools/) \n",
"and supports several methods for defining your own [custom tools](/docs/modules/tools/custom_tools). \n",
"Tool-calling is extremely useful for building [tool-using chains and agents](/docs/use_cases/tool_use), \n",
"and for getting structured outputs from models more generally.\n",
"\n",
"Providers adopt different conventions for formatting tool schemas and tool calls. \n",
"For instance, Anthropic returns tool calls as parsed structures within a larger content block:\n",
"```python\n",
"[\n",
" {\n",
" \"text\": \"<thinking>\\nI should use a tool.\\n</thinking>\",\n",
" \"type\": \"text\"\n",
" },\n",
" {\n",
" \"id\": \"id_value\",\n",
" \"input\": {\"arg_name\": \"arg_value\"},\n",
" \"name\": \"tool_name\",\n",
" \"type\": \"tool_use\"\n",
" }\n",
"]\n",
"```\n",
"whereas OpenAI separates tool calls into a distinct parameter, with arguments as JSON strings:\n",
"```python\n",
"{\n",
" \"tool_calls\": [\n",
" {\n",
" \"id\": \"id_value\",\n",
" \"function\": {\n",
" \"arguments\": '{\"arg_name\": \"arg_value\"}',\n",
" \"name\": \"tool_name\"\n",
" },\n",
" \"type\": \"function\"\n",
" }\n",
" ]\n",
"}\n",
"```\n",
"LangChain implements standard interfaces for defining tools, passing them to LLMs, \n",
"and representing tool calls.\n",
"\n",
"## Passing tools to LLMs\n",
"\n",
"Chat models supporting tool calling features implement a `.bind_tools` method, which \n",
"receives a list of LangChain [tool objects](https://api.python.langchain.com/en/latest/tools/langchain_core.tools.BaseTool.html#langchain_core.tools.BaseTool) \n",
"and binds them to the chat model in its expected format. Subsequent invocations of the \n",
"chat model will include tool schemas in its calls to the LLM.\n",
"\n",
"For example, we can define the schema for custom tools using the `@tool` decorator \n",
"on Python functions:"
]
},
{
"cell_type": "code",
"execution_count": 22,
"id": "841dca72-1b57-4a42-8e22-da4835c4cfe0",
"metadata": {},
"outputs": [],
"source": [
"from langchain_core.tools import tool\n",
"\n",
"\n",
"@tool\n",
"def add(a: int, b: int) -> int:\n",
" \"\"\"Adds a and b.\"\"\"\n",
" return a + b\n",
"\n",
"\n",
"@tool\n",
"def multiply(a: int, b: int) -> int:\n",
" \"\"\"Multiplies a and b.\"\"\"\n",
" return a * b\n",
"\n",
"\n",
"tools = [add, multiply]"
]
},
{
"cell_type": "markdown",
"id": "48058b7d-048d-48e6-a272-3931ad7ad146",
"metadata": {},
"source": [
"Or below, we define the schema using Pydantic:\n"
]
},
{
"cell_type": "code",
"execution_count": 23,
"id": "fca56328-85e4-4839-97b7-b5dc55920602",
"metadata": {},
"outputs": [],
"source": [
"from langchain_core.pydantic_v1 import BaseModel, Field\n",
"\n",
"\n",
"# Note that the docstrings here are crucial, as they will be passed along\n",
"# to the model along with the class name.\n",
"class Add(BaseModel):\n",
" \"\"\"Add two integers together.\"\"\"\n",
"\n",
" a: int = Field(..., description=\"First integer\")\n",
" b: int = Field(..., description=\"Second integer\")\n",
"\n",
"\n",
"class Multiply(BaseModel):\n",
" \"\"\"Multiply two integers together.\"\"\"\n",
"\n",
" a: int = Field(..., description=\"First integer\")\n",
" b: int = Field(..., description=\"Second integer\")\n",
"\n",
"\n",
"tools = [Add, Multiply]"
]
},
{
"cell_type": "markdown",
"id": "ead9068d-11f6-42f3-a508-3c1830189947",
"metadata": {},
"source": [
"We can bind them to chat models as follows:\n",
"\n",
"```{=mdx}\n",
"import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
"\n",
"<ChatModelTabs\n",
" customVarName=\"llm\"\n",
" fireworksParams={`model=\"accounts/fireworks/models/firefunction-v1\", temperature=0`}\n",
"/>\n",
"```\n",
"\n",
"We can use the `bind_tools()` method to handle converting\n",
"`Multiply` to a \"tool\" and binding it to the model (i.e.,\n",
"passing it in each time the model is invoked)."
]
},
{
"cell_type": "code",
"execution_count": 67,
"id": "44eb8327-a03d-4c7c-945e-30f13f455346",
"metadata": {},
"outputs": [],
"source": [
"# | echo: false\n",
"# | output: false\n",
"\n",
"from langchain_openai import ChatOpenAI\n",
"\n",
"llm = ChatOpenAI(model=\"gpt-3.5-turbo-0125\", temperature=0)"
]
},
{
"cell_type": "code",
"execution_count": 68,
"id": "af2a83ac-e43f-43ce-b107-9ed8376bfb75",
"metadata": {},
"outputs": [],
"source": [
"llm_with_tools = llm.bind_tools(tools)"
]
},
{
"cell_type": "markdown",
"id": "16208230-f64f-4935-9aa1-280a91f34ba3",
"metadata": {},
"source": [
"## Tool calls\n",
"\n",
"If tool calls are included in a LLM response, they are attached to the corresponding \n",
"[message](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessage.html#langchain_core.messages.ai.AIMessage) \n",
"or [message chunk](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessageChunk.html#langchain_core.messages.ai.AIMessageChunk) \n",
"as a list of [tool call](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.ToolCall.html#langchain_core.messages.tool.ToolCall) \n",
"objects in the `.tool_calls` attribute. A `ToolCall` is a typed dict that includes a \n",
"tool name, dict of argument values, and (optionally) an identifier. Messages with no \n",
"tool calls default to an empty list for this attribute.\n",
"\n",
"Example:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "1640a4b4-c201-4b23-b257-738d854fb9fd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'name': 'Multiply',\n",
" 'args': {'a': 3, 'b': 12},\n",
" 'id': 'call_1Tdp5wUXbYQzpkBoagGXqUTo'},\n",
" {'name': 'Add',\n",
" 'args': {'a': 11, 'b': 49},\n",
" 'id': 'call_k9v09vYioS3X0Qg35zESuUKI'}]"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"query = \"What is 3 * 12? Also, what is 11 + 49?\"\n",
"\n",
"llm_with_tools.invoke(query).tool_calls"
]
},
{
"cell_type": "markdown",
"id": "ac3ff0fe-5119-46b8-a578-530245bff23f",
"metadata": {},
"source": [
"The `.tool_calls` attribute should contain valid tool calls. Note that on occasion, \n",
"model providers may output malformed tool calls (e.g., arguments that are not \n",
"valid JSON). When parsing fails in these cases, instances \n",
"of [InvalidToolCall](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.InvalidToolCall.html#langchain_core.messages.tool.InvalidToolCall) \n",
"are populated in the `.invalid_tool_calls` attribute. An `InvalidToolCall` can have \n",
"a name, string arguments, identifier, and error message.\n",
"\n",
"If desired, [output parsers](/docs/modules/model_io/output_parsers) can further \n",
"process the output. For example, we can convert back to the original Pydantic class:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "ca15fcad-74fe-4109-a1b1-346c3eefe238",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Multiply(a=3, b=12), Add(a=11, b=49)]"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_core.output_parsers.openai_tools import PydanticToolsParser\n",
"\n",
"chain = llm_with_tools | PydanticToolsParser(tools=[Multiply, Add])\n",
"chain.invoke(query)"
]
},
{
"cell_type": "markdown",
"id": "0ba3505d-f405-43ba-93c4-7fbd84f6464b",
"metadata": {},
"source": [
"### Streaming\n",
"\n",
"When tools are called in a streaming context, \n",
"[message chunks](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessageChunk.html#langchain_core.messages.ai.AIMessageChunk) \n",
"will be populated with [tool call chunk](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.ToolCallChunk.html#langchain_core.messages.tool.ToolCallChunk) \n",
"objects in a list via the `.tool_call_chunks` attribute. A `ToolCallChunk` includes \n",
"optional string fields for the tool `name`, `args`, and `id`, and includes an optional \n",
"integer field `index` that can be used to join chunks together. Fields are optional \n",
"because portions of a tool call may be streamed across different chunks (e.g., a chunk \n",
"that includes a substring of the arguments may have null values for the tool name and id).\n",
"\n",
"Because message chunks inherit from their parent message class, an \n",
"[AIMessageChunk](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessageChunk.html#langchain_core.messages.ai.AIMessageChunk) \n",
"with tool call chunks will also include `.tool_calls` and `.invalid_tool_calls` fields. \n",
"These fields are parsed best-effort from the message's tool call chunks.\n",
"\n",
"Note that not all providers currently support streaming for tool calls.\n",
"\n",
"Example:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "4f54a0de-74c7-4f2d-86c5-660aed23840d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[]\n",
"[{'name': 'Multiply', 'args': '', 'id': 'call_d39MsxKM5cmeGJOoYKdGBgzc', 'index': 0}]\n",
"[{'name': None, 'args': '{\"a\"', 'id': None, 'index': 0}]\n",
"[{'name': None, 'args': ': 3, ', 'id': None, 'index': 0}]\n",
"[{'name': None, 'args': '\"b\": 1', 'id': None, 'index': 0}]\n",
"[{'name': None, 'args': '2}', 'id': None, 'index': 0}]\n",
"[{'name': 'Add', 'args': '', 'id': 'call_QJpdxD9AehKbdXzMHxgDMMhs', 'index': 1}]\n",
"[{'name': None, 'args': '{\"a\"', 'id': None, 'index': 1}]\n",
"[{'name': None, 'args': ': 11,', 'id': None, 'index': 1}]\n",
"[{'name': None, 'args': ' \"b\": ', 'id': None, 'index': 1}]\n",
"[{'name': None, 'args': '49}', 'id': None, 'index': 1}]\n",
"[]\n"
]
}
],
"source": [
"async for chunk in llm_with_tools.astream(query):\n",
" print(chunk.tool_call_chunks)"
]
},
{
"cell_type": "markdown",
"id": "55046320-3466-4ec1-a1f8-336234ba9019",
"metadata": {},
"source": [
"Note that adding message chunks will merge their corresponding tool call chunks. This is the principle by which LangChain's various [tool output parsers](/docs/modules/model_io/output_parsers/types/openai_tools/) support streaming.\n",
"\n",
"For example, below we accumulate tool call chunks:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "0a944af0-eedd-43c8-8ff3-f4301f129d9b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[]\n",
"[{'name': 'Multiply', 'args': '', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}]\n",
"[{'name': 'Multiply', 'args': '{\"a\"', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, ', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, \"b\": 1', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, \"b\": 12}', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, \"b\": 12}', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}, {'name': 'Add', 'args': '', 'id': 'call_tYHYdEV2YBvzDcSCiFCExNvw', 'index': 1}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, \"b\": 12}', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}, {'name': 'Add', 'args': '{\"a\"', 'id': 'call_tYHYdEV2YBvzDcSCiFCExNvw', 'index': 1}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, \"b\": 12}', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}, {'name': 'Add', 'args': '{\"a\": 11,', 'id': 'call_tYHYdEV2YBvzDcSCiFCExNvw', 'index': 1}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, \"b\": 12}', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}, {'name': 'Add', 'args': '{\"a\": 11, \"b\": ', 'id': 'call_tYHYdEV2YBvzDcSCiFCExNvw', 'index': 1}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, \"b\": 12}', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}, {'name': 'Add', 'args': '{\"a\": 11, \"b\": 49}', 'id': 'call_tYHYdEV2YBvzDcSCiFCExNvw', 'index': 1}]\n",
"[{'name': 'Multiply', 'args': '{\"a\": 3, \"b\": 12}', 'id': 'call_erKtz8z3e681cmxYKbRof0NS', 'index': 0}, {'name': 'Add', 'args': '{\"a\": 11, \"b\": 49}', 'id': 'call_tYHYdEV2YBvzDcSCiFCExNvw', 'index': 1}]\n"
]
}
],
"source": [
"first = True\n",
"async for chunk in llm_with_tools.astream(query):\n",
" if first:\n",
" gathered = chunk\n",
" first = False\n",
" else:\n",
" gathered = gathered + chunk\n",
"\n",
" print(gathered.tool_call_chunks)"
]
},
{
"cell_type": "code",
"execution_count": 19,
"id": "db4e3e3a-3553-44dc-bd31-149c0981a06a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'str'>\n"
]
}
],
"source": [
"print(type(gathered.tool_call_chunks[0][\"args\"]))"
]
},
{
"cell_type": "markdown",
"id": "95e92826-6e55-4684-9498-556f357f73ac",
"metadata": {},
"source": [
"And below we accumulate tool calls to demonstrate partial parsing:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"id": "e9402bde-d4b5-4564-a99e-f88c9b46b28a",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[]\n",
"[]\n",
"[{'name': 'Multiply', 'args': {}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3, 'b': 1}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}, {'name': 'Add', 'args': {}, 'id': 'call_UjSHJKROSAw2BDc8cp9cSv4i'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}, {'name': 'Add', 'args': {'a': 11}, 'id': 'call_UjSHJKROSAw2BDc8cp9cSv4i'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}, {'name': 'Add', 'args': {'a': 11}, 'id': 'call_UjSHJKROSAw2BDc8cp9cSv4i'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}, {'name': 'Add', 'args': {'a': 11, 'b': 49}, 'id': 'call_UjSHJKROSAw2BDc8cp9cSv4i'}]\n",
"[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_BXqUtt6jYCwR1DguqpS2ehP0'}, {'name': 'Add', 'args': {'a': 11, 'b': 49}, 'id': 'call_UjSHJKROSAw2BDc8cp9cSv4i'}]\n"
]
}
],
"source": [
"first = True\n",
"async for chunk in llm_with_tools.astream(query):\n",
" if first:\n",
" gathered = chunk\n",
" first = False\n",
" else:\n",
" gathered = gathered + chunk\n",
"\n",
" print(gathered.tool_calls)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"id": "8c2f21cc-0c6d-416a-871f-e854621c96e2",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"<class 'dict'>\n"
]
}
],
"source": [
"print(type(gathered.tool_calls[0][\"args\"]))"
]
},
{
"cell_type": "markdown",
"id": "97a0c977-0c3c-4011-b49b-db98c609d0ce",
"metadata": {},
"source": [
"## Passing tool outputs to model\n",
"\n",
"If we're using the model-generated tool invocations to actually call tools and want to pass the tool results back to the model, we can do so using `ToolMessage`s."
]
},
{
"cell_type": "code",
"execution_count": 117,
"id": "48049192-be28-42ab-9a44-d897924e67cd",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[HumanMessage(content='What is 3 * 12? Also, what is 11 + 49?'),\n",
" AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_K5DsWEmgt6D08EI9AFu9NaL1', 'function': {'arguments': '{\"a\": 3, \"b\": 12}', 'name': 'Multiply'}, 'type': 'function'}, {'id': 'call_qywVrsplg0ZMv7LHYYMjyG81', 'function': {'arguments': '{\"a\": 11, \"b\": 49}', 'name': 'Add'}, 'type': 'function'}]}, response_metadata={'token_usage': {'completion_tokens': 50, 'prompt_tokens': 105, 'total_tokens': 155}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_b28b39ffa8', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-1a0b8cdd-9221-4d94-b2ed-5701f67ce9fe-0', tool_calls=[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_K5DsWEmgt6D08EI9AFu9NaL1'}, {'name': 'Add', 'args': {'a': 11, 'b': 49}, 'id': 'call_qywVrsplg0ZMv7LHYYMjyG81'}]),\n",
" ToolMessage(content='36', tool_call_id='call_K5DsWEmgt6D08EI9AFu9NaL1'),\n",
" ToolMessage(content='60', tool_call_id='call_qywVrsplg0ZMv7LHYYMjyG81')]"
]
},
"execution_count": 117,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_core.messages import HumanMessage, ToolMessage\n",
"\n",
"messages = [HumanMessage(query)]\n",
"ai_msg = llm_with_tools.invoke(messages)\n",
"messages.append(ai_msg)\n",
"for tool_call in ai_msg.tool_calls:\n",
" selected_tool = {\"add\": add, \"multiply\": multiply}[tool_call[\"name\"].lower()]\n",
" tool_output = selected_tool.invoke(tool_call[\"args\"])\n",
" messages.append(ToolMessage(tool_output, tool_call_id=tool_call[\"id\"]))\n",
"messages"
]
},
{
"cell_type": "code",
"execution_count": 118,
"id": "611e0f36-d736-48d1-bca1-1cec51d223f3",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"AIMessage(content='3 * 12 is 36 and 11 + 49 is 60.', response_metadata={'token_usage': {'completion_tokens': 18, 'prompt_tokens': 171, 'total_tokens': 189}, 'model_name': 'gpt-3.5-turbo', 'system_fingerprint': 'fp_b28b39ffa8', 'finish_reason': 'stop', 'logprobs': None}, id='run-a6c8093c-b16a-4c92-8308-7c9ac998118c-0')"
]
},
"execution_count": 118,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"llm_with_tools.invoke(messages)"
]
},
{
"cell_type": "markdown",
"id": "a5937498-d6fe-400a-b192-ef35c314168e",
"metadata": {},
"source": [
"## Few-shot prompting\n",
"\n",
"For more complex tool use it's very useful to add few-shot examples to the prompt. We can do this by adding `AIMessage`s with `ToolCall`s and corresponding `ToolMessage`s to our prompt.\n",
"\n",
"For example, even with some special instructions our model can get tripped up by order of operations:"
]
},
{
"cell_type": "code",
"execution_count": 112,
"id": "5ef2e7c3-0925-49da-ab8f-e42c4fa40f29",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'name': 'Multiply',\n",
" 'args': {'a': 119, 'b': 8},\n",
" 'id': 'call_Dl3FXRVkQCFW4sUNYOe4rFr7'},\n",
" {'name': 'Add',\n",
" 'args': {'a': 952, 'b': -20},\n",
" 'id': 'call_n03l4hmka7VZTCiP387Wud2C'}]"
]
},
"execution_count": 112,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"llm_with_tools.invoke(\n",
" \"Whats 119 times 8 minus 20. Don't do any math yourself, only use tools for math. Respect order of operations\"\n",
").tool_calls"
]
},
{
"cell_type": "markdown",
"id": "a5249069-b5f8-40ac-ae74-30d67c4e9168",
"metadata": {},
"source": [
"The model shouldn't be trying to add anything yet, since it technically can't know the results of 119 * 8 yet.\n",
"\n",
"By adding a prompt with some examples we can correct this behavior:"
]
},
{
"cell_type": "code",
"execution_count": 107,
"id": "7b2e8b19-270f-4e1a-8be7-7aad704c1cf4",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'name': 'Multiply',\n",
" 'args': {'a': 119, 'b': 8},\n",
" 'id': 'call_MoSgwzIhPxhclfygkYaKIsGZ'}]"
]
},
"execution_count": 107,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from langchain_core.messages import AIMessage\n",
"from langchain_core.prompts import ChatPromptTemplate\n",
"from langchain_core.runnables import RunnablePassthrough\n",
"\n",
"examples = [\n",
" HumanMessage(\n",
" \"What's the product of 317253 and 128472 plus four\", name=\"example_user\"\n",
" ),\n",
" AIMessage(\n",
" \"\",\n",
" name=\"example_assistant\",\n",
" tool_calls=[\n",
" {\"name\": \"Multiply\", \"args\": {\"x\": 317253, \"y\": 128472}, \"id\": \"1\"}\n",
" ],\n",
" ),\n",
" ToolMessage(\"16505054784\", tool_call_id=\"1\"),\n",
" AIMessage(\n",
" \"\",\n",
" name=\"example_assistant\",\n",
" tool_calls=[{\"name\": \"Add\", \"args\": {\"x\": 16505054784, \"y\": 4}, \"id\": \"2\"}],\n",
" ),\n",
" ToolMessage(\"16505054788\", tool_call_id=\"2\"),\n",
" AIMessage(\n",
" \"The product of 317253 and 128472 plus four is 16505054788\",\n",
" name=\"example_assistant\",\n",
" ),\n",
"]\n",
"\n",
"system = \"\"\"You are bad at math but are an expert at using a calculator. \n",
"\n",
"Use past tool usage as an example of how to correctly use the tools.\"\"\"\n",
"few_shot_prompt = ChatPromptTemplate.from_messages(\n",
" [\n",
" (\"system\", system),\n",
" *examples,\n",
" (\"human\", \"{query}\"),\n",
" ]\n",
")\n",
"\n",
"chain = {\"query\": RunnablePassthrough()} | few_shot_prompt | llm_with_tools\n",
"chain.invoke(\"Whats 119 times 8 minus 20\").tool_calls"
]
},
{
"cell_type": "markdown",
"id": "19160e3e-3eb5-4e9a-ae56-74a2dce0af32",
"metadata": {},
"source": [
"Seems like we get the correct output this time.\n",
"\n",
"Here's what the [LangSmith trace](https://smith.langchain.com/public/f70550a1-585f-4c9d-a643-13148ab1616f/r) looks like."
]
},
{
"cell_type": "markdown",
"id": "020cfd3b-0838-49d0-96bb-7cd919921833",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"- **Output parsing**: See [OpenAI Tools output\n",
" parsers](/docs/modules/model_io/output_parsers/types/openai_tools/)\n",
" and [OpenAI Functions output\n",
" parsers](/docs/modules/model_io/output_parsers/types/openai_functions/)\n",
" to learn about extracting the function calling API responses into\n",
" various formats.\n",
"- **Structured output chains**: [Some models have constructors](/docs/modules/model_io/chat/structured_output/) that\n",
" handle creating a structured output chain for you.\n",
"- **Tool use**: See how to construct chains and agents that\n",
" call the invoked tools in [these\n",
" guides](/docs/use_cases/tool_use/)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "poetry-venv-2",
"language": "python",
"name": "poetry-venv-2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
}
},
"nbformat": 4,
"nbformat_minor": 5
}

324
docs/docs/modules/model_io/chat/function_calling.mdx
Unescape Escape View File

@ -1,324 +0,0 @@
---
sidebar_position: 2
title: Tool/function calling
---
# Tool calling
:::info
We use the term tool calling interchangeably with function calling. Although
function calling is sometimes meant to refer to invocations of a single function,
we treat all models as though they can return multiple tool or function calls in
each message.
:::
# Calling Tools
Tool calling allows a model to respond to a given prompt by generating output that
matches a user-defined schema. While the name implies that the model is performing
some action, this is actually not the case! The model is coming up with the
arguments to a tool, and actually running the tool (or not) is up to the user -
for example, if you want to [extract output matching some schema](/docs/use_cases/extraction/)
from unstructured text, you could give the model an "extraction" tool that takes
parameters matching the desired schema, then treat the generated output as your final
result.
A tool call includes a name, arguments dict, and an optional identifier. The
arguments dict is structured `{argument_name: argument_value}`.
Many LLM providers, including [Anthropic](https://www.anthropic.com/),
[Cohere](https://cohere.com/), [Google](https://cloud.google.com/vertex-ai),
[Mistral](https://mistral.ai/), [OpenAI](https://openai.com/), and others,
support variants of a tool calling feature. These features typically allow requests
to the LLM to include available tools and their schemas, and for responses to include
calls to these tools. For instance, given a search engine tool, an LLM might handle a
query by first issuing a call to the search engine. The system calling the LLM can
receive the tool call, execute it, and return the output to the LLM to inform its
response. LangChain includes a suite of [built-in tools](/docs/integrations/tools/)
and supports several methods for defining your own [custom tools](/docs/modules/tools/custom_tools).
Tool-calling is extremely useful for building [tool-using chains and agents](/docs/use_cases/tool_use),
and for getting structured outputs from models more generally.
Providers adopt different conventions for formatting tool schemas and tool calls.
For instance, Anthropic returns tool calls as parsed structures within a larger content block:
```
[
{
"text": "<thinking>\nI should use a tool.\n</thinking>",
"type": "text"
},
{
"id": "id_value",
"input": {"arg_name": "arg_value"},
"name": "tool_name",
"type": "tool_use"
}
]
```
whereas OpenAI separates tool calls into a distinct parameter, with arguments as JSON strings:
```
{
"tool_calls": [
{
"id": "id_value",
"function": {
"arguments": '{"arg_name": "arg_value"}',
"name": "tool_name"
},
"type": "function"
}
]
}
```
LangChain implements standard interfaces for defining tools, passing them to LLMs,
and representing tool calls.
## Passing tools to LLMs
Chat models supporting tool calling features implement a `.bind_tools` method, which
receives a list of LangChain [tool objects](https://api.python.langchain.com/en/latest/tools/langchain_core.tools.BaseTool.html#langchain_core.tools.BaseTool)
and binds them to the chat model in its expected format. Subsequent invocations of the
chat model will include tool schemas in its calls to the LLM.
For example, we can define the schema for custom tools using the `@tool` decorator
on Python functions:
```python
from langchain.tools import tool
@tool
def add(a: int, b: int) -> int:
"""Adds a and b."""
return a + b
@tool
def multiply(a: int, b: int) -> int:
"""Multiplies a and b."""
return a * b
tools = [add, multiply]
```
Or below, we define the schema using Pydantic:
```python
from langchain_core.pydantic_v1 import BaseModel, Field
# Note that the docstrings here are crucial, as they will be passed along
# to the model along with the class name.
class Add(BaseModel):
"""Add two integers together."""
a: int = Field(..., description="First integer")
b: int = Field(..., description="Second integer")
class Multiply(BaseModel):
"""Multiply two integers together."""
a: int = Field(..., description="First integer")
b: int = Field(..., description="Second integer")
tools = [Add, Multiply]
```
We can bind them to chat models as follows:
import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
import ChatModelTabs from "@theme/ChatModelTabs";
<ChatModelTabs
customVarName="llm"
fireworksParams={`model="accounts/fireworks/models/firefunction-v1", temperature=0`}
/>
We can use the `bind_tools()` method to handle converting
`Multiply` to a "tool" and binding it to the model (i.e.,
passing it in each time the model is invoked).
```python
llm_with_tools = llm.bind_tools(tools)
```
## Tool calls
If tool calls are included in a LLM response, they are attached to the corresponding
[message](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessage.html#langchain_core.messages.ai.AIMessage)
or [message chunk](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessageChunk.html#langchain_core.messages.ai.AIMessageChunk)
as a list of [tool call](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.ToolCall.html#langchain_core.messages.tool.ToolCall)
objects in the `.tool_calls` attribute. A `ToolCall` is a typed dict that includes a
tool name, dict of argument values, and (optionally) an identifier. Messages with no
tool calls default to an empty list for this attribute.
Example:
```python
query = "What is 3 * 12? Also, what is 11 + 49?"
llm_with_tools.invoke(query).tool_calls
```
```text
[{'name': 'Multiply',
'args': {'a': 3, 'b': 12},
'id': 'call_viACG45wBz9jYzljHIwHamXw'},
{'name': 'Add',
'args': {'a': 11, 'b': 49},
'id': 'call_JMFUqoi5L27rGeMuII4MJMWo'}]
```
The `.tool_calls` attribute should contain valid tool calls. Note that on occasion,
model providers may output malformed tool calls (e.g., arguments that are not
valid JSON). When parsing fails in these cases, instances
of [InvalidToolCall](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.InvalidToolCall.html#langchain_core.messages.tool.InvalidToolCall)
are populated in the `.invalid_tool_calls` attribute. An `InvalidToolCall` can have
a name, string arguments, identifier, and error message.
If desired, [output parsers](/docs/modules/model_io/output_parsers) can further
process the output. For example, we can convert back to the original Pydantic class:
```python
from langchain_core.output_parsers.openai_tools import PydanticToolsParser
chain = llm_with_tools | PydanticToolsParser(tools=[Multiply, Add])
chain.invoke(query)
```
```text
[Multiply(a=3, b=12), Add(a=11, b=49)]
```
### Streaming
When tools are called in a streaming context,
[message chunks](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessageChunk.html#langchain_core.messages.ai.AIMessageChunk)
will be populated with [tool call chunk](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.ToolCallChunk.html#langchain_core.messages.tool.ToolCallChunk)
objects in a list via the `.tool_call_chunks` attribute. A `ToolCallChunk` includes
optional string fields for the tool `name`, `args`, and `id`, and includes an optional
integer field `index` that can be used to join chunks together. Fields are optional
because portions of a tool call may be streamed across different chunks (e.g., a chunk
that includes a substring of the arguments may have null values for the tool name and id).
Because message chunks inherit from their parent message class, an
[AIMessageChunk](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessageChunk.html#langchain_core.messages.ai.AIMessageChunk)
with tool call chunks will also include `.tool_calls` and `.invalid_tool_calls` fields.
These fields are parsed best-effort from the message's tool call chunks.
Note that not all providers currently support streaming for tool calls.
Example:
```python
async for chunk in llm_with_tools.astream(query):
print(chunk.tool_call_chunks)
```
```text
[]
[{'name': 'Multiply', 'args': '', 'id': 'call_Al2xpR4uFPXQUDzGTSawMOah', 'index': 0}]
[{'name': None, 'args': '{"a"', 'id': None, 'index': 0}]
[{'name': None, 'args': ': 3, ', 'id': None, 'index': 0}]
[{'name': None, 'args': '"b": 1', 'id': None, 'index': 0}]
[{'name': None, 'args': '2}', 'id': None, 'index': 0}]
[{'name': 'Add', 'args': '', 'id': 'call_VV6ck8JSQ6joKtk2xGtNKgXf', 'index': 1}]
[{'name': None, 'args': '{"a"', 'id': None, 'index': 1}]
[{'name': None, 'args': ': 11,', 'id': None, 'index': 1}]
[{'name': None, 'args': ' "b": ', 'id': None, 'index': 1}]
[{'name': None, 'args': '49}', 'id': None, 'index': 1}]
[]
```
Note that adding message chunks will merge their corresponding tool call chunks. This is the principle by which LangChain's various [tool output parsers](/docs/modules/model_io/output_parsers/types/openai_tools/) support streaming.
For example, below we accumulate tool call chunks:
```python
first = True
async for chunk in llm_with_tools.astream(query):
if first:
gathered = chunk
first = False
else:
gathered = gathered + chunk
print(gathered.tool_call_chunks)
```
```text
[]
[{'name': 'Multiply', 'args': '', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}]
[{'name': 'Multiply', 'args': '{"a"', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}]
[{'name': 'Multiply', 'args': '{"a": 3, ', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}]
[{'name': 'Multiply', 'args': '{"a": 3, "b": 1', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}]
[{'name': 'Multiply', 'args': '{"a": 3, "b": 12}', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}]
[{'name': 'Multiply', 'args': '{"a": 3, "b": 12}', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}, {'name': 'Add', 'args': '', 'id': 'call_uGot9MOHDcz67Bj0h13c7QA5', 'index': 1}]
[{'name': 'Multiply', 'args': '{"a": 3, "b": 12}', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}, {'name': 'Add', 'args': '{"a"', 'id': 'call_uGot9MOHDcz67Bj0h13c7QA5', 'index': 1}]
[{'name': 'Multiply', 'args': '{"a": 3, "b": 12}', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}, {'name': 'Add', 'args': '{"a": 11,', 'id': 'call_uGot9MOHDcz67Bj0h13c7QA5', 'index': 1}]
[{'name': 'Multiply', 'args': '{"a": 3, "b": 12}', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}, {'name': 'Add', 'args': '{"a": 11, "b": ', 'id': 'call_uGot9MOHDcz67Bj0h13c7QA5', 'index': 1}]
[{'name': 'Multiply', 'args': '{"a": 3, "b": 12}', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}, {'name': 'Add', 'args': '{"a": 11, "b": 49}', 'id': 'call_uGot9MOHDcz67Bj0h13c7QA5', 'index': 1}]
[{'name': 'Multiply', 'args': '{"a": 3, "b": 12}', 'id': 'call_2MG1IGft6WmgMooqZgJ07JX6', 'index': 0}, {'name': 'Add', 'args': '{"a": 11, "b": 49}', 'id': 'call_uGot9MOHDcz67Bj0h13c7QA5', 'index': 1}]
```
```python
print(type(gathered.tool_call_chunks[0]["args"]))
```
```text
<class 'str'>
```
And below we accumulate tool calls to demonstrate partial parsing:
```python
first = True
async for chunk in llm_with_tools.astream(query):
if first:
gathered = chunk
first = False
else:
gathered = gathered + chunk
print(gathered.tool_calls)
```
```text
[]
[]
[{'name': 'Multiply', 'args': {}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}]
[{'name': 'Multiply', 'args': {'a': 3}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}]
[{'name': 'Multiply', 'args': {'a': 3, 'b': 1}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}]
[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}]
[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}]
[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}, {'name': 'Add', 'args': {}, 'id': 'call_zPAyMWr8hN1q083GWGX2dSiB'}]
[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}, {'name': 'Add', 'args': {'a': 11}, 'id': 'call_zPAyMWr8hN1q083GWGX2dSiB'}]
[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}, {'name': 'Add', 'args': {'a': 11}, 'id': 'call_zPAyMWr8hN1q083GWGX2dSiB'}]
[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}, {'name': 'Add', 'args': {'a': 11, 'b': 49}, 'id': 'call_zPAyMWr8hN1q083GWGX2dSiB'}]
[{'name': 'Multiply', 'args': {'a': 3, 'b': 12}, 'id': 'call_z3B4o82SQDY5NCnmrXIcVQo4'}, {'name': 'Add', 'args': {'a': 11, 'b': 49}, 'id': 'call_zPAyMWr8hN1q083GWGX2dSiB'}]
```
```python
print(type(gathered.tool_calls[0]["args"]))
```
```text
<class 'dict'>
```
## Next steps
- **Output parsing**: See [OpenAI Tools output
parsers](/docs/modules/model_io/output_parsers/types/openai_tools/)
and [OpenAI Functions output
parsers](/docs/modules/model_io/output_parsers/types/openai_functions/)
to learn about extracting the function calling API responses into
various formats.
- **Structured output chains**: [Some models have constructors](/docs/modules/model_io/chat/structured_output/) that
handle creating a structured output chain for you.
- **Tool use**: See how to construct chains and agents that actually
call the invoked tools in [these
guides](/docs/use_cases/tool_use/).
Write Preview
Loading…
Cancel
Save