"The above cell assumes that your OpenAI API key is set in your environment variables. If you would rather manually specify your API key and/or organization ID, use the following code:\n",
" content=\"You are a helpful assistant that translates English to French.\"\n",
" ),\n",
" HumanMessage(\n",
" content=\"Translate this sentence from English to French. I love programming.\"\n",
" ),\n",
" (\"system\", \"You are a helpful assistant that translates English to French.\"),\n",
" (\"human\", \"Translate this sentence from English to French. I love programming.\"),\n",
"]\n",
"chat.invoke(messages)"
"llm.invoke(messages)"
]
},
{
@ -99,56 +91,154 @@
"id": "778f912a-66ea-4a5d-b3de-6c7db4baba26",
"metadata": {},
"source": [
"You can make use of templating by using a `MessagePromptTemplate`. You can build a `ChatPromptTemplate` from one or more `MessagePromptTemplates`. You can use `ChatPromptTemplate`'s `format_prompt` -- this returns a `PromptValue`, which you can convert to a string or Message object, depending on whether you want to use the formatted value as input to an llm or chat model.\n",
"## Chaining\n",
"\n",
"For convenience, there is a `from_template` method exposed on the template. If you were to use this template, this is what it would look like:"
"We can chain our model with a prompt template like so:"
" \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n",
" ),\n",
" (\"human\", \"{input}\"),\n",
" ]\n",
")\n",
"\n",
"chain = prompt | llm\n",
"chain.invoke(\n",
" {\n",
" \"input_language\": \"English\",\n",
" \"output_language\": \"German\",\n",
" \"input\": \"I love programming.\",\n",
" }\n",
")"
]
},
{
"cell_type": "markdown",
"id": "0b1b52a5-b58d-40c9-bcdd-88eb8fb351e2",
"metadata": {},
"source": [
"## Tool calling\n",
"\n",
"OpenAI has a [tool calling](https://platform.openai.com/docs/guides/function-calling) (we use \"tool calling\" and \"function calling\" interchangeably here) API that lets you describe tools and their arguments, and have the model return a JSON object with a tool to invoke and the inputs to that tool. tool-calling is extremely useful for building tool-using chains and agents, and for getting structured outputs from models more generally.\n",
"\n",
"### ChatOpenAI.bind_tools()\n",
"\n",
"With `ChatAnthropic.bind_tools`, we can easily pass in Pydantic classes, dict schemas, LangChain tools, or even functions as tools to the model. Under the hood these are converted to an Anthropic tool schemas, which looks like:\n",
"```\n",
"{\n",
" \"name\": \"...\",\n",
" \"description\": \"...\",\n",
" \"parameters\": {...} # JSONSchema\n",
"}\n",
"```\n",
"and passed in every model invocation."
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "b7ea7690-ec7a-4337-b392-e87d1f39a6ec",
"metadata": {},
"outputs": [],
"source": [
"template = (\n",
" \"You are a helpful assistant that translates {input_language} to {output_language}.\"\n",
@ -33,8 +33,9 @@ Our commentary on when you should consider using this agent type.
| Agent Type | Intended Model Type | Supports Chat History | Supports Multi-Input Tools | Supports Parallel Function Calling | Required Model Params | When to Use | API |
| [OpenAI Tools](./openai_tools) | Chat | ✅ | ✅ | ✅ | `tools` | If you are using a recent OpenAI model (`1106` onwards) | [Ref](https://api.python.langchain.com/en/latest/agents/langchain.agents.openai_tools.base.create_openai_tools_agent.html) |
| [OpenAI Functions](./openai_functions_agent)| Chat | ✅ | ✅ | | `functions` | If you are using an OpenAI model, or an open-source model that has been finetuned for function calling and exposes the same `functions` parameters as OpenAI | [Ref](https://api.python.langchain.com/en/latest/agents/langchain.agents.openai_functions_agent.base.create_openai_functions_agent.html) |
| [Tool Calling](/docs/modules/agents/agent_types/tool_calling) | Chat | ✅ | ✅ | ✅ | `tools` | If you are using a tool-calling model | TODO: Ref |
| [OpenAI Tools](./openai_tools) | Chat | ✅ | ✅ | ✅ | `tools` | [Legacy] If you are using a recent OpenAI model (`1106` onwards). Generic Tool Calling agent recommended instead. | [Ref](https://api.python.langchain.com/en/latest/agents/langchain.agents.openai_tools.base.create_openai_tools_agent.html) |
| [OpenAI Functions](./openai_functions_agent)| Chat | ✅ | ✅ | | `functions` | [Legacy] If you are using an OpenAI model, or an open-source model that has been finetuned for function calling and exposes the same `functions` parameters as OpenAI. Generic Tool Calling agent recommended instead | [Ref](https://api.python.langchain.com/en/latest/agents/langchain.agents.openai_functions_agent.base.create_openai_functions_agent.html) |
| [XML](./xml_agent) | LLM | ✅ | | | | If you are using Anthropic models, or other models good at XML | [Ref](https://api.python.langchain.com/en/latest/agents/langchain.agents.xml.base.create_xml_agent.html) |
| [Structured Chat](./structured_chat) | Chat | ✅ | ✅ | | | If you need to support tools with multiple inputs | [Ref](https://api.python.langchain.com/en/latest/agents/langchain.agents.structured_chat.base.create_structured_chat_agent.html) |
| [JSON Chat](./json_agent) | Chat | ✅ | | | | If you are using a model good at JSON | [Ref](https://api.python.langchain.com/en/latest/agents/langchain.agents.json_chat.base.create_json_chat_agent.html) |
"[Tool calling](/docs/modules/model_io/chat/function_calling) allows a model to detect when one or more tools should be called and respond with the inputs that should be passed to those tools. In an API call, you can describe tools and have the model intelligently choose to output a structured object like JSON containing arguments to call these tools. The goal of tools APIs is to more reliably return valid and useful tool calls than what can be done using a generic text completion or chat API.\n",
"\n",
"We can take advantage of this structured output, combined with the fact that you can bind multiple tools to a [tool calling chat model](/docs/integrations/chat/) and\n",
"allow the model to choose which one to call, to create an agent that repeatedly calls tools and receives results until a query is resolved.\n",
"\n",
"This is a more generalized version of the [OpenAI tools agent](/docs/modules/agents/agent_types/openai_tools/), which was designed for OpenAI's specific style of\n",
"tool calling. It uses LangChain's ToolCall interface to support a wider range of\n",
"provider implementations, such as [Anthropic](/docs/integrations/chat/anthropic/), [Google Gemini](/docs/integrations/chat/google_vertex_ai_palm/), and [Mistral](/docs/integrations/chat/mistralai/)\n",
"in addition to [OpenAI](/docs/integrations/chat/openai/).\n",
"\n",
"## Setup\n",
"\n",
"Any models that support tool calling can be used in this agent. [TODO ADD WHICH]\n",
"\n",
"This demo uses [Tavily](https://app.tavily.com), but you can also swap in any other [built-in tool](/docs/integrations/tools) or add [custom tools](/docs/modules/tools/custom_tools/).\n",
"You'll need to sign up for an API key and set it as `process.env.TAVILY_API_KEY`.\n",
"\n",
"```{=mdx}\n",
"import ChatModelTabs from \"@theme/ChatModelTabs\";\n",
"Now, let's initialize the executor that will run our agent and invoke it!"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/bagatur/langchain/libs/partners/anthropic/langchain_anthropic/chat_models.py:347: UserWarning: stream: Tool use is not yet supported in streaming mode.\n",
" warnings.warn(\"stream: Tool use is not yet supported in streaming mode.\")\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[32;1m\u001b[1;3m\n",
"Invoking: `tavily_search_results_json` with `{'query': 'LangChain'}`\n",
"\u001b[0m\u001b[36;1m\u001b[1;3m[{'url': 'https://github.com/langchain-ai/langchain', 'content': 'About\\n⚡ Building applications with LLMs through composability ⚡\\nResources\\nLicense\\nCode of conduct\\nSecurity policy\\nStars\\nWatchers\\nForks\\nReleases\\n291\\nPackages\\n0\\nUsed by 39k\\nContributors\\n1,848\\nLanguages\\nFooter\\nFooter navigation Latest commit\\nGit stats\\nFiles\\nREADME.md\\n🦜️🔗 LangChain\\n⚡ Building applications with LLMs through composability ⚡\\nLooking for the JS/TS library? ⚡ Building applications with LLMs through composability ⚡\\nLicense\\nlangchain-ai/langchain\\nName already in use\\nUse Git or checkout with SVN using the web URL.\\n 📖 Documentation\\nPlease see here for full documentation, which includes:\\n💁 Contributing\\nAs an open-source project in a rapidly developing field, we are extremely open to contributions, whether it be in the form of a new feature, improved infrastructure, or better documentation.\\n What can you build with LangChain?\\n❓ Retrieval augmented generation\\n💬 Analyzing structured data\\n🤖 Chatbots\\nAnd much more!'}]\u001b[0m"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/bagatur/langchain/libs/partners/anthropic/langchain_anthropic/chat_models.py:347: UserWarning: stream: Tool use is not yet supported in streaming mode.\n",
" warnings.warn(\"stream: Tool use is not yet supported in streaming mode.\")\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[32;1m\u001b[1;3mLangChain is an open-source Python library that helps developers build applications with large language models (LLMs) through composability. Some key features of LangChain include:\n",
"\n",
"- Retrieval augmented generation - Allowing LLMs to retrieve and utilize external data sources when generating outputs.\n",
"\n",
"- Analyzing structured data - Tools for working with structured data like databases, APIs, PDFs, etc. and allowing LLMs to reason over this data.\n",
"\n",
"- Building chatbots and agents - Frameworks for building conversational AI applications.\n",
"\n",
"- Composability - LangChain allows you to chain together different LLM capabilities and data sources in a modular and reusable way.\n",
"\n",
"The library aims to make it easier to build real-world applications that leverage the power of large language models in a scalable and robust way. It provides abstractions and primitives for working with LLMs from different providers like OpenAI, Anthropic, Cohere, etc. LangChain is open-source and has an active community contributing new features and improvements.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
},
{
"data": {
"text/plain": [
"{'input': 'what is LangChain?',\n",
" 'output': 'LangChain is an open-source Python library that helps developers build applications with large language models (LLMs) through composability. Some key features of LangChain include:\\n\\n- Retrieval augmented generation - Allowing LLMs to retrieve and utilize external data sources when generating outputs.\\n\\n- Analyzing structured data - Tools for working with structured data like databases, APIs, PDFs, etc. and allowing LLMs to reason over this data.\\n\\n- Building chatbots and agents - Frameworks for building conversational AI applications.\\n\\n- Composability - LangChain allows you to chain together different LLM capabilities and data sources in a modular and reusable way.\\n\\nThe library aims to make it easier to build real-world applications that leverage the power of large language models in a scalable and robust way. It provides abstractions and primitives for working with LLMs from different providers like OpenAI, Anthropic, Cohere, etc. LangChain is open-source and has an active community contributing new features and improvements.'}"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Create an agent executor by passing in the agent and tools\n",
"This type of agent can optionally take chat messages representing previous conversation turns. It can use that previous history to respond conversationally. For more details, see [this section of the agent quickstart](/docs/modules/agents/quick_start#adding-in-memory)."
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/bagatur/langchain/libs/partners/anthropic/langchain_anthropic/chat_models.py:347: UserWarning: stream: Tool use is not yet supported in streaming mode.\n",
" warnings.warn(\"stream: Tool use is not yet supported in streaming mode.\")\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[32;1m\u001b[1;3mBased on what you told me, your name is Bob. I don't need to use any tools to look that up since you directly provided your name.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
},
{
"data": {
"text/plain": [
"{'input': \"what's my name? Don't use tools to look this up unless you NEED to\",\n",
" 'chat_history': [HumanMessage(content='hi! my name is bob'),\n",
" AIMessage(content='Hello Bob! How can I assist you today?')],\n",
" 'output': \"Based on what you told me, your name is Bob. I don't need to use any tools to look that up since you directly provided your name.\"}"
" 'content': 'Thu 11 Jan Thu 11 Jan Seven day forecast for San Francisco San Francisco (United States of America) weather Find a forecast Sat 6 Jan Sat 6 Jan Sun 7 Jan Sun 7 Jan Mon 8 Jan Mon 8 Jan Tue 9 Jan Tue 9 Jan Wed 10 Jan Wed 10 Jan Thu 11 Jan Find a forecast Please choose your location from the nearest places to : Forecast days Today Today Sat 6 Jan Sat 6 JanSan Francisco 7 day weather forecast including weather warnings, temperature, rain, wind, visibility, humidity and UV ... (11 January 2024) Time 00:00 01:00 02:00 03:00 04:00 05:00 06:00 07:00 08:00 09:00 10:00 11:00 12:00 ... Oakland Int. 11.5 miles; San Francisco International 11.5 miles; Corte Madera 12.3 miles; Redwood City 23.4 miles;'},\n",
" 'content': \"May 18, 2023 Jan. 4, 2024 Subscribe for unlimited accessSite Map Follow Us MORE FROM THE L.A. TIMES Jan. 8, 2024 Travel & Experiences This must be Elysian Valley (a.k.a. Frogtown) Jan. 5, 2024 Food June 30, 2023The East Brother Light Station in the San Francisco Bay is not a destination for everyone. ... Jan. 11, 2024 3 AM PT ... Champagne and hors d'oeuvres are served in late afternoon — outdoors if ...\"}]"
" 'content': \"2024 NBA Mock Draft 6.0: Projections for every pick following March Madness With the NCAA tournament behind us, here's an updated look at Yahoo Sports' first- and second-round projections for the ...\"},\n",
" 'content': 'Extended weather forecast in San Francisco. Hourly Week 10 days 14 days 30 days Year. Detailed ⚡ San Francisco Weather Forecast for April 2024 - day/night 🌡️ temperatures, precipitations - World-Weather.info.'},\n",
" 'content': 'San Francisco Temperature History 2024\\nHourly Temperature in 2024 in San Francisco\\nCompare San Francisco to another city:\\nCloud Cover in 2024 in San Francisco\\nDaily Precipitation in 2024 in San Francisco\\nObserved Weather in 2024 in San Francisco\\nHours of Daylight and Twilight in 2024 in San Francisco\\nSunrise & Sunset with Twilight and Daylight Saving Time in 2024 in San Francisco\\nSolar Elevation and Azimuth in 2024 in San Francisco\\nMoon Rise, Set & Phases in 2024 in San Francisco\\nHumidity Comfort Levels in 2024 in San Francisco\\nWind Speed in 2024 in San Francisco\\nHourly Wind Speed in 2024 in San Francisco\\nHourly Wind Direction in 2024 in San Francisco\\nAtmospheric Pressure in 2024 in San Francisco\\nData Sources\\n See all nearby weather stations\\nLatest Report — 3:56 PM\\nWed, Jan 24, 2024\\xa0\\xa0\\xa0\\xa013 min ago\\xa0\\xa0\\xa0\\xa0UTC 23:56\\nCall Sign KSFO\\nTemp.\\n60.1°F\\nPrecipitation\\nNo Report\\nWind\\n6.9 mph\\nCloud Cover\\nMostly Cloudy\\n1,800 ft\\nRaw: KSFO 242356Z 18006G19KT 10SM FEW015 BKN018 BKN039 16/12 A3004 RMK AO2 SLP171 T01560122 10156 20122 55001\\n While having the tremendous advantages of temporal and spatial completeness, these reconstructions: (1) are based on computer models that may have model-based errors, (2) are coarsely sampled on a 50 km grid and are therefore unable to reconstruct the local variations of many microclimates, and (3) have particular difficulty with the weather in some coastal areas, especially small islands.\\n We further caution that our travel scores are only as good as the data that underpin them, that weather conditions at any given location and time are unpredictable and variable, and that the definition of the scores reflects a particular set of preferences that may not agree with those of any particular reader.\\n 2024 Weather History in San Francisco California, United States\\nThe data for this report comes from the San Francisco International Airport.'}]"
]
},
"execution_count": 24,
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
@ -140,7 +146,7 @@
{
"data": {
"text/plain": [
"Document(page_content=\"dataset uploading.Once we have a dataset, how can we use it to test changes to a prompt or chain? The most basic approach is to run the chain over the data points and visualize the outputs. Despite technological advancements, there still is no substitute for looking at outputs by eye. Currently, running the chain over the data points needs to be done client-side. The LangSmith client makes it easy to pull down a dataset and then run a chain over them, logging the results to a new project associated with the dataset. From there, you can review them. We've made it easy to assign feedback to runs and mark them as correct or incorrect directly in the web app, displaying aggregate statistics for each test project.We also make it easier to evaluate these runs. To that end, we've added a set of evaluators to the open-source LangChain library. These evaluators can be specified when initiating a test run and will evaluate the results once the test run completes. If we’re being honest, most of\", metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'})"
"Document(page_content='import Clientfrom langsmith.evaluation import evaluateclient = Client()# Define dataset: these are your test casesdataset_name = \"Sample Dataset\"dataset = client.create_dataset(dataset_name, description=\"A sample dataset in LangSmith.\")client.create_examples( inputs=[ {\"postfix\": \"to LangSmith\"}, {\"postfix\": \"to Evaluations in LangSmith\"}, ], outputs=[ {\"output\": \"Welcome to LangSmith\"}, {\"output\": \"Welcome to Evaluations in LangSmith\"}, ], dataset_id=dataset.id,)# Define your evaluatordef exact_match(run, example): return {\"score\": run.outputs[\"output\"] == example.outputs[\"output\"]}experiment_results = evaluate( lambda input: \"Welcome \" + input[\\'postfix\\'], # Your AI system goes here data=dataset_name, # The data to predict and grade over evaluators=[exact_match], # The evaluators to score the results experiment_prefix=\"sample-experiment\", # The name of the experiment metadata={ \"version\": \"1.0.0\", \"revision_id\":', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'Getting started with LangSmith | 🦜️🛠️ LangSmith', 'description': 'Introduction', 'language': 'en'})"
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3m\n",
"Invoking: `langsmith_search` with `{'query': 'LangSmith testing'}`\n",
"Invoking: `langsmith_search` with `{'query': 'how can LangSmith help with testing'}`\n",
"\n",
"\n",
"\u001b[0m\u001b[33;1m\u001b[1;3mGetting started with LangSmith | 🦜️🛠️ LangSmith\n",
"\n",
"Skip to main contentLangSmith API DocsSearchGo to AppQuick StartUser GuideTracingEvaluationProduction Monitoring & AutomationsPrompt HubProxyPricingSelf-HostingCookbookQuick StartOn this pageGetting started with LangSmithIntroductionLangSmith is a platform for building production-grade LLM applications. It allows you to closely monitor and evaluate your application, so you can ship quickly and with confidence. Use of LangChain is not necessary - LangSmith works on its own!Install LangSmithWe offer Python and Typescript SDKs for all your LangSmith needs.PythonTypeScriptpip install -U langsmithyarn add langchain langsmithCreate an API keyTo create an API key head to the setting pages. Then click Create API Key.Setup your environmentShellexport LANGCHAIN_TRACING_V2=trueexport LANGCHAIN_API_KEY=<your-api-key># The below examples use the OpenAI API, though it's not necessary in generalexport OPENAI_API_KEY=<your-openai-api-key>Log your first traceWe provide multiple ways to log traces\n",
"\n",
"Learn about the workflows LangSmith supports at each stage of the LLM application lifecycle.Pricing: Learn about the pricing model for LangSmith.Self-Hosting: Learn about self-hosting options for LangSmith.Proxy: Learn about the proxy capabilities of LangSmith.Tracing: Learn about the tracing capabilities of LangSmith.Evaluation: Learn about the evaluation capabilities of LangSmith.Prompt Hub Learn about the Prompt Hub, a prompt management tool built into LangSmith.Additional ResourcesLangSmith Cookbook: A collection of tutorials and end-to-end walkthroughs using LangSmith.LangChain Python: Docs for the Python LangChain library.LangChain Python API Reference: documentation to review the core APIs of LangChain.LangChain JS: Docs for the TypeScript LangChain libraryDiscord: Join us on our Discord to discuss all things LangChain!FAQHow do I migrate projects between organizations?Currently we do not support project migration betwen organizations. While you can manually imitate this by\n",
"\n",
"\u001b[0m\u001b[33;1m\u001b[1;3m[Document(page_content='LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content='Skip to main content🦜️🛠️ LangSmith DocsPython DocsJS/TS DocsSearchGo to AppLangSmithOverviewTracingTesting & EvaluationOrganizationsHubLangSmith CookbookOverviewOn this pageLangSmith Overview and User GuideBuilding reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.Over the past two months, we at LangChain have been building and using LangSmith with the goal of bridging this gap. This is our tactical user guide to outline effective ways to use LangSmith and maximize its benefits.On by default\\u200bAt LangChain, all of us have LangSmith’s tracing running in the background by default. On the Python side, this is achieved by setting environment variables, which we establish whenever we launch a virtual environment or open our bash shell and leave them set. The same principle applies to most JavaScript', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content='You can also quickly edit examples and add them to datasets to expand the surface area of your evaluation sets or to fine-tune a model for improved quality or reduced costs.Monitoring\\u200bAfter all this, your app might finally ready to go in production. LangSmith can also be used to monitor your application in much the same way that you used for debugging. You can log all traces, visualize latency and token usage statistics, and troubleshoot specific issues as they arise. Each run can also be assigned string tags or key-value metadata, allowing you to attach correlation ids or AB test variants, and filter runs accordingly.We’ve also made it possible to associate feedback programmatically with runs. This means that if your application has a thumbs up/down button on it, you can use that to log feedback back to LangSmith. This can be used to track performance over time and pinpoint under performing data points, which you can subsequently add to a dataset for future testing — mirroring the', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'}), Document(page_content='inputs, and see what happens. At some point though, our application is performing\\nwell and we want to be more rigorous about testing changes. We can use a dataset\\nthat we’ve constructed along the way (see above). Alternatively, we could spend some\\ntime constructing a small dataset by hand. For these situations, LangSmith simplifies', metadata={'source': 'https://docs.smith.langchain.com/overview', 'title': 'LangSmith Overview and User Guide | 🦜️🛠️ LangSmith', 'description': 'Building reliable LLM applications can be challenging. LangChain simplifies the initial setup, but there is still work needed to bring the performance of prompts, chains and agents up the level where they are reliable enough to be used in production.', 'language': 'en'})]\u001b[0m\u001b[32;1m\u001b[1;3mLangSmith can help with testing in several ways. Here are some ways LangSmith can assist with testing:\n",
"1. Tracing: LangSmith provides tracing capabilities that can be used to monitor and debug your application during testing. You can log all traces, visualize latency and token usage statistics, and troubleshoot specific issues as they arise.\n",
"1. **Tracing**: LangSmith provides tracing capabilities that allow you to closely monitor and evaluate your application during testing. You can log traces to track the behavior of your application and identify any issues.\n",
"\n",
"2. Evaluation: LangSmith allows you to quickly edit examples and add them to datasets to expand the surface area of your evaluation sets. This can help you test and fine-tune your models for improved quality or reduced costs.\n",
"2. **Evaluation**: LangSmith offers evaluation capabilities that enable you to assess the performance of your application during testing. This helps you ensure that your application functions as expected and meets the required standards.\n",
"\n",
"3. Monitoring: Once your application is ready for production, LangSmith can be used to monitor your application. You can log feedback programmatically with runs, track performance over time, and pinpoint underperforming data points. This information can be used to improve your application and add to datasets for future testing.\n",
"3. **Production Monitoring & Automations**: LangSmith allows you to monitor your application in production and automate certain processes, which can be beneficial for testing different scenarios and ensuring the stability of your application.\n",
"\n",
"4. Rigorous Testing: When your application is performing well and you want to be more rigorous about testing changes, LangSmith can simplify the process. You can use existing datasets or construct small datasets by hand to test different scenarios and evaluate the performance of your application.\n",
"4. **Prompt Hub**: LangSmith includes a Prompt Hub, a prompt management tool that can streamline the testing process by providing a centralized location for managing prompts and inputs for your application.\n",
"\n",
"For more detailed information on how to use LangSmith for testing, you can refer to the [LangSmith Overview and User Guide](https://docs.smith.langchain.com/overview).\u001b[0m\n",
"Overall, LangSmith can assist with testing by providing tools for monitoring, evaluating, and automating processes to ensure the reliability and performance of your application during testing phases.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
@ -389,7 +401,7 @@
"data": {
"text/plain": [
"{'input': 'how can langsmith help with testing?',\n",
" 'output': 'LangSmith can help with testing in several ways. Here are some ways LangSmith can assist with testing:\\n\\n1. Tracing: LangSmith provides tracing capabilities that can be used to monitor and debug your application during testing. You can log all traces, visualize latency and token usage statistics, and troubleshoot specific issues as they arise.\\n\\n2. Evaluation: LangSmith allows you to quickly edit examples and add them to datasets to expand the surface area of your evaluation sets. This can help you test and fine-tune your models for improved quality or reduced costs.\\n\\n3. Monitoring: Once your application is ready for production, LangSmith can be used to monitor your application. You can log feedback programmatically with runs, track performance over time, and pinpoint underperforming data points. This information can be used to improve your application and add to datasets for future testing.\\n\\n4. Rigorous Testing: When your application is performing well and you want to be more rigorous about testing changes, LangSmith can simplify the process. You can use existing datasets or construct small datasets by hand to test different scenarios and evaluate the performance of your application.\\n\\nFor more detailed information on how to use LangSmith for testing, you can refer to the [LangSmith Overview and User Guide](https://docs.smith.langchain.com/overview).'}"
" 'output': 'LangSmith is a platform for building production-grade LLM applications that can help with testing in the following ways:\\n\\n1. **Tracing**: LangSmith provides tracing capabilities that allow you to closely monitor and evaluate your application during testing. You can log traces to track the behavior of your application and identify any issues.\\n\\n2. **Evaluation**: LangSmith offers evaluation capabilities that enable you to assess the performance of your application during testing. This helps you ensure that your application functions as expected and meets the required standards.\\n\\n3. **Production Monitoring & Automations**: LangSmith allows you to monitor your application in production and automate certain processes, which can be beneficial for testing different scenarios and ensuring the stability of your application.\\n\\n4. **Prompt Hub**: LangSmith includes a Prompt Hub, a prompt management tool that can streamline the testing process by providing a centralized location for managing prompts and inputs for your application.\\n\\nOverall, LangSmith can assist with testing by providing tools for monitoring, evaluating, and automating processes to ensure the reliability and performance of your application during testing phases.'}"
]
},
"execution_count": 14,
@ -418,7 +430,7 @@
"Invoking: `tavily_search_results_json` with `{'query': 'weather in San Francisco'}`\n",
"\n",
"\n",
"\u001b[0m\u001b[36;1m\u001b[1;3m[{'url': 'https://www.whereandwhen.net/when/north-america/california/san-francisco-ca/january/', 'content': 'Best time to go to San Francisco? Weather in San Francisco in january 2024 How was the weather last january? Here is the day by day recorded weather in San Francisco in january 2023: Seasonal average climate and temperature of San Francisco in january 8% 46% 29% 12% 8% Evolution of daily average temperature and precipitation in San Francisco in januaryWeather in San Francisco in january 2024. The weather in San Francisco in january comes from statistical datas on the past years. You can view the weather statistics the entire month, but also by using the tabs for the beginning, the middle and the end of the month. ... 11-01-2023 50°F to 54°F. 12-01-2023 50°F to 59°F. 13-01-2023 54°F to ...'}, {'url': 'https://www.latimes.com/travel/story/2024-01-11/east-brother-light-station-lighthouse-california', 'content': \"May 18, 2023 Jan. 4, 2024 Subscribe for unlimited accessSite Map Follow Us MORE FROM THE L.A. TIMES Jan. 8, 2024 Travel & Experiences This must be Elysian Valley (a.k.a. Frogtown) Jan. 5, 2024 Food June 30, 2023The East Brother Light Station in the San Francisco Bay is not a destination for everyone. ... Jan. 11, 2024 3 AM PT ... Champagne and hors d'oeuvres are served in late afternoon — outdoors if ...\"}]\u001b[0m\u001b[32;1m\u001b[1;3mI'm sorry, I couldn't find the current weather in San Francisco. However, you can check the weather in San Francisco by visiting a reliable weather website or using a weather app on your phone.\u001b[0m\n",
"\u001b[0m\u001b[36;1m\u001b[1;3m[{'url': 'https://www.weatherapi.com/', 'content': \"{'location': {'name': 'San Francisco', 'region': 'California', 'country': 'United States of America', 'lat': 37.78, 'lon': -122.42, 'tz_id': 'America/Los_Angeles', 'localtime_epoch': 1712847697, 'localtime': '2024-04-11 8:01'}, 'current': {'last_updated_epoch': 1712847600, 'last_updated': '2024-04-11 08:00', 'temp_c': 11.1, 'temp_f': 52.0, 'is_day': 1, 'condition': {'text': 'Partly cloudy', 'icon': '//cdn.weatherapi.com/weather/64x64/day/116.png', 'code': 1003}, 'wind_mph': 2.2, 'wind_kph': 3.6, 'wind_degree': 10, 'wind_dir': 'N', 'pressure_mb': 1015.0, 'pressure_in': 29.98, 'precip_mm': 0.0, 'precip_in': 0.0, 'humidity': 97, 'cloud': 25, 'feelslike_c': 11.5, 'feelslike_f': 52.6, 'vis_km': 14.0, 'vis_miles': 8.0, 'uv': 4.0, 'gust_mph': 2.8, 'gust_kph': 4.4}}\"}, {'url': 'https://www.yahoo.com/news/april-11-2024-san-francisco-122026435.html', 'content': \"2024 NBA Mock Draft 6.0: Projections for every pick following March Madness With the NCAA tournament behind us, here's an updated look at Yahoo Sports' first- and second-round projections for the ...\"}, {'url': 'https://www.weathertab.com/en/c/e/04/united-states/california/san-francisco/', 'content': 'Explore comprehensive April 2024 weather forecasts for San Francisco, including daily high and low temperatures, precipitation risks, and monthly temperature trends. Featuring detailed day-by-day forecasts, dynamic graphs of daily rain probabilities, and temperature trends to help you plan ahead. ... 11 65°F 49°F 18°C 9°C 29% 12 64°F 49°F ...'}, {'url': 'https://weatherspark.com/h/y/557/2024/Historical-Weather-during-2024-in-San-Francisco-California-United-States', 'content': 'San Francisco Temperature History 2024\\nHourly Temperature in 2024 in San Francisco\\nCompare San Francisco to another city:\\nCloud Cover in 2024 in San Francisco\\nDaily Precipitation in 2024 in San Francisco\\nObserved Weather in 2024 in San Francisco\\nHours of Daylight and Twilight in 2024 in San Francisco\\nSunrise & Sunset with Twilight and Daylight Saving Time in 2024 in San Francisco\\nSolar Elevation and Azimuth in 2024 in San Francisco\\nMoon Rise, Set & Phases in 2024 in San Francisco\\nHumidity Comfort Levels in 2024 in San Francisco\\nWind Speed in 2024 in San Francisco\\nHourly Wind Speed in 2024 in San Francisco\\nHourly Wind Direction in 2024 in San Francisco\\nAtmospheric Pressure in 2024 in San Francisco\\nData Sources\\n See all nearby weather stations\\nLatest Report — 3:56 PM\\nWed, Jan 24, 2024\\xa0\\xa0\\xa0\\xa013 min ago\\xa0\\xa0\\xa0\\xa0UTC 23:56\\nCall Sign KSFO\\nTemp.\\n60.1°F\\nPrecipitation\\nNo Report\\nWind\\n6.9 mph\\nCloud Cover\\nMostly Cloudy\\n1,800 ft\\nRaw: KSFO 242356Z 18006G19KT 10SM FEW015 BKN018 BKN039 16/12 A3004 RMK AO2 SLP171 T01560122 10156 20122 55001\\n While having the tremendous advantages of temporal and spatial completeness, these reconstructions: (1) are based on computer models that may have model-based errors, (2) are coarsely sampled on a 50 km grid and are therefore unable to reconstruct the local variations of many microclimates, and (3) have particular difficulty with the weather in some coastal areas, especially small islands.\\n We further caution that our travel scores are only as good as the data that underpin them, that weather conditions at any given location and time are unpredictable and variable, and that the definition of the scores reflects a particular set of preferences that may not agree with those of any particular reader.\\n 2024 Weather History in San Francisco California, United States\\nThe data for this report comes from the San Francisco International Airport.'}, {'url': 'https://www.msn.com/en-us/weather/topstories/april-11-2024-san-francisco-bay-area-weather-forecast/vi-BB1lrXDb', 'content': 'April 11, 2024 San Francisco Bay Area weather forecast. Posted: April 11, 2024 | Last updated: April 11, 2024 ...'}]\u001b[0m\u001b[32;1m\u001b[1;3mThe current weather in San Francisco is partly cloudy with a temperature of 52.0°F (11.1°C). The wind speed is 3.6 kph coming from the north, and the humidity is at 97%.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
@ -427,7 +439,7 @@
"data": {
"text/plain": [
"{'input': 'whats the weather in sf?',\n",
" 'output': \"I'm sorry, I couldn't find the current weather in San Francisco. However, you can check the weather in San Francisco by visiting a reliable weather website or using a weather app on your phone.\"}"
" 'output': 'The current weather in San Francisco is partly cloudy with a temperature of 52.0°F (11.1°C). The wind speed is 3.6 kph coming from the north, and the humidity is at 97%.'}"
]
},
"execution_count": 15,
@ -508,7 +520,7 @@
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mYour name is Bob.\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mYour name is Bob. How can I assist you, Bob?\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
@ -519,7 +531,7 @@
"{'chat_history': [HumanMessage(content='hi! my name is bob'),\n",
" AIMessage(content='Hello Bob! How can I assist you today?')],\n",
" 'input': \"what's my name?\",\n",
" 'output': 'Your name is Bob.'}"
" 'output': 'Your name is Bob. How can I assist you, Bob?'}"
]
},
"execution_count": 18,
@ -549,7 +561,7 @@
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": 20,
"id": "8edd96e6",
"metadata": {},
"outputs": [],
@ -560,7 +572,7 @@
},
{
"cell_type": "code",
"execution_count": 20,
"execution_count": 21,
"id": "6e76552a",
"metadata": {},
"outputs": [],
@ -570,7 +582,7 @@
},
{
"cell_type": "code",
"execution_count": 21,
"execution_count": 22,
"id": "828d1e95",
"metadata": {},
"outputs": [],
@ -587,7 +599,7 @@
},
{
"cell_type": "code",
"execution_count": 22,
"execution_count": 23,
"id": "1f5932b6",
"metadata": {},
"outputs": [
@ -611,7 +623,7 @@
" 'output': 'Hello Bob! How can I assist you today?'}"
]
},
"execution_count": 22,
"execution_count": 23,
"metadata": {},
"output_type": "execute_result"
}
@ -627,7 +639,7 @@
},
{
"cell_type": "code",
"execution_count": 23,
"execution_count": 24,
"id": "ae627966",
"metadata": {},
"outputs": [
@ -638,7 +650,7 @@
"\n",
"\n",
"\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mYour name is Bob.\u001b[0m\n",
"\u001b[32;1m\u001b[1;3mYour name is Bob! How can I help you, Bob?\u001b[0m\n",
[Mistral](https://mistral.ai/), [OpenAI](https://openai.com/), and others,
support variants of a tool calling feature. These features typically allow requests
to the LLM to include available tools and their schemas, and for responses to include
calls to these tools. For instance, given a search engine tool, an LLM might handle a
query by first issuing a call to the search engine. The system calling the LLM can
receive the tool call, execute it, and return the output to the LLM to inform its
response. LangChain includes a suite of [built-in tools](/docs/integrations/tools/)
and supports several methods for defining your own [custom tools](/docs/modules/tools/custom_tools).
Tool-calling is extremely useful for building [tool-using chains and agents](/docs/use_cases/tool_use),
and for getting structured outputs from models more generally.
Providers adopt different conventions for formatting tool schemas and tool calls.
For instance, Anthropic returns tool calls as parsed structures within a larger content block:
```
[
{
"text": "<thinking>\nI should use a tool.\n</thinking>",
"type": "text"
},
{
"id": "id_value",
"input": {"arg_name": "arg_value"},
"name": "tool_name",
"type": "tool_use"
}
]
```
whereas OpenAI separates tool calls into a distinct parameter, with arguments as JSON strings:
```
{
"tool_calls": [
{
"id": "id_value",
"function": {
"arguments": '{"arg_name": "arg_value"}',
"name": "tool_name"
},
"type": "function"
}
]
}
```
LangChain implements standard interfaces for defining tools, passing them to LLMs,
and representing tool calls.
## Passing tools to LLMs
Chat models supporting tool calling features implement a `.bind_tools` method, which
receives a list of LangChain [tool objects](https://api.python.langchain.com/en/latest/tools/langchain_core.tools.BaseTool.html#langchain_core.tools.BaseTool)
and binds them to the chat model in its expected format. Subsequent invocations of the
chat model will include tool schemas in its calls to the LLM.
For example, we can define the schema for custom tools using the `@tool` decorator
on Python functions:
```python
import getpass
import os
```
from langchain.tools import tool
## Binding functions
@tool
def add(a: int, b: int) -> int:
"""Adds a and b."""
return a + b
A number of models implement helper methods that will take care of
formatting and binding different function-like objects to the model.
Let’s take a look at how we might take the following Pydantic function
schema and get different models to invoke it:
@tool
def multiply(a: int, b: int) -> int:
"""Multiplies a and b."""
return a * b
tools = [add, multiply]
```
Or below, we define the schema using Pydantic:
```python
from langchain_core.pydantic_v1 import BaseModel, Field
# Note that the docstrings here are crucial, as they will be passed along
# to the model along with the class name.
class Add(BaseModel):
"""Add two integers together."""
a: int = Field(..., description="First integer")
b: int = Field(..., description="Second integer")
class Multiply(BaseModel):
"""Multiply two integers together."""
a: int = Field(..., description="First integer")
b: int = Field(..., description="Second integer")
tools = [Add, Multiply]
```
We can bind them to chat models as follows:
import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
@ -72,230 +137,180 @@ import ChatModelTabs from "@theme/ChatModelTabs";
or [message chunk](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.ai.AIMessageChunk.html#langchain_core.messages.ai.AIMessageChunk)
as a list of [tool call](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.ToolCall.html#langchain_core.messages.tool.ToolCall)
objects in the `.tool_calls` attribute. A `ToolCall` is a typed dict that includes a
tool name, dict of argument values, and (optionally) an identifier. Messages with no
tool calls default to an empty list for this attribute.
We can add a tool parser to extract the tool calls from the generated
message to JSON:
Example:
```python
from langchain_core.output_parsers.openai_tools import JsonOutputToolsParser
The `.tool_calls` attribute should contain valid tool calls. Note that on occasion,
model providers may output malformed tool calls (e.g., arguments that are not
valid JSON). When parsing fails in these cases, instances
of [InvalidToolCall](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.InvalidToolCall.html#langchain_core.messages.tool.InvalidToolCall)
are populated in the `.invalid_tool_calls` attribute. An `InvalidToolCall` can have
a name, string arguments, identifier, and error message.
If desired, [output parsers](/docs/modules/model_io/output_parsers) can further
process the output. For example, we can convert back to the original Pydantic class:
```python
from langchain_core.output_parsers.openai_tools import PydanticToolsParser
will be populated with [tool call chunk](https://api.python.langchain.com/en/latest/messages/langchain_core.messages.tool.ToolCallChunk.html#langchain_core.messages.tool.ToolCallChunk)
objects in a list via the `.tool_call_chunks` attribute. A `ToolCallChunk` includes
optional string fields for the tool `name`, `args`, and `id`, and includes an optional
integer field `index` that can be used to join chunks together. Fields are optional
because portions of a tool call may be streamed across different chunks (e.g., a chunk
that includes a substring of the arguments may have null values for the tool name and id).
Note that adding message chunks will merge their corresponding tool call chunks. This is the principle by which LangChain's various [tool output parsers](/docs/modules/model_io/output_parsers/types/openai_tools/) support streaming.
## Defining functions schemas
In case you need to access function schemas directly, LangChain has a built-in converter that can turn
Python functions, Pydantic classes, and LangChain Tools into the OpenAI format JSON schema:
### Python function
For example, below we accumulate tool call chunks:
```python
import json
from langchain_core.utils.function_calling import convert_to_openai_tool
Eugene Yurtsev
2 months ago
committed by
GitHub