"[](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/extras/guides/langsmith/walkthrough.ipynb)\n",
"LangChain makes it easy to prototype LLM applications and Agents. However, delivering LLM applications to production can be deceptively difficult. You will likely have to heavily customize and iterate on your prompts, chains, and other components to create a high-quality product.\n",
"\n",
"To aid in this process, we've launched LangSmith, a unified platform for debugging, testing, and monitoring your LLM applications.\n",
"\n",
"When might this come in handy? You may find it useful when you want to:\n",
"\n",
"- Quickly debug a new chain, agent, or set of tools\n",
"- Visualize how components (chains, llms, retrievers, etc.) relate and are used\n",
"- Evaluate different prompts and LLMs for a single component\n",
"- Run a given chain several times over a dataset to ensure it consistently meets a quality bar\n",
"- Capture usage traces and using LLMs or analytics pipelines to generate insights"
]
},
{
"cell_type": "markdown",
"id": "138fbb8f-960d-4d26-9dd5-6d6acab3ee55",
"metadata": {},
"source": [
"## Prerequisites\n",
"\n",
"**[Create a LangSmith account](https://smith.langchain.com/) and create an API key (see bottom left corner). Familiarize yourself with the platform by looking through the [docs](https://docs.smith.langchain.com/)**\n",
"\n",
"Note LangSmith is in closed beta; we're in the process of rolling it out to more users. However, you can fill out the form on the website for expedited access.\n",
"\n",
"Now, let's get started!"
]
},
{
"cell_type": "markdown",
"id": "2d77d064-41b4-41fb-82e6-2d16461269ec",
"metadata": {
"tags": []
},
"source": [
"## Log runs to LangSmith\n",
"\n",
"First, configure your environment variables to tell LangChain to log traces. This is done by setting the `LANGCHAIN_TRACING_V2` environment variable to true.\n",
"You can tell LangChain which project to log to by setting the `LANGCHAIN_PROJECT` environment variable (if this isn't set, runs will be logged to the `default` project). This will automatically create the project for you if it doesn't exist. You must also set the `LANGCHAIN_ENDPOINT` and `LANGCHAIN_API_KEY` environment variables.\n",
"\n",
"For more information on other ways to set up tracing, please reference the [LangSmith documentation](https://docs.smith.langchain.com/docs/).\n",
"\n",
"**NOTE:** You must also set your `OPENAI_API_KEY` environment variables in order to run the following tutorial.\n",
"\n",
"**NOTE:** You can only access an API key when you first create it. Keep it somewhere safe.\n",
"\n",
"**NOTE:** You can also use a context manager in python to log traces using\n",
"Create the langsmith client to interact with the API"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "510b5ca0",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"from langsmith import Client\n",
"\n",
"client = Client()"
]
},
{
"cell_type": "markdown",
"id": "ca27fa11-ddce-4af0-971e-c5c37d5b92ef",
"metadata": {},
"source": [
"Create a LangChain component and log runs to the platform. In this example, we will create a ReAct-style agent with access to a general search tool (DuckDuckGo). The agent's prompt can be viewed in the [Hub here](https://smith.langchain.com/hub/wfh/langsmith-agent-prompt)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "a0fbfbba-3c82-4298-a312-9cec016d9d2e",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/wfh/code/lc/langchain/libs/langchain/langchain/__init__.py:24: UserWarning: Importing hub from langchain root module is no longer supported.\n",
"We are running the agent concurrently on multiple inputs to reduce latency. Runs get logged to LangSmith in the background so execution latency is unaffected."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "19537902-b95c-4390-80a4-f6c9a937081e",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"inputs = [\n",
" \"What is LangChain?\",\n",
" \"What's LangSmith?\",\n",
" \"When was Llama-v2 released?\",\n",
" \"Who trained Llama-v2?\",\n",
" \"What is the langsmith cookbook?\",\n",
" \"When did langchain first announce the hub?\",\n",
"]\n",
"\n",
"results = agent_executor.batch([{\"input\": x} for x in inputs], return_exceptions=True)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "9a6a764c-5d7a-4de7-a916-3ecc987d5bb6",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'input': 'What is LangChain?',\n",
" 'output': 'I\\'m sorry, but I couldn\\'t find any information about \"LangChain\". Could you please provide more context or clarify your question?'},\n",
" {'input': \"What's LangSmith?\",\n",
" 'output': 'I\\'m sorry, but I couldn\\'t find any information about \"LangSmith\". It could be a company, a product, or a person. Can you provide more context or details about what you are referring to?'}]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"results[:2]"
]
},
{
"cell_type": "markdown",
"id": "9decb964-be07-4b6c-9802-9825c8be7b64",
"metadata": {},
"source": [
"Assuming you've successfully set up your environment, your agent traces should show up in the `Projects` section in the [app](https://smith.langchain.com/). Congrats!\n",
"\n",
"\n",
"\n",
"It looks like the agent isn't effectively using the tools though. Let's evaluate this so we have a baseline."
]
},
{
"cell_type": "markdown",
"id": "6c43c311-4e09-4d57-9ef3-13afb96ff430",
"metadata": {},
"source": [
"## Evaluate Agent\n",
"\n",
"In addition to logging runs, LangSmith also allows you to test and evaluate your LLM applications.\n",
"\n",
"In this section, you will leverage LangSmith to create a benchmark dataset and run AI-assisted evaluators on an agent. You will do so in a few steps:\n",
"\n",
"1. Create a dataset\n",
"2. Initialize a new agent to benchmark\n",
"3. Configure evaluators to grade an agent's output\n",
"4. Run the agent over the dataset and evaluate the results"
]
},
{
"cell_type": "markdown",
"id": "beab1a29-b79d-4a99-b5b1-0870c2d772b1",
"metadata": {},
"source": [
"### 1. Create a LangSmith dataset\n",
"\n",
"Below, we use the LangSmith client to create a dataset from the input questions from above and a list labels. You will use these later to measure performance for a new agent. A dataset is a collection of examples, which are nothing more than input-output pairs you can use as test cases to your application.\n",
"\n",
"For more information on datasets, including how to create them from CSVs or other files or how to create them in the platform, please refer to the [LangSmith documentation](https://docs.smith.langchain.com/)."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "43fd40b2-3f02-4e51-9343-705aafe90a36",
"metadata": {},
"outputs": [],
"source": [
"outputs = [\n",
" \"LangChain is an open-source framework for building applications using large language models. It is also the name of the company building LangSmith.\",\n",
" \"LangSmith is a unified platform for debugging, testing, and monitoring language model applications and agents powered by LangChain\",\n",
" \"July 18, 2023\",\n",
" \"The langsmith cookbook is a github repository containing detailed examples of how to use LangSmith to debug, evaluate, and monitor large language model-powered applications.\",\n",
" \"September 5, 2023\",\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "17580c4b-bd04-4dde-9d21-9d4edd25b00d",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"dataset_name = f\"agent-qa-{unique_id}\"\n",
"\n",
"dataset = client.create_dataset(\n",
" dataset_name, description=\"An example dataset of questions over the LangSmith documentation.\"\n",
"LangSmith lets you evaluate any LLM, chain, agent, or even a custom function. Conversational agents are stateful (they have memory); to ensure that this state isn't shared between dataset runs, we will pass in a `chain_factory` (aka a `constructor`) function to initialize for each call.\n",
"\n",
"In this case, we will test an agent that uses OpenAI's function calling endpoints."
" # Both the Criteria and LabeledCriteria evaluators can be configured with a dictionary of custom criteria.\n",
" RunEvalConfig.Criteria(\n",
" {\n",
" \"fifth-grader-score\": \"Do you have to be smarter than a fifth grader to answer this question? Y if so.\"\n",
" }\n",
" ),\n",
" ],\n",
" # You can add custom StringEvaluator or RunEvaluator objects here as well, which will automatically be\n",
" # applied to each prediction. Check out the docs for examples.\n",
" custom_evaluators=[],\n",
")"
]
},
{
"cell_type": "markdown",
"id": "07885b10",
"metadata": {
"tags": []
},
"source": [
"### 4. Run the agent and evaluators\n",
"\n",
"Use the [run_on_dataset](https://api.python.langchain.com/en/latest/smith/langchain.smith.evaluation.runner_utils.run_on_dataset.html#langchain.smith.evaluation.runner_utils.run_on_dataset) (or asynchronous [arun_on_dataset](https://api.python.langchain.com/en/latest/smith/langchain.smith.evaluation.runner_utils.arun_on_dataset.html#langchain.smith.evaluation.runner_utils.arun_on_dataset)) function to evaluate your model. This will:\n",
"1. Fetch example rows from the specified dataset.\n",
"2. Run your agent (or any custom function) on each example.\n",
"3. Apply evalutors to the resulting run traces and corresponding reference examples to generate automated feedback.\n",
"\n",
"The results will be visible in the LangSmith app."
" tags=[\"testing-notebook\", \"prompt:5d466cbc\"], # Optional, adds a tag to the resulting chain runs\n",
")\n",
"\n",
"# Sometimes, the agent will error due to parsing issues, incompatible tool inputs, etc.\n",
"# These are logged as warnings here and captured as errors in the tracing UI."
]
},
{
"cell_type": "markdown",
"id": "cdacd159-eb4d-49e9-bb2a-c55322c40ed4",
"metadata": {
"tags": []
},
"source": [
"### Review the test results\n",
"\n",
"You can review the test results tracing UI below by clicking the URL in the output above or navigating to the \"Testing & Datasets\" page in LangSmith **\"agent-qa-{unique_id}\"** dataset. \n",
"\n",
"\n",
"\n",
"This will show the new runs and the feedback logged from the selected evaluators. You can also explore a summary of the results in tabular format below."
"244cff41-198e-4441-8248-d095bebbfd16 {'output': 'LangSmith is a unified platform fo... \n",
"2272dbe6-0b6f-476b-b9d4-48c38f481a12 {'output': 'LangChain is an open-source framew... "
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"chain_results.to_dataframe()"
]
},
{
"cell_type": "markdown",
"id": "13aad317-73ff-46a7-a5a0-60b5b5295f02",
"metadata": {},
"source": [
"### (Optional) Compare to another prompt\n",
"\n",
"Now that we have our test run results, we can make changes to our agent and benchmark them. Let's try this again with a different prompt and see the results."
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "5eeb023f-ded2-4d0f-b910-2a57d9675853",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"View the evaluation results for project 'runnable-agent-test-39f3bbd0-1d7fb5e9' at:\n",
" tags=[\"testing-notebook\", \"prompt:39f3bbd0\"], # Optional, adds a tag to the resulting chain runs\n",
")"
]
},
{
"cell_type": "markdown",
"id": "591c819e-9932-45cf-adab-63727dd49559",
"metadata": {},
"source": [
"## Exporting datasets and runs\n",
"\n",
"LangSmith lets you export data to common formats such as CSV or JSONL directly in the web app. You can also use the client to fetch runs for further analysis, to store in your own database, or to share with others. Let's fetch the run traces from the evaluation run.\n",
"\n",
"**Note: It may be a few moments before all the runs are accessible.**"
"Congratulations! You have succesfully traced and evaluated an agent using LangSmith!\n",
"\n",
"This was a quick guide to get started, but there are many more ways to use LangSmith to speed up your developer flow and produce better results.\n",
"\n",
"For more information on how you can get the most out of LangSmith, check out [LangSmith documentation](https://docs.smith.langchain.com/), and please reach out with questions, feature requests, or feedback at [support@langchain.dev](mailto:support@langchain.dev)."
