forked from Archives/langchain
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
765 lines
28 KiB
Plaintext
765 lines
28 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "084ee2f0",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Output Parsers\n",
|
|
"\n",
|
|
"Language models output text. But many times you may want to get more structured information than just text back. This is where output parsers come in.\n",
|
|
"\n",
|
|
"Output parsers are classes that help structure language model responses. There are two main methods an output parser must implement:\n",
|
|
"\n",
|
|
"- `get_format_instructions() -> str`: A method which returns a string containing instructions for how the output of a language model should be formatted.\n",
|
|
"- `parse(str) -> Any`: A method which takes in a string (assumed to be the response from a language model) and parses it into some structure.\n",
|
|
"\n",
|
|
"And then one optional one:\n",
|
|
"\n",
|
|
"- `parse_with_prompt(str) -> Any`: A method which takes in a string (assumed to be the response from a language model) and a prompt (assumed to the prompt that generated such a response) and parses it into some structure. The prompt is largely provided in the event the OutputParser wants to retry or fix the output in some way, and needs information from the prompt to do so.\n",
|
|
"\n",
|
|
"\n",
|
|
"Below we go over some examples of output parsers."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "5f0c8a33",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.prompts import PromptTemplate, ChatPromptTemplate, HumanMessagePromptTemplate\n",
|
|
"from langchain.llms import OpenAI\n",
|
|
"from langchain.chat_models import ChatOpenAI"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a1ae632a",
|
|
"metadata": {},
|
|
"source": [
|
|
"## PydanticOutputParser\n",
|
|
"This output parser allows users to specify an arbitrary JSON schema and query LLMs for JSON outputs that conform to that schema.\n",
|
|
"\n",
|
|
"Keep in mind that large language models are leaky abstractions! You'll have to use an LLM with sufficient capacity to generate well-formed JSON. In the OpenAI family, DaVinci can do reliably but Curie's ability already drops off dramatically. \n",
|
|
"\n",
|
|
"Use Pydantic to declare your data model. Pydantic's BaseModel like a Python dataclass, but with actual type checking + coercion."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "cba6d8e3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.output_parsers import PydanticOutputParser\n",
|
|
"from pydantic import BaseModel, Field, validator\n",
|
|
"from typing import List"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "0a203100",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"model_name = 'text-davinci-003'\n",
|
|
"temperature = 0.0\n",
|
|
"model = OpenAI(model_name=model_name, temperature=temperature)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "b3f16168",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Joke(setup='Why did the chicken cross the road?', punchline='To get to the other side!')"
|
|
]
|
|
},
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Define your desired data structure.\n",
|
|
"class Joke(BaseModel):\n",
|
|
" setup: str = Field(description=\"question to set up a joke\")\n",
|
|
" punchline: str = Field(description=\"answer to resolve the joke\")\n",
|
|
" \n",
|
|
" # You can add custom validation logic easily with Pydantic.\n",
|
|
" @validator('setup')\n",
|
|
" def question_ends_with_question_mark(cls, field):\n",
|
|
" if field[-1] != '?':\n",
|
|
" raise ValueError(\"Badly formed question!\")\n",
|
|
" return field\n",
|
|
"\n",
|
|
"# And a query intented to prompt a language model to populate the data structure.\n",
|
|
"joke_query = \"Tell me a joke.\"\n",
|
|
"\n",
|
|
"# Set up a parser + inject instructions into the prompt template.\n",
|
|
"parser = PydanticOutputParser(pydantic_object=Joke)\n",
|
|
"\n",
|
|
"prompt = PromptTemplate(\n",
|
|
" template=\"Answer the user query.\\n{format_instructions}\\n{query}\\n\",\n",
|
|
" input_variables=[\"query\"],\n",
|
|
" partial_variables={\"format_instructions\": parser.get_format_instructions()}\n",
|
|
")\n",
|
|
"\n",
|
|
"_input = prompt.format_prompt(query=joke_query)\n",
|
|
"\n",
|
|
"output = model(_input.to_string())\n",
|
|
"\n",
|
|
"parser.parse(output)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "03049f88",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Actor(name='Tom Hanks', film_names=['Forrest Gump', 'Saving Private Ryan', 'The Green Mile', 'Cast Away', 'Toy Story'])"
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# Here's another example, but with a compound typed field.\n",
|
|
"class Actor(BaseModel):\n",
|
|
" name: str = Field(description=\"name of an actor\")\n",
|
|
" film_names: List[str] = Field(description=\"list of names of films they starred in\")\n",
|
|
" \n",
|
|
"actor_query = \"Generate the filmography for a random actor.\"\n",
|
|
"\n",
|
|
"parser = PydanticOutputParser(pydantic_object=Actor)\n",
|
|
"\n",
|
|
"prompt = PromptTemplate(\n",
|
|
" template=\"Answer the user query.\\n{format_instructions}\\n{query}\\n\",\n",
|
|
" input_variables=[\"query\"],\n",
|
|
" partial_variables={\"format_instructions\": parser.get_format_instructions()}\n",
|
|
")\n",
|
|
"\n",
|
|
"_input = prompt.format_prompt(query=actor_query)\n",
|
|
"\n",
|
|
"output = model(_input.to_string())\n",
|
|
"\n",
|
|
"parser.parse(output)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4d6c0c86",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Fixing Output Parsing Mistakes\n",
|
|
"\n",
|
|
"The above guardrail simply tries to parse the LLM response. If it does not parse correctly, then it errors.\n",
|
|
"\n",
|
|
"But we can do other things besides throw errors. Specifically, we can pass the misformatted output, along with the formatted instructions, to the model and ask it to fix it.\n",
|
|
"\n",
|
|
"For this example, we'll use the above OutputParser. Here's what happens if we pass it a result that does not comply with the schema:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "73beb20d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"misformatted = \"{'name': 'Tom Hanks', 'film_names': ['Forrest Gump']}\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "f0e5ba80",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"ename": "OutputParserException",
|
|
"evalue": "Failed to parse Actor from completion {'name': 'Tom Hanks', 'film_names': ['Forrest Gump']}. Got: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mJSONDecodeError\u001b[0m Traceback (most recent call last)",
|
|
"File \u001b[0;32m~/workplace/langchain/langchain/output_parsers/pydantic.py:23\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m 22\u001b[0m json_str \u001b[38;5;241m=\u001b[39m match\u001b[38;5;241m.\u001b[39mgroup()\n\u001b[0;32m---> 23\u001b[0m json_object \u001b[38;5;241m=\u001b[39m \u001b[43mjson\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mloads\u001b[49m\u001b[43m(\u001b[49m\u001b[43mjson_str\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 24\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpydantic_object\u001b[38;5;241m.\u001b[39mparse_obj(json_object)\n",
|
|
"File \u001b[0;32m~/.pyenv/versions/3.9.1/lib/python3.9/json/__init__.py:346\u001b[0m, in \u001b[0;36mloads\u001b[0;34m(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)\u001b[0m\n\u001b[1;32m 343\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m (\u001b[38;5;28mcls\u001b[39m \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m object_hook \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m\n\u001b[1;32m 344\u001b[0m parse_int \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m parse_float \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m\n\u001b[1;32m 345\u001b[0m parse_constant \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m object_pairs_hook \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m \u001b[38;5;129;01mand\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m kw):\n\u001b[0;32m--> 346\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43m_default_decoder\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdecode\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 347\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mcls\u001b[39m \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n",
|
|
"File \u001b[0;32m~/.pyenv/versions/3.9.1/lib/python3.9/json/decoder.py:337\u001b[0m, in \u001b[0;36mJSONDecoder.decode\u001b[0;34m(self, s, _w)\u001b[0m\n\u001b[1;32m 333\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Return the Python representation of ``s`` (a ``str`` instance\u001b[39;00m\n\u001b[1;32m 334\u001b[0m \u001b[38;5;124;03mcontaining a JSON document).\u001b[39;00m\n\u001b[1;32m 335\u001b[0m \n\u001b[1;32m 336\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m--> 337\u001b[0m obj, end \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mraw_decode\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43midx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43m_w\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mend\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 338\u001b[0m end \u001b[38;5;241m=\u001b[39m _w(s, end)\u001b[38;5;241m.\u001b[39mend()\n",
|
|
"File \u001b[0;32m~/.pyenv/versions/3.9.1/lib/python3.9/json/decoder.py:353\u001b[0m, in \u001b[0;36mJSONDecoder.raw_decode\u001b[0;34m(self, s, idx)\u001b[0m\n\u001b[1;32m 352\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m--> 353\u001b[0m obj, end \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mscan_once\u001b[49m\u001b[43m(\u001b[49m\u001b[43ms\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43midx\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 354\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mStopIteration\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n",
|
|
"\u001b[0;31mJSONDecodeError\u001b[0m: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)",
|
|
"\nDuring handling of the above exception, another exception occurred:\n",
|
|
"\u001b[0;31mOutputParserException\u001b[0m Traceback (most recent call last)",
|
|
"Cell \u001b[0;32mIn[7], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mparser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmisformatted\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/workplace/langchain/langchain/output_parsers/pydantic.py:29\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m 27\u001b[0m name \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpydantic_object\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m\n\u001b[1;32m 28\u001b[0m msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFailed to parse \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m from completion \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mtext\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. Got: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00me\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m---> 29\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(msg)\n",
|
|
"\u001b[0;31mOutputParserException\u001b[0m: Failed to parse Actor from completion {'name': 'Tom Hanks', 'film_names': ['Forrest Gump']}. Got: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"parser.parse(misformatted)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6c7c82b6",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now we can construct and use a `OutputFixingParser`. This output parser takes as an argument another output parser but also an LLM with which to try to correct any formatting mistakes."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "39b1a5ce",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.output_parsers import OutputFixingParser\n",
|
|
"\n",
|
|
"new_parser = OutputFixingParser.from_llm(parser=parser, llm=ChatOpenAI())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "0fd96d68",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Actor(name='Tom Hanks', film_names=['Forrest Gump'])"
|
|
]
|
|
},
|
|
"execution_count": 10,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"new_parser.parse(misformatted)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ea34eeaa",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Fixing Output Parsing Mistakes with the original prompt\n",
|
|
"\n",
|
|
"While in some cases it is possible to fix any parsing mistakes by only looking at the output, in other cases it can't. An example of this is when the output is not just in the incorrect format, but is partially complete. Consider the below example."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "67c5e1ac",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"template = \"\"\"Based on the user question, provide an Action and Action Input for what step should be taken.\n",
|
|
"{format_instructions}\n",
|
|
"Question: {query}\n",
|
|
"Response:\"\"\"\n",
|
|
"class Action(BaseModel):\n",
|
|
" action: str = Field(description=\"action to take\")\n",
|
|
" action_input: str = Field(description=\"input to the action\")\n",
|
|
" \n",
|
|
"parser = PydanticOutputParser(pydantic_object=Action)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "007aa87f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = PromptTemplate(\n",
|
|
" template=\"Answer the user query.\\n{format_instructions}\\n{query}\\n\",\n",
|
|
" input_variables=[\"query\"],\n",
|
|
" partial_variables={\"format_instructions\": parser.get_format_instructions()}\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "10d207ff",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt_value = prompt.format_prompt(query=\"who is leo di caprios gf?\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "68622837",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"bad_response = '{\"action\": \"search\"}'"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "25631465",
|
|
"metadata": {},
|
|
"source": [
|
|
"If we try to parse this response as is, we will get an error"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "894967c1",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"ename": "OutputParserException",
|
|
"evalue": "Failed to parse Action from completion {\"action\": \"search\"}. Got: 1 validation error for Action\naction_input\n field required (type=value_error.missing)",
|
|
"output_type": "error",
|
|
"traceback": [
|
|
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
|
|
"\u001b[0;31mValidationError\u001b[0m Traceback (most recent call last)",
|
|
"File \u001b[0;32m~/workplace/langchain/langchain/output_parsers/pydantic.py:24\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m 23\u001b[0m json_object \u001b[38;5;241m=\u001b[39m json\u001b[38;5;241m.\u001b[39mloads(json_str)\n\u001b[0;32m---> 24\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mpydantic_object\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse_obj\u001b[49m\u001b[43m(\u001b[49m\u001b[43mjson_object\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m 26\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m (json\u001b[38;5;241m.\u001b[39mJSONDecodeError, ValidationError) \u001b[38;5;28;01mas\u001b[39;00m e:\n",
|
|
"File \u001b[0;32m~/.pyenv/versions/3.9.1/envs/langchain/lib/python3.9/site-packages/pydantic/main.py:527\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.parse_obj\u001b[0;34m()\u001b[0m\n",
|
|
"File \u001b[0;32m~/.pyenv/versions/3.9.1/envs/langchain/lib/python3.9/site-packages/pydantic/main.py:342\u001b[0m, in \u001b[0;36mpydantic.main.BaseModel.__init__\u001b[0;34m()\u001b[0m\n",
|
|
"\u001b[0;31mValidationError\u001b[0m: 1 validation error for Action\naction_input\n field required (type=value_error.missing)",
|
|
"\nDuring handling of the above exception, another exception occurred:\n",
|
|
"\u001b[0;31mOutputParserException\u001b[0m Traceback (most recent call last)",
|
|
"Cell \u001b[0;32mIn[15], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mparser\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mparse\u001b[49m\u001b[43m(\u001b[49m\u001b[43mbad_response\u001b[49m\u001b[43m)\u001b[49m\n",
|
|
"File \u001b[0;32m~/workplace/langchain/langchain/output_parsers/pydantic.py:29\u001b[0m, in \u001b[0;36mPydanticOutputParser.parse\u001b[0;34m(self, text)\u001b[0m\n\u001b[1;32m 27\u001b[0m name \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mpydantic_object\u001b[38;5;241m.\u001b[39m\u001b[38;5;18m__name__\u001b[39m\n\u001b[1;32m 28\u001b[0m msg \u001b[38;5;241m=\u001b[39m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFailed to parse \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mname\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m from completion \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mtext\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m. Got: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00me\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m---> 29\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m OutputParserException(msg)\n",
|
|
"\u001b[0;31mOutputParserException\u001b[0m: Failed to parse Action from completion {\"action\": \"search\"}. Got: 1 validation error for Action\naction_input\n field required (type=value_error.missing)"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"parser.parse(bad_response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f6b64696",
|
|
"metadata": {},
|
|
"source": [
|
|
"If we try to use the `OutputFixingParser` to fix this error, it will be confused - namely, it doesn't know what to actually put for action input."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "78b2b40d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"fix_parser = OutputFixingParser.from_llm(parser=parser, llm=ChatOpenAI())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "4fe1301d",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Action(action='search', action_input='keyword')"
|
|
]
|
|
},
|
|
"execution_count": 17,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"fix_parser.parse(bad_response)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9bd9ea7d",
|
|
"metadata": {},
|
|
"source": [
|
|
"Instead, we can use the RetryOutputParser, which passes in the prompt (as well as the original output) to try again to get a better response."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "7e8a8a28",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.output_parsers import RetryWithErrorOutputParser"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "5c86e141",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"retry_parser = RetryWithErrorOutputParser.from_llm(parser=parser, llm=ChatOpenAI())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "9c04f731",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Action(action='search', action_input='leo di caprios girlfriend')"
|
|
]
|
|
},
|
|
"execution_count": 21,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"retry_parser.parse_with_prompt(bad_response, prompt_value)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "61f67890",
|
|
"metadata": {},
|
|
"source": [
|
|
"<br>\n",
|
|
"<br>\n",
|
|
"<br>\n",
|
|
"<br>\n",
|
|
"<br>\n",
|
|
"<br>\n",
|
|
"<br>\n",
|
|
"<br>\n",
|
|
"\n",
|
|
"---"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "64bf525a",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Older, less powerful parsers"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "91871002",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Structured Output Parser\n",
|
|
"\n",
|
|
"While the Pydantic/JSON parser is more powerful, we initially experimented data structures having text fields only."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "b492997a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.output_parsers import StructuredOutputParser, ResponseSchema"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "09473dce",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here we define the response schema we want to receive."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "432ac44a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"response_schemas = [\n",
|
|
" ResponseSchema(name=\"answer\", description=\"answer to the user's question\"),\n",
|
|
" ResponseSchema(name=\"source\", description=\"source used to answer the user's question, should be a website.\")\n",
|
|
"]\n",
|
|
"output_parser = StructuredOutputParser.from_response_schemas(response_schemas)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7b92ce96",
|
|
"metadata": {},
|
|
"source": [
|
|
"We now get a string that contains instructions for how the response should be formatted, and we then insert that into our prompt."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "593cfc25",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"format_instructions = output_parser.get_format_instructions()\n",
|
|
"prompt = PromptTemplate(\n",
|
|
" template=\"answer the users question as best as possible.\\n{format_instructions}\\n{question}\",\n",
|
|
" input_variables=[\"question\"],\n",
|
|
" partial_variables={\"format_instructions\": format_instructions}\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "0943e783",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can now use this to format a prompt to send to the language model, and then parse the returned result."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "106f1ba6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"model = OpenAI(temperature=0)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "86d9d24f",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"_input = prompt.format_prompt(question=\"what's the capital of france\")\n",
|
|
"output = model(_input.to_string())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 21,
|
|
"id": "956bdc99",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'answer': 'Paris', 'source': 'https://en.wikipedia.org/wiki/Paris'}"
|
|
]
|
|
},
|
|
"execution_count": 21,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"output_parser.parse(output)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "da639285",
|
|
"metadata": {},
|
|
"source": [
|
|
"And here's an example of using this in a chat model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 22,
|
|
"id": "8f483d7d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"chat_model = ChatOpenAI(temperature=0)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 23,
|
|
"id": "f761cbf1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"prompt = ChatPromptTemplate(\n",
|
|
" messages=[\n",
|
|
" HumanMessagePromptTemplate.from_template(\"answer the users question as best as possible.\\n{format_instructions}\\n{question}\") \n",
|
|
" ],\n",
|
|
" input_variables=[\"question\"],\n",
|
|
" partial_variables={\"format_instructions\": format_instructions}\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 24,
|
|
"id": "edd73ae3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"_input = prompt.format_prompt(question=\"what's the capital of france\")\n",
|
|
"output = chat_model(_input.to_messages())"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 25,
|
|
"id": "a3c8b91e",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'answer': 'Paris', 'source': 'https://en.wikipedia.org/wiki/Paris'}"
|
|
]
|
|
},
|
|
"execution_count": 25,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"output_parser.parse(output.content)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9936fa27",
|
|
"metadata": {},
|
|
"source": [
|
|
"## CommaSeparatedListOutputParser\n",
|
|
"\n",
|
|
"Here's another parser strictly less powerful than Pydantic/JSON parsing."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 26,
|
|
"id": "872246d7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.output_parsers import CommaSeparatedListOutputParser"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 27,
|
|
"id": "c3f9aee6",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"output_parser = CommaSeparatedListOutputParser()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 28,
|
|
"id": "e77871b7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"format_instructions = output_parser.get_format_instructions()\n",
|
|
"prompt = PromptTemplate(\n",
|
|
" template=\"List five {subject}.\\n{format_instructions}\",\n",
|
|
" input_variables=[\"subject\"],\n",
|
|
" partial_variables={\"format_instructions\": format_instructions}\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 29,
|
|
"id": "a71cb5d3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"model = OpenAI(temperature=0)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 30,
|
|
"id": "783d7d98",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"_input = prompt.format(subject=\"ice cream flavors\")\n",
|
|
"output = model(_input)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 31,
|
|
"id": "fcb81344",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"['Vanilla',\n",
|
|
" 'Chocolate',\n",
|
|
" 'Strawberry',\n",
|
|
" 'Mint Chocolate Chip',\n",
|
|
" 'Cookies and Cream']"
|
|
]
|
|
},
|
|
"execution_count": 31,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"output_parser.parse(output)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.1"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|