Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-04 06:00:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
193f107cb5
langchain/cookbook/stepback-qa.ipynb

351 lines
11 KiB
Plaintext
Raw Normal View History

add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
{
"cells": [
{
"cell_type": "markdown",
"id": "83ef724e",
"metadata": {},
"source": [
"# Step-Back Prompting (Question-Answering)\n",
"\n",
"One prompting technique called \"Step-Back\" prompting can improve performance on complex questions by first asking a \"step back\" question. This can be combined with regular question-answering applications by then doing retrieval on both the original and step-back question.\n",
"\n",
"Read the paper [here](https://arxiv.org/abs/2310.06117)\n",
"\n",
fix typo in langchain/cookbook/stepback-qa.ipynb (#12204)
2023-10-24 19:51:51 +00:00
"See an excellent blog post on this by Cobus Greyling [here](https://cobusgreyling.medium.com/a-new-prompt-engineering-technique-has-been-introduced-called-step-back-prompting-b00e8954cacb)\n",
add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
"\n",
"In this cookbook we will replicate this technique. We modify the prompts used slightly to work better with chat models."
]
},
{
"cell_type": "code",
"execution_count": 85,
"id": "67b5cdac",
"metadata": {},
"outputs": [],
"source": [
"from langchain.chat_models import ChatOpenAI\n",
"from langchain.prompts import ChatPromptTemplate, FewShotChatMessagePromptTemplate\n",
docs[patch], templates[patch]: Import from core (#14575) Update imports to use core for the low-hanging fruit changes. Ran following ```bash git grep -l 'langchain.schema.runnable' {docs,templates,cookbook} | xargs sed -i '' 's/langchain\.schema\.runnable/langchain_core.runnables/g' git grep -l 'langchain.schema.output_parser' {docs,templates,cookbook} | xargs sed -i '' 's/langchain\.schema\.output_parser/langchain_core.output_parsers/g' git grep -l 'langchain.schema.messages' {docs,templates,cookbook} | xargs sed -i '' 's/langchain\.schema\.messages/langchain_core.messages/g' git grep -l 'langchain.schema.chat_histry' {docs,templates,cookbook} | xargs sed -i '' 's/langchain\.schema\.chat_history/langchain_core.chat_history/g' git grep -l 'langchain.schema.prompt_template' {docs,templates,cookbook} | xargs sed -i '' 's/langchain\.schema\.prompt_template/langchain_core.prompts/g' git grep -l 'from langchain.pydantic_v1' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.pydantic_v1/from langchain_core.pydantic_v1/g' git grep -l 'from langchain.tools.base' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.tools\.base/from langchain_core.tools/g' git grep -l 'from langchain.chat_models.base' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.chat_models.base/from langchain_core.language_models.chat_models/g' git grep -l 'from langchain.llms.base' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.llms\.base\ /from langchain_core.language_models.llms\ /g' git grep -l 'from langchain.embeddings.base' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.embeddings\.base/from langchain_core.embeddings/g' git grep -l 'from langchain.vectorstores.base' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.vectorstores\.base/from langchain_core.vectorstores/g' git grep -l 'from langchain.agents.tools' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.agents\.tools/from langchain_core.tools/g' git grep -l 'from langchain.schema.output' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.schema\.output\ /from langchain_core.outputs\ /g' git grep -l 'from langchain.schema.embeddings' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.schema\.embeddings/from langchain_core.embeddings/g' git grep -l 'from langchain.schema.document' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.schema\.document/from langchain_core.documents/g' git grep -l 'from langchain.schema.agent' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.schema\.agent/from langchain_core.agents/g' git grep -l 'from langchain.schema.prompt ' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.schema\.prompt\ /from langchain_core.prompt_values /g' git grep -l 'from langchain.schema.language_model' {docs,templates,cookbook} | xargs sed -i '' 's/from langchain\.schema\.language_model/from langchain_core.language_models/g' ```
2023-12-12 00:49:10 +00:00
"from langchain_core.output_parsers import StrOutputParser\n",
"from langchain_core.runnables import RunnableLambda"
add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
]
},
{
"cell_type": "code",
"execution_count": 86,
"id": "7e017c44",
"metadata": {},
"outputs": [],
"source": [
"# Few Shot Examples\n",
"examples = [\n",
" {\n",
" \"input\": \"Could the members of The Police perform lawful arrests?\",\n",
notebook fmt (#12498)
2023-10-29 22:50:09 +00:00
" \"output\": \"what can the members of The Police do?\",\n",
add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
" },\n",
" {\n",
notebook fmt (#12498)
2023-10-29 22:50:09 +00:00
" \"input\": \"Jan Sindel’s was born in what country?\",\n",
" \"output\": \"what is Jan Sindel’s personal history?\",\n",
add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
" },\n",
"]\n",
"# We now transform these to example messages\n",
"example_prompt = ChatPromptTemplate.from_messages(\n",
" [\n",
" (\"human\", \"{input}\"),\n",
" (\"ai\", \"{output}\"),\n",
" ]\n",
")\n",
"few_shot_prompt = FewShotChatMessagePromptTemplate(\n",
" example_prompt=example_prompt,\n",
" examples=examples,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 87,
"id": "206415ee",
"metadata": {},
"outputs": [],
"source": [
notebook fmt (#12498)
2023-10-29 22:50:09 +00:00
"prompt = ChatPromptTemplate.from_messages(\n",
" [\n",
" (\n",
" \"system\",\n",
" \"\"\"You are an expert at world knowledge. Your task is to step back and paraphrase a question to a more generic step-back question, which is easier to answer. Here are a few examples:\"\"\",\n",
" ),\n",
" # Few shot examples\n",
" few_shot_prompt,\n",
" # New question\n",
" (\"user\", \"{question}\"),\n",
" ]\n",
")"
add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
]
},
{
"cell_type": "code",
"execution_count": 88,
"id": "d643a85c",
"metadata": {},
"outputs": [],
"source": [
"question_gen = prompt | ChatOpenAI(temperature=0) | StrOutputParser()"
]
},
{
"cell_type": "code",
"execution_count": 182,
"id": "5ba21b2a",
"metadata": {},
"outputs": [],
"source": [
"question = \"was chatgpt around while trump was president?\""
]
},
{
"cell_type": "code",
"execution_count": 183,
"id": "5992c8ca",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'when was ChatGPT developed?'"
]
},
"execution_count": 183,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"question_gen.invoke({\"question\": question})"
]
},
{
"cell_type": "code",
"execution_count": 190,
"id": "32667424",
"metadata": {},
"outputs": [],
"source": [
"from langchain.utilities import DuckDuckGoSearchAPIWrapper\n",
"\n",
"search = DuckDuckGoSearchAPIWrapper(max_results=4)\n",
"\n",
notebook fmt (#12498)
2023-10-29 22:50:09 +00:00
"\n",
add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
"def retriever(query):\n",
" return search.run(query)"
]
},
{
"cell_type": "code",
"execution_count": 191,
"id": "ffc28c91",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'This includes content about former President Donald Trump. According to further tests, ChatGPT successfully wrote poems admiring all recent U.S. presidents, but failed when we entered a query for ... On Wednesday, a Twitter user posted screenshots of him asking OpenAI\\'s chatbot, ChatGPT, to write a positive poem about former President Donald Trump, to which the chatbot declined, citing it ... While impressive in many respects, ChatGPT also has some major flaws. ... [President\\'s Name],\" refused to write a poem about ex-President Trump, but wrote one about President Biden ... During the Trump administration, Altman gained new attention as a vocal critic of the president. It was against that backdrop that he was rumored to be considering a run for California governor.'"
]
},
"execution_count": 191,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"retriever(question)"
]
},
{
"cell_type": "code",
"execution_count": 192,
"id": "00c77443",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"Will Douglas Heaven March 3, 2023 Stephanie Arnett/MITTR | Envato When OpenAI launched ChatGPT, with zero fanfare, in late November 2022, the San Francisco-based artificial-intelligence company... ChatGPT, which stands for Chat Generative Pre-trained Transformer, is a large language model -based chatbot developed by OpenAI and launched on November 30, 2022, which enables users to refine and steer a conversation towards a desired length, format, style, level of detail, and language. ChatGPT is an artificial intelligence (AI) chatbot built on top of OpenAI's foundational large language models (LLMs) like GPT-4 and its predecessors. This chatbot has redefined the standards of... June 4, 2023 ⋅ 4 min read 124 SHARES 13K At the end of 2022, OpenAI introduced the world to ChatGPT. Since its launch, ChatGPT hasn't shown significant signs of slowing down in developing new...\""
]
},
"execution_count": 192,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"retriever(question_gen.invoke({\"question\": question}))"
]
},
{
"cell_type": "code",
"execution_count": 193,
"id": "b257bc06",
"metadata": {},
"outputs": [],
"source": [
"# response_prompt_template = \"\"\"You are an expert of world knowledge. I am going to ask you a question. Your response should be comprehensive and not contradicted with the following context if they are relevant. Otherwise, ignore them if they are not relevant.\n",
"\n",
"# {normal_context}\n",
"# {step_back_context}\n",
"\n",
"# Original Question: {question}\n",
"# Answer:\"\"\"\n",
"# response_prompt = ChatPromptTemplate.from_template(response_prompt_template)"
]
},
{
"cell_type": "code",
"execution_count": 203,
"id": "f48c65b2",
"metadata": {},
"outputs": [],
"source": [
"from langchain import hub\n",
"\n",
"response_prompt = hub.pull(\"langchain-ai/stepback-answer\")"
]
},
{
"cell_type": "code",
"execution_count": 204,
"id": "97a6d5ab",
"metadata": {},
"outputs": [],
"source": [
notebook fmt (#12498)
2023-10-29 22:50:09 +00:00
"chain = (\n",
" {\n",
" # Retrieve context using the normal question\n",
" \"normal_context\": RunnableLambda(lambda x: x[\"question\"]) | retriever,\n",
" # Retrieve context using the step-back question\n",
" \"step_back_context\": question_gen | retriever,\n",
" # Pass on the question\n",
" \"question\": lambda x: x[\"question\"],\n",
" }\n",
" | response_prompt\n",
" | ChatOpenAI(temperature=0)\n",
" | StrOutputParser()\n",
")"
add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
]
},
{
"cell_type": "code",
"execution_count": 205,
"id": "ce554cb0",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"No, ChatGPT was not around while Donald Trump was president. ChatGPT was launched on November 30, 2022, which is after Donald Trump's presidency. The context provided mentions that during the Trump administration, Altman, the CEO of OpenAI, gained attention as a vocal critic of the president. This suggests that ChatGPT was not developed or available during that time.\""
]
},
"execution_count": 205,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"chain.invoke({\"question\": question})"
]
},
{
"cell_type": "markdown",
"id": "a9fb8dd2",
"metadata": {},
"source": [
"## Baseline"
]
},
{
"cell_type": "code",
"execution_count": 206,
"id": "00db8a15",
"metadata": {},
"outputs": [],
"source": [
"response_prompt_template = \"\"\"You are an expert of world knowledge. I am going to ask you a question. Your response should be comprehensive and not contradicted with the following context if they are relevant. Otherwise, ignore them if they are not relevant.\n",
"\n",
"{normal_context}\n",
"\n",
"Original Question: {question}\n",
"Answer:\"\"\"\n",
"response_prompt = ChatPromptTemplate.from_template(response_prompt_template)"
]
},
{
"cell_type": "code",
"execution_count": 207,
"id": "06335ebb",
"metadata": {},
"outputs": [],
"source": [
notebook fmt (#12498)
2023-10-29 22:50:09 +00:00
"chain = (\n",
" {\n",
" # Retrieve context using the normal question (only the first 3 results)\n",
" \"normal_context\": RunnableLambda(lambda x: x[\"question\"]) | retriever,\n",
" # Pass on the question\n",
" \"question\": lambda x: x[\"question\"],\n",
" }\n",
" | response_prompt\n",
" | ChatOpenAI(temperature=0)\n",
" | StrOutputParser()\n",
")"
add step back notebook (#11953)
2023-10-21 17:05:52 +00:00
]
},
{
"cell_type": "code",
"execution_count": 208,
"id": "15e0e741",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"\"Yes, ChatGPT was around while Donald Trump was president. However, it is important to note that the specific context you provided mentions that ChatGPT refused to write a positive poem about former President Donald Trump. This suggests that while ChatGPT was available during Trump's presidency, it may have had limitations or biases in its responses regarding him.\""
]
},
"execution_count": 208,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"chain.invoke({\"question\": question})"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "e7b9e5d6",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.1"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue Copy Permalink