blob42
/
langchain
forked from Archives/langchain
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Added other evaluation metrics for data-augmented QA (#1521)

Browse Source 
This PR adds additional evaluation metrics for data-augmented QA,
resulting in a report like this at the end of the notebook:

![Screen Shot 2023-03-08 at 8 53 23
AM](https://user-images.githubusercontent.com/398875/223731199-8eb8e77f-5ff3-40a2-a23e-f3bede623344.png)

The score calculation is based on the
[Critique](https://docs.inspiredco.ai/critique/) toolkit, an API-based
toolkit (like OpenAI) that has minimal dependencies, so it should be
easy for people to run if they choose.

The code could further be simplified by actually adding a chain that
calls Critique directly, but that probably should be saved for another
PR if necessary. Any comments or change requests are welcome!
fix-searx
Graham Neubig 1 year ago committed by GitHub
parent 494c9d341a
commit 31303d0b11
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 183 additions and 38 deletions
Show Stats Download Patch File Download Diff File

221
docs/use_cases/evaluation/data_augmented_question_answering.ipynb
Unescape Escape View File

@ -28,7 +28,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 2,
"id": "4fdc211d",
"metadata": {},
"outputs": [
@ -67,7 +67,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 3,
"id": "3459b001",
"metadata": {},
"outputs": [],
@ -87,7 +87,7 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 4,
"id": "b9c3fa75",
"metadata": {},
"outputs": [],
@ -99,7 +99,7 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 5,
"id": "c24543a9",
"metadata": {},
"outputs": [],
@ -116,16 +116,16 @@
{
"data": {
"text/plain": [
"[{'query': 'What did Vladimir Putin miscalculate when he sought to shake the foundations of the free world? ',\n",
" 'answer': 'He miscalculated that the world would roll over and that he could roll into Ukraine without facing resistance.'},\n",
" {'query': 'What is the purpose of NATO?',\n",
" 'answer': 'The purpose of NATO is to secure peace and stability in Europe after World War 2.'},\n",
" {'query': \"What did the author do to prepare for Putin's attack on Ukraine?\",\n",
" 'answer': \"The author spent months building a coalition of freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin, shared with the world in advance what they knew Putin was planning, and countered Russia's lies with truth.\"},\n",
" {'query': 'What are the US and its allies doing to isolate Russia from the world?',\n",
" 'answer': \"Enforcing powerful economic sanctions, cutting off Russia's largest banks from the international financial system, preventing Russia's central bank from defending the Russian Ruble, choking off Russia's access to technology, and joining with European allies to find and seize assets of Russian oligarchs.\"},\n",
" {'query': 'How much direct assistance is the U.S. providing to Ukraine?',\n",
" 'answer': 'The U.S. is providing more than $1 Billion in direct assistance to Ukraine.'}]"
"[{'query': 'According to the document, what did Vladimir Putin miscalculate?',\n",
" 'answer': 'He miscalculated that he could roll into Ukraine and the world would roll over.'},\n",
" {'query': 'Who is the Ukrainian Ambassador to the United States?',\n",
" 'answer': 'The Ukrainian Ambassador to the United States is here tonight.'},\n",
" {'query': 'How many countries were part of the coalition formed to confront Putin?',\n",
" 'answer': '27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.'},\n",
" {'query': 'What action is the U.S. Department of Justice taking to target Russian oligarchs?',\n",
" 'answer': 'The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.'},\n",
" {'query': 'How much direct assistance is the United States providing to Ukraine?',\n",
" 'answer': 'The United States is providing more than $1 Billion in direct assistance to Ukraine.'}]"
]
},
"execution_count": 6,
@ -211,44 +211,43 @@
"Example 0:\n",
"Question: What did the president say about Ketanji Brown Jackson\n",
"Real Answer: He praised her legal ability and said he nominated her for the supreme court.\n",
"Predicted Answer: The president said that Ketanji Brown Jackson is one of the nation's top legal minds and that she will continue Justice Breyer's legacy of excellence.\n",
"Predicted Answer: The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 1:\n",
"Question: What did the president say about Michael Jackson\n",
"Real Answer: Nothing\n",
"Predicted Answer: \n",
"The president did not mention Michael Jackson in this context.\n",
"Predicted Answer: The president did not mention Michael Jackson in this speech.\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 2:\n",
"Question: What did Vladimir Putin miscalculate when he sought to shake the foundations of the free world? \n",
"Real Answer: He miscalculated that the world would roll over and that he could roll into Ukraine without facing resistance.\n",
"Predicted Answer: Putin miscalculated that the West and NATO wouldn't respond to his attack on Ukraine and that he could divide the US and its allies.\n",
"Question: According to the document, what did Vladimir Putin miscalculate?\n",
"Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.\n",
"Predicted Answer: Putin miscalculated that the world would roll over when he rolled into Ukraine.\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 3:\n",
"Question: What is the purpose of NATO?\n",
"Real Answer: The purpose of NATO is to secure peace and stability in Europe after World War 2.\n",
"Predicted Answer: The purpose of NATO is to secure peace and stability in Europe after World War 2.\n",
"Predicted Grade: CORRECT\n",
"Question: Who is the Ukrainian Ambassador to the United States?\n",
"Real Answer: The Ukrainian Ambassador to the United States is here tonight.\n",
"Predicted Answer: I don't know.\n",
"Predicted Grade: INCORRECT\n",
"\n",
"Example 4:\n",
"Question: What did the author do to prepare for Putin's attack on Ukraine?\n",
"Real Answer: The author spent months building a coalition of freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin, shared with the world in advance what they knew Putin was planning, and countered Russia's lies with truth.\n",
"Predicted Answer: The author prepared extensively and carefully. They spent months building a coalition of other freedom-loving nations from Europe and the Americas to Asia and Africa to confront Putin, and they spent countless hours unifying their European allies. They also shared with the world in advance what they knew Putin was planning and precisely how he would try to falsely justify his aggression. They countered Russia’s lies with truth.\n",
"Predicted Grade: CORRECT\n",
"Question: How many countries were part of the coalition formed to confront Putin?\n",
"Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
"Predicted Answer: The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
"Predicted Grade: INCORRECT\n",
"\n",
"Example 5:\n",
"Question: What are the US and its allies doing to isolate Russia from the world?\n",
"Real Answer: Enforcing powerful economic sanctions, cutting off Russia's largest banks from the international financial system, preventing Russia's central bank from defending the Russian Ruble, choking off Russia's access to technology, and joining with European allies to find and seize assets of Russian oligarchs.\n",
"Predicted Answer: The US and its allies are enforcing economic sanctions on Russia, cutting off its largest banks from the international financial system, preventing its central bank from defending the Russian Ruble, choking off Russia's access to technology, closing American airspace to all Russian flights, and providing support to Ukraine.\n",
"Predicted Grade: CORRECT\n",
"Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?\n",
"Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.\n",
"Predicted Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.\n",
"Predicted Grade: INCORRECT\n",
"\n",
"Example 6:\n",
"Question: How much direct assistance is the U.S. providing to Ukraine?\n",
"Real Answer: The U.S. is providing more than $1 Billion in direct assistance to Ukraine.\n",
"Predicted Answer: The U.S. is providing more than $1 Billion in direct assistance to Ukraine.\n",
"Question: How much direct assistance is the United States providing to Ukraine?\n",
"Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.\n",
"Predicted Answer: The United States is providing more than $1 billion in direct assistance to Ukraine.\n",
"Predicted Grade: CORRECT\n",
"\n"
]
@ -264,13 +263,159 @@
" print()"
]
},
{
"cell_type": "markdown",
"id": "50a9e845",
"metadata": {},
"source": [
"## Evaluate with Other Metrics\n",
"\n",
"In addition to predicting whether the answer is correct or incorrect using a language model, we can also use other metrics to get a more nuanced view on the quality of the answers. To do so, we can use the [Critique](https://docs.inspiredco.ai/critique/) library, which allows for simple calculation of various metrics over generated text.\n",
"\n",
"First you can get an API key from the [Inspired Cognition Dashboard](https://dashboard.inspiredco.ai) and do some setup:\n",
"\n",
"```bash\n",
"export INSPIREDCO_API_KEY=\"...\"\n",
"pip install inspiredco\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 13,
"id": "bd0b01dc",
"metadata": {},
"outputs": [],
"source": []
"source": [
"import inspiredco.critique\n",
"import os\n",
"critique = inspiredco.critique.Critique(api_key=os.environ['INSPIREDCO_API_KEY'])"
]
},
{
"cell_type": "markdown",
"id": "4f52629e",
"metadata": {},
"source": [
"Then run the following code to set up the configuration and calculate the [ROUGE](https://docs.inspiredco.ai/critique/metric_rouge.html), [chrf](https://docs.inspiredco.ai/critique/metric_chrf.html), [BERTScore](https://docs.inspiredco.ai/critique/metric_bert_score.html), and [UniEval](https://docs.inspiredco.ai/critique/metric_uni_eval.html) (you can choose [other metrics](https://docs.inspiredco.ai/critique/metrics.html) too):"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "84a0ba21",
"metadata": {},
"outputs": [],
"source": [
"metrics = {\n",
" \"rouge\": {\n",
" \"metric\": \"rouge\",\n",
" \"config\": {\"variety\": \"rouge_l\"},\n",
" },\n",
" \"chrf\": {\n",
" \"metric\": \"chrf\",\n",
" \"config\": {},\n",
" },\n",
" \"bert_score\": {\n",
" \"metric\": \"bert_score\",\n",
" \"config\": {\"model\": \"bert-base-uncased\"},\n",
" },\n",
" \"uni_eval\": {\n",
" \"metric\": \"uni_eval\",\n",
" \"config\": {\"task\": \"summarization\", \"evaluation_aspect\": \"relevance\"},\n",
" },\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "3b9a4056",
"metadata": {},
"outputs": [],
"source": [
"critique_data = [\n",
" {\"target\": pred['result'], \"references\": [pred['answer']]} for pred in predictions\n",
"]\n",
"eval_results = {\n",
" k: critique.evaluate(dataset=critique_data, metric=v[\"metric\"], config=v[\"config\"])\n",
" for k, v in metrics.items()\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "6f0ae799",
"metadata": {},
"source": [
"Finally, we can print out the results. We can see that overall the scores are higher when the output is semantically correct, and also when the output closely matches with the gold-standard answer."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "b51edcf4",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Example 0:\n",
"Question: What did the president say about Ketanji Brown Jackson\n",
"Real Answer: He praised her legal ability and said he nominated her for the supreme court.\n",
"Predicted Answer: The president said that she is one of the nation's top legal minds, a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He also said that she is a consensus builder and that she has received a broad range of support from the Fraternal Order of Police to former judges appointed by both Democrats and Republicans.\n",
"Predicted Scores: rouge=0.0941, chrf=0.2001, bert_score=0.5219, uni_eval=0.9043\n",
"\n",
"Example 1:\n",
"Question: What did the president say about Michael Jackson\n",
"Real Answer: Nothing\n",
"Predicted Answer: The president did not mention Michael Jackson in this speech.\n",
"Predicted Scores: rouge=0.0000, chrf=0.1087, bert_score=0.3486, uni_eval=0.7802\n",
"\n",
"Example 2:\n",
"Question: According to the document, what did Vladimir Putin miscalculate?\n",
"Real Answer: He miscalculated that he could roll into Ukraine and the world would roll over.\n",
"Predicted Answer: Putin miscalculated that the world would roll over when he rolled into Ukraine.\n",
"Predicted Scores: rouge=0.5185, chrf=0.6955, bert_score=0.8421, uni_eval=0.9578\n",
"\n",
"Example 3:\n",
"Question: Who is the Ukrainian Ambassador to the United States?\n",
"Real Answer: The Ukrainian Ambassador to the United States is here tonight.\n",
"Predicted Answer: I don't know.\n",
"Predicted Scores: rouge=0.0000, chrf=0.0375, bert_score=0.3159, uni_eval=0.7493\n",
"\n",
"Example 4:\n",
"Question: How many countries were part of the coalition formed to confront Putin?\n",
"Real Answer: 27 members of the European Union, France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
"Predicted Answer: The coalition included freedom-loving nations from Europe and the Americas to Asia and Africa, 27 members of the European Union including France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
"Predicted Scores: rouge=0.7419, chrf=0.8602, bert_score=0.8388, uni_eval=0.0669\n",
"\n",
"Example 5:\n",
"Question: What action is the U.S. Department of Justice taking to target Russian oligarchs?\n",
"Real Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and joining with European allies to find and seize their yachts, luxury apartments, and private jets.\n",
"Predicted Answer: The U.S. Department of Justice is assembling a dedicated task force to go after the crimes of Russian oligarchs and to find and seize their yachts, luxury apartments, and private jets.\n",
"Predicted Scores: rouge=0.9412, chrf=0.8687, bert_score=0.9607, uni_eval=0.9718\n",
"\n",
"Example 6:\n",
"Question: How much direct assistance is the United States providing to Ukraine?\n",
"Real Answer: The United States is providing more than $1 Billion in direct assistance to Ukraine.\n",
"Predicted Answer: The United States is providing more than $1 billion in direct assistance to Ukraine.\n",
"Predicted Scores: rouge=1.0000, chrf=0.9483, bert_score=1.0000, uni_eval=0.9734\n",
"\n"
]
}
],
"source": [
"for i, eg in enumerate(examples):\n",
" score_string = \", \".join([f\"{k}={v['examples'][i]['value']:.4f}\" for k, v in eval_results.items()])\n",
" print(f\"Example {i}:\")\n",
" print(\"Question: \" + predictions[i]['query'])\n",
" print(\"Real Answer: \" + predictions[i]['answer'])\n",
" print(\"Predicted Answer: \" + predictions[i]['result'])\n",
" print(\"Predicted Scores: \" + score_string)\n",
" print()"
]
}
],
"metadata": {
@ -289,7 +434,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
"version": "3.10.9"
}
},
"nbformat": 4,

Write Preview
Loading…
Cancel
Save