"[{'query': 'What did Vladimir Putin seek to do according to the document?',\n",
" 'answer': 'Vladimir Putin sought to shake the foundations of the free world and make it bend to his menacing ways.'},\n",
"[{'query': \"Who did Russia's Vladimir Putin miscalculate when he attempted to shake the foundations of the free world?\",\n",
" 'answer': 'Putin miscalculated the Ukrainian people.'},\n",
" {'query': 'What did President Zelenskyy say in his speech to the European Parliament?',\n",
" 'answer': 'President Zelenskyy said \"Light will win over darkness.\"'},\n",
" {'query': \"How many countries joined the European Union in opposing Putin's attack on Ukraine?\",\n",
" 'answer': '27'},\n",
" {'query': 'What is the U.S. Department of Justice assembling in response to the Russian oligarchs?',\n",
" 'answer': 'A dedicated task force.'},\n",
" {'query': 'How much direct assistance is the US providing to Ukraine?',\n",
" 'answer': 'The US is providing more than $1 Billion in direct assistance to Ukraine.'}]"
" 'answer': '\"Light will win over darkness.\"'},\n",
" {'query': 'What countries joined the coalition to confront Putin?',\n",
" 'answer': 'France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.'},\n",
" {'query': 'What measures is the US taking to punish Russia and its oligarchs?',\n",
" 'answer': \"The US is enforcing powerful economic sanctions, cutting off Russia's largest banks from the international financial system, preventing the Russian Ruble from being defended, choking off Russia's access to technology, and joining with European allies to seize yachts, luxury apartments, and private jets.\"},\n",
" {'query': 'What is the total amount of direct assistance being provided to Ukraine?',\n",
" 'answer': '$1 Billion'}]"
]
},
"execution_count": 6,
@ -190,29 +190,67 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 14,
"id": "32fac2dc",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'text': ' CORRECT'},\n",
" {'text': ' CORRECT'},\n",
" {'text': ' INCORRECT'},\n",
" {'text': ' CORRECT'},\n",
" {'text': ' CORRECT'},\n",
" {'text': ' CORRECT'},\n",
" {'text': ' CORRECT'}]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
"name": "stdout",
"output_type": "stream",
"text": [
"Example 0:\n",
"Question: What did the president say about Ketanji Brown Jackson\n",
"Real Answer: He praised her legal ability and said he nominated her for the supreme court.\n",
"Predicted Answer: The president said that Ketanji Brown Jackson is one of the nation's top legal minds and is a former top litigator in private practice, a former federal public defender, and from a family of public school educators and police officers. He said she is a consensus builder and has received a broad range of support since being nominated.\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 1:\n",
"Question: What did the president say about Michael Jackson\n",
"Real Answer: Nothing\n",
"Predicted Answer: The president did not mention Michael Jackson.\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 2:\n",
"Question: Who did Russia's Vladimir Putin miscalculate when he attempted to shake the foundations of the free world?\n",
"Real Answer: Putin miscalculated the Ukrainian people.\n",
"Predicted Answer: Putin miscalculated the strength of the Ukrainian people, the support of freedom-loving nations from Europe and the Americas to Asia and Africa, and the resolve of the United States and its allies.\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 3:\n",
"Question: What did President Zelenskyy say in his speech to the European Parliament?\n",
"Real Answer: \"Light will win over darkness.\"\n",
"Predicted Answer: President Zelenskyy said in his speech to the European Parliament “Light will win over darkness.”\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 4:\n",
"Question: What countries joined the coalition to confront Putin?\n",
"Real Answer: France, Germany, Italy, the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
"Predicted Answer: The coalition included members of the European Union, including France, Germany, Italy, as well as countries like the United Kingdom, Canada, Japan, Korea, Australia, New Zealand, and many others, even Switzerland.\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 5:\n",
"Question: What measures is the US taking to punish Russia and its oligarchs?\n",
"Real Answer: The US is enforcing powerful economic sanctions, cutting off Russia's largest banks from the international financial system, preventing the Russian Ruble from being defended, choking off Russia's access to technology, and joining with European allies to seize yachts, luxury apartments, and private jets.\n",
"Predicted Answer: The US is enforcing economic sanctions, cutting off Russia's access to international financial systems, preventing Russia's central bank from defending the Ruble and making Putin's \"war fund\" worthless, and targeting the crimes of Russian oligarchs by assembling a dedicated task force to seize their yachts, luxury apartments, and private jets. The US is also closing off American airspace to all Russian flights, providing aid to Ukraine, and mobilizing ground forces, air squadrons, and ship deployments to protect NATO countries.\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 6:\n",
"Question: What is the total amount of direct assistance being provided to Ukraine?\n",
"Real Answer: $1 Billion\n",
"Predicted Answer: The total amount of direct assistance being provided to Ukraine is $1 billion.\n",
" {'text': ' No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not likely that he would be catching a screen pass in the NFC championship.'}]"
" {'text': ' No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not possible for him to catch a screen pass in the NFC championship.'}]"
]
},
"execution_count": 6,
@ -157,23 +157,37 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 10,
"id": "63780020",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'text': ' CORRECT'}, {'text': ' CORRECT'}]"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
"name": "stdout",
"output_type": "stream",
"text": [
"Example 0:\n",
"Question: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls does he have now?\n",
"Real Answer: 11\n",
"Predicted Answer: 11 tennis balls\n",
"Predicted Grade: CORRECT\n",
"\n",
"Example 1:\n",
"Question: Is the following sentence plausible? \"Joao Moutinho caught the screen pass in the NFC championship.\"\n",
"Real Answer: No\n",
"Predicted Answer: No, this sentence is not plausible. Joao Moutinho is a professional soccer player, not an American football player, so it is not possible for him to catch a screen pass in the NFC championship.\n",
Harrison Chase
2 years ago
committed by
GitHub