mirror of
https://github.com/hwchase17/langchain
synced 2024-10-31 15:20:26 +00:00
87e502c6bc
Co-authored-by: jacoblee93 <jacoblee93@gmail.com> Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
1130 lines
52 KiB
Plaintext
1130 lines
52 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Summarization checker chain\n",
|
|
"This notebook shows some examples of LLMSummarizationCheckerChain in use with different types of texts. It has a few distinct differences from the `LLMCheckerChain`, in that it doesn't have any assumptions to the format of the input text (or summary).\n",
|
|
"Additionally, as the LLMs like to hallucinate when fact checking or get confused by context, it is sometimes beneficial to run the checker multiple times. It does this by feeding the rewritten \"True\" result back on itself, and checking the \"facts\" for truth. As you can see from the examples below, this can be very effective in arriving at a generally true body of text.\n",
|
|
"\n",
|
|
"You can control the number of times the checker runs by setting the `max_checks` parameter. The default is 2, but you can set it to 1 if you don't want any double-checking."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMSummarizationCheckerChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new SequentialChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mGiven some text, extract a list of facts from the text.\n",
|
|
"\n",
|
|
"Format your output as a bulleted list.\n",
|
|
"\n",
|
|
"Text:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Your 9-year old might like these recent discoveries made by The James Webb Space Telescope (JWST):\n",
|
|
"• In 2023, The JWST spotted a number of galaxies nicknamed \"green peas.\" They were given this name because they are small, round, and green, like peas.\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old. This means that the light from these galaxies has been traveling for over 13 billion years to reach us.\n",
|
|
"• JWST took the very first pictures of a planet outside of our own solar system. These distant worlds are called \"exoplanets.\" Exo means \"from outside.\"\n",
|
|
"These discoveries can spark a child's imagination about the infinite wonders of the universe.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Facts:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mYou are an expert fact checker. You have been hired by a major news organization to fact check a very important story.\n",
|
|
"\n",
|
|
"Here is a bullet point list of facts:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"• The James Webb Space Telescope (JWST) spotted a number of galaxies nicknamed \"green peas.\"\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old.\n",
|
|
"• JWST took the very first pictures of a planet outside of our own solar system.\n",
|
|
"• These distant worlds are called \"exoplanets.\"\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"For each fact, determine whether it is true or false about the subject. If you are unable to determine whether the fact is true or false, output \"Undetermined\".\n",
|
|
"If the fact is false, explain why.\n",
|
|
"\n",
|
|
"\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true of false. If the answer is false, a suggestion is given for a correction.\n",
|
|
"\n",
|
|
"Checked Assertions:\n",
|
|
"\"\"\"\n",
|
|
"• The James Webb Space Telescope (JWST) spotted a number of galaxies nicknamed \"green peas.\" - True \n",
|
|
"\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old. - True \n",
|
|
"\n",
|
|
"• JWST took the very first pictures of a planet outside of our own solar system. - False. The first exoplanet was discovered in 1992, before the JWST was launched. \n",
|
|
"\n",
|
|
"• These distant worlds are called \"exoplanets.\" - True\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Original Summary:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Your 9-year old might like these recent discoveries made by The James Webb Space Telescope (JWST):\n",
|
|
"• In 2023, The JWST spotted a number of galaxies nicknamed \"green peas.\" They were given this name because they are small, round, and green, like peas.\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old. This means that the light from these galaxies has been traveling for over 13 billion years to reach us.\n",
|
|
"• JWST took the very first pictures of a planet outside of our own solar system. These distant worlds are called \"exoplanets.\" Exo means \"from outside.\"\n",
|
|
"These discoveries can spark a child's imagination about the infinite wonders of the universe.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Using these checked assertions, rewrite the original summary to be completely true.\n",
|
|
"\n",
|
|
"The output should have the same structure and formatting as the original summary.\n",
|
|
"\n",
|
|
"Summary:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true or false.\n",
|
|
"\n",
|
|
"If all of the assertions are true, return \"True\". If any of the assertions are false, return \"False\".\n",
|
|
"\n",
|
|
"Here are some examples:\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is red: False\n",
|
|
"- Water is made of lava: False\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue: True\n",
|
|
"- Water is wet: True\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: True\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue - True\n",
|
|
"- Water is made of lava- False\n",
|
|
"- The sun is a star - True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions:\"\"\"\n",
|
|
"• The James Webb Space Telescope (JWST) spotted a number of galaxies nicknamed \"green peas.\" - True \n",
|
|
"\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old. - True \n",
|
|
"\n",
|
|
"• JWST took the very first pictures of a planet outside of our own solar system. - False. The first exoplanet was discovered in 1992, before the JWST was launched. \n",
|
|
"\n",
|
|
"• These distant worlds are called \"exoplanets.\" - True\n",
|
|
"\"\"\"\n",
|
|
"Result:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"Your 9-year old might like these recent discoveries made by The James Webb Space Telescope (JWST):\n",
|
|
"• In 2023, The JWST spotted a number of galaxies nicknamed \"green peas.\" They were given this name because they are small, round, and green, like peas.\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old. This means that the light from these galaxies has been traveling for over 13 billion years to reach us.\n",
|
|
"• JWST has provided us with the first images of exoplanets, which are planets outside of our own solar system. These distant worlds were first discovered in 1992, and the JWST has allowed us to see them in greater detail.\n",
|
|
"These discoveries can spark a child's imagination about the infinite wonders of the universe.\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new SequentialChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mGiven some text, extract a list of facts from the text.\n",
|
|
"\n",
|
|
"Format your output as a bulleted list.\n",
|
|
"\n",
|
|
"Text:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"\n",
|
|
"Your 9-year old might like these recent discoveries made by The James Webb Space Telescope (JWST):\n",
|
|
"• In 2023, The JWST spotted a number of galaxies nicknamed \"green peas.\" They were given this name because they are small, round, and green, like peas.\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old. This means that the light from these galaxies has been traveling for over 13 billion years to reach us.\n",
|
|
"• JWST has provided us with the first images of exoplanets, which are planets outside of our own solar system. These distant worlds were first discovered in 1992, and the JWST has allowed us to see them in greater detail.\n",
|
|
"These discoveries can spark a child's imagination about the infinite wonders of the universe.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Facts:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mYou are an expert fact checker. You have been hired by a major news organization to fact check a very important story.\n",
|
|
"\n",
|
|
"Here is a bullet point list of facts:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"• The James Webb Space Telescope (JWST) spotted a number of galaxies nicknamed \"green peas.\"\n",
|
|
"• The light from these galaxies has been traveling for over 13 billion years to reach us.\n",
|
|
"• JWST has provided us with the first images of exoplanets, which are planets outside of our own solar system.\n",
|
|
"• Exoplanets were first discovered in 1992.\n",
|
|
"• The JWST has allowed us to see exoplanets in greater detail.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"For each fact, determine whether it is true or false about the subject. If you are unable to determine whether the fact is true or false, output \"Undetermined\".\n",
|
|
"If the fact is false, explain why.\n",
|
|
"\n",
|
|
"\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true of false. If the answer is false, a suggestion is given for a correction.\n",
|
|
"\n",
|
|
"Checked Assertions:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"• The James Webb Space Telescope (JWST) spotted a number of galaxies nicknamed \"green peas.\" - True \n",
|
|
"\n",
|
|
"• The light from these galaxies has been traveling for over 13 billion years to reach us. - True \n",
|
|
"\n",
|
|
"• JWST has provided us with the first images of exoplanets, which are planets outside of our own solar system. - False. The first exoplanet was discovered in 1992, but the first images of exoplanets were taken by the Hubble Space Telescope in 2004. \n",
|
|
"\n",
|
|
"• Exoplanets were first discovered in 1992. - True \n",
|
|
"\n",
|
|
"• The JWST has allowed us to see exoplanets in greater detail. - Undetermined. The JWST has not yet been launched, so it is not yet known how much detail it will be able to provide.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Original Summary:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"\n",
|
|
"Your 9-year old might like these recent discoveries made by The James Webb Space Telescope (JWST):\n",
|
|
"• In 2023, The JWST spotted a number of galaxies nicknamed \"green peas.\" They were given this name because they are small, round, and green, like peas.\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old. This means that the light from these galaxies has been traveling for over 13 billion years to reach us.\n",
|
|
"• JWST has provided us with the first images of exoplanets, which are planets outside of our own solar system. These distant worlds were first discovered in 1992, and the JWST has allowed us to see them in greater detail.\n",
|
|
"These discoveries can spark a child's imagination about the infinite wonders of the universe.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Using these checked assertions, rewrite the original summary to be completely true.\n",
|
|
"\n",
|
|
"The output should have the same structure and formatting as the original summary.\n",
|
|
"\n",
|
|
"Summary:\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true or false.\n",
|
|
"\n",
|
|
"If all of the assertions are true, return \"True\". If any of the assertions are false, return \"False\".\n",
|
|
"\n",
|
|
"Here are some examples:\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is red: False\n",
|
|
"- Water is made of lava: False\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue: True\n",
|
|
"- Water is wet: True\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: True\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue - True\n",
|
|
"- Water is made of lava- False\n",
|
|
"- The sun is a star - True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions:\"\"\"\n",
|
|
"\n",
|
|
"• The James Webb Space Telescope (JWST) spotted a number of galaxies nicknamed \"green peas.\" - True \n",
|
|
"\n",
|
|
"• The light from these galaxies has been traveling for over 13 billion years to reach us. - True \n",
|
|
"\n",
|
|
"• JWST has provided us with the first images of exoplanets, which are planets outside of our own solar system. - False. The first exoplanet was discovered in 1992, but the first images of exoplanets were taken by the Hubble Space Telescope in 2004. \n",
|
|
"\n",
|
|
"• Exoplanets were first discovered in 1992. - True \n",
|
|
"\n",
|
|
"• The JWST has allowed us to see exoplanets in greater detail. - Undetermined. The JWST has not yet been launched, so it is not yet known how much detail it will be able to provide.\n",
|
|
"\"\"\"\n",
|
|
"Result:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"Your 9-year old might like these recent discoveries made by The James Webb Space Telescope (JWST):\n",
|
|
"• In 2023, The JWST will spot a number of galaxies nicknamed \"green peas.\" They were given this name because they are small, round, and green, like peas.\n",
|
|
"• The telescope will capture images of galaxies that are over 13 billion years old. This means that the light from these galaxies has been traveling for over 13 billion years to reach us.\n",
|
|
"• Exoplanets, which are planets outside of our own solar system, were first discovered in 1992. The JWST will allow us to see them in greater detail when it is launched in 2023.\n",
|
|
"These discoveries can spark a child's imagination about the infinite wonders of the universe.\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'Your 9-year old might like these recent discoveries made by The James Webb Space Telescope (JWST):\\n• In 2023, The JWST will spot a number of galaxies nicknamed \"green peas.\" They were given this name because they are small, round, and green, like peas.\\n• The telescope will capture images of galaxies that are over 13 billion years old. This means that the light from these galaxies has been traveling for over 13 billion years to reach us.\\n• Exoplanets, which are planets outside of our own solar system, were first discovered in 1992. The JWST will allow us to see them in greater detail when it is launched in 2023.\\nThese discoveries can spark a child\\'s imagination about the infinite wonders of the universe.'"
|
|
]
|
|
},
|
|
"execution_count": 1,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain.chains import LLMSummarizationCheckerChain\n",
|
|
"from langchain.llms import OpenAI\n",
|
|
"\n",
|
|
"llm = OpenAI(temperature=0)\n",
|
|
"checker_chain = LLMSummarizationCheckerChain.from_llm(llm, verbose=True, max_checks=2)\n",
|
|
"text = \"\"\"\n",
|
|
"Your 9-year old might like these recent discoveries made by The James Webb Space Telescope (JWST):\n",
|
|
"• In 2023, The JWST spotted a number of galaxies nicknamed \"green peas.\" They were given this name because they are small, round, and green, like peas.\n",
|
|
"• The telescope captured images of galaxies that are over 13 billion years old. This means that the light from these galaxies has been traveling for over 13 billion years to reach us.\n",
|
|
"• JWST took the very first pictures of a planet outside of our own solar system. These distant worlds are called \"exoplanets.\" Exo means \"from outside.\"\n",
|
|
"These discoveries can spark a child's imagination about the infinite wonders of the universe.\"\"\"\n",
|
|
"checker_chain.run(text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMSummarizationCheckerChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new SequentialChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mGiven some text, extract a list of facts from the text.\n",
|
|
"\n",
|
|
"Format your output as a bulleted list.\n",
|
|
"\n",
|
|
"Text:\n",
|
|
"\"\"\"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is one of five oceans in the world, alongside the Pacific Ocean, Atlantic Ocean, Indian Ocean, and the Southern Ocean. It is the smallest of the five oceans and is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the island of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Norwegian Sea.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Facts:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mYou are an expert fact checker. You have been hired by a major news organization to fact check a very important story.\n",
|
|
"\n",
|
|
"Here is a bullet point list of facts:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland.\n",
|
|
"- It has an area of 465,000 square miles.\n",
|
|
"- It is one of five oceans in the world, alongside the Pacific Ocean, Atlantic Ocean, Indian Ocean, and the Southern Ocean.\n",
|
|
"- It is the smallest of the five oceans.\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs.\n",
|
|
"- The sea is named after the island of Greenland.\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic.\n",
|
|
"- It is often frozen over so navigation is limited.\n",
|
|
"- It is considered the northern branch of the Norwegian Sea.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"For each fact, determine whether it is true or false about the subject. If you are unable to determine whether the fact is true or false, output \"Undetermined\".\n",
|
|
"If the fact is false, explain why.\n",
|
|
"\n",
|
|
"\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true of false. If the answer is false, a suggestion is given for a correction.\n",
|
|
"\n",
|
|
"Checked Assertions:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. True\n",
|
|
"\n",
|
|
"- It has an area of 465,000 square miles. True\n",
|
|
"\n",
|
|
"- It is one of five oceans in the world, alongside the Pacific Ocean, Atlantic Ocean, Indian Ocean, and the Southern Ocean. False - The Greenland Sea is not an ocean, it is an arm of the Arctic Ocean.\n",
|
|
"\n",
|
|
"- It is the smallest of the five oceans. False - The Greenland Sea is not an ocean, it is an arm of the Arctic Ocean.\n",
|
|
"\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. True\n",
|
|
"\n",
|
|
"- The sea is named after the island of Greenland. True\n",
|
|
"\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic. True\n",
|
|
"\n",
|
|
"- It is often frozen over so navigation is limited. True\n",
|
|
"\n",
|
|
"- It is considered the northern branch of the Norwegian Sea. True\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Original Summary:\n",
|
|
"\"\"\"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is one of five oceans in the world, alongside the Pacific Ocean, Atlantic Ocean, Indian Ocean, and the Southern Ocean. It is the smallest of the five oceans and is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the island of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Norwegian Sea.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Using these checked assertions, rewrite the original summary to be completely true.\n",
|
|
"\n",
|
|
"The output should have the same structure and formatting as the original summary.\n",
|
|
"\n",
|
|
"Summary:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true or false.\n",
|
|
"\n",
|
|
"If all of the assertions are true, return \"True\". If any of the assertions are false, return \"False\".\n",
|
|
"\n",
|
|
"Here are some examples:\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is red: False\n",
|
|
"- Water is made of lava: False\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue: True\n",
|
|
"- Water is wet: True\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: True\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue - True\n",
|
|
"- Water is made of lava- False\n",
|
|
"- The sun is a star - True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions:\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. True\n",
|
|
"\n",
|
|
"- It has an area of 465,000 square miles. True\n",
|
|
"\n",
|
|
"- It is one of five oceans in the world, alongside the Pacific Ocean, Atlantic Ocean, Indian Ocean, and the Southern Ocean. False - The Greenland Sea is not an ocean, it is an arm of the Arctic Ocean.\n",
|
|
"\n",
|
|
"- It is the smallest of the five oceans. False - The Greenland Sea is not an ocean, it is an arm of the Arctic Ocean.\n",
|
|
"\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. True\n",
|
|
"\n",
|
|
"- The sea is named after the island of Greenland. True\n",
|
|
"\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic. True\n",
|
|
"\n",
|
|
"- It is often frozen over so navigation is limited. True\n",
|
|
"\n",
|
|
"- It is considered the northern branch of the Norwegian Sea. True\n",
|
|
"\"\"\"\n",
|
|
"Result:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is an arm of the Arctic Ocean. It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the island of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Norwegian Sea.\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new SequentialChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mGiven some text, extract a list of facts from the text.\n",
|
|
"\n",
|
|
"Format your output as a bulleted list.\n",
|
|
"\n",
|
|
"Text:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is an arm of the Arctic Ocean. It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the island of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Norwegian Sea.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Facts:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mYou are an expert fact checker. You have been hired by a major news organization to fact check a very important story.\n",
|
|
"\n",
|
|
"Here is a bullet point list of facts:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland.\n",
|
|
"- It has an area of 465,000 square miles.\n",
|
|
"- It is an arm of the Arctic Ocean.\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs.\n",
|
|
"- It is named after the island of Greenland.\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic.\n",
|
|
"- It is often frozen over so navigation is limited.\n",
|
|
"- It is considered the northern branch of the Norwegian Sea.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"For each fact, determine whether it is true or false about the subject. If you are unable to determine whether the fact is true or false, output \"Undetermined\".\n",
|
|
"If the fact is false, explain why.\n",
|
|
"\n",
|
|
"\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true of false. If the answer is false, a suggestion is given for a correction.\n",
|
|
"\n",
|
|
"Checked Assertions:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. True\n",
|
|
"\n",
|
|
"- It has an area of 465,000 square miles. True\n",
|
|
"\n",
|
|
"- It is an arm of the Arctic Ocean. True\n",
|
|
"\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. True\n",
|
|
"\n",
|
|
"- It is named after the island of Greenland. False - It is named after the country of Greenland.\n",
|
|
"\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic. True\n",
|
|
"\n",
|
|
"- It is often frozen over so navigation is limited. True\n",
|
|
"\n",
|
|
"- It is considered the northern branch of the Norwegian Sea. False - It is considered the northern branch of the Atlantic Ocean.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Original Summary:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is an arm of the Arctic Ocean. It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the island of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Norwegian Sea.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Using these checked assertions, rewrite the original summary to be completely true.\n",
|
|
"\n",
|
|
"The output should have the same structure and formatting as the original summary.\n",
|
|
"\n",
|
|
"Summary:\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true or false.\n",
|
|
"\n",
|
|
"If all of the assertions are true, return \"True\". If any of the assertions are false, return \"False\".\n",
|
|
"\n",
|
|
"Here are some examples:\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is red: False\n",
|
|
"- Water is made of lava: False\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue: True\n",
|
|
"- Water is wet: True\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: True\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue - True\n",
|
|
"- Water is made of lava- False\n",
|
|
"- The sun is a star - True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions:\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. True\n",
|
|
"\n",
|
|
"- It has an area of 465,000 square miles. True\n",
|
|
"\n",
|
|
"- It is an arm of the Arctic Ocean. True\n",
|
|
"\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. True\n",
|
|
"\n",
|
|
"- It is named after the island of Greenland. False - It is named after the country of Greenland.\n",
|
|
"\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic. True\n",
|
|
"\n",
|
|
"- It is often frozen over so navigation is limited. True\n",
|
|
"\n",
|
|
"- It is considered the northern branch of the Norwegian Sea. False - It is considered the northern branch of the Atlantic Ocean.\n",
|
|
"\"\"\"\n",
|
|
"Result:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is an arm of the Arctic Ocean. It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the country of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Atlantic Ocean.\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new SequentialChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mGiven some text, extract a list of facts from the text.\n",
|
|
"\n",
|
|
"Format your output as a bulleted list.\n",
|
|
"\n",
|
|
"Text:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is an arm of the Arctic Ocean. It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the country of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Atlantic Ocean.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Facts:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mYou are an expert fact checker. You have been hired by a major news organization to fact check a very important story.\n",
|
|
"\n",
|
|
"Here is a bullet point list of facts:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland.\n",
|
|
"- It has an area of 465,000 square miles.\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs.\n",
|
|
"- The sea is named after the country of Greenland.\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic.\n",
|
|
"- It is often frozen over so navigation is limited.\n",
|
|
"- It is considered the northern branch of the Atlantic Ocean.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"For each fact, determine whether it is true or false about the subject. If you are unable to determine whether the fact is true or false, output \"Undetermined\".\n",
|
|
"If the fact is false, explain why.\n",
|
|
"\n",
|
|
"\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true of false. If the answer is false, a suggestion is given for a correction.\n",
|
|
"\n",
|
|
"Checked Assertions:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. True\n",
|
|
"\n",
|
|
"- It has an area of 465,000 square miles. True\n",
|
|
"\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. True\n",
|
|
"\n",
|
|
"- The sea is named after the country of Greenland. True\n",
|
|
"\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic. False - The Arctic Ocean's main outlet to the Atlantic is the Barents Sea.\n",
|
|
"\n",
|
|
"- It is often frozen over so navigation is limited. True\n",
|
|
"\n",
|
|
"- It is considered the northern branch of the Atlantic Ocean. False - The Greenland Sea is considered part of the Arctic Ocean, not the Atlantic Ocean.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Original Summary:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is an arm of the Arctic Ocean. It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the country of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Atlantic Ocean.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Using these checked assertions, rewrite the original summary to be completely true.\n",
|
|
"\n",
|
|
"The output should have the same structure and formatting as the original summary.\n",
|
|
"\n",
|
|
"Summary:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true or false.\n",
|
|
"\n",
|
|
"If all of the assertions are true, return \"True\". If any of the assertions are false, return \"False\".\n",
|
|
"\n",
|
|
"Here are some examples:\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is red: False\n",
|
|
"- Water is made of lava: False\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue: True\n",
|
|
"- Water is wet: True\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: True\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue - True\n",
|
|
"- Water is made of lava- False\n",
|
|
"- The sun is a star - True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions:\"\"\"\n",
|
|
"\n",
|
|
"- The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. True\n",
|
|
"\n",
|
|
"- It has an area of 465,000 square miles. True\n",
|
|
"\n",
|
|
"- It is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. True\n",
|
|
"\n",
|
|
"- The sea is named after the country of Greenland. True\n",
|
|
"\n",
|
|
"- It is the Arctic Ocean's main outlet to the Atlantic. False - The Arctic Ocean's main outlet to the Atlantic is the Barents Sea.\n",
|
|
"\n",
|
|
"- It is often frozen over so navigation is limited. True\n",
|
|
"\n",
|
|
"- It is considered the northern branch of the Atlantic Ocean. False - The Greenland Sea is considered part of the Arctic Ocean, not the Atlantic Ocean.\n",
|
|
"\"\"\"\n",
|
|
"Result:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the country of Greenland, and is the Arctic Ocean's main outlet to the Barents Sea. It is often frozen over so navigation is limited, and is considered part of the Arctic Ocean.\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"\"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the country of Greenland, and is the Arctic Ocean's main outlet to the Barents Sea. It is often frozen over so navigation is limited, and is considered part of the Arctic Ocean.\""
|
|
]
|
|
},
|
|
"execution_count": 2,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain.chains import LLMSummarizationCheckerChain\n",
|
|
"from langchain.llms import OpenAI\n",
|
|
"\n",
|
|
"llm = OpenAI(temperature=0)\n",
|
|
"checker_chain = LLMSummarizationCheckerChain.from_llm(llm, verbose=True, max_checks=3)\n",
|
|
"text = \"The Greenland Sea is an outlying portion of the Arctic Ocean located between Iceland, Norway, the Svalbard archipelago and Greenland. It has an area of 465,000 square miles and is one of five oceans in the world, alongside the Pacific Ocean, Atlantic Ocean, Indian Ocean, and the Southern Ocean. It is the smallest of the five oceans and is covered almost entirely by water, some of which is frozen in the form of glaciers and icebergs. The sea is named after the island of Greenland, and is the Arctic Ocean's main outlet to the Atlantic. It is often frozen over so navigation is limited, and is considered the northern branch of the Norwegian Sea.\"\n",
|
|
"checker_chain.run(text)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMSummarizationCheckerChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new SequentialChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mGiven some text, extract a list of facts from the text.\n",
|
|
"\n",
|
|
"Format your output as a bulleted list.\n",
|
|
"\n",
|
|
"Text:\n",
|
|
"\"\"\"\n",
|
|
"Mammals can lay eggs, birds can lay eggs, therefore birds are mammals.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Facts:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mYou are an expert fact checker. You have been hired by a major news organization to fact check a very important story.\n",
|
|
"\n",
|
|
"Here is a bullet point list of facts:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- Mammals can lay eggs\n",
|
|
"- Birds can lay eggs\n",
|
|
"- Birds are mammals\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"For each fact, determine whether it is true or false about the subject. If you are unable to determine whether the fact is true or false, output \"Undetermined\".\n",
|
|
"If the fact is false, explain why.\n",
|
|
"\n",
|
|
"\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true of false. If the answer is false, a suggestion is given for a correction.\n",
|
|
"\n",
|
|
"Checked Assertions:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- Mammals can lay eggs: False. Mammals are not capable of laying eggs, as they give birth to live young.\n",
|
|
"\n",
|
|
"- Birds can lay eggs: True. Birds are capable of laying eggs.\n",
|
|
"\n",
|
|
"- Birds are mammals: False. Birds are not mammals, they are a class of their own.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Original Summary:\n",
|
|
"\"\"\"\n",
|
|
"Mammals can lay eggs, birds can lay eggs, therefore birds are mammals.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Using these checked assertions, rewrite the original summary to be completely true.\n",
|
|
"\n",
|
|
"The output should have the same structure and formatting as the original summary.\n",
|
|
"\n",
|
|
"Summary:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true or false.\n",
|
|
"\n",
|
|
"If all of the assertions are true, return \"True\". If any of the assertions are false, return \"False\".\n",
|
|
"\n",
|
|
"Here are some examples:\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is red: False\n",
|
|
"- Water is made of lava: False\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue: True\n",
|
|
"- Water is wet: True\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: True\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue - True\n",
|
|
"- Water is made of lava- False\n",
|
|
"- The sun is a star - True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions:\"\"\"\n",
|
|
"\n",
|
|
"- Mammals can lay eggs: False. Mammals are not capable of laying eggs, as they give birth to live young.\n",
|
|
"\n",
|
|
"- Birds can lay eggs: True. Birds are capable of laying eggs.\n",
|
|
"\n",
|
|
"- Birds are mammals: False. Birds are not mammals, they are a class of their own.\n",
|
|
"\"\"\"\n",
|
|
"Result:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
" Birds and mammals are both capable of laying eggs, however birds are not mammals, they are a class of their own.\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new SequentialChain chain...\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mGiven some text, extract a list of facts from the text.\n",
|
|
"\n",
|
|
"Format your output as a bulleted list.\n",
|
|
"\n",
|
|
"Text:\n",
|
|
"\"\"\"\n",
|
|
" Birds and mammals are both capable of laying eggs, however birds are not mammals, they are a class of their own.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Facts:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mYou are an expert fact checker. You have been hired by a major news organization to fact check a very important story.\n",
|
|
"\n",
|
|
"Here is a bullet point list of facts:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- Birds and mammals are both capable of laying eggs.\n",
|
|
"- Birds are not mammals.\n",
|
|
"- Birds are a class of their own.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"For each fact, determine whether it is true or false about the subject. If you are unable to determine whether the fact is true or false, output \"Undetermined\".\n",
|
|
"If the fact is false, explain why.\n",
|
|
"\n",
|
|
"\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true of false. If the answer is false, a suggestion is given for a correction.\n",
|
|
"\n",
|
|
"Checked Assertions:\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"- Birds and mammals are both capable of laying eggs: False. Mammals give birth to live young, while birds lay eggs.\n",
|
|
"\n",
|
|
"- Birds are not mammals: True. Birds are a class of their own, separate from mammals.\n",
|
|
"\n",
|
|
"- Birds are a class of their own: True. Birds are a class of their own, separate from mammals.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Original Summary:\n",
|
|
"\"\"\"\n",
|
|
" Birds and mammals are both capable of laying eggs, however birds are not mammals, they are a class of their own.\n",
|
|
"\"\"\"\n",
|
|
"\n",
|
|
"Using these checked assertions, rewrite the original summary to be completely true.\n",
|
|
"\n",
|
|
"The output should have the same structure and formatting as the original summary.\n",
|
|
"\n",
|
|
"Summary:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\n",
|
|
"\u001b[1m> Entering new LLMChain chain...\u001b[0m\n",
|
|
"Prompt after formatting:\n",
|
|
"\u001b[32;1m\u001b[1;3mBelow are some assertions that have been fact checked and are labeled as true or false.\n",
|
|
"\n",
|
|
"If all of the assertions are true, return \"True\". If any of the assertions are false, return \"False\".\n",
|
|
"\n",
|
|
"Here are some examples:\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is red: False\n",
|
|
"- Water is made of lava: False\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue: True\n",
|
|
"- Water is wet: True\n",
|
|
"- The sun is a star: True\n",
|
|
"\"\"\"\n",
|
|
"Result: True\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions: \"\"\"\n",
|
|
"- The sky is blue - True\n",
|
|
"- Water is made of lava- False\n",
|
|
"- The sun is a star - True\n",
|
|
"\"\"\"\n",
|
|
"Result: False\n",
|
|
"\n",
|
|
"===\n",
|
|
"\n",
|
|
"Checked Assertions:\"\"\"\n",
|
|
"\n",
|
|
"- Birds and mammals are both capable of laying eggs: False. Mammals give birth to live young, while birds lay eggs.\n",
|
|
"\n",
|
|
"- Birds are not mammals: True. Birds are a class of their own, separate from mammals.\n",
|
|
"\n",
|
|
"- Birds are a class of their own: True. Birds are a class of their own, separate from mammals.\n",
|
|
"\"\"\"\n",
|
|
"Result:\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n",
|
|
"\n",
|
|
"\u001b[1m> Finished chain.\u001b[0m\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'Birds are not mammals, but they are a class of their own. They lay eggs, unlike mammals which give birth to live young.'"
|
|
]
|
|
},
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain.chains import LLMSummarizationCheckerChain\n",
|
|
"from langchain.llms import OpenAI\n",
|
|
"\n",
|
|
"llm = OpenAI(temperature=0)\n",
|
|
"checker_chain = LLMSummarizationCheckerChain.from_llm(llm, max_checks=3, verbose=True)\n",
|
|
"text = \"Mammals can lay eggs, birds can lay eggs, therefore birds are mammals.\"\n",
|
|
"checker_chain.run(text)"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.3"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 4
|
|
}
|