Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-10-29 17:07:25 +00:00
Code Issues Packages Projects Releases Wiki Activity
c087ce74f7
langchain/tests/integration_tests/chains/test_cpal.py
Boris 9129318466
CPAL (#6255) 
# Causal program-aided language (CPAL) chain

## Motivation

This builds on the recent [PAL](https://arxiv.org/abs/2211.10435) to
stop LLM hallucination. The problem with the
[PAL](https://arxiv.org/abs/2211.10435) approach is that it hallucinates
on a math problem with a nested chain of dependence. The innovation here
is that this new CPAL approach includes causal structure to fix
hallucination.

For example, using the below word problem, PAL answers with 5, and CPAL
answers with 13.

    "Tim buys the same number of pets as Cindy and Boris."
    "Cindy buys the same number of pets as Bill plus Bob."
    "Boris buys the same number of pets as Ben plus Beth."
    "Bill buys the same number of pets as Obama."
    "Bob buys the same number of pets as Obama."
    "Ben buys the same number of pets as Obama."
    "Beth buys the same number of pets as Obama."
    "If Obama buys one pet, how many pets total does everyone buy?"

The CPAL chain represents the causal structure of the above narrative as
a causal graph or DAG, which it can also plot, as shown below.


![complex-graph](https://github.com/hwchase17/langchain/assets/367522/d938db15-f941-493d-8605-536ad530f576)

.

The two major sections below are:

1. Technical overview
2. Future application

Also see [this jupyter
notebook](https://github.com/borisdev/langchain/blob/master/docs/extras/modules/chains/additional/cpal.ipynb)
doc.


## 1. Technical overview

### CPAL versus PAL

Like [PAL](https://arxiv.org/abs/2211.10435), CPAL intends to reduce
large language model (LLM) hallucination.

The CPAL chain is different from the PAL chain for a couple of reasons. 

* CPAL adds a causal structure (or DAG) to link entity actions (or math
expressions).
* The CPAL math expressions are modeling a chain of cause and effect
relations, which can be intervened upon, whereas for the PAL chain math
expressions are projected math identities.

PAL's generated python code is wrong. It hallucinates when complexity
increases.

```python
def solution():
    """Tim buys the same number of pets as Cindy and Boris.Cindy buys the same number of pets as Bill plus Bob.Boris buys the same number of pets as Ben plus Beth.Bill buys the same number of pets as Obama.Bob buys the same number of pets as Obama.Ben buys the same number of pets as Obama.Beth buys the same number of pets as Obama.If Obama buys one pet, how many pets total does everyone buy?"""
    obama_pets = 1
    tim_pets = obama_pets
    cindy_pets = obama_pets + obama_pets
    boris_pets = obama_pets + obama_pets
    total_pets = tim_pets + cindy_pets + boris_pets
    result = total_pets
    return result  # math result is 5
```

CPAL's generated python code is correct.

```python
story outcome data
    name                                   code  value      depends_on
0  obama                                   pass    1.0              []
1   bill               bill.value = obama.value    1.0         [obama]
2    bob                bob.value = obama.value    1.0         [obama]
3    ben                ben.value = obama.value    1.0         [obama]
4   beth               beth.value = obama.value    1.0         [obama]
5  cindy   cindy.value = bill.value + bob.value    2.0     [bill, bob]
6  boris   boris.value = ben.value + beth.value    2.0     [ben, beth]
7    tim  tim.value = cindy.value + boris.value    4.0  [cindy, boris]

query data
{
    "question": "how many pets total does everyone buy?",
    "expression": "SELECT SUM(value) FROM df",
    "llm_error_msg": ""
}
# query result is 13
```

Based on the comments below, CPAL's intended location in the library is
`experimental/chains/cpal` and PAL's location is`chains/pal`.

### CPAL vs Graph QA

Both the CPAL chain and the Graph QA chain extract entity-action-entity
relations into a DAG.

The CPAL chain is different from the Graph QA chain for a few reasons.

* Graph QA does not connect entities to math expressions
* Graph QA does not associate actions in a sequence of dependence.
* Graph QA does not decompose the narrative into these three parts:
  1. Story plot or causal model
  4. Hypothetical question
  5. Hypothetical condition 

### Evaluation

Preliminary evaluation on simple math word problems shows that this CPAL
chain generates less hallucination than the PAL chain on answering
questions about a causal narrative. Two examples are in [this jupyter
notebook](https://github.com/borisdev/langchain/blob/master/docs/extras/modules/chains/additional/cpal.ipynb)
doc.

## 2. Future application

### "Describe as Narrative, Test as Code"

The thesis here is that the Describe as Narrative, Test as Code approach
allows you to represent a causal mental model both as code and as a
narrative, giving you the best of both worlds.

#### Why describe a causal mental mode as a narrative?

The narrative form is quick. At a consensus building meeting, people use
narratives to persuade others of their causal mental model, aka. plan.
You can share, version control and index a narrative.

#### Why test a causal mental model as a code?

Code is testable, complex narratives are not. Though fast, narratives
are problematic as their complexity increases. The problem is LLMs and
humans are prone to hallucination when predicting the outcomes of a
narrative. The cost of building a consensus around the validity of a
narrative outcome grows as its narrative complexity increases. Code does
not require tribal knowledge or social power to validate.

Code is composable, complex narratives are not. The answer of one CPAL
chain can be the hypothetical conditions of another CPAL Chain. For
stochastic simulations, a composable plan can be integrated with the
[DoWhy library](https://github.com/py-why/dowhy). Lastly, for the
futuristic folk, a composable plan as code allows ordinary community
folk to design a plan that can be integrated with a blockchain for
funding.

An explanation of a dependency planning application is
[here.](https://github.com/borisdev/cpal-llm-chain-demo)

--- 
Twitter handle: @boris_dev

---------

Co-authored-by: Boris Dev <borisdev@Boriss-MacBook-Air.local>
2023-07-11 10:11:21 -04:00

555 lines
21 KiB
Python
Raw Blame History

"""Test CPAL chain."""
import json
import unittest
from typing import Type
from unittest import mock
import pydantic
import pytest
from langchain import OpenAI
from langchain.experimental.cpal.base import (
CausalChain,
CPALChain,
InterventionChain,
NarrativeChain,
QueryChain,
)
from langchain.experimental.cpal.constants import Constant
from langchain.experimental.cpal.models import (
CausalModel,
EntityModel,
EntitySettingModel,
InterventionModel,
NarrativeModel,
QueryModel,
)
from langchain.experimental.cpal.templates.univariate.causal import (
template as causal_template,
)
from langchain.experimental.cpal.templates.univariate.intervention import (
template as intervention_template,
)
from langchain.experimental.cpal.templates.univariate.narrative import (
template as narrative_template,
)
from langchain.experimental.cpal.templates.univariate.query import (
template as query_template,
)
from langchain.output_parsers import PydanticOutputParser
from langchain.prompts.prompt import PromptTemplate
from tests.unit_tests.llms.fake_llm import FakeLLM
class TestUnitCPALChain_MathWordProblems(unittest.TestCase):
"""Unit Test the CPAL chain and its component chains on math word problems.
These tests can't run in the standard unit test directory because of
this issue, https://github.com/hwchase17/langchain/issues/7451
"""
def setUp(self) -> None:
self.fake_llm = self.make_fake_llm()
def make_fake_llm(self) -> FakeLLM:
"""
Fake LLM service for testing CPAL chain and its components chains
on univariate math examples.
"""
class LLMMockData(pydantic.BaseModel):
question: str
completion: str
template: str
data_model: Type[pydantic.BaseModel]
@property
def prompt(self) -> str:
"""Create LLM prompt with the question."""
prompt_template = PromptTemplate(
input_variables=[Constant.narrative_input.value],
template=self.template,
partial_variables={
"format_instructions": PydanticOutputParser(
pydantic_object=self.data_model
).get_format_instructions()
},
)
prompt = prompt_template.format(narrative_input=self.question)
return prompt
narrative = LLMMockData(
**{
"question": (
"jan has three times the number of pets as marcia. "
"marcia has two more pets than cindy."
"if cindy has ten pets, how many pets does jan have? "
),
"completion": json.dumps(
{
"story_outcome_question": "how many pets does jan have? ",
"story_hypothetical": "if cindy has ten pets",
"story_plot": "jan has three times the number of pets as marcia. marcia has two more pets than cindy.", # noqa: E501
}
),
"template": narrative_template,
"data_model": NarrativeModel,
}
)
causal_model = LLMMockData(
**{
"question": (
"jan has three times the number of pets as marcia. "
"marcia has two more pets than cindy."
),
"completion": (
"\n"
"{\n"
' "attribute": "pet_count",\n'
' "entities": [\n'
" {\n"
' "name": "cindy",\n'
' "value": 0,\n'
' "depends_on": [],\n'
' "code": "pass"\n'
" },\n"
" {\n"
' "name": "marcia",\n'
' "value": 0,\n'
' "depends_on": ["cindy"],\n'
' "code": "marcia.value = cindy.value + 2"\n'
" },\n"
" {\n"
' "name": "jan",\n'
' "value": 0,\n'
' "depends_on": ["marcia"],\n'
' "code": "jan.value = marcia.value * 3"\n'
" }\n"
" ]\n"
"}"
),
"template": causal_template,
"data_model": CausalModel,
}
)
intervention = LLMMockData(
**{
"question": ("if cindy has ten pets"),
"completion": (
"{\n"
' "entity_settings" : [\n'
' { "name": "cindy", "attribute": "pet_count", "value": "10" }\n' # noqa: E501
" ]\n"
"}"
),
"template": intervention_template,
"data_model": InterventionModel,
}
)
query = LLMMockData(
**{
"question": ("how many pets does jan have? "),
"completion": (
"{\n"
' "narrative_input": "how many pets does jan have? ",\n'
' "llm_error_msg": "",\n'
' "expression": "SELECT name, value FROM df WHERE name = \'jan\'"\n' # noqa: E501
"}"
),
"template": query_template,
"data_model": QueryModel,
}
)
fake_llm = FakeLLM()
fake_llm.queries = {}
for mock_data in [narrative, causal_model, intervention, query]:
fake_llm.queries.update({mock_data.prompt: mock_data.completion})
return fake_llm
def test_narrative_chain(self) -> None:
"""Test narrative chain returns the three main elements of the causal
narrative as a pydantic object.
"""
narrative_chain = NarrativeChain.from_univariate_prompt(llm=self.fake_llm)
output = narrative_chain(
(
"jan has three times the number of pets as marcia. "
"marcia has two more pets than cindy."
"if cindy has ten pets, how many pets does jan have? "
)
)
expected_output = {
"chain_answer": None,
"chain_data": NarrativeModel(
story_outcome_question="how many pets does jan have? ",
story_hypothetical="if cindy has ten pets",
story_plot="jan has three times the number of pets as marcia. marcia has two more pets than cindy.", # noqa: E501
),
"narrative_input": "jan has three times the number of pets as marcia. marcia " # noqa: E501
"has two more pets than cindy.if cindy has ten pets, how "
"many pets does jan have? ",
}
assert output == expected_output
def test_causal_chain(self) -> None:
"""
Test causal chain returns a DAG as a pydantic object.
"""
causal_chain = CausalChain.from_univariate_prompt(llm=self.fake_llm)
output = causal_chain(
(
"jan has three times the number of pets as "
"marcia. marcia has two more pets than cindy."
)
)
expected_output = {
"chain_answer": None,
"chain_data": CausalModel(
attribute="pet_count",
entities=[
EntityModel(name="cindy", code="pass", value=0.0, depends_on=[]),
EntityModel(
name="marcia",
code="marcia.value = cindy.value + 2",
value=0.0,
depends_on=["cindy"],
),
EntityModel(
name="jan",
code="jan.value = marcia.value * 3",
value=0.0,
depends_on=["marcia"],
),
],
),
"narrative_input": "jan has three times the number of pets as marcia. marcia " # noqa: E501
"has two more pets than cindy.",
}
assert output == expected_output
def test_intervention_chain(self) -> None:
"""
Test intervention chain correctly transforms
the LLM's text completion into a setting-like object.
"""
intervention_chain = InterventionChain.from_univariate_prompt(llm=self.fake_llm)
output = intervention_chain("if cindy has ten pets")
expected_output = {
"chain_answer": None,
"chain_data": InterventionModel(
entity_settings=[
EntitySettingModel(name="cindy", attribute="pet_count", value=10),
]
),
"narrative_input": "if cindy has ten pets",
}
assert output == expected_output
def test_query_chain(self) -> None:
"""
Test query chain correctly transforms
the LLM's text completion into a query-like object.
"""
query_chain = QueryChain.from_univariate_prompt(llm=self.fake_llm)
output = query_chain("how many pets does jan have? ")
expected_output = {
"chain_answer": None,
"chain_data": QueryModel(
narrative_input="how many pets does jan have? ",
llm_error_msg="",
expression="SELECT name, value FROM df WHERE name = 'jan'",
),
"narrative_input": "how many pets does jan have? ",
}
assert output == expected_output
def test_cpal_chain(self) -> None:
"""
patch required since `networkx` package is not part of unit test environment
"""
with mock.patch(
"langchain.experimental.cpal.models.NetworkxEntityGraph"
) as mock_networkx:
graph_instance = mock_networkx.return_value
graph_instance.get_topological_sort.return_value = [
"cindy",
"marcia",
"jan",
]
cpal_chain = CPALChain.from_univariate_prompt(
llm=self.fake_llm, verbose=True
)
cpal_chain.run(
(
"jan has three times the number of pets as "
"marcia. marcia has two more pets than cindy."
"if cindy has ten pets, how many pets does jan have? "
)
)
class TestCPALChain_MathWordProblems(unittest.TestCase):
"""Test the CPAL chain and its component chains on math word problems."""
def test_causal_chain(self) -> None:
"""Test CausalChain can translate a narrative's plot into a causal model
containing operations linked by a DAG."""
llm = OpenAI(temperature=0, max_tokens=512)
casual_chain = CausalChain.from_univariate_prompt(llm)
narrative_plot = (
"Jan has three times the number of pets as Marcia. "
"Marcia has two more pets than Cindy. "
)
output = casual_chain(narrative_plot)
expected_output = {
"chain_answer": None,
"chain_data": CausalModel(
attribute="pet_count",
entities=[
EntityModel(name="cindy", code="pass", value=0.0, depends_on=[]),
EntityModel(
name="marcia",
code="marcia.value = cindy.value + 2",
value=0.0,
depends_on=["cindy"],
),
EntityModel(
name="jan",
code="jan.value = marcia.value * 3",
value=0.0,
depends_on=["marcia"],
),
],
),
"narrative_input": "Jan has three times the number of pets as Marcia. Marcia " # noqa: E501
"has two more pets than Cindy. ",
}
self.assertDictEqual(output, expected_output)
self.assertEqual(
isinstance(output[Constant.chain_data.value], CausalModel), True
)
def test_intervention_chain(self) -> None:
"""Test InterventionChain translates a hypothetical into a new value setting."""
llm = OpenAI(temperature=0, max_tokens=512)
story_conditions_chain = InterventionChain.from_univariate_prompt(llm)
question = "if cindy has ten pets"
data = story_conditions_chain(question)[Constant.chain_data.value]
self.assertEqual(type(data), InterventionModel)
def test_intervention_chain_2(self) -> None:
"""Test InterventionChain translates a hypothetical into a new value setting."""
llm = OpenAI(temperature=0, max_tokens=512)
story_conditions_chain = InterventionChain.from_univariate_prompt(llm)
narrative_condition = "What if Cindy has ten pets and Boris has 5 pets? "
data = story_conditions_chain(narrative_condition)[Constant.chain_data.value]
self.assertEqual(type(data), InterventionModel)
def test_query_chain(self) -> None:
"""Test QueryChain translates a question into a query expression."""
llm = OpenAI(temperature=0, max_tokens=512)
query_chain = QueryChain.from_univariate_prompt(llm)
narrative_question = "How many pets will Marcia end up with? "
data = query_chain(narrative_question)[Constant.chain_data.value]
self.assertEqual(type(data), QueryModel)
def test_narrative_chain(self) -> None:
"""Test NarrativeChain decomposes a human's narrative into three story elements:
- causal model
- intervention model
- query model
"""
narrative = (
"Jan has three times the number of pets as Marcia. "
"Marcia has two more pets than Cindy. "
"If Cindy has ten pets, how many pets does Jan have? "
)
llm = OpenAI(temperature=0, max_tokens=512)
narrative_chain = NarrativeChain.from_univariate_prompt(llm)
data = narrative_chain(narrative)[Constant.chain_data.value]
self.assertEqual(type(data), NarrativeModel)
out = narrative_chain(narrative)
expected_narrative_out = {
"chain_answer": None,
"chain_data": NarrativeModel(
story_outcome_question="how many pets does Jan have?",
story_hypothetical="If Cindy has ten pets",
story_plot="Jan has three times the number of pets as Marcia. Marcia has two more pets than Cindy.", # noqa: E501
),
"narrative_input": "Jan has three times the number of pets as Marcia. Marcia " # noqa: E501
"has two more pets than Cindy. If Cindy has ten pets, how "
"many pets does Jan have? ",
}
self.assertDictEqual(out, expected_narrative_out)
def test_against_pal_chain_doc(self) -> None:
"""
Test CPAL chain against the first example in the PAL chain notebook doc:
https://github.com/hwchase17/langchain/blob/master/docs/extras/modules/chains/additional/pal.ipynb
"""
narrative_input = (
"Jan has three times the number of pets as Marcia."
" Marcia has two more pets than Cindy."
" If Cindy has four pets, how many total pets do the three have?"
)
llm = OpenAI(temperature=0, max_tokens=512)
cpal_chain = CPALChain.from_univariate_prompt(llm=llm, verbose=True)
answer = cpal_chain.run(narrative_input)
"""
>>> story._outcome_table
name code value depends_on
0 cindy pass 4.0 []
1 marcia marcia.value = cindy.value + 2 6.0 [cindy]
2 jan jan.value = marcia.value * 3 18.0 [marcia]
"""
self.assertEqual(answer, 28.0)
def test_simple(self) -> None:
"""
Given a simple math word problem here we are test and illustrate the
the data structures that are produced by the CPAL chain.
"""
narrative_input = (
"jan has three times the number of pets as marcia."
"marcia has two more pets than cindy."
"If cindy has ten pets, how many pets does jan have?"
)
llm = OpenAI(temperature=0, max_tokens=512)
cpal_chain = CPALChain.from_univariate_prompt(llm=llm, verbose=True)
output = cpal_chain(narrative_input)
data = output[Constant.chain_data.value]
expected_output = {
"causal_operations": {
"attribute": "pet_count",
"entities": [
{"code": "pass", "depends_on": [], "name": "cindy", "value": 10.0},
{
"code": "marcia.value = cindy.value + 2",
"depends_on": ["cindy"],
"name": "marcia",
"value": 12.0,
},
{
"code": "jan.value = marcia.value * 3",
"depends_on": ["marcia"],
"name": "jan",
"value": 36.0,
},
],
},
"intervention": {
"entity_settings": [
{"attribute": "pet_count", "name": "cindy", "value": 10.0}
],
"system_settings": None,
},
"query": {
"expression": "SELECT name, value FROM df WHERE name = 'jan'",
"llm_error_msg": "",
"question": "how many pets does jan have?",
},
}
self.assertDictEqual(data.dict(), expected_output)
"""
Illustrate the query model's result table as a printed pandas dataframe
>>> data._outcome_table
name code value depends_on
0 cindy pass 10.0 []
1 marcia marcia.value = cindy.value + 2 12.0 [cindy]
2 jan jan.value = marcia.value * 3 36.0 [marcia]
"""
expected_output = {
"code": {
0: "pass",
1: "marcia.value = cindy.value + 2",
2: "jan.value = marcia.value * 3",
},
"depends_on": {0: [], 1: ["cindy"], 2: ["marcia"]},
"name": {0: "cindy", 1: "marcia", 2: "jan"},
"value": {0: 10.0, 1: 12.0, 2: 36.0},
}
self.assertDictEqual(data._outcome_table.to_dict(), expected_output)
expected_output = {"name": {0: "jan"}, "value": {0: 36.0}}
self.assertDictEqual(data.query._result_table.to_dict(), expected_output)
# TODO: use an LLM chain to translate numbers to words
df = data.query._result_table
expr = "name == 'jan'"
answer = df.query(expr).iloc[0]["value"]
self.assertEqual(float(answer), 36.0)
def test_hallucinating(self) -> None:
"""
Test CPAL approach does not hallucinate when given
an invalid entity in the question.
The PAL chain would hallucinates here!
"""
narrative_input = (
"Jan has three times the number of pets as Marcia."
"Marcia has two more pets than Cindy."
"If Cindy has ten pets, how many pets does Barak have?"
)
llm = OpenAI(temperature=0, max_tokens=512)
cpal_chain = CPALChain.from_univariate_prompt(llm=llm, verbose=True)
with pytest.raises(Exception) as e_info:
print(e_info)
cpal_chain.run(narrative_input)
def test_causal_mediator(self) -> None:
"""
Test CPAL approach on causal mediator.
"""
narrative_input = (
"jan has three times the number of pets as marcia."
"marcia has two more pets than cindy."
"If marcia has ten pets, how many pets does jan have?"
)
llm = OpenAI(temperature=0, max_tokens=512)
cpal_chain = CPALChain.from_univariate_prompt(llm=llm, verbose=True)
answer = cpal_chain.run(narrative_input)
self.assertEqual(answer, 30.0)
@pytest.mark.skip(reason="requires manual install of debian and py packages")
def test_draw(self) -> None:
"""
Test CPAL chain can draw its resulting DAG.
"""
import os
narrative_input = (
"Jan has three times the number of pets as Marcia."
"Marcia has two more pets than Cindy."
"If Marcia has ten pets, how many pets does Jan have?"
)
llm = OpenAI(temperature=0, max_tokens=512)
cpal_chain = CPALChain.from_univariate_prompt(llm=llm, verbose=True)
cpal_chain.run(narrative_input)
path = "graph.svg"
cpal_chain.draw(path=path)
self.assertTrue(os.path.exists(path))
Reference in New Issue View Git Blame Copy Permalink