2023-10-17 21:24:51 +00:00
|
|
|
# ruff: noqa: E501
|
CPAL (#6255)
# Causal program-aided language (CPAL) chain
## Motivation
This builds on the recent [PAL](https://arxiv.org/abs/2211.10435) to
stop LLM hallucination. The problem with the
[PAL](https://arxiv.org/abs/2211.10435) approach is that it hallucinates
on a math problem with a nested chain of dependence. The innovation here
is that this new CPAL approach includes causal structure to fix
hallucination.
For example, using the below word problem, PAL answers with 5, and CPAL
answers with 13.
"Tim buys the same number of pets as Cindy and Boris."
"Cindy buys the same number of pets as Bill plus Bob."
"Boris buys the same number of pets as Ben plus Beth."
"Bill buys the same number of pets as Obama."
"Bob buys the same number of pets as Obama."
"Ben buys the same number of pets as Obama."
"Beth buys the same number of pets as Obama."
"If Obama buys one pet, how many pets total does everyone buy?"
The CPAL chain represents the causal structure of the above narrative as
a causal graph or DAG, which it can also plot, as shown below.
.
The two major sections below are:
1. Technical overview
2. Future application
Also see [this jupyter
notebook](https://github.com/borisdev/langchain/blob/master/docs/extras/modules/chains/additional/cpal.ipynb)
doc.
## 1. Technical overview
### CPAL versus PAL
Like [PAL](https://arxiv.org/abs/2211.10435), CPAL intends to reduce
large language model (LLM) hallucination.
The CPAL chain is different from the PAL chain for a couple of reasons.
* CPAL adds a causal structure (or DAG) to link entity actions (or math
expressions).
* The CPAL math expressions are modeling a chain of cause and effect
relations, which can be intervened upon, whereas for the PAL chain math
expressions are projected math identities.
PAL's generated python code is wrong. It hallucinates when complexity
increases.
```python
def solution():
"""Tim buys the same number of pets as Cindy and Boris.Cindy buys the same number of pets as Bill plus Bob.Boris buys the same number of pets as Ben plus Beth.Bill buys the same number of pets as Obama.Bob buys the same number of pets as Obama.Ben buys the same number of pets as Obama.Beth buys the same number of pets as Obama.If Obama buys one pet, how many pets total does everyone buy?"""
obama_pets = 1
tim_pets = obama_pets
cindy_pets = obama_pets + obama_pets
boris_pets = obama_pets + obama_pets
total_pets = tim_pets + cindy_pets + boris_pets
result = total_pets
return result # math result is 5
```
CPAL's generated python code is correct.
```python
story outcome data
name code value depends_on
0 obama pass 1.0 []
1 bill bill.value = obama.value 1.0 [obama]
2 bob bob.value = obama.value 1.0 [obama]
3 ben ben.value = obama.value 1.0 [obama]
4 beth beth.value = obama.value 1.0 [obama]
5 cindy cindy.value = bill.value + bob.value 2.0 [bill, bob]
6 boris boris.value = ben.value + beth.value 2.0 [ben, beth]
7 tim tim.value = cindy.value + boris.value 4.0 [cindy, boris]
query data
{
"question": "how many pets total does everyone buy?",
"expression": "SELECT SUM(value) FROM df",
"llm_error_msg": ""
}
# query result is 13
```
Based on the comments below, CPAL's intended location in the library is
`experimental/chains/cpal` and PAL's location is`chains/pal`.
### CPAL vs Graph QA
Both the CPAL chain and the Graph QA chain extract entity-action-entity
relations into a DAG.
The CPAL chain is different from the Graph QA chain for a few reasons.
* Graph QA does not connect entities to math expressions
* Graph QA does not associate actions in a sequence of dependence.
* Graph QA does not decompose the narrative into these three parts:
1. Story plot or causal model
4. Hypothetical question
5. Hypothetical condition
### Evaluation
Preliminary evaluation on simple math word problems shows that this CPAL
chain generates less hallucination than the PAL chain on answering
questions about a causal narrative. Two examples are in [this jupyter
notebook](https://github.com/borisdev/langchain/blob/master/docs/extras/modules/chains/additional/cpal.ipynb)
doc.
## 2. Future application
### "Describe as Narrative, Test as Code"
The thesis here is that the Describe as Narrative, Test as Code approach
allows you to represent a causal mental model both as code and as a
narrative, giving you the best of both worlds.
#### Why describe a causal mental mode as a narrative?
The narrative form is quick. At a consensus building meeting, people use
narratives to persuade others of their causal mental model, aka. plan.
You can share, version control and index a narrative.
#### Why test a causal mental model as a code?
Code is testable, complex narratives are not. Though fast, narratives
are problematic as their complexity increases. The problem is LLMs and
humans are prone to hallucination when predicting the outcomes of a
narrative. The cost of building a consensus around the validity of a
narrative outcome grows as its narrative complexity increases. Code does
not require tribal knowledge or social power to validate.
Code is composable, complex narratives are not. The answer of one CPAL
chain can be the hypothetical conditions of another CPAL Chain. For
stochastic simulations, a composable plan can be integrated with the
[DoWhy library](https://github.com/py-why/dowhy). Lastly, for the
futuristic folk, a composable plan as code allows ordinary community
folk to design a plan that can be integrated with a blockchain for
funding.
An explanation of a dependency planning application is
[here.](https://github.com/borisdev/cpal-llm-chain-demo)
---
Twitter handle: @boris_dev
---------
Co-authored-by: Boris Dev <borisdev@Boriss-MacBook-Air.local>
2023-07-11 14:11:21 +00:00
|
|
|
|
|
|
|
|
|
|
|
|
|
# fmt: off
|
|
|
|
|
template = (
|
|
|
|
|
"""
|
|
|
|
|
Transform the narrative_input into an SQL expression. If you are
|
|
|
|
|
unsure, then do not guess, instead add a llm_error_msg that explains why you are unsure.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
{format_instructions}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: how much money will boris have?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "how much money will boris have?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT name, value FROM df WHERE name = 'boris'"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: How much money does ted have?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "How much money does ted have?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT name, value FROM df WHERE name = 'ted'"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what is the sum of pet count for all the people?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what is the sum of pet count for all the people?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT SUM(value) FROM df"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the average of the pet counts for all the people?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the average of the pet counts for all the people?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT AVG(value) FROM df"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the maximum of the pet counts for all the people?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the maximum of the pet counts for all the people?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT MAX(value) FROM df"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the minimum of the pet counts for all the people?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the minimum of the pet counts for all the people?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT MIN(value) FROM df"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the number of people with pet counts greater than 10?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the number of people with pet counts greater than 10?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT COUNT(*) FROM df WHERE value > 10"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the pet count for boris?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the pet count for boris?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT name, value FROM df WHERE name = 'boris'"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the pet count for cindy and marcia?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the pet count for cindy and marcia?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT name, value FROM df WHERE name IN ('cindy', 'marcia')"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the total pet count for cindy and marcia?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the total pet count for cindy and marcia?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT SUM(value) FROM df WHERE name IN ('cindy', 'marcia')"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the total pet count for TED?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the total pet count for TED?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT SUM(value) FROM df WHERE name = 'TED'"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the total dollar count for TED and cindy?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the total dollar count for TED and cindy?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT SUM(value) FROM df WHERE name IN ('TED', 'cindy')"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the total pet count for TED and cindy?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the total pet count for TED and cindy?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT SUM(value) FROM df WHERE name IN ('TED', 'cindy')"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the best for TED and cindy?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the best for TED and cindy?",
|
|
|
|
|
"llm_error_msg": "ambiguous narrative_input, not sure what 'best' means",
|
|
|
|
|
"expression": ""
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: what's the value?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "what's the value?",
|
|
|
|
|
"llm_error_msg": "ambiguous narrative_input, not sure what entity is being asked about",
|
|
|
|
|
"expression": ""
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: how many total pets do the three have?
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
|
|
|
|
|
{{
|
|
|
|
|
"narrative_input": "how many total pets do the three have?",
|
|
|
|
|
"llm_error_msg": "",
|
|
|
|
|
"expression": "SELECT SUM(value) FROM df"
|
|
|
|
|
}}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
narrative_input: {narrative_input}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
# JSON:
|
|
|
|
|
""".strip()
|
|
|
|
|
+ "\n\n\n"
|
|
|
|
|
)
|
|
|
|
|
# fmt: on
|
