|
|
# Self-Consistency
|
|
|
|
|
|
Perhaps one of the more advanced techniques out there for prompt engineering is self-consistency. Proposed by [Wang et al. (2022)](https://arxiv.org/pdf/2203.11171.pdf), self-consistency aims "to replace the naive greedy decoding used in chain-of-thought prompting". The idea is to sample multiple, diverse reasoning paths through few-shot CoT, and use the generations to select the most consistent answer. This helps to boost the performance of CoT prompting on tasks involving arithmetic and commonsense reasoning.
|
|
|
|
|
|
Let's try the following example for arithmetic reasoning:
|
|
|
|
|
|
*Prompt:*
|
|
|
```
|
|
|
When I was 6 my sister was half my age. Now
|
|
|
I’m 70 how old is my sister?
|
|
|
```
|
|
|
|
|
|
*Output:*
|
|
|
```
|
|
|
35
|
|
|
```
|
|
|
|
|
|
The output is obviously wrong! How may we improve this with self-consistency? Let's try it out. We will use the few-shot exemplars from Wang et al. 2022 (Table 17):
|
|
|
|
|
|
*Prompt:*
|
|
|
```
|
|
|
Q: There are 15 trees in the grove. Grove workers will plant trees in the grove today. After they are done,
|
|
|
there will be 21 trees. How many trees did the grove workers plant today?
|
|
|
A: We start with 15 trees. Later we have 21 trees. The difference must be the number of trees they planted.
|
|
|
So, they must have planted 21 - 15 = 6 trees. The answer is 6.
|
|
|
|
|
|
Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot?
|
|
|
A: There are 3 cars in the parking lot already. 2 more arrive. Now there are 3 + 2 = 5 cars. The answer is 5.
|
|
|
|
|
|
Q: Leah had 32 chocolates and her sister had 42. If they ate 35, how many pieces do they have left in total?
|
|
|
A: Leah had 32 chocolates and Leah’s sister had 42. That means there were originally 32 + 42 = 74
|
|
|
chocolates. 35 have been eaten. So in total they still have 74 - 35 = 39 chocolates. The answer is 39.
|
|
|
|
|
|
Q: Jason had 20 lollipops. He gave Denny some lollipops. Now Jason has 12 lollipops. How many lollipops
|
|
|
did Jason give to Denny?
|
|
|
A: Jason had 20 lollipops. Since he only has 12 now, he must have given the rest to Denny. The number of
|
|
|
lollipops he has given to Denny must have been 20 - 12 = 8 lollipops. The answer is 8.
|
|
|
|
|
|
Q: Shawn has five toys. For Christmas, he got two toys each from his mom and dad. How many toys does
|
|
|
he have now?
|
|
|
A: He has 5 toys. He got 2 from mom, so after that he has 5 + 2 = 7 toys. Then he got 2 more from dad, so
|
|
|
in total he has 7 + 2 = 9 toys. The answer is 9.
|
|
|
|
|
|
Q: There were nine computers in the server room. Five more computers were installed each day, from
|
|
|
monday to thursday. How many computers are now in the server room?
|
|
|
A: There are 4 days from monday to thursday. 5 computers were added each day. That means in total 4 * 5 =
|
|
|
20 computers were added. There were 9 computers in the beginning, so now there are 9 + 20 = 29 computers.
|
|
|
The answer is 29.
|
|
|
|
|
|
Q: Michael had 58 golf balls. On tuesday, he lost 23 golf balls. On wednesday, he lost 2 more. How many
|
|
|
golf balls did he have at the end of wednesday?
|
|
|
A: Michael initially had 58 balls. He lost 23 on Tuesday, so after that he has 58 - 23 = 35 balls. On
|
|
|
Wednesday he lost 2 more so now he has 35 - 2 = 33 balls. The answer is 33.
|
|
|
|
|
|
Q: Olivia has $23. She bought five bagels for $3 each. How much money does she have left?
|
|
|
A: She bought 5 bagels for $3 each. This means she spent 5
|
|
|
|
|
|
Q: When I was 6 my sister was half my age. Now I’m 70 how old is my sister?
|
|
|
A:
|
|
|
```
|
|
|
|
|
|
*Output 1:*
|
|
|
```
|
|
|
When I was 6 my sister was half my age, so she was 3. Now I am 70, so she is 70 - 3 = 67. The answer is 67.
|
|
|
```
|
|
|
|
|
|
*Output 2:*
|
|
|
```
|
|
|
When the narrator was 6, his sister was half his age, which is 3. Now that the narrator is 70, his sister would be 70 - 3 = 67 years old. The answer is 67.
|
|
|
```
|
|
|
|
|
|
*Output 3:*
|
|
|
```
|
|
|
When I was 6 my sister was half my age, so she was 3. Now I am 70, so she is 70/2 = 35. The answer is 35.
|
|
|
```
|
|
|
|
|
|
Computing for the final answer involves a few steps (check out the paper for the details) but for the sake of simplicity, we can clearly see that there is already a majority answer emerging so that would essentially become the final answer. |