mirror of
https://github.com/dair-ai/Prompt-Engineering-Guide
synced 2024-11-08 07:10:41 +00:00
88 lines
5.8 KiB
Plaintext
88 lines
5.8 KiB
Plaintext
# Identify Hallucination in LLM Responses
|
||
|
||
import { Tabs, Tab } from 'nextra/components'
|
||
import {Callout} from 'nextra/components'
|
||
|
||
## Background
|
||
The following prompt tests an LLM's capabilities to identify hallucination in the context of [closed-domain question answering](/question-answering/closed-domain). Bubeck et al. (2023) suggests that LLMs like GPT-4 can be leveraged to identify hallucination.
|
||
|
||
<Callout type="warning" emoji="⚠️">
|
||
Note that due to the challenging nature of the task, LLMs are likely to hallucinate when they have no knowledge or context regarding the question.
|
||
</Callout>
|
||
|
||
The prompt has been slightly modified to include indicators that show which sections of the prompt corresponds to what information.
|
||
|
||
## Prompt
|
||
```markdown
|
||
Patient’s facts:
|
||
- 20 year old female
|
||
- with a history of anerxia nervosa and depression
|
||
- blood pressure 100/50, pulse 50, height 5’5’’
|
||
- referred by her nutrionist but is in denial of her illness
|
||
- reports eating fine but is severely underweight
|
||
|
||
Question: Please rewrite the data above into a medical note, using exclusively the information above.
|
||
|
||
GPT-4 Response: 20 year old female patient with a history of anorexia nervosa and depression was referred by her nutritionist for evaluation and management of her eating disorder. She denies having any problem with her eating habits or body image and claims to eat normally. However, she is severely underweight for her height (BMI 16.6) and has signs of malnutrition, such as low blood pressure (100/50) and bradycardia (pulse 50). She also reports feeling depressed and hopeless. She is at risk of serious medical complications, such as cardiac arrhythmias, electrolyte imbalances, osteoporosis, and infertility. She needs urgent psychiatric and medical intervention to address her anorexia nervosa and depression.
|
||
|
||
Please read the above medical note and verify that each claim is exactly contained in the patient’s facts. Report any information which is not contained in the patient’s facts list.
|
||
```
|
||
|
||
## Code / API
|
||
|
||
<Tabs items={['GPT-4 (OpenAI)', 'Mixtral MoE 8x7B Instruct (Fireworks)']}>
|
||
<Tab>
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
client = OpenAI()
|
||
|
||
response = client.chat.completions.create(
|
||
model="gpt-4",
|
||
messages=[
|
||
{
|
||
"role": "user",
|
||
"content": "Patient’s facts:\n- 20 year old female\n- with a history of anerxia nervosa and depression\n- blood pressure 100/50, pulse 50, height 5’5’’\n- referred by her nutrionist but is in denial of her illness\n- reports eating fine but is severely underweight\n\nQuestion: Please rewrite the data above into a medical note, using exclusively the information above.\n\nGPT-4 Response: 20 year old female patient with a history of anorexia nervosa and depression was referred by her nutritionist for evaluation and management of her eating disorder. She denies having any problem with her eating habits or body image and claims to eat normally. However, she is severely underweight for her height (BMI 16.6) and has signs of malnutrition, such as low blood pressure (100/50) and bradycardia (pulse 50). She also reports feeling depressed and hopeless. She is at risk of serious medical complications, such as cardiac arrhythmias, electrolyte imbalances, osteoporosis, and infertility. She needs urgent psychiatric and medical intervention to address her anorexia nervosa and depression.\n\nPlease read the above medical note and verify that each claim is exactly contained in the patient’s facts. Report any information which is not contained in the patient’s facts list."
|
||
}
|
||
],
|
||
temperature=1,
|
||
max_tokens=250,
|
||
top_p=1,
|
||
frequency_penalty=0,
|
||
presence_penalty=0
|
||
)
|
||
```
|
||
</Tab>
|
||
|
||
<Tab>
|
||
```python
|
||
import fireworks.client
|
||
fireworks.client.api_key = "<FIREWORKS_API_KEY>"
|
||
completion = fireworks.client.ChatCompletion.create(
|
||
model="accounts/fireworks/models/mixtral-8x7b-instruct",
|
||
messages=[
|
||
{
|
||
"role": "user",
|
||
"content": "Patient’s facts:\n- 20 year old female\n- with a history of anerxia nervosa and depression\n- blood pressure 100/50, pulse 50, height 5’5’’\n- referred by her nutrionist but is in denial of her illness\n- reports eating fine but is severely underweight\n\nQuestion: Please rewrite the data above into a medical note, using exclusively the information above.\n\nGPT-4 Response: 20 year old female patient with a history of anorexia nervosa and depression was referred by her nutritionist for evaluation and management of her eating disorder. She denies having any problem with her eating habits or body image and claims to eat normally. However, she is severely underweight for her height (BMI 16.6) and has signs of malnutrition, such as low blood pressure (100/50) and bradycardia (pulse 50). She also reports feeling depressed and hopeless. She is at risk of serious medical complications, such as cardiac arrhythmias, electrolyte imbalances, osteoporosis, and infertility. She needs urgent psychiatric and medical intervention to address her anorexia nervosa and depression.\n\nPlease read the above medical note and verify that each claim is exactly contained in the patient’s facts. Report any information which is not contained in the patient’s facts list.",
|
||
}
|
||
],
|
||
stop=["<|im_start|>","<|im_end|>","<|endoftext|>"],
|
||
stream=True,
|
||
n=1,
|
||
top_p=1,
|
||
top_k=40,
|
||
presence_penalty=0,
|
||
frequency_penalty=0,
|
||
prompt_truncate_len=1024,
|
||
context_length_exceeded_behavior="truncate",
|
||
temperature=0.9,
|
||
max_tokens=4000
|
||
)
|
||
```
|
||
</Tab>
|
||
|
||
|
||
</Tabs>
|
||
|
||
## Reference
|
||
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://arxiv.org/abs/2303.12712) (13 April 2023) |