mirror of
https://github.com/hwchase17/langchain
synced 2024-10-31 15:20:26 +00:00
3179ee3a56
Still don't have good "how to's", and the guides / examples section could be further pruned and improved, but this PR adds a couple examples for each of the common evaluator interfaces. - [x] Example docs for each implemented evaluator - [x] "how to make a custom evalutor" notebook for each low level APIs (comparison, string, agent) - [x] Move docs to modules area - [x] Link to reference docs for more information - [X] Still need to finish the evaluation index page - ~[ ] Don't have good data generation section~ - ~[ ] Don't have good how to section for other common scenarios / FAQs like regression testing, testing over similar inputs to measure sensitivity, etc.~
373 lines
8.6 KiB
Plaintext
373 lines
8.6 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "984169ca",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Question Answering Benchmarking: Paul Graham Essay\n",
|
|
"\n",
|
|
"Here we go over how to benchmark performance on a question answering task over a Paul Graham essay.\n",
|
|
"\n",
|
|
"It is highly recommended that you do any evaluation/benchmarking with tracing enabled. See [here](https://python.langchain.com/docs/modules/callbacks/how_to/tracing) for an explanation of what tracing is and how to set it up."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "8a16b75d",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Loading the data\n",
|
|
"First, let's load the data."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "5b2d5e98",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stderr",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Found cached dataset json (/Users/harrisonchase/.cache/huggingface/datasets/LangChainDatasets___json/LangChainDatasets--question-answering-paul-graham-76e8f711e038d742/0.0.0/0f7e3662623656454fcd2b650f34e886a7db4b9104504885bd462096cc7a9f51)\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"application/vnd.jupyter.widget-view+json": {
|
|
"model_id": "9264acfe710b4faabf060f0fcf4f7308",
|
|
"version_major": 2,
|
|
"version_minor": 0
|
|
},
|
|
"text/plain": [
|
|
" 0%| | 0/1 [00:00<?, ?it/s]"
|
|
]
|
|
},
|
|
"metadata": {},
|
|
"output_type": "display_data"
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain.evaluation.loading import load_dataset\n",
|
|
"\n",
|
|
"dataset = load_dataset(\"question-answering-paul-graham\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4ab6a716",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Setting up a chain\n",
|
|
"Now we need to create some pipelines for doing question answering. Step one in that is creating an index over the data in question."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "c18680b5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.document_loaders import TextLoader\n",
|
|
"\n",
|
|
"loader = TextLoader(\"../../modules/paul_graham_essay.txt\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "7f0de2b3",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.indexes import VectorstoreIndexCreator"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "ef84ff99",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Running Chroma using direct local API.\n",
|
|
"Using DuckDB in-memory for database. Data will be transient.\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"vectorstore = VectorstoreIndexCreator().from_loaders([loader]).vectorstore"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f0b5d8f6",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now we can create a question answering chain."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "8843cb0c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.chains import RetrievalQA\n",
|
|
"from langchain.llms import OpenAI"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "573719a0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"chain = RetrievalQA.from_chain_type(\n",
|
|
" llm=OpenAI(),\n",
|
|
" chain_type=\"stuff\",\n",
|
|
" retriever=vectorstore.as_retriever(),\n",
|
|
" input_key=\"question\",\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "53b5aa23",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Make a prediction\n",
|
|
"\n",
|
|
"First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "3f81d951",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'question': 'What were the two main things the author worked on before college?',\n",
|
|
" 'answer': 'The two main things the author worked on before college were writing and programming.',\n",
|
|
" 'result': ' Writing and programming.'}"
|
|
]
|
|
},
|
|
"execution_count": 18,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"chain(dataset[0])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d0c16cd7",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Make many predictions\n",
|
|
"Now we can make predictions"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "24b4c66e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"predictions = chain.apply(dataset)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "49d969fb",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Evaluate performance\n",
|
|
"Now we can evaluate the predictions. The first thing we can do is look at them by eye."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "1d583f03",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'question': 'What were the two main things the author worked on before college?',\n",
|
|
" 'answer': 'The two main things the author worked on before college were writing and programming.',\n",
|
|
" 'result': ' Writing and programming.'}"
|
|
]
|
|
},
|
|
"execution_count": 10,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"predictions[0]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4783344b",
|
|
"metadata": {},
|
|
"source": [
|
|
"Next, we can use a language model to score them programatically"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "d0a9341d",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.evaluation.qa import QAEvalChain"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "1612dec1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"llm = OpenAI(temperature=0)\n",
|
|
"eval_chain = QAEvalChain.from_llm(llm)\n",
|
|
"graded_outputs = eval_chain.evaluate(\n",
|
|
" dataset, predictions, question_key=\"question\", prediction_key=\"result\"\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "79587806",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can add in the graded output to the `predictions` dict and then get a count of the grades."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "2a689df5",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"for i, prediction in enumerate(predictions):\n",
|
|
" prediction[\"grade\"] = graded_outputs[i][\"text\"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "27b61215",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"Counter({' CORRECT': 12, ' INCORRECT': 10})"
|
|
]
|
|
},
|
|
"execution_count": 14,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"from collections import Counter\n",
|
|
"\n",
|
|
"Counter([pred[\"grade\"] for pred in predictions])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "12fe30f4",
|
|
"metadata": {},
|
|
"source": [
|
|
"We can also filter the datapoints to the incorrect examples and look at them."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "47c692a1",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"incorrect = [pred for pred in predictions if pred[\"grade\"] == \" INCORRECT\"]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "0ef976c1",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'question': 'What did the author write their dissertation on?',\n",
|
|
" 'answer': 'The author wrote their dissertation on applications of continuations.',\n",
|
|
" 'result': ' The author does not mention what their dissertation was on, so it is not known.',\n",
|
|
" 'grade': ' INCORRECT'}"
|
|
]
|
|
},
|
|
"execution_count": 16,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"incorrect[0]"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "7710401a",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.11.2"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|