"This is an example notebook that can be used to create a benchmarking notebook for a task of your choice. Evaluation is really hard, and so we greatly welcome any contributions that can make it easier for people to experiment"
]
},
{
"cell_type": "markdown",
"id": "984169ca",
"metadata": {},
"source": [
"It is highly reccomended that you do any evaluation/benchmarking with tracing enabled. See [here](https://langchain.readthedocs.io/en/latest/tracing.html) for an explanation of what tracing is and how to set it up."
]
},
{
"cell_type": "code",
"execution_count": 28,
"id": "9fe4d1b4",
"metadata": {},
"outputs": [],
"source": [
"# Comment this out if you are NOT using tracing\n",
"This next section should have an example of setting up a chain that can be run on this dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a2661ce0",
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "markdown",
"id": "6c0062e7",
"metadata": {},
"source": [
"## Make a prediction\n",
"\n",
"First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d28c5e7d",
"metadata": {},
"outputs": [],
"source": [
"# Example of running the chain on a single datapoint (`dataset[0]`) goes here"
]
},
{
"cell_type": "markdown",
"id": "d0c16cd7",
"metadata": {},
"source": [
"## Make many predictions\n",
"Now we can make predictions."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "24b4c66e",
"metadata": {},
"outputs": [],
"source": [
"# Example of running the chain on many predictions goes here\n",
"\n",
"# Sometimes its as simple as `chain.apply(dataset)`\n",
"\n",
"# Othertimes you may want to write a for loop to catch errors"
]
},
{
"cell_type": "markdown",
"id": "4783344b",
"metadata": {},
"source": [
"## Evaluate performance\n",
"\n",
"Any guide to evaluating performance in a more systematic manner goes here."