UpTrain has a new dashboard now that makes it easier to view projects
and evaluations. Using this requires specifying both project_name and
evaluation_name when performing evaluations. I have updated the code to
support it.
"This key is required to perform the evaluations. UpTrain uses the GPT models to evaluate the responses generated by the LLM."
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [],
"source": [
"OPENAI_API_KEY = getpass()"
]
},
{
{
"cell_type": "markdown",
"cell_type": "markdown",
"metadata": {},
"metadata": {},
"source": [
"source": [
"## Setup\n",
"## Setup\n",
"\n",
"\n",
"For each of the retrievers below, it is better to define the callback handler again to avoid interference. You can choose between the following options for evaluating using UpTrain:\n",
"UpTrain provides you with:\n",
"1. Dashboards with advanced drill-down and filtering options\n",
"1. Insights and common topics among failing cases\n",
"1. Observability and real-time monitoring of production data\n",
"1. Regression testing via seamless integration with your CI/CD pipelines\n",
"\n",
"\n",
"You can choose between the following options for evaluating using UpTrain:\n",
"You can use the open-source evaluation service to evaluate your model.\n",
"You can use the open-source evaluation service to evaluate your model. In this case, you will need to provie an OpenAI API key. UpTrain uses the GPT models to evaluate the responses generated by the LLM. You can get yours [here](https://platform.openai.com/account/api-keys).\n",
"In this case, you will need to provie an OpenAI API key. You can get yours [here](https://platform.openai.com/account/api-keys).\n",
"\n",
"In order to view your evaluations in the UpTrain dashboard, you will need to set it up by running the following commands in your terminal:\n",
"### 2. **UpTrain Managed Service and Dashboards**:\n",
"### 2. **UpTrain Managed Service and Dashboards**:\n",
"You can create a free UpTrain account [here](https://uptrain.ai/) and get free trial credits. If you want more trial credits, [book a call with the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min).\n",
"Alternatively, you can use UpTrain's managed service to evaluate your model. You can create a free UpTrain account [here](https://uptrain.ai/) and get free trial credits. If you want more trial credits, [book a call with the maintainers of UpTrain here](https://calendly.com/uptrain-sourabh/30min).\n",
"\n",
"\n",
"UpTrain Managed service provides:\n",
"The benefits of using the managed service are:\n",
"1. Dashboards with advanced drill-down and filtering options\n",
"1. No need to set up the UpTrain dashboard on your local machine.\n",
"1. Insights and common topics among failing cases\n",
"1. Access to many LLMs without needing their API keys.\n",
"1. Observability and real-time monitoring of production data\n",
"1. Regression testing via seamless integration with your CI/CD pipelines\n",
"\n",
"\n",
"The notebook contains some screenshots of the dashboards and the insights that you can get from the UpTrain managed service.\n",
"Once you perform the evaluations, you can view them in the UpTrain dashboard at `https://dashboard.uptrain.ai/dashboard`\n",
"**Note:** The `project_name_prefix` will be used as prefix for the project names in the UpTrain dashboard. These will be different for different types of evals. For example, if you set project_name_prefix=\"langchain\" and perform the multi_query evaluation, the project name will be \"langchain_multi_query\"."
"**Note:** The `project_name` will be the project name under which the evaluations performed will be shown in the UpTrain dashboard."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set the API key\n",
"\n",
"The notebook will prompt you to enter the API key. You can choose between the OpenAI API key or the UpTrain API key by changing the `key_type` parameter in the cell below."