diff --git a/examples/How_to_finetune_chat_models.ipynb b/examples/How_to_finetune_chat_models.ipynb new file mode 100644 index 00000000..c368790b --- /dev/null +++ b/examples/How_to_finetune_chat_models.ipynb @@ -0,0 +1,686 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "7bcdad0e-b67c-4927-b00e-3a4d950cd8ce", + "metadata": {}, + "source": [ + "# How to fine-tune chat models\n", + "\n", + "This notebook provides a step-by-step guide for our new `gpt-3.5-turbo` fine-tuning. We'll perform entity extraction using the [RecipeNLG dataset](https://github.com/Glorf/recipenlg), which provides various recipes and a list of extracted generic ingredients for each. This is a common dataset for named entity recognition (NER) tasks.\n", + "\n", + "We will go through the following steps:\n", + "\n", + "1. **Setup:** Loading our dataset and filtering down to one domain to fine-tune on.\n", + "2. **Data preparation:** Preparing your data for fine-tuning by creating training and validation examples, and uploading them to the `Files` endpoint.\n", + "3. **Create the fine-tune:** Creating your fine-tuned model.\n", + "4. **Use model for inference:** Using your fine-tuned model for inference on new inputs.\n", + "\n", + "By the end of this you should be able to train, evaluate and deploy a fine-tuned `gpt-3.5-turbo` model.\n" + ] + }, + { + "cell_type": "markdown", + "id": "6f49cb10-f895-41f4-aa97-da606d0084d4", + "metadata": {}, + "source": [ + "## Setup\n", + "\n", + "First we will import any required libraries and prepare our data.\n", + "\n", + "Fine tuning works best when focused on a particular domain. It's important to make sure your dataset is both focused enough for the model to learn, but general enough that unseen examples won't be missed. Having this in mind, we have already extracted a subset from the RecipesNLG dataset to only contain documents from www.cookbooks.com.\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "32036e70", + "metadata": {}, + "outputs": [], + "source": [ + "# make sure to use the latest version of the openai python package\n", + "!pip install --upgrade openai " + ] + }, + { + "cell_type": "code", + "execution_count": 76, + "id": "6e1f4403-37e1-4115-a215-12fd7daa1eb6", + "metadata": {}, + "outputs": [], + "source": [ + "import json\n", + "import openai\n", + "import os\n", + "import pandas as pd\n", + "import requests\n", + "from pprint import pprint\n", + "\n", + "OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\", \"\")" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "f57ebc23-14b7-47f9-90b8-1d791ccfc9bc", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
titleingredientsdirectionslinksourceNER
0No-Bake Nut Cookies[\"1 c. firmly packed brown sugar\", \"1/2 c. eva...[\"In a heavy 2-quart saucepan, mix brown sugar...www.cookbooks.com/Recipe-Details.aspx?id=44874www.cookbooks.com[\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"bu...
1Jewell Ball'S Chicken[\"1 small jar chipped beef, cut up\", \"4 boned ...[\"Place chipped beef on bottom of baking dish....www.cookbooks.com/Recipe-Details.aspx?id=699419www.cookbooks.com[\"beef\", \"chicken breasts\", \"cream of mushroom...
2Creamy Corn[\"2 (16 oz.) pkg. frozen corn\", \"1 (8 oz.) pkg...[\"In a slow cooker, combine all ingredients. C...www.cookbooks.com/Recipe-Details.aspx?id=10570www.cookbooks.com[\"frozen corn\", \"cream cheese\", \"butter\", \"gar...
3Chicken Funny[\"1 large whole chicken\", \"2 (10 1/2 oz.) cans...[\"Boil and debone chicken.\", \"Put bite size pi...www.cookbooks.com/Recipe-Details.aspx?id=897570www.cookbooks.com[\"chicken\", \"chicken gravy\", \"cream of mushroo...
4Reeses Cups(Candy)[\"1 c. peanut butter\", \"3/4 c. graham cracker ...[\"Combine first four ingredients and press in ...www.cookbooks.com/Recipe-Details.aspx?id=659239www.cookbooks.com[\"peanut butter\", \"graham cracker crumbs\", \"bu...
\n", + "
" + ], + "text/plain": [ + " title ingredients \\\n", + "0 No-Bake Nut Cookies [\"1 c. firmly packed brown sugar\", \"1/2 c. eva... \n", + "1 Jewell Ball'S Chicken [\"1 small jar chipped beef, cut up\", \"4 boned ... \n", + "2 Creamy Corn [\"2 (16 oz.) pkg. frozen corn\", \"1 (8 oz.) pkg... \n", + "3 Chicken Funny [\"1 large whole chicken\", \"2 (10 1/2 oz.) cans... \n", + "4 Reeses Cups(Candy) [\"1 c. peanut butter\", \"3/4 c. graham cracker ... \n", + "\n", + " directions \\\n", + "0 [\"In a heavy 2-quart saucepan, mix brown sugar... \n", + "1 [\"Place chipped beef on bottom of baking dish.... \n", + "2 [\"In a slow cooker, combine all ingredients. C... \n", + "3 [\"Boil and debone chicken.\", \"Put bite size pi... \n", + "4 [\"Combine first four ingredients and press in ... \n", + "\n", + " link source \\\n", + "0 www.cookbooks.com/Recipe-Details.aspx?id=44874 www.cookbooks.com \n", + "1 www.cookbooks.com/Recipe-Details.aspx?id=699419 www.cookbooks.com \n", + "2 www.cookbooks.com/Recipe-Details.aspx?id=10570 www.cookbooks.com \n", + "3 www.cookbooks.com/Recipe-Details.aspx?id=897570 www.cookbooks.com \n", + "4 www.cookbooks.com/Recipe-Details.aspx?id=659239 www.cookbooks.com \n", + "\n", + " NER \n", + "0 [\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"bu... \n", + "1 [\"beef\", \"chicken breasts\", \"cream of mushroom... \n", + "2 [\"frozen corn\", \"cream cheese\", \"butter\", \"gar... \n", + "3 [\"chicken\", \"chicken gravy\", \"cream of mushroo... \n", + "4 [\"peanut butter\", \"graham cracker crumbs\", \"bu... " + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Read in the dataset we'll use for this task.\n", + "# This will be the RecipesNLG dataset, which we've cleaned to only contain documents from www.cookbooks.com\n", + "recipe_df = pd.read_csv(\"data/cookbook_recipes_nlg_10k.csv\")\n", + "\n", + "recipe_df.head()" + ] + }, + { + "cell_type": "markdown", + "id": "2b3151e9-8715-47bd-a153-195d6a0d0a70", + "metadata": {}, + "source": [ + "## Data preparation\n", + "\n", + "We'll begin by preparing our data. When fine-tuning with the `ChatCompletion` format, each training example is a simple list of `messages`. For example, an entry could look like:\n", + "\n", + "```\n", + "[{'role': 'system',\n", + " 'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'},\n", + "\n", + " {'role': 'user',\n", + " 'content': 'Title: No-Bake Nut Cookies\\n\\nIngredients: [\"1 c. firmly packed brown sugar\", \"1/2 c. evaporated milk\", \"1/2 tsp. vanilla\", \"1/2 c. broken nuts (pecans)\", \"2 Tbsp. butter or margarine\", \"3 1/2 c. bite size shredded rice biscuits\"]\\n\\nGeneric ingredients: '},\n", + "\n", + " {'role': 'assistant',\n", + " 'content': '[\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"butter\", \"bite size shredded rice biscuits\"]'}]\n", + "```\n", + "\n", + "During the training process this conversation will be split, with the final entry being the `completion` that the model will produce, and the remainder of the `messages` acting as the prompt. Consider this when building your training examples - if your model will act on multi-turn conversations, then please provide representative examples so it doesn't perform poorly when the conversation starts to expand.\n", + "\n", + "For fine-tuning with `ChatCompletion` you can begin with even 30-50 well-pruned examples. You should see performance continue to scale linearly as you increase the size of the training set, but your jobs will also take longer.\n", + "\n", + "Please note that currently there is a 4096 token limit for each training example. Anything longer than this will be truncated at 4096 tokens.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "9a8216b0-d1dc-472d-b07d-1be03acd70a5", + "metadata": {}, + "outputs": [], + "source": [ + "training_data = []\n", + "\n", + "system_message = \"You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.\"\n", + "\n", + "\n", + "def create_user_message(row):\n", + " return f\"\"\"Title: {row['title']}\\n\\nIngredients: {row['ingredients']}\\n\\nGeneric ingredients: \"\"\"\n", + "\n", + "\n", + "# Take first 100 records for training\n", + "for x, y in recipe_df.head(100).iterrows():\n", + " training_message = []\n", + " training_message.append({\"role\": \"system\", \"content\": system_message})\n", + "\n", + " user_message = create_user_message(y)\n", + " training_message.append({\"role\": \"user\", \"content\": user_message})\n", + "\n", + " training_message.append({\"role\": \"assistant\", \"content\": y[\"NER\"]})\n", + " training_message_dict = {\"messages\": training_message}\n", + " training_data.append(training_message_dict)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "5b853efa-dfea-4770-ab88-9b7e17794421", + "metadata": {}, + "outputs": [], + "source": [ + "validation_data = []\n", + "\n", + "# We'll pick a test set from further on in the dataset\n", + "test_df = recipe_df.loc[100:200]\n", + "\n", + "for x, y in test_df.iterrows():\n", + " validation_message = []\n", + " validation_message.append({\"role\": \"system\", \"content\": system_message})\n", + "\n", + " user_message = create_user_message(y)\n", + " validation_message.append({\"role\": \"user\", \"content\": user_message})\n", + "\n", + " validation_message.append({\"role\": \"assistant\", \"content\": y[\"NER\"]})\n", + " validation_message_dict = {\"messages\": validation_message}\n", + " validation_data.append(validation_message_dict)" + ] + }, + { + "cell_type": "markdown", + "id": "1d5e7bfe-f6c8-4a23-a951-3df3f3791d7f", + "metadata": {}, + "source": [ + "We then need to export these as `.jsonl` files, with each row being one training example.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "8d2eb207-2c2b-43f6-a613-64a7e92d494d", + "metadata": {}, + "outputs": [], + "source": [ + "def dicts_to_jsonl(data_list: list, filename: str) -> None:\n", + " \"\"\"\n", + " Method saves list of dicts into jsonl file.\n", + " :param data: (list) list of dicts to be stored,\n", + " :param filename: (str) path to the output file. If suffix .jsonl is not given then methods appends\n", + " .jsonl suffix into the file.\n", + " \"\"\"\n", + " sjsonl = \".jsonl\"\n", + "\n", + " # Check filename\n", + " if not filename.endswith(sjsonl):\n", + " filename = filename + \".jsonl\"\n", + "\n", + " # Save data\n", + " with open(filename, \"w\") as out:\n", + " for ddict in data_list:\n", + " jout = json.dumps(ddict) + \"\\n\"\n", + " out.write(jout)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "8b53e7a2-1cac-4c5f-8ba4-3292ba2a0770", + "metadata": {}, + "outputs": [], + "source": [ + "# Save training_data to JSONL\n", + "training_file_name = \"tmp_recipe_finetune_training\"\n", + "dicts_to_jsonl(training_data, training_file_name)\n", + "\n", + "# Save validation_data to JSONL\n", + "validation_file_name = \"tmp_recipe_finetune_validation\"\n", + "dicts_to_jsonl(validation_data, validation_file_name)" + ] + }, + { + "cell_type": "markdown", + "id": "0d149e2e-50dd-45c1-bd8d-1291975670b4", + "metadata": {}, + "source": [ + "### Upload files\n", + "\n", + "You can then upload the files to our `Files` endpoint to be used by the fine-tuned model.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "69462d9e-e6bd-49b9-a064-9eae4ea5b7a8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Training file id: file-XMhftmLXyyTvvmiLpMRMIAcL\n", + "Validation file id: file-kZz433aerpPIMADZF7xC5NKd\n" + ] + } + ], + "source": [ + "training_response = openai.File.create(\n", + " file=open(training_file_name + \".jsonl\", \"rb\"), purpose=\"fine-tune\"\n", + ")\n", + "training_file_id = training_response[\"id\"]\n", + "\n", + "validation_response = openai.File.create(\n", + " file=open(validation_file_name + \".jsonl\", \"rb\"), purpose=\"fine-tune\"\n", + ")\n", + "validation_file_id = validation_response[\"id\"]\n", + "\n", + "print(\"Training file id:\", training_file_id)\n", + "print(\"Validation file id:\", validation_file_id)" + ] + }, + { + "cell_type": "markdown", + "id": "d61cd381-63ad-4ed9-b0be-47a438891028", + "metadata": {}, + "source": [ + "### Create fine-tune job\n", + "\n", + "Now we can create our fine-tuning job with the generated files and an optional suffix to identify the model. The response will contain an `id` which you can use to retrieve updates on the job.\n", + "\n", + "Note: The files have to first be processed by our system, so you might get a `File is not ready` error. In that case, simply retry a few minutes later.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "05541ceb-5628-447e-962d-7e57c112439c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"object\": \"fine_tuning.job\",\n", + " \"id\": \"ftjob-ksOzx7zjpsrADZfhB5eyfB0Z\",\n", + " \"model\": \"gpt-3.5-turbo-0613\",\n", + " \"created_at\": 1692734343,\n", + " \"finished_at\": null,\n", + " \"fine_tuned_model\": null,\n", + " \"organization_id\": \"org-l89177bnhkme4a44292n5r3j\",\n", + " \"result_files\": [],\n", + " \"status\": \"created\",\n", + " \"validation_file\": \"file-kZz433aerpPIMADZF7xC5NKd\",\n", + " \"training_file\": \"file-XMhftmLXyyTvvmiLpMRMIAcL\",\n", + " \"hyperparameters\": {\n", + " \"n_epochs\": 3\n", + " },\n", + " \"trained_tokens\": null\n", + "}\n" + ] + } + ], + "source": [ + "suffix_name = \"recipe-ner\"\n", + "\n", + "\n", + "response = openai.FineTuningJob.create(\n", + " training_file=training_file_id,\n", + " validation_file=validation_file_id,\n", + " model=\"gpt-3.5-turbo\",\n", + " suffix=suffix_name,\n", + ")\n", + "\n", + "job_id = response[\"id\"]\n", + "\n", + "print(response)" + ] + }, + { + "cell_type": "markdown", + "id": "1de3ed71-f2d4-4138-95a3-70da187a007e", + "metadata": {}, + "source": [ + "#### Check job status\n", + "\n", + "You can make a `GET` request to the `https://api.openai.com/v1/alpha/fine-tunes` endpoint to list your alpha fine-tune jobs. In this instance you'll want to check that the ID you got from the previous step ends up as `status: succeeded`.\n", + "\n", + "Once it is completed, you can use the `result_files` to sample the results from the validation set (if you uploaded one), and use the ID from the `fine_tuned_model` parameter to invoke your trained model.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 59, + "id": "d7392f48", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"object\": \"fine_tuning.job\",\n", + " \"id\": \"ftjob-ksOzx7zjpsrADZfhB5eyfB0Z\",\n", + " \"model\": \"gpt-3.5-turbo-0613\",\n", + " \"created_at\": 1692734343,\n", + " \"finished_at\": 1692735182,\n", + " \"fine_tuned_model\": \"ft:gpt-3.5-turbo-0613:openai:recipe-ner:7qS2GFaX\",\n", + " \"organization_id\": \"org-l89177bnhkme4a44292n5r3j\",\n", + " \"result_files\": [\n", + " \"file-Tjt0E6BvQ846m75gYxtWiRzb\"\n", + " ],\n", + " \"status\": \"succeeded\",\n", + " \"validation_file\": \"file-kZz433aerpPIMADZF7xC5NKd\",\n", + " \"training_file\": \"file-XMhftmLXyyTvvmiLpMRMIAcL\",\n", + " \"hyperparameters\": {\n", + " \"n_epochs\": 3\n", + " },\n", + " \"trained_tokens\": 39687\n", + "}\n" + ] + } + ], + "source": [ + "response = openai.FineTuningJob.retrieve(job_id)\n", + "print(response)" + ] + }, + { + "cell_type": "markdown", + "id": "30a57fbb", + "metadata": {}, + "source": [ + "We can track the progress of the fine-tune with the events endpoint. You can rerun the cell below a few times until the fine-tune is ready. \n" + ] + }, + { + "cell_type": "code", + "execution_count": 69, + "id": "08cace28", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Created fine-tune: ftjob-ksOzx7zjpsrADZfhB5eyfB0Z\n", + "Fine tuning job started\n", + "Step 10: training loss=0.16\n", + "Step 20: training loss=0.14\n", + "Step 30: training loss=0.30\n", + "Step 40: training loss=0.17\n", + "Step 50: training loss=0.03\n", + "Step 60: training loss=0.55\n", + "Step 70: training loss=0.09\n", + "Step 80: training loss=0.00\n", + "Step 90: training loss=0.15\n", + "Step 100: training loss=0.06\n", + "Step 110: training loss=0.03\n", + "Step 120: training loss=0.04\n", + "Step 130: training loss=0.21\n", + "Step 140: training loss=0.00\n", + "Step 150: training loss=0.02\n", + "Step 160: training loss=0.00\n", + "Step 170: training loss=0.00\n", + "Step 180: training loss=0.19\n", + "Step 190: training loss=0.55\n", + "Step 200: training loss=0.01\n", + "Step 210: training loss=0.00\n", + "Step 220: training loss=0.00\n", + "Step 230: training loss=0.00\n", + "Step 240: training loss=0.00\n", + "Step 250: training loss=0.00\n", + "Step 260: training loss=0.36\n", + "Step 270: training loss=0.16\n", + "Step 280: training loss=0.01\n", + "Step 290: training loss=0.04\n", + "Step 300: training loss=0.53\n", + "New fine-tuned model created: ft:gpt-3.5-turbo-0613:openai:recipe-ner:7qS2GFaX\n", + "Fine-tuning job successfully completed\n" + ] + } + ], + "source": [ + "response = openai.FineTuningJob.list_events(id=job_id, limit=50)\n", + "\n", + "events = response[\"data\"]\n", + "events.reverse()\n", + "\n", + "for event in events:\n", + " print(event[\"message\"])" + ] + }, + { + "cell_type": "markdown", + "id": "d0da4e32", + "metadata": {}, + "source": [ + "Now that it's done, we can get a fine-tuned model ID from the job\n" + ] + }, + { + "cell_type": "code", + "execution_count": 81, + "id": "40b28c26", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"object\": \"fine_tuning.job\",\n", + " \"id\": \"ftjob-ksOzx7zjpsrADZfhB5eyfB0Z\",\n", + " \"model\": \"gpt-3.5-turbo-0613\",\n", + " \"created_at\": 1692734343,\n", + " \"finished_at\": 1692735182,\n", + " \"fine_tuned_model\": \"ft:gpt-3.5-turbo-0613:openai:recipe-ner:7qS2GFaX\",\n", + " \"organization_id\": \"org-l89177bnhkme4a44292n5r3j\",\n", + " \"result_files\": [\n", + " \"file-Tjt0E6BvQ846m75gYxtWiRzb\"\n", + " ],\n", + " \"status\": \"succeeded\",\n", + " \"validation_file\": \"file-kZz433aerpPIMADZF7xC5NKd\",\n", + " \"training_file\": \"file-XMhftmLXyyTvvmiLpMRMIAcL\",\n", + " \"hyperparameters\": {\n", + " \"n_epochs\": 3\n", + " },\n", + " \"trained_tokens\": 39687\n", + "}\n", + "\n", + "Fine-tuned model id: ft:gpt-3.5-turbo-0613:openai:recipe-ner:7qS2GFaX\n" + ] + } + ], + "source": [ + "response = openai.FineTuningJob.retrieve(job_id)\n", + "fine_tuned_model_id = response[\"fine_tuned_model\"]\n", + "\n", + "print(response)\n", + "print(\"\\nFine-tuned model id:\", fine_tuned_model_id)" + ] + }, + { + "cell_type": "markdown", + "id": "0025e392-84cd-4566-a384-ea31ca43e567", + "metadata": {}, + "source": [ + "## Generate with the fine-tuned model" + ] + }, + { + "cell_type": "markdown", + "id": "0ab9ac11", + "metadata": {}, + "source": [ + "The last step is to use your fine-tuned model for inference. Similar to the classic `FineTuning`, you simply call `ChatCompletions` with your new fine-tuned model name filling the `model` parameter." + ] + }, + { + "cell_type": "code", + "execution_count": 87, + "id": "1c7de631-b68f-4eff-9ae7-051641579c2b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[{'content': 'You are a helpful recipe assistant. You are to extract the '\n", + " 'generic ingredients from each of the recipes provided.',\n", + " 'role': 'system'},\n", + " {'content': 'Title: Pancakes\\n'\n", + " '\\n'\n", + " 'Ingredients: [\"1 c. flour\", \"1 tsp. soda\", \"1 tsp. salt\", \"1 '\n", + " 'Tbsp. sugar\", \"1 egg\", \"3 Tbsp. margarine, melted\", \"1 c. '\n", + " 'buttermilk\"]\\n'\n", + " '\\n'\n", + " 'Generic ingredients: ',\n", + " 'role': 'user'}]\n" + ] + } + ], + "source": [ + "test_row = test_df.iloc[0]\n", + "test_messages = []\n", + "test_messages.append({\"role\": \"system\", \"content\": system_message})\n", + "user_message = create_user_message(test_row)\n", + "test_messages.append({\"role\": \"user\", \"content\": create_user_message(test_row)})\n", + "\n", + "pprint(test_messages)" + ] + }, + { + "cell_type": "code", + "execution_count": 88, + "id": "1a1d2589", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\"flour\", \"soda\", \"salt\", \"sugar\", \"egg\", \"margarine\", \"buttermilk\"]\n" + ] + } + ], + "source": [ + "response = openai.ChatCompletion.create(\n", + " model=fine_tuned_model_id, messages=test_messages, temperature=0, max_tokens=500\n", + ")\n", + "print(response[\"choices\"][0][\"message\"][\"content\"])" + ] + }, + { + "cell_type": "markdown", + "id": "07799909-3f2a-4274-b81e-dabc048be28f", + "metadata": {}, + "source": [ + "## Conclusion\n", + "\n", + "Congratulations, you are now ready to fine-tune your own models using the `ChatCompletion` format! We look forward to seeing what you build\n" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.13" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/How_to_finetune_chat_models_for_structured_data_extraction.ipynb b/examples/How_to_finetune_chat_models_for_structured_data_extraction.ipynb deleted file mode 100644 index 6ba6e2f5..00000000 --- a/examples/How_to_finetune_chat_models_for_structured_data_extraction.ipynb +++ /dev/null @@ -1,1598 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "7bcdad0e-b67c-4927-b00e-3a4d950cd8ce", - "metadata": {}, - "source": [ - "# How to fine-tune chat models for structured data extraction\n", - "\n", - "This notebook walks through using our new fine-tuning feature in `ChatCompletion` format. To illustrate this example we'll perform entity extraction using the [RecipeNLG dataset](https://github.com/Glorf/recipenlg), which provides the title, ingredients and directions of a food dish and extracts generic ingredients from it.\n", - "\n", - "We will walk through the end-to-end process which consists of:\n", - "- **Setup:** Loading our dataset and filtering down to one domain to fine-tune on.\n", - "- **Few-shot predictions:** Run a first attempt using few-shot examples, so we have a baseline to measure how well our fine-tune performs.\n", - "- **Data preparation:** Prepare your data for fine-tuning by creating training/validation examples and uploading to the `Files` endpoint.\n", - "- **Create and evaluate fine-tune:** Create your model and evaluate it against a validation set to measure its performance.\n", - "- **Use model for inference:** Close off the process by using your model for inference on new inputs.\n", - "\n", - "By the end of this you should be able to train, evaluate and deploy your fine-tuned models confidently. \n" - ] - }, - { - "cell_type": "markdown", - "id": "6f49cb10-f895-41f4-aa97-da606d0084d4", - "metadata": {}, - "source": [ - "## Setup\n", - "\n", - "Import any required libraries and prepare our data.\n", - "\n", - "Our data preparation includes breaking our source data up by where it was sourced from, and then focusing on one source only. This is an important consideration when fine-tuning - it will work best when tuning to a particular domain, so make sure your dataset is both focused enough for the model to learn, but general enough that unseen examples won't be missed." - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "32036e70", - "metadata": {}, - "outputs": [], - "source": [ - "!pip install matplotlib \n", - "!pip install openai \n", - "!pip install pandas\n", - "!pip install requests" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "6e1f4403-37e1-4115-a215-12fd7daa1eb6", - "metadata": {}, - "outputs": [], - "source": [ - "import ast\n", - "import json\n", - "import matplotlib\n", - "import openai\n", - "import os\n", - "import pandas as pd\n", - "import requests\n", - "\n", - "GPT_MODEL = 'gpt-3.5-turbo'\n" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "f57ebc23-14b7-47f9-90b8-1d791ccfc9bc", - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
titleingredientsdirectionslinksourceNER
0No-Bake Nut Cookies[\"1 c. firmly packed brown sugar\", \"1/2 c. eva...[\"In a heavy 2-quart saucepan, mix brown sugar...www.cookbooks.com/Recipe-Details.aspx?id=44874www.cookbooks.com[\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"bu...
1Jewell Ball'S Chicken[\"1 small jar chipped beef, cut up\", \"4 boned ...[\"Place chipped beef on bottom of baking dish....www.cookbooks.com/Recipe-Details.aspx?id=699419www.cookbooks.com[\"beef\", \"chicken breasts\", \"cream of mushroom...
2Creamy Corn[\"2 (16 oz.) pkg. frozen corn\", \"1 (8 oz.) pkg...[\"In a slow cooker, combine all ingredients. C...www.cookbooks.com/Recipe-Details.aspx?id=10570www.cookbooks.com[\"frozen corn\", \"cream cheese\", \"butter\", \"gar...
3Chicken Funny[\"1 large whole chicken\", \"2 (10 1/2 oz.) cans...[\"Boil and debone chicken.\", \"Put bite size pi...www.cookbooks.com/Recipe-Details.aspx?id=897570www.cookbooks.com[\"chicken\", \"chicken gravy\", \"cream of mushroo...
4Reeses Cups(Candy)[\"1 c. peanut butter\", \"3/4 c. graham cracker ...[\"Combine first four ingredients and press in ...www.cookbooks.com/Recipe-Details.aspx?id=659239www.cookbooks.com[\"peanut butter\", \"graham cracker crumbs\", \"bu...
\n", - "
" - ], - "text/plain": [ - " title ingredients \\\n", - "0 No-Bake Nut Cookies [\"1 c. firmly packed brown sugar\", \"1/2 c. eva... \n", - "1 Jewell Ball'S Chicken [\"1 small jar chipped beef, cut up\", \"4 boned ... \n", - "2 Creamy Corn [\"2 (16 oz.) pkg. frozen corn\", \"1 (8 oz.) pkg... \n", - "3 Chicken Funny [\"1 large whole chicken\", \"2 (10 1/2 oz.) cans... \n", - "4 Reeses Cups(Candy) [\"1 c. peanut butter\", \"3/4 c. graham cracker ... \n", - "\n", - " directions \\\n", - "0 [\"In a heavy 2-quart saucepan, mix brown sugar... \n", - "1 [\"Place chipped beef on bottom of baking dish.... \n", - "2 [\"In a slow cooker, combine all ingredients. C... \n", - "3 [\"Boil and debone chicken.\", \"Put bite size pi... \n", - "4 [\"Combine first four ingredients and press in ... \n", - "\n", - " link source \\\n", - "0 www.cookbooks.com/Recipe-Details.aspx?id=44874 www.cookbooks.com \n", - "1 www.cookbooks.com/Recipe-Details.aspx?id=699419 www.cookbooks.com \n", - "2 www.cookbooks.com/Recipe-Details.aspx?id=10570 www.cookbooks.com \n", - "3 www.cookbooks.com/Recipe-Details.aspx?id=897570 www.cookbooks.com \n", - "4 www.cookbooks.com/Recipe-Details.aspx?id=659239 www.cookbooks.com \n", - "\n", - " NER \n", - "0 [\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"bu... \n", - "1 [\"beef\", \"chicken breasts\", \"cream of mushroom... \n", - "2 [\"frozen corn\", \"cream cheese\", \"butter\", \"gar... \n", - "3 [\"chicken\", \"chicken gravy\", \"cream of mushroo... \n", - "4 [\"peanut butter\", \"graham cracker crumbs\", \"bu... " - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# Read in the dataset we'll use for this task.\n", - "# This will be the RecipesNLG dataset, which we've cleaned to only contain documents from www.cookbooks.com\n", - "recipe_df = pd.read_csv('data/cookbook_recipes_nlg_10k.csv')\n", - "\n", - "recipe_df.head()\n" - ] - }, - { - "cell_type": "markdown", - "id": "a2ad00de-18fe-4454-b27d-9758c6f73365", - "metadata": {}, - "source": [ - "## Few-shot\n", - "\n", - "First we'll try to solve this problem few-shot by giving 5 examples and then measuring the proportion of entities that were accurately tagged for each entry.\n", - "\n", - "First we set a basic system prompt that outlines our task, to extract generic ingredients from each recipe provided." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "e2ba5fc2-8d5d-43ad-b51b-d625dcd4cfc9", - "metadata": {}, - "outputs": [], - "source": [ - "messages = []\n", - "system_prompt = 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'\n", - "messages.append({\"role\":\"system\",\"content\":system_prompt})\n" - ] - }, - { - "cell_type": "markdown", - "id": "120fb541-a0bb-4fbc-87eb-419b9356b6e8", - "metadata": {}, - "source": [ - "Next we'll build up our user prompt with this `create_prompt` function. As with the old `FineTuning` endpoint, this is a critical point - you need to make sure your training examples match the format the model will see in production, otherwise your results may be poor." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "7d5cf5bd-618f-4f89-9649-8cd2c286dcb4", - "metadata": {}, - "outputs": [], - "source": [ - "def create_prompt(row):\n", - " \"\"\"Simple function to take in one of our observations and build it into a prompt\"\"\"\n", - " title = row['title']\n", - " ingredients = row['ingredients']\n", - " directions = row['directions']\n", - "\n", - " user_prompt = f'''Title: {title}\\n\\nIngredients: {ingredients}\\n\\nDirections: {directions}\\n\\nGeneric ingredients: '''\n", - " return user_prompt" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "6ec43c84-56e7-4c1b-9f66-795f55b4c7c5", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "User Prompt: Title: No-Bake Nut Cookies\n", - "\n", - "Ingredients: [\"1 c. firmly packed brown sugar\", \"1/2 c. evaporated milk\", \"1/2 tsp. vanilla\", \"1/2 c. broken nuts (pecans)\", \"2 Tbsp. butter or margarine\", \"3 1/2 c. bite size shredded rice biscuits\"]\n", - "\n", - "Directions: [\"In a heavy 2-quart saucepan, mix brown sugar, nuts, evaporated milk and butter or margarine.\", \"Stir over medium heat until mixture bubbles all over top.\", \"Boil and stir 5 minutes more. Take off heat.\", \"Stir in vanilla and cereal; mix well.\", \"Using 2 teaspoons, drop and shape into 30 clusters on wax paper.\", \"Let stand until firm, about 30 minutes.\"]\n", - "\n", - "Generic ingredients: \n", - "Assistant Response: ['brown sugar', 'milk', 'vanilla', 'nuts', 'butter', 'bite size shredded rice biscuits']\n", - "User Prompt: Title: Jewell Ball'S Chicken\n", - "\n", - "Ingredients: [\"1 small jar chipped beef, cut up\", \"4 boned chicken breasts\", \"1 can cream of mushroom soup\", \"1 carton sour cream\"]\n", - "\n", - "Directions: [\"Place chipped beef on bottom of baking dish.\", \"Place chicken on top of beef.\", \"Mix soup and cream together; pour over chicken. Bake, uncovered, at 275\\u00b0 for 3 hours.\"]\n", - "\n", - "Generic ingredients: \n", - "Assistant Response: ['beef', 'chicken breasts', 'cream of mushroom soup', 'sour cream']\n", - "User Prompt: Title: Creamy Corn\n", - "\n", - "Ingredients: [\"2 (16 oz.) pkg. frozen corn\", \"1 (8 oz.) pkg. cream cheese, cubed\", \"1/3 c. butter, cubed\", \"1/2 tsp. garlic powder\", \"1/2 tsp. salt\", \"1/4 tsp. pepper\"]\n", - "\n", - "Directions: [\"In a slow cooker, combine all ingredients. Cover and cook on low for 4 hours or until heated through and cheese is melted. Stir well before serving. Yields 6 servings.\"]\n", - "\n", - "Generic ingredients: \n", - "Assistant Response: ['frozen corn', 'cream cheese', 'butter', 'garlic powder', 'salt', 'pepper']\n", - "User Prompt: Title: Chicken Funny\n", - "\n", - "Ingredients: [\"1 large whole chicken\", \"2 (10 1/2 oz.) cans chicken gravy\", \"1 (10 1/2 oz.) can cream of mushroom soup\", \"1 (6 oz.) box Stove Top stuffing\", \"4 oz. shredded cheese\"]\n", - "\n", - "Directions: [\"Boil and debone chicken.\", \"Put bite size pieces in average size square casserole dish.\", \"Pour gravy and cream of mushroom soup over chicken; level.\", \"Make stuffing according to instructions on box (do not make too moist).\", \"Put stuffing on top of chicken and gravy; level.\", \"Sprinkle shredded cheese on top and bake at 350\\u00b0 for approximately 20 minutes or until golden and bubbly.\"]\n", - "\n", - "Generic ingredients: \n", - "Assistant Response: ['chicken', 'chicken gravy', 'cream of mushroom soup', 'shredded cheese']\n", - "User Prompt: Title: Reeses Cups(Candy) \n", - "\n", - "Ingredients: [\"1 c. peanut butter\", \"3/4 c. graham cracker crumbs\", \"1 c. melted butter\", \"1 lb. (3 1/2 c.) powdered sugar\", \"1 large pkg. chocolate chips\"]\n", - "\n", - "Directions: [\"Combine first four ingredients and press in 13 x 9-inch ungreased pan.\", \"Melt chocolate chips and spread over mixture. Refrigerate for about 20 minutes and cut into pieces before chocolate gets hard.\", \"Keep in refrigerator.\"]\n", - "\n", - "Generic ingredients: \n", - "Assistant Response: ['peanut butter', 'graham cracker crumbs', 'butter', 'powdered sugar', 'chocolate chips']\n" - ] - } - ], - "source": [ - "# Pick the top 5 entries to be our few-shot examples\n", - "for x,y in recipe_df.head(5).iterrows():\n", - " \n", - " user_prompt = create_prompt(y)\n", - " print(f\"User Prompt: {user_prompt}\")\n", - "\n", - " messages.append({\"role\":\"user\",\"content\":user_prompt})\n", - "\n", - " results = ast.literal_eval(y['NER'])\n", - " messages.append({\"role\":\"assistant\",\"content\": y['NER']})\n", - " print(f\"Assistant Response: {results}\")\n", - " " - ] - }, - { - "cell_type": "markdown", - "id": "542386a1-e5e3-4222-9f58-bc96ddbbfb63", - "metadata": {}, - "source": [ - "### Test few-shot approach\n", - "\n", - "We'll pick 300 test cases from our dataset and measure how many of the entities were extracted correctly.\n", - "\n", - "If a result has a 100% match then all entities were extracted - if it has less than some were missed by the model." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "f8d81b7f-366f-4a77-a0f5-599d45bbc183", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "300" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# We'll pick a test set from further on in the dataset\n", - "test_df = recipe_df.loc[1500:1799]\n", - "len(test_df)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "22d98bfd-2f76-4540-89fb-6e67fc336082", - "metadata": {}, - "outputs": [], - "source": [ - "results = []\n", - "for x,y in test_df.iterrows():\n", - " test_messages = messages.copy()\n", - " user_prompt = create_prompt(y)\n", - " test_messages.append({\"role\":\"user\",\"content\": user_prompt})\n", - "\n", - " try:\n", - " response = openai.ChatCompletion.create(\n", - " model=GPT_MODEL,\n", - " messages=test_messages,\n", - " temperature=0,\n", - " max_tokens=500\n", - " )\n", - " results.append((user_prompt,y['NER'],response['choices'][0]['message']['content']))\n", - "\n", - " except Exception as e:\n", - " print(e)\n", - " " - ] - }, - { - "cell_type": "markdown", - "id": "37dc6d6d-b1cf-423e-a998-4837f1b69ca2", - "metadata": {}, - "source": [ - "### Evaluate results\n", - "\n", - "Let's evaluate how well the model performed with 5 few-shot examples" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "e451b7ff-e474-4dfb-aeaa-08ae55cb86fd", - "metadata": {}, - "outputs": [], - "source": [ - "def evaluate_ner(row):\n", - " actual_results = ast.literal_eval(row['actual'])\n", - " predicted_results = ast.literal_eval(row['prediction'])\n", - " actual_set = set(actual_results)\n", - "\n", - " accuracy = 0\n", - " for x in predicted_results:\n", - " if x in actual_set:\n", - " accuracy += 1\n", - "\n", - " if len(actual_results) == 0:\n", - " score = 1\n", - " else:\n", - " score = accuracy / len(actual_results)\n", - " return score\n", - " " - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "cd9b5b56-7570-4910-8b35-5c9d2340b2c1", - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
inputactualpredictionscore
0Title: Pretzel Candy\\n\\nIngredients: [\"1 lb. w...[\"white chocolate\", \"pretzel sticks\", \"peanuts\"][\"white chocolate\", \"pretzel sticks\", \"salted ...0.666667
1Title: Salmon Party Ball\\n\\nIngredients: [\"8 o...[\"cream cheese\", \"salmon\", \"lemon juice\", \"hor...[\"cream cheese\", \"salmon\", \"lemon juice\", \"hor...0.777778
2Title: Fancy Fried Green Tomatoes\\n\\nIngredien...[\"sour cream\", \"green onion\", \"salt\", \"eggs\", ...[\"sour cream\", \"green onion\", \"salt\", \"eggs\", ...0.800000
3Title: Potluck Potatoes\\n\\nIngredients: [\"1 (2...[\"frozen hash brown potatoes\", \"onions\", \"salt...[\"frozen hash brown potatoes\", \"onions\", \"salt...1.000000
4Title: Old-Fashioned Sweet-Sour Cole Slaw\\n\\nI...[\"shredded green cabbage\", \"salt\", \"sugar\", \"c...[\"shredded green cabbage\", \"salt\", \"sugar\", \"c...1.000000
5Title: Peanut Brittle\\n\\nIngredients: [\"3 c. w...[\"white sugar\", \"water\", \"butter\", \"soda\", \"wh...[\"white sugar\", \"water\", \"butter\", \"soda\", \"wh...0.857143
6Title: Chicken Inspiration\\n\\nIngredients: [\"1...[\"chicken breast\", \"fresh mushrooms\", \"Provolo...[\"chicken breast\", \"mushrooms\", \"Provolone che...0.636364
7Title: Down East Blueberry Cake\\n\\nIngredients...[\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue...[\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue...1.000000
8Title: Cranberry-Pecan Bars\\n\\nIngredients: [\"...[\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan...[\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan...0.833333
9Title: Stacked Twinkie Cake\\n\\nIngredients: [\"...[\"coconut\", \"Twinkies\", \"pineapple\", \"pecans\",...[\"coconut\", \"instant French vanilla pudding\", ...0.600000
\n", - "
" - ], - "text/plain": [ - " input \\\n", - "0 Title: Pretzel Candy\\n\\nIngredients: [\"1 lb. w... \n", - "1 Title: Salmon Party Ball\\n\\nIngredients: [\"8 o... \n", - "2 Title: Fancy Fried Green Tomatoes\\n\\nIngredien... \n", - "3 Title: Potluck Potatoes\\n\\nIngredients: [\"1 (2... \n", - "4 Title: Old-Fashioned Sweet-Sour Cole Slaw\\n\\nI... \n", - "5 Title: Peanut Brittle\\n\\nIngredients: [\"3 c. w... \n", - "6 Title: Chicken Inspiration\\n\\nIngredients: [\"1... \n", - "7 Title: Down East Blueberry Cake\\n\\nIngredients... \n", - "8 Title: Cranberry-Pecan Bars\\n\\nIngredients: [\"... \n", - "9 Title: Stacked Twinkie Cake\\n\\nIngredients: [\"... \n", - "\n", - " actual \\\n", - "0 [\"white chocolate\", \"pretzel sticks\", \"peanuts\"] \n", - "1 [\"cream cheese\", \"salmon\", \"lemon juice\", \"hor... \n", - "2 [\"sour cream\", \"green onion\", \"salt\", \"eggs\", ... \n", - "3 [\"frozen hash brown potatoes\", \"onions\", \"salt... \n", - "4 [\"shredded green cabbage\", \"salt\", \"sugar\", \"c... \n", - "5 [\"white sugar\", \"water\", \"butter\", \"soda\", \"wh... \n", - "6 [\"chicken breast\", \"fresh mushrooms\", \"Provolo... \n", - "7 [\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue... \n", - "8 [\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan... \n", - "9 [\"coconut\", \"Twinkies\", \"pineapple\", \"pecans\",... \n", - "\n", - " prediction score \n", - "0 [\"white chocolate\", \"pretzel sticks\", \"salted ... 0.666667 \n", - "1 [\"cream cheese\", \"salmon\", \"lemon juice\", \"hor... 0.777778 \n", - "2 [\"sour cream\", \"green onion\", \"salt\", \"eggs\", ... 0.800000 \n", - "3 [\"frozen hash brown potatoes\", \"onions\", \"salt... 1.000000 \n", - "4 [\"shredded green cabbage\", \"salt\", \"sugar\", \"c... 1.000000 \n", - "5 [\"white sugar\", \"water\", \"butter\", \"soda\", \"wh... 0.857143 \n", - "6 [\"chicken breast\", \"mushrooms\", \"Provolone che... 0.636364 \n", - "7 [\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue... 1.000000 \n", - "8 [\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan... 0.833333 \n", - "9 [\"coconut\", \"instant French vanilla pudding\", ... 0.600000 " - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "results_df = pd.DataFrame(results)\n", - "results_df.columns = ['input','actual','prediction']\n", - "results_df['score'] = results_df.apply(lambda x: evaluate_ner(x),axis=1)\n", - "results_df.head(10)" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "ff94c184-ce76-46b0-9cdd-8bec041ee18f", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "0.806756229007777" - ] - }, - "execution_count": 10, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "results_df['score'].mean()" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "24ee164a-b619-4e14-a15f-ee361259329c", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGdCAYAAAD0e7I1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAj4UlEQVR4nO3dfVSUdf7/8dcgzIAGeLcMkBhkmpaVqxaR+m1NNkpzNT1brVZkrNaKpVKZpuZ6U6gVmaZSrqKe1Sx31a00usGyNfEmxG4NMy0pBW0VUFwQ4fr90Wl+O6klw8AMn56Pc+aPueaaizefSJ7nmmsYm2VZlgAAAAwV4OsBAAAA6hOxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBogb4ewB/U1NTo4MGDCg0Nlc1m8/U4AADgPFiWpePHjys6OloBAec+f0PsSDp48KBiYmJ8PQYAAPBAYWGh2rRpc87HiR1JoaGhkn5YrLCwMB9PAwAAzkdZWZliYmJcv8fPhdiRXC9dhYWFETsAADQyv3QJChcoAwAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaIG+HgAAAJy/2PHrfT1CrX09s59Pvz5ndgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEbzaey8//776t+/v6Kjo2Wz2bRu3Tq3xy3L0uOPP66oqCiFhIQoMTFRX375pds+R48e1dChQxUWFqbmzZsrJSVFJ06caMDvAgAA+DOfxk55ebmuuuoqzZ8//6yPz549W3PnzlVmZqa2bdumZs2aKSkpSRUVFa59hg4dqs8++0xvv/22Xn/9db3//vsaMWJEQ30LAADAzwX68ovffPPNuvnmm8/6mGVZmjNnjiZNmqQBAwZIkpYvXy6n06l169bpjjvu0O7du5Wdna0dO3aoe/fukqR58+apb9++evrppxUdHd1g3wsAAPBPfnvNzv79+1VUVKTExETXtvDwcMXHxys3N1eSlJubq+bNm7tCR5ISExMVEBCgbdu2nfPYlZWVKisrc7sBAAAz+W3sFBUVSZKcTqfbdqfT6XqsqKhIERERbo8HBgaqZcuWrn3OJj09XeHh4a5bTEyMl6cHAAD+wm9jpz5NmDBBpaWlrlthYaGvRwIAAPXEb2MnMjJSklRcXOy2vbi42PVYZGSkDh8+7Pb46dOndfToUdc+Z+NwOBQWFuZ2AwAAZvLb2ImLi1NkZKRycnJc28rKyrRt2zYlJCRIkhISElRSUqK8vDzXPhs3blRNTY3i4+MbfGYAAOB/fPpurBMnTmjv3r2u+/v379euXbvUsmVLtW3bVmPGjNGMGTPUvn17xcXFafLkyYqOjtbAgQMlSZ06ddJNN92k4cOHKzMzU1VVVRo1apTuuOMO3okFAAAk+Th2PvzwQ/Xu3dt1Py0tTZKUnJyspUuXaty4cSovL9eIESNUUlKinj17Kjs7W8HBwa7nrFixQqNGjVKfPn0UEBCgwYMHa+7cuQ3+vQAAAP9ksyzL8vUQvlZWVqbw8HCVlpZy/Q4AwK/Fjl/v6xFq7euZ/erluOf7+9tvr9kBAADwBmIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABjNr2OnurpakydPVlxcnEJCQtSuXTtNnz5dlmW59rEsS48//riioqIUEhKixMREffnllz6cGgAA+BO/jp1Zs2Zp4cKFev7557V7927NmjVLs2fP1rx581z7zJ49W3PnzlVmZqa2bdumZs2aKSkpSRUVFT6cHAAA+ItAXw/wc7Zs2aIBAwaoX79+kqTY2Fi99NJL2r59u6QfzurMmTNHkyZN0oABAyRJy5cvl9Pp1Lp163THHXf4bHYAAOAf/PrMznXXXaecnBzt2bNHkvTRRx9p8+bNuvnmmyVJ+/fvV1FRkRITE13PCQ8PV3x8vHJzc8953MrKSpWVlbndAACAmfz6zM748eNVVlamjh07qkmTJqqurtYTTzyhoUOHSpKKiookSU6n0+15TqfT9djZpKena+rUqfU3OAAA8Bt+fWbnlVde0YoVK7Ry5Urt3LlTy5Yt09NPP61ly5bV6bgTJkxQaWmp61ZYWOiliQEAgL/x6zM7jzzyiMaPH++69uaKK67QN998o/T0dCUnJysyMlKSVFxcrKioKNfziouL1aVLl3Me1+FwyOFw1OvsAADAP/j1mZ2TJ08qIMB9xCZNmqimpkaSFBcXp8jISOXk5LgeLysr07Zt25SQkNCgswIAAP/k12d2+vfvryeeeEJt27bV5Zdfrvz8fGVkZOjee++VJNlsNo0ZM0YzZsxQ+/btFRcXp8mTJys6OloDBw707fAAAMAv+HXszJs3T5MnT9bIkSN1+PBhRUdH67777tPjjz/u2mfcuHEqLy/XiBEjVFJSop49eyo7O1vBwcE+nBwAAPgLm/W/f474V6qsrEzh4eEqLS1VWFiYr8cBAOCcYsev9/UItfb1zH71ctzz/f3t19fsAAAA1BWxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMFqgrwcAAMBXYsev9/UIaACc2QEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYzaPY2bdvn7fnAAAAqBcexc4ll1yi3r176+9//7sqKiq8PRMAAIDXeBQ7O3fu1JVXXqm0tDRFRkbqvvvu0/bt2709GwAAQJ15FDtdunTRc889p4MHD2rJkiU6dOiQevbsqc6dOysjI0NHjhzx9pwAAAAeqdMFyoGBgRo0aJBWr16tWbNmae/evXr44YcVExOju+++W4cOHfLWnAAAAB6pU+x8+OGHGjlypKKiopSRkaGHH35YX331ld5++20dPHhQAwYM8NacAAAAHgn05EkZGRnKyspSQUGB+vbtq+XLl6tv374KCPihneLi4rR06VLFxsZ6c1YAAIBa8yh2Fi5cqHvvvVf33HOPoqKizrpPRESEFi9eXKfhAAAA6sqj2Pnyyy9/cR+73a7k5GRPDg8AAOA1Hl2zk5WVpdWrV5+xffXq1Vq2bFmdhwIAAPAWj2InPT1drVu3PmN7RESEnnzyyToPBQAA4C0exc6BAwcUFxd3xvaLLrpIBw4cqPNQAAAA3uJR7EREROjjjz8+Y/tHH32kVq1a1XkoAAAAb/Eodv70pz/pwQcf1Lvvvqvq6mpVV1dr48aNGj16tO644w5vzwgAAOAxj96NNX36dH399dfq06ePAgN/OERNTY3uvvturtkBAAB+xaPYsdvtevnllzV9+nR99NFHCgkJ0RVXXKGLLrrI2/MBAADUiUex86MOHTqoQ4cO3poFAADA6zyKnerqai1dulQ5OTk6fPiwampq3B7fuHGjV4YDAACoK49iZ/To0Vq6dKn69eunzp07y2azeXsuAAAAr/AodlatWqVXXnlFffv29fY8Z/juu+/06KOP6o033tDJkyd1ySWXKCsrS927d5ckWZalKVOmaNGiRSopKVGPHj20cOFCtW/fvt5nAwAA/s+jt57b7XZdcskl3p7lDMeOHVOPHj0UFBSkN954Q59//rmeeeYZtWjRwrXP7NmzNXfuXGVmZmrbtm1q1qyZkpKSVFFRUe/zAQAA/+fRmZ2HHnpIzz33nJ5//vl6fQlr1qxZiomJUVZWlmvb//7lZsuyNGfOHE2aNEkDBgyQJC1fvlxOp1Pr1q3jb/4AAADPYmfz5s1699139cYbb+jyyy9XUFCQ2+Nr1qzxynCvvvqqkpKS9Mc//lGbNm3ShRdeqJEjR2r48OGSpP3796uoqEiJiYmu54SHhys+Pl65ubnEDgAA8Cx2mjdvrltvvdXbs5xh3759WrhwodLS0vTYY49px44devDBB2W325WcnKyioiJJktPpdHue0+l0PXY2lZWVqqysdN0vKyurn28AAAD4nEex878vK9Wnmpoade/e3fVXmX/729/q008/VWZmppKTkz0+bnp6uqZOneqtMQEAgB/z6AJlSTp9+rTeeecdvfDCCzp+/Lgk6eDBgzpx4oTXhouKitJll13mtq1Tp06uT1aPjIyUJBUXF7vtU1xc7HrsbCZMmKDS0lLXrbCw0GszAwAA/+LRmZ1vvvlGN910kw4cOKDKykr9/ve/V2hoqGbNmqXKykplZmZ6ZbgePXqooKDAbduePXtcH0sRFxenyMhI5eTkqEuXLpJ+eElq27Zt+stf/nLO4zocDjkcDq/MCAAA/JtHZ3ZGjx6t7t2769ixYwoJCXFtv/XWW5WTk+O14caOHautW7fqySef1N69e7Vy5Uq9+OKLSk1NlSTZbDaNGTNGM2bM0KuvvqpPPvlEd999t6KjozVw4ECvzQEAABovj87s/Pvf/9aWLVtkt9vdtsfGxuq7777zymCSdPXVV2vt2rWaMGGCpk2bpri4OM2ZM0dDhw517TNu3DiVl5drxIgRKikpUc+ePZWdna3g4GCvzQEAABovj2KnpqZG1dXVZ2z/9ttvFRoaWueh/tctt9yiW2655ZyP22w2TZs2TdOmTfPq1wUAAGbw6GWsG2+8UXPmzHHdt9lsOnHihKZMmdIgHyEBAABwvjw6s/PMM88oKSlJl112mSoqKjRkyBB9+eWXat26tV566SVvzwgAAOAxj2KnTZs2+uijj7Rq1Sp9/PHHOnHihFJSUjR06FC3C5YBAAB8zaPYkaTAwEDdeeed3pwFAADA6zyKneXLl//s43fffbdHwwAAAHibR7EzevRot/tVVVU6efKk7Ha7mjZtSuwAAAC/4dG7sY4dO+Z2O3HihAoKCtSzZ08uUAYAAH7F48/G+qn27dtr5syZZ5z1AQAA8CWvxY70w0XLBw8e9OYhAQAA6sSja3ZeffVVt/uWZenQoUN6/vnn1aNHD68MBgAA4A0exc5PP2TTZrPpN7/5jW644QY988wz3pgLAADAKzz+bCwAAIDGwKvX7AAAAPgbj87spKWlnfe+GRkZnnwJAAAAr/AodvLz85Wfn6+qqipdeumlkqQ9e/aoSZMm6tq1q2s/m83mnSkBAAA85FHs9O/fX6GhoVq2bJlatGgh6Yc/NDhs2DD16tVLDz30kFeHBAAA8JRH1+w888wzSk9Pd4WOJLVo0UIzZszg3VgAAMCveBQ7ZWVlOnLkyBnbjxw5ouPHj9d5KAAAAG/xKHZuvfVWDRs2TGvWrNG3336rb7/9Vv/85z+VkpKiQYMGeXtGAAAAj3l0zU5mZqYefvhhDRkyRFVVVT8cKDBQKSkpeuqpp7w6IAAAQF14FDtNmzbVggUL9NRTT+mrr76SJLVr107NmjXz6nAAAAB1Vac/Knjo0CEdOnRI7du3V7NmzWRZlrfmAgAA8AqPYuc///mP+vTpow4dOqhv3746dOiQJCklJYW3nQMAAL/iUeyMHTtWQUFBOnDggJo2berafvvttys7O9trwwEAANSVR9fsvPXWW3rzzTfVpk0bt+3t27fXN99845XBAAAAvMGjMzvl5eVuZ3R+dPToUTkcjjoPBQAA4C0exU6vXr20fPly132bzaaamhrNnj1bvXv39tpwAAAAdeXRy1izZ89Wnz599OGHH+rUqVMaN26cPvvsMx09elQffPCBt2cEAADwmEdndjp37qw9e/aoZ8+eGjBggMrLyzVo0CDl5+erXbt23p4RAADAY7U+s1NVVaWbbrpJmZmZmjhxYn3MBAAA4DW1PrMTFBSkjz/+uD5mAQAA8DqPXsa68847tXjxYm/PAgAA4HUeXaB8+vRpLVmyRO+88466det2xmdiZWRkeGU4AACAuqpV7Ozbt0+xsbH69NNP1bVrV0nSnj173Pax2Wzemw4AAKCOahU77du316FDh/Tuu+9K+uHjIebOnSun01kvwwEAANRVra7Z+emnmr/xxhsqLy/36kAAAADe5NEFyj/6afwAAAD4m1rFjs1mO+OaHK7RAQAA/qxW1+xYlqV77rnH9WGfFRUVuv/++894N9aaNWu8NyEAAEAd1Cp2kpOT3e7feeedXh0GAADA22oVO1lZWfU1BwAAQL2o0wXKAAAA/o7YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRavV3dgAAOJfY8et9PQJwVpzZAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGa1SxM3PmTNlsNo0ZM8a1raKiQqmpqWrVqpUuuOACDR48WMXFxb4bEgAA+JVGEzs7duzQCy+8oCuvvNJt+9ixY/Xaa69p9erV2rRpkw4ePKhBgwb5aEoAAOBvGkXsnDhxQkOHDtWiRYvUokUL1/bS0lItXrxYGRkZuuGGG9StWzdlZWVpy5Yt2rp1qw8nBgAA/qJRxE5qaqr69eunxMREt+15eXmqqqpy296xY0e1bdtWubm55zxeZWWlysrK3G4AAMBMfv8XlFetWqWdO3dqx44dZzxWVFQku92u5s2bu213Op0qKio65zHT09M1depUb48KAAD8kF+f2SksLNTo0aO1YsUKBQcHe+24EyZMUGlpqetWWFjotWMDAAD/4texk5eXp8OHD6tr164KDAxUYGCgNm3apLlz5yowMFBOp1OnTp1SSUmJ2/OKi4sVGRl5zuM6HA6FhYW53QAAgJn8+mWsPn366JNPPnHbNmzYMHXs2FGPPvqoYmJiFBQUpJycHA0ePFiSVFBQoAMHDighIcEXIwMAAD/j17ETGhqqzp07u21r1qyZWrVq5dqekpKitLQ0tWzZUmFhYXrggQeUkJCga6+91hcjAwAAP+PXsXM+nn32WQUEBGjw4MGqrKxUUlKSFixY4OuxAACAn7BZlmX5eghfKysrU3h4uEpLS7l+BwA8FDt+va9HgJ/6ema/ejnu+f7+9usLlAEAAOqK2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGC0Rv+p5wBgIj5UE/AezuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIwW6OsBAKC+xY5f7+sRAPgQZ3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABGI3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDRiBwAAGI3YAQAARiN2AACA0YgdAABgNGIHAAAYjdgBAABG8+vYSU9P19VXX63Q0FBFRERo4MCBKigocNunoqJCqampatWqlS644AINHjxYxcXFPpoYAAD4G7+OnU2bNik1NVVbt27V22+/raqqKt14440qLy937TN27Fi99tprWr16tTZt2qSDBw9q0KBBPpwaAAD4k0BfD/BzsrOz3e4vXbpUERERysvL0//93/+ptLRUixcv1sqVK3XDDTdIkrKystSpUydt3bpV1157rS/GBgAAfsSvz+z8VGlpqSSpZcuWkqS8vDxVVVUpMTHRtU/Hjh3Vtm1b5ebmnvM4lZWVKisrc7sBAAAzNZrYqamp0ZgxY9SjRw917txZklRUVCS73a7mzZu77et0OlVUVHTOY6Wnpys8PNx1i4mJqc/RAQCADzWa2ElNTdWnn36qVatW1flYEyZMUGlpqetWWFjohQkBAIA/8utrdn40atQovf7663r//ffVpk0b1/bIyEidOnVKJSUlbmd3iouLFRkZec7jORwOORyO+hwZAAD4Cb8+s2NZlkaNGqW1a9dq48aNiouLc3u8W7duCgoKUk5OjmtbQUGBDhw4oISEhIYeFwAA+CG/PrOTmpqqlStX6l//+pdCQ0Nd1+GEh4crJCRE4eHhSklJUVpamlq2bKmwsDA98MADSkhI4J1YAABAkp/HzsKFCyVJv/vd79y2Z2Vl6Z577pEkPfvsswoICNDgwYNVWVmppKQkLViwoIEnBQAA/sqvY8eyrF/cJzg4WPPnz9f8+fMbYCIAANDY+PU1OwAAAHVF7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMFqgrwcA0LjEjl/v6xEAoFY4swMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAoxE7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGiBvh4A+LWKHb/e1yMAwK8CZ3YAAIDRiB0AAGA0YgcAABiN2AEAAEYjdgAAgNGIHQAAYDTeel7PGuPbi7+e2c/XIwAA4DWc2QEAAEYzJnbmz5+v2NhYBQcHKz4+Xtu3b/f1SAAAwA8YETsvv/yy0tLSNGXKFO3cuVNXXXWVkpKSdPjwYV+PBgAAfMyI2MnIyNDw4cM1bNgwXXbZZcrMzFTTpk21ZMkSX48GAAB8rNFfoHzq1Cnl5eVpwoQJrm0BAQFKTExUbm7uWZ9TWVmpyspK1/3S0lJJUllZmdfnq6k86fVj1rf6WAecqTH+bACAJ+rr98qPx7Us62f3a/Sx8/3336u6ulpOp9Ntu9Pp1BdffHHW56Snp2vq1KlnbI+JiamXGRub8Dm+ngAAYJL6/r1y/PhxhYeHn/PxRh87npgwYYLS0tJc92tqanT06FG1atVKNpvNa1+nrKxMMTExKiwsVFhYmNeOC3esc8NhrRsG69wwWOeGUZ/rbFmWjh8/rujo6J/dr9HHTuvWrdWkSRMVFxe7bS8uLlZkZORZn+NwOORwONy2NW/evL5GVFhYGP8jNQDWueGw1g2DdW4YrHPDqK91/rkzOj9q9Bco2+12devWTTk5Oa5tNTU1ysnJUUJCgg8nAwAA/qDRn9mRpLS0NCUnJ6t79+665pprNGfOHJWXl2vYsGG+Hg0AAPiYEbFz++2368iRI3r88cdVVFSkLl26KDs7+4yLlhuaw+HQlClTznjJDN7FOjcc1rphsM4Ng3VuGP6wzjbrl96vBQAA0Ig1+mt2AAAAfg6xAwAAjEbsAAAAoxE7AADAaMROHc2fP1+xsbEKDg5WfHy8tm/f/rP7r169Wh07dlRwcLCuuOIKbdiwoYEmbdxqs86LFi1Sr1691KJFC7Vo0UKJiYm/+N8FP6jtz/OPVq1aJZvNpoEDB9bvgAap7VqXlJQoNTVVUVFRcjgc6tChA/9+nIfarvOcOXN06aWXKiQkRDExMRo7dqwqKioaaNrG6f3331f//v0VHR0tm82mdevW/eJz3nvvPXXt2lUOh0OXXHKJli5dWr9DWvDYqlWrLLvdbi1ZssT67LPPrOHDh1vNmze3iouLz7r/Bx98YDVp0sSaPXu29fnnn1uTJk2ygoKCrE8++aSBJ29carvOQ4YMsebPn2/l5+dbu3fvtu655x4rPDzc+vbbbxt48saltuv8o/3791sXXnih1atXL2vAgAENM2wjV9u1rqystLp372717dvX2rx5s7V//37rvffes3bt2tXAkzcutV3nFStWWA6Hw1qxYoW1f/9+680337SioqKssWPHNvDkjcuGDRusiRMnWmvWrLEkWWvXrv3Z/fft22c1bdrUSktLsz7//HNr3rx5VpMmTazs7Ox6m5HYqYNrrrnGSk1Ndd2vrq62oqOjrfT09LPuf9ttt1n9+vVz2xYfH2/dd9999TpnY1fbdf6p06dPW6GhodayZcvqa0QjeLLOp0+ftq677jrrb3/7m5WcnEzsnKfarvXChQutiy++2Dp16lRDjWiE2q5zamqqdcMNN7htS0tLs3r06FGvc5rkfGJn3Lhx1uWXX+627fbbb7eSkpLqbS5exvLQqVOnlJeXp8TERNe2gIAAJSYmKjc396zPyc3NddtfkpKSks65Pzxb5586efKkqqqq1LJly/oas9HzdJ2nTZumiIgIpaSkNMSYRvBkrV999VUlJCQoNTVVTqdTnTt31pNPPqnq6uqGGrvR8WSdr7vuOuXl5ble6tq3b582bNigvn37NsjMvxa++F1oxF9Q9oXvv/9e1dXVZ/yVZqfTqS+++OKszykqKjrr/kVFRfU2Z2PnyTr/1KOPPqro6Ogz/ufC/+fJOm/evFmLFy/Wrl27GmBCc3iy1vv27dPGjRs1dOhQbdiwQXv37tXIkSNVVVWlKVOmNMTYjY4n6zxkyBB9//336tmzpyzL0unTp3X//ffrsccea4iRfzXO9buwrKxM//3vfxUSEuL1r8mZHRht5syZWrVqldauXavg4GBfj2OM48eP66677tKiRYvUunVrX49jvJqaGkVEROjFF19Ut27ddPvtt2vixInKzMz09WhGee+99/Tkk09qwYIF2rlzp9asWaP169dr+vTpvh4NdcSZHQ+1bt1aTZo0UXFxsdv24uJiRUZGnvU5kZGRtdofnq3zj55++mnNnDlT77zzjq688sr6HLPRq+06f/XVV/r666/Vv39/17aamhpJUmBgoAoKCtSuXbv6HbqR8uRnOioqSkFBQWrSpIlrW6dOnVRUVKRTp07JbrfX68yNkSfrPHnyZN11113685//LEm64oorVF5erhEjRmjixIkKCOD8gDec63dhWFhYvZzVkTiz4zG73a5u3bopJyfHta2mpkY5OTlKSEg463MSEhLc9pekt99++5z7w7N1lqTZs2dr+vTpys7OVvfu3Rti1EattuvcsWNHffLJJ9q1a5fr9oc//EG9e/fWrl27FBMT05DjNyqe/Ez36NFDe/fudQWlJO3Zs0dRUVGEzjl4ss4nT548I2h+DEyLj5H0Gp/8Lqy3S59/BVatWmU5HA5r6dKl1ueff26NGDHCat68uVVUVGRZlmXddddd1vjx4137f/DBB1ZgYKD19NNPW7t377amTJnCW8/PQ23XeebMmZbdbrf+8Y9/WIcOHXLdjh8/7qtvoVGo7Tr/FO/GOn+1XesDBw5YoaGh1qhRo6yCggLr9ddftyIiIqwZM2b46ltoFGq7zlOmTLFCQ0Otl156ydq3b5/11ltvWe3atbNuu+02X30LjcLx48et/Px8Kz8/35JkZWRkWPn5+dY333xjWZZljR8/3rrrrrtc+//41vNHHnnE2r17tzV//nzeeu7v5s2bZ7Vt29ay2+3WNddcY23dutX12PXXX28lJye77f/KK69YHTp0sOx2u3X55Zdb69evb+CJG6farPNFF11kSTrjNmXKlIYfvJGp7c/z/yJ2aqe2a71lyxYrPj7ecjgc1sUXX2w98cQT1unTpxt46sanNutcVVVl/fWvf7XatWtnBQcHWzExMdbIkSOtY8eONfzgjci777571n9zf1zb5ORk6/rrrz/jOV26dLHsdrt18cUXW1lZWfU6o82yODcHAADMxTU7AADAaMQOAAAwGrEDAACMRuwAAACjETsAAMBoxA4AADAasQMAAIxG7AAAAKMROwAAwGjEDgAAMBqxAwAAjEbsAAAAo/0/6ZyBZxAfhiIAAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "results_df['score'].plot.hist(bins=10)" - ] - }, - { - "cell_type": "markdown", - "id": "d3f012fc-0891-4ea6-8fc5-ae49398cc8c7", - "metadata": {}, - "source": [ - "Results are ok, with 80% of entities extracted correctly - the histogram doesn't look great though, with under half of results being 100% accurate. Lets see if baking learning into the model using fine-tuning gets us to a better place!" - ] - }, - { - "cell_type": "markdown", - "id": "8dd6be2c", - "metadata": {}, - "source": [ - "## OPTIONAL: Few-shot with GPT-4\n", - "\n", - "For some use cases the increased cost and latency of GPT-4 will be worth the trade-off of increased accuracy. In these cases, you may want to extend the few-shot approach to GPT-4 before diving in with fine-tuning.\n", - "\n", - "We'll perform the above test again with GPT-4 this time to give us another data point to compare our fine-tuning performance to." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "62bb8702", - "metadata": {}, - "outputs": [], - "source": [ - "# If you don't run this section, run this cell so the evaluation at the bottom works\n", - "gpt_4_score = 0.8201281042085995" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "aeec72b7", - "metadata": {}, - "outputs": [], - "source": [ - "results_gpt4 = []\n", - "for x,y in test_df.iterrows():\n", - " test_messages = messages.copy()\n", - " user_prompt = create_prompt(y)\n", - " test_messages.append({\"role\":\"user\",\"content\": user_prompt})\n", - "\n", - " try:\n", - " response = openai.ChatCompletion.create(\n", - " model='gpt-4',\n", - " messages=test_messages,\n", - " temperature=0,\n", - " max_tokens=500\n", - " )\n", - " results_gpt4.append((user_prompt,y['NER'],response['choices'][0]['message']['content']))\n", - "\n", - " except Exception as e:\n", - " print(e)\n", - " " - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "e5e6e8fe", - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
inputactualpredictionscore
0Title: Pretzel Candy\\n\\nIngredients: [\"1 lb. w...[\"white chocolate\", \"pretzel sticks\", \"peanuts\"][\"white chocolate\", \"pretzel sticks\", \"salted ...0.666667
1Title: Salmon Party Ball\\n\\nIngredients: [\"8 o...[\"cream cheese\", \"salmon\", \"lemon juice\", \"hor...[\"cream cheese\", \"salmon\", \"lemon juice\", \"hor...0.888889
2Title: Fancy Fried Green Tomatoes\\n\\nIngredien...[\"sour cream\", \"green onion\", \"salt\", \"eggs\", ...[\"sour cream\", \"green onion\", \"salt\", \"eggs\", ...0.800000
3Title: Potluck Potatoes\\n\\nIngredients: [\"1 (2...[\"frozen hash brown potatoes\", \"onions\", \"salt...[\"hash brown potatoes\", \"onions\", \"salt\", \"pep...0.900000
4Title: Old-Fashioned Sweet-Sour Cole Slaw\\n\\nI...[\"shredded green cabbage\", \"salt\", \"sugar\", \"c...[\"green cabbage\", \"salt\", \"sugar\", \"cider vine...0.800000
5Title: Peanut Brittle\\n\\nIngredients: [\"3 c. w...[\"white sugar\", \"water\", \"butter\", \"soda\", \"wh...[\"white sugar\", \"water\", \"butter\", \"soda\", \"wh...0.857143
6Title: Chicken Inspiration\\n\\nIngredients: [\"1...[\"chicken breast\", \"fresh mushrooms\", \"Provolo...[\"chicken breast\", \"fresh mushrooms\", \"Provolo...0.636364
7Title: Down East Blueberry Cake\\n\\nIngredients...[\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue...[\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue...1.000000
8Title: Cranberry-Pecan Bars\\n\\nIngredients: [\"...[\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan...[\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan...0.833333
9Title: Stacked Twinkie Cake\\n\\nIngredients: [\"...[\"coconut\", \"Twinkies\", \"pineapple\", \"pecans\",...[\"coconut\", \"instant French vanilla pudding\", ...0.800000
\n", - "
" - ], - "text/plain": [ - " input \\\n", - "0 Title: Pretzel Candy\\n\\nIngredients: [\"1 lb. w... \n", - "1 Title: Salmon Party Ball\\n\\nIngredients: [\"8 o... \n", - "2 Title: Fancy Fried Green Tomatoes\\n\\nIngredien... \n", - "3 Title: Potluck Potatoes\\n\\nIngredients: [\"1 (2... \n", - "4 Title: Old-Fashioned Sweet-Sour Cole Slaw\\n\\nI... \n", - "5 Title: Peanut Brittle\\n\\nIngredients: [\"3 c. w... \n", - "6 Title: Chicken Inspiration\\n\\nIngredients: [\"1... \n", - "7 Title: Down East Blueberry Cake\\n\\nIngredients... \n", - "8 Title: Cranberry-Pecan Bars\\n\\nIngredients: [\"... \n", - "9 Title: Stacked Twinkie Cake\\n\\nIngredients: [\"... \n", - "\n", - " actual \\\n", - "0 [\"white chocolate\", \"pretzel sticks\", \"peanuts\"] \n", - "1 [\"cream cheese\", \"salmon\", \"lemon juice\", \"hor... \n", - "2 [\"sour cream\", \"green onion\", \"salt\", \"eggs\", ... \n", - "3 [\"frozen hash brown potatoes\", \"onions\", \"salt... \n", - "4 [\"shredded green cabbage\", \"salt\", \"sugar\", \"c... \n", - "5 [\"white sugar\", \"water\", \"butter\", \"soda\", \"wh... \n", - "6 [\"chicken breast\", \"fresh mushrooms\", \"Provolo... \n", - "7 [\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue... \n", - "8 [\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan... \n", - "9 [\"coconut\", \"Twinkies\", \"pineapple\", \"pecans\",... \n", - "\n", - " prediction score \n", - "0 [\"white chocolate\", \"pretzel sticks\", \"salted ... 0.666667 \n", - "1 [\"cream cheese\", \"salmon\", \"lemon juice\", \"hor... 0.888889 \n", - "2 [\"sour cream\", \"green onion\", \"salt\", \"eggs\", ... 0.800000 \n", - "3 [\"hash brown potatoes\", \"onions\", \"salt\", \"pep... 0.900000 \n", - "4 [\"green cabbage\", \"salt\", \"sugar\", \"cider vine... 0.800000 \n", - "5 [\"white sugar\", \"water\", \"butter\", \"soda\", \"wh... 0.857143 \n", - "6 [\"chicken breast\", \"fresh mushrooms\", \"Provolo... 0.636364 \n", - "7 [\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue... 1.000000 \n", - "8 [\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan... 0.833333 \n", - "9 [\"coconut\", \"instant French vanilla pudding\", ... 0.800000 " - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "results_gpt4_df = pd.DataFrame(results_gpt4)\n", - "results_gpt4_df.columns = ['input','actual','prediction']\n", - "results_gpt4_df['score'] = results_gpt4_df.apply(lambda x: evaluate_ner(x),axis=1)\n", - "results_gpt4_df.head(10)" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "b01d1a12", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "0.8183172576477531" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "results_gpt4_df['score'].mean()" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "9fc1a911", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 16, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGdCAYAAAD0e7I1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAjwUlEQVR4nO3dfVSUdf7/8dcgcqMB3sUAKwaZpqltpUWkfVuVE6XH1fRsupqpsVkblkrm6nq3pYVaEWkqm+vtWctyV91Kowzv1iQt024NLS01BW1NRnFBhM/vD0/z20ktGQZm+PR8nDN/zDXXXL75ZPI811wz4zDGGAEAAFgqyN8DAAAA1CRiBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDVgv09QCCorKzU4cOHFRERIYfD4e9xAADAJTDG6OTJk4qLi1NQ0MXP3xA7kg4fPqz4+Hh/jwEAALxw8OBBNW/e/KKPEzuSIiIiJJ1brMjISD9PAwAALoXL5VJ8fLz79/jFEDuS+6WryMhIYgcAgDrm5y5B4QJlAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYLdjfAwAAgEuXMG6Nv0eosq+n9/Trn8+ZHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYza+xs3nzZvXq1UtxcXFyOBxavXq1x+PGGE2ePFmxsbEKDw9XSkqK9u7d67HP8ePHNWjQIEVGRqpRo0ZKS0vTqVOnavGnAAAAgcyvsVNSUqJf//rXmjNnzgUfnzlzpmbNmqWcnBxt27ZNDRs2VGpqqkpLS937DBo0SJ999pnWrVunN954Q5s3b9bw4cNr60cAAAABLtiff/idd96pO++884KPGWOUnZ2tiRMnqnfv3pKkpUuXyul0avXq1RowYIB2796t3Nxcvf/+++rUqZMkafbs2erRo4eeeeYZxcXF1drPAgAAAlPAXrOzf/9+FRYWKiUlxb0tKipKSUlJys/PlyTl5+erUaNG7tCRpJSUFAUFBWnbtm0XPXZZWZlcLpfHDQAA2ClgY6ewsFCS5HQ6PbY7nU73Y4WFhYqOjvZ4PDg4WE2aNHHvcyGZmZmKiopy3+Lj4308PQAACBQBGzs1afz48SouLnbfDh486O+RAABADQnY2ImJiZEkFRUVeWwvKipyPxYTE6OjR496PH727FkdP37cvc+FhIaGKjIy0uMGAADsFLCxk5iYqJiYGOXl5bm3uVwubdu2TcnJyZKk5ORknThxQjt27HDvs379elVWViopKanWZwYAAIHHr+/GOnXqlL788kv3/f3792vXrl1q0qSJWrRooVGjRmnatGlq1aqVEhMTNWnSJMXFxalPnz6SpLZt2+qOO+7Q/fffr5ycHJWXl2vEiBEaMGAA78QCAACS/Bw7H3zwgbp27eq+n5GRIUkaMmSIFi9erLFjx6qkpETDhw/XiRMn1KVLF+Xm5iosLMz9nGXLlmnEiBHq3r27goKC1K9fP82aNavWfxYAABCYHMYY4+8h/M3lcikqKkrFxcVcvwMACGgJ49b4e4Qq+3p6zxo57qX+/g7Ya3YAAAB8gdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YL9PQAAAP6SMG6Nv0dALeDMDgAAsBqxAwAArEbsAAAAqxE7AADAagEdOxUVFZo0aZISExMVHh6uli1baurUqTLGuPcxxmjy5MmKjY1VeHi4UlJStHfvXj9ODQAAAklAx86MGTM0b948vfDCC9q9e7dmzJihmTNnavbs2e59Zs6cqVmzZiknJ0fbtm1Tw4YNlZqaqtLSUj9ODgAAAkVAv/V869at6t27t3r27ClJSkhI0Msvv6zt27dLOndWJzs7WxMnTlTv3r0lSUuXLpXT6dTq1as1YMAAv80OAAACQ0Cf2bnllluUl5enPXv2SJI++ugjbdmyRXfeeackaf/+/SosLFRKSor7OVFRUUpKSlJ+fr5fZgYAAIEloM/sjBs3Ti6XS23atFG9evVUUVGhJ598UoMGDZIkFRYWSpKcTqfH85xOp/uxCykrK1NZWZn7vsvlqoHpAQBAIAjoMzuvvvqqli1bppdeekkffvihlixZomeeeUZLliyp1nEzMzMVFRXlvsXHx/toYgAAEGgCOnYee+wxjRs3TgMGDFCHDh00ePBgjR49WpmZmZKkmJgYSVJRUZHH84qKityPXcj48eNVXFzsvh08eLDmfggAAOBXAR07p0+fVlCQ54j16tVTZWWlJCkxMVExMTHKy8tzP+5yubRt2zYlJydf9LihoaGKjIz0uAEAADsF9DU7vXr10pNPPqkWLVqoXbt22rlzp7KysnTfffdJkhwOh0aNGqVp06apVatWSkxM1KRJkxQXF6c+ffr4d3gAABAQAjp2Zs+erUmTJumhhx7S0aNHFRcXpwceeECTJ0927zN27FiVlJRo+PDhOnHihLp06aLc3FyFhYX5cXIAABAoHOZ/P474F8rlcikqKkrFxcW8pAUAvyAJ49b4e4RfhK+n96yR417q7++AvmYHAACguogdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1r2Jn3759vp4DAACgRngVO1dddZW6du2qv//97yotLfX1TAAAAD7jVex8+OGHuvbaa5WRkaGYmBg98MAD2r59u69nAwAAqDavYue6667T888/r8OHD2vhwoU6cuSIunTpovbt2ysrK0vHjh3z9ZwAAABeqdYFysHBwerbt69WrFihGTNm6Msvv9SYMWMUHx+ve++9V0eOHPHVnAAAAF6pVux88MEHeuihhxQbG6usrCyNGTNGX331ldatW6fDhw+rd+/evpoTAADAK8HePCkrK0uLFi1SQUGBevTooaVLl6pHjx4KCjrXTomJiVq8eLESEhJ8OSsAAECVeRU78+bN03333aehQ4cqNjb2gvtER0drwYIF1RoOAACguryKnb179/7sPiEhIRoyZIg3hwcAAPAZr67ZWbRokVasWHHe9hUrVmjJkiXVHgoAAMBXvIqdzMxMNWvW7Lzt0dHReuqpp6o9FAAAgK94FTsHDhxQYmLieduvuOIKHThwoNpDAQAA+IpXsRMdHa2PP/74vO0fffSRmjZtWu2hAAAAfMWr2Pn973+vRx55RBs2bFBFRYUqKiq0fv16jRw5UgMGDPD1jAAAAF7z6t1YU6dO1ddff63u3bsrOPjcISorK3XvvfdyzQ4AAAgoXsVOSEiIXnnlFU2dOlUfffSRwsPD1aFDB11xxRW+ng8AAKBavIqdH7Ru3VqtW7f21SwAAAA+51XsVFRUaPHixcrLy9PRo0dVWVnp8fj69et9MhwAAEB1eRU7I0eO1OLFi9WzZ0+1b99eDofD13MBAAD4hFexs3z5cr366qvq0aOHr+cBAADwKa/eeh4SEqKrrrrK17MAAAD4nFex8+ijj+r555+XMcbX8wAAAPiUVy9jbdmyRRs2bNCbb76pdu3aqX79+h6Pr1y50ifDAQAAVJdXZ3YaNWqku+66S7fddpuaNWumqKgoj5svffvtt7rnnnvUtGlT9+f5fPDBB+7HjTGaPHmyYmNjFR4erpSUFO3du9enMwAAgLrLqzM7ixYt8vUcF/T999+rc+fO6tq1q958801dfvnl2rt3rxo3buzeZ+bMmZo1a5aWLFmixMRETZo0Sampqfr8888VFhZWK3MCAIDA5fWHCp49e1YbN27UV199pYEDByoiIkKHDx9WZGSkLrvsMp8MN2PGDMXHx3vE1f9+27oxRtnZ2Zo4caJ69+4tSVq6dKmcTqdWr17N93QBAADvXsb65ptv1KFDB/Xu3Vvp6ek6duyYpHNxMmbMGJ8N99prr6lTp0763e9+p+joaF1//fWaP3+++/H9+/ersLBQKSkp7m1RUVFKSkpSfn7+RY9bVlYml8vlcQMAAHbyKnZGjhypTp066fvvv1d4eLh7+1133aW8vDyfDbdv3z7NmzdPrVq10ltvvaU//vGPeuSRR7RkyRJJUmFhoSTJ6XR6PM/pdLofu5DMzEyPa4zi4+N9NjMAAAgsXr2M9e9//1tbt25VSEiIx/aEhAR9++23PhlMOvdN6p06dXJ/k/r111+vTz/9VDk5ORoyZIjXxx0/frwyMjLc910uF8EDAIClvDqzU1lZqYqKivO2Hzp0SBEREdUe6gexsbG65pprPLa1bdtWBw4ckCTFxMRIkoqKijz2KSoqcj92IaGhoYqMjPS4AQAAO3kVO7fffruys7Pd9x0Oh06dOqUpU6b49CskOnfurIKCAo9te/bs0RVXXCHp3MXKMTExHi+duVwubdu2TcnJyT6bAwAA1F1evYz17LPPKjU1Vddcc41KS0s1cOBA7d27V82aNdPLL7/ss+FGjx6tW265RU899ZTuvvtubd++XS+++KJefPFFSecia9SoUZo2bZpatWrlfut5XFyc+vTp47M5AABA3eVV7DRv3lwfffSRli9fro8//linTp1SWlqaBg0a5HHBcnXdeOONWrVqlcaPH68nnnhCiYmJys7O1qBBg9z7jB07ViUlJRo+fLhOnDihLl26KDc3l8/YAQAAkiSH4Quu5HK5FBUVpeLiYq7fAYBfkIRxa/w9wi/C19N71shxL/X3t1dndpYuXfqTj997773eHBYAAMDnvIqdkSNHetwvLy/X6dOnFRISogYNGhA7AAAgYHj1bqzvv//e43bq1CkVFBSoS5cuPr1AGQAAoLq8ip0LadWqlaZPn37eWR8AAAB/8lnsSFJwcLAOHz7sy0MCAABUi1fX7Lz22mse940xOnLkiF544QV17tzZJ4MBAAD4glex8+MP7HM4HLr88svVrVs3Pfvss76YCwAAwCe8ip3KykpfzwEAAFAjfHrNDgAAQKDx6sxORkbGJe+blZXlzR8BAADgE17Fzs6dO7Vz506Vl5fr6quvlnTu28jr1aunG264wb2fw+HwzZQAAABe8ip2evXqpYiICC1ZskSNGzeWdO6DBocNG6Zbb71Vjz76qE+HBAAA8JZX1+w8++yzyszMdIeOJDVu3FjTpk3j3VgAACCgeBU7LpdLx44dO2/7sWPHdPLkyWoPBQAA4Ctexc5dd92lYcOGaeXKlTp06JAOHTqkf/7zn0pLS1Pfvn19PSMAAIDXvLpmJycnR2PGjNHAgQNVXl5+7kDBwUpLS9PTTz/t0wEBAACqw6vYadCggebOnaunn35aX331lSSpZcuWatiwoU+HAwAAqK5qfajgkSNHdOTIEbVq1UoNGzaUMcZXcwEAAPiEV7Hzn//8R927d1fr1q3Vo0cPHTlyRJKUlpbG284BAEBA8Sp2Ro8erfr16+vAgQNq0KCBe3v//v2Vm5vrs+EAAACqy6trdt5++2299dZbat68ucf2Vq1a6ZtvvvHJYAAAAL7g1ZmdkpISjzM6Pzh+/LhCQ0OrPRQAAICveBU7t956q5YuXeq+73A4VFlZqZkzZ6pr164+Gw4AAKC6vHoZa+bMmerevbs++OADnTlzRmPHjtVnn32m48eP69133/X1jAAAAF7z6sxO+/bttWfPHnXp0kW9e/dWSUmJ+vbtq507d6ply5a+nhEAAMBrVT6zU15erjvuuEM5OTmaMGFCTcwEAADgM1U+s1O/fn19/PHHNTELAACAz3n1MtY999yjBQsW+HoWAAAAn/PqAuWzZ89q4cKFeuedd9SxY8fzvhMrKyvLJ8MBAABUV5ViZ9++fUpISNCnn36qG264QZK0Z88ej30cDofvpgMAAKimKsVOq1atdOTIEW3YsEHSua+HmDVrlpxOZ40MBwAAUF1Vumbnx99q/uabb6qkpMSnAwEAAPiSVxco/+DH8QMAABBoqhQ7DofjvGtyuEYHAAAEsipds2OM0dChQ91f9llaWqoHH3zwvHdjrVy50ncTAgAAVEOVYmfIkCEe9++55x6fDgMAAOBrVYqdRYsW1dQcAAAANaJaFygDAAAEOmIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWK1KXxcBAMDFJIxb4+8RgAvizA4AALAasQMAAKxG7AAAAKsROwAAwGp1KnamT58uh8OhUaNGubeVlpYqPT1dTZs21WWXXaZ+/fqpqKjIf0MCAICAUmdi5/3339df//pXXXvttR7bR48erddff10rVqzQpk2bdPjwYfXt29dPUwIAgEBTJ2Ln1KlTGjRokObPn6/GjRu7txcXF2vBggXKyspSt27d1LFjRy1atEhbt27Ve++958eJAQBAoKgTsZOenq6ePXsqJSXFY/uOHTtUXl7usb1NmzZq0aKF8vPzL3q8srIyuVwujxsAALBTwH+o4PLly/Xhhx/q/fffP++xwsJChYSEqFGjRh7bnU6nCgsLL3rMzMxMPf74474eFQAABKCAPrNz8OBBjRw5UsuWLVNYWJjPjjt+/HgVFxe7bwcPHvTZsQEAQGAJ6NjZsWOHjh49qhtuuEHBwcEKDg7Wpk2bNGvWLAUHB8vpdOrMmTM6ceKEx/OKiooUExNz0eOGhoYqMjLS4wYAAOwU0C9jde/eXZ988onHtmHDhqlNmzb605/+pPj4eNWvX195eXnq16+fJKmgoEAHDhxQcnKyP0YGAAABJqBjJyIiQu3bt/fY1rBhQzVt2tS9PS0tTRkZGWrSpIkiIyP18MMPKzk5WTfffLM/RgYAAAEmoGPnUjz33HMKCgpSv379VFZWptTUVM2dO9ffYwEAgADhMMYYfw/hby6XS1FRUSouLub6HQDwUsK4Nf4eAQHq6+k9a+S4l/r7O6AvUAYAAKguYgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAVgv29wAAgPMljFvj7xEAa3BmBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFgtoGMnMzNTN954oyIiIhQdHa0+ffqooKDAY5/S0lKlp6eradOmuuyyy9SvXz8VFRX5aWIAABBoAjp2Nm3apPT0dL333ntat26dysvLdfvtt6ukpMS9z+jRo/X6669rxYoV2rRpkw4fPqy+ffv6cWoAABBIgv09wE/Jzc31uL948WJFR0drx44d+r//+z8VFxdrwYIFeumll9StWzdJ0qJFi9S2bVu99957uvnmm/0xNgAACCABfWbnx4qLiyVJTZo0kSTt2LFD5eXlSklJce/Tpk0btWjRQvn5+Rc9TllZmVwul8cNAADYKaDP7PyvyspKjRo1Sp07d1b79u0lSYWFhQoJCVGjRo089nU6nSosLLzosTIzM/X444/X5LgAAkjCuDX+HgGAH9WZMzvp6en69NNPtXz58mofa/z48SouLnbfDh486IMJAQBAIKoTZ3ZGjBihN954Q5s3b1bz5s3d22NiYnTmzBmdOHHC4+xOUVGRYmJiLnq80NBQhYaG1uTIAAAgQAT0mR1jjEaMGKFVq1Zp/fr1SkxM9Hi8Y8eOql+/vvLy8tzbCgoKdODAASUnJ9f2uAAAIAAF9Jmd9PR0vfTSS/rXv/6liIgI93U4UVFRCg8PV1RUlNLS0pSRkaEmTZooMjJSDz/8sJKTk3knFgAAkBTgsTNv3jxJ0m9+8xuP7YsWLdLQoUMlSc8995yCgoLUr18/lZWVKTU1VXPnzq3lSQEAQKAK6NgxxvzsPmFhYZozZ47mzJlTCxMBAIC6JqCv2QEAAKguYgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFYjdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYLdjfAwCoWxLGrfH3CABQJZzZAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFbjE5QBP+GTiAGgdnBmBwAAWI3YAQAAVuNlLFiBl4QAABfDmR0AAGA1YgcAAFiN2AEAAFYjdgAAgNWsiZ05c+YoISFBYWFhSkpK0vbt2/09EgAACABWvBvrlVdeUUZGhnJycpSUlKTs7GylpqaqoKBA0dHRfp2tLr5L6OvpPf09AgAAPmPFmZ2srCzdf//9GjZsmK655hrl5OSoQYMGWrhwob9HAwAAflbnz+ycOXNGO3bs0Pjx493bgoKClJKSovz8/As+p6ysTGVlZe77xcXFkiSXy+Xz+SrLTvv8mDWtJtahptXFdQaAX4qa+r3yw3GNMT+5X52Pne+++04VFRVyOp0e251Op7744osLPiczM1OPP/74edvj4+NrZMa6Jirb3xMAAGxS079XTp48qaioqIs+Xudjxxvjx49XRkaG+35lZaWOHz+upk2byuFw+OzPcblcio+P18GDBxUZGemz48IT61x7WOvawTrXDta5dtTkOhtjdPLkScXFxf3kfnU+dpo1a6Z69eqpqKjIY3tRUZFiYmIu+JzQ0FCFhoZ6bGvUqFFNjajIyEj+R6oFrHPtYa1rB+tcO1jn2lFT6/xTZ3R+UOcvUA4JCVHHjh2Vl5fn3lZZWam8vDwlJyf7cTIAABAI6vyZHUnKyMjQkCFD1KlTJ910003Kzs5WSUmJhg0b5u/RAACAn1kRO/3799exY8c0efJkFRYW6rrrrlNubu55Fy3XttDQUE2ZMuW8l8zgW6xz7WGtawfrXDtY59oRCOvsMD/3fi0AAIA6rM5fswMAAPBTiB0AAGA1YgcAAFiN2AEAAFYjdqppzpw5SkhIUFhYmJKSkrR9+/af3H/FihVq06aNwsLC1KFDB61du7aWJq3bqrLO8+fP16233qrGjRurcePGSklJ+dn/Ljinqn+ff7B8+XI5HA716dOnZge0SFXX+sSJE0pPT1dsbKxCQ0PVunVr/v24BFVd5+zsbF199dUKDw9XfHy8Ro8erdLS0lqatm7avHmzevXqpbi4ODkcDq1evfpnn7Nx40bdcMMNCg0N1VVXXaXFixfX7JAGXlu+fLkJCQkxCxcuNJ999pm5//77TaNGjUxRUdEF93/33XdNvXr1zMyZM83nn39uJk6caOrXr28++eSTWp68bqnqOg8cONDMmTPH7Ny50+zevdsMHTrUREVFmUOHDtXy5HVLVdf5B/v37ze/+tWvzK233mp69+5dO8PWcVVd67KyMtOpUyfTo0cPs2XLFrN//36zceNGs2vXrlqevG6p6jovW7bMhIaGmmXLlpn9+/ebt956y8TGxprRo0fX8uR1y9q1a82ECRPMypUrjSSzatWqn9x/3759pkGDBiYjI8N8/vnnZvbs2aZevXomNze3xmYkdqrhpptuMunp6e77FRUVJi4uzmRmZl5w/7vvvtv07NnTY1tSUpJ54IEHanTOuq6q6/xjZ8+eNREREWbJkiU1NaIVvFnns2fPmltuucX87W9/M0OGDCF2LlFV13revHnmyiuvNGfOnKmtEa1Q1XVOT0833bp189iWkZFhOnfuXKNz2uRSYmfs2LGmXbt2Htv69+9vUlNTa2wuXsby0pkzZ7Rjxw6lpKS4twUFBSklJUX5+fkXfE5+fr7H/pKUmpp60f3h3Tr/2OnTp1VeXq4mTZrU1Jh1nrfr/MQTTyg6OlppaWm1MaYVvFnr1157TcnJyUpPT5fT6VT79u311FNPqaKiorbGrnO8WedbbrlFO3bscL/UtW/fPq1du1Y9evSolZl/Kfzxu9CKT1D2h++++04VFRXnfUqz0+nUF198ccHnFBYWXnD/wsLCGpuzrvNmnX/sT3/6k+Li4s77nwv/nzfrvGXLFi1YsEC7du2qhQnt4c1a79u3T+vXr9egQYO0du1affnll3rooYdUXl6uKVOm1MbYdY436zxw4EB999136tKli4wxOnv2rB588EH9+c9/ro2RfzEu9rvQ5XLpv//9r8LDw33+Z3JmB1abPn26li9frlWrViksLMzf41jj5MmTGjx4sObPn69mzZr5exzrVVZWKjo6Wi+++KI6duyo/v37a8KECcrJyfH3aFbZuHGjnnrqKc2dO1cffvihVq5cqTVr1mjq1Kn+Hg3VxJkdLzVr1kz16tVTUVGRx/aioiLFxMRc8DkxMTFV2h/erfMPnnnmGU2fPl3vvPOOrr322pocs86r6jp/9dVX+vrrr9WrVy/3tsrKSklScHCwCgoK1LJly5oduo7y5u90bGys6tevr3r16rm3tW3bVoWFhTpz5oxCQkJqdOa6yJt1njRpkgYPHqw//OEPkqQOHTqopKREw4cP14QJExQUxPkBX7jY78LIyMgaOasjcWbHayEhIerYsaPy8vLc2yorK5WXl6fk5OQLPic5Odljf0lat27dRfeHd+ssSTNnztTUqVOVm5urTp061caodVpV17lNmzb65JNPtGvXLvftt7/9rbp27apdu3YpPj6+NsevU7z5O925c2d9+eWX7qCUpD179ig2NpbQuQhv1vn06dPnBc0PgWn4Gkmf8cvvwhq79PkXYPny5SY0NNQsXrzYfP7552b48OGmUaNGprCw0BhjzODBg824cePc+7/77rsmODjYPPPMM2b37t1mypQpvPX8ElR1nadPn25CQkLMP/7xD3PkyBH37eTJk/76EeqEqq7zj/FurEtX1bU+cOCAiYiIMCNGjDAFBQXmjTfeMNHR0WbatGn++hHqhKqu85QpU0xERIR5+eWXzb59+8zbb79tWrZsae6++25//Qh1wsmTJ83OnTvNzp07jSSTlZVldu7cab755htjjDHjxo0zgwcPdu//w1vPH3vsMbN7924zZ84c3noe6GbPnm1atGhhQkJCzE033WTee+8992O33XabGTJkiMf+r776qmndurUJCQkx7dq1M2vWrKnlieumqqzzFVdcYSSdd5syZUrtD17HVPXv8/8idqqmqmu9detWk5SUZEJDQ82VV15pnnzySXP27Nlanrruqco6l5eXm7/85S+mZcuWJiwszMTHx5uHHnrIfP/997U/eB2yYcOGC/6b+8PaDhkyxNx2223nPee6664zISEh5sorrzSLFi2q0RkdxnBuDgAA2ItrdgAAgNWIHQAAYDViBwAAWI3YAQAAViN2AACA1YgdAABgNWIHAABYjdgBAABWI3YAAIDViB0AAGA1YgcAAFiN2AEAAFb7f8V8bHl2G19GAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "results_gpt4_df['score'].plot.hist(bins=10)" - ] - }, - { - "cell_type": "markdown", - "id": "22f175e0-4613-4e70-ace9-b7f1bba51459", - "metadata": {}, - "source": [ - "## Fine-tuning\n", - "\n", - "Our results so far are **81%** correct classifications for `gpt-3.5-turbo` and **X%** for `gpt-4`. `gpt-3.5-turbo` is far cheaper and quicker, but is much less accurate.\n", - "\n", - "Lets see if we can achieve the best of both worlds, by fine-tuning `gpt-3.5-turbo` on a domain-specific training set of 1500 examples." - ] - }, - { - "cell_type": "markdown", - "id": "2b3151e9-8715-47bd-a153-195d6a0d0a70", - "metadata": {}, - "source": [ - "### Data preparation\n", - "\n", - "We'll begin by preparing our data. When fine-tuning with the `ChatCompletion` format, each training example is a simple list of `messages`. For example, an entry could look like:\n", - "```\n", - "[{'role': 'system',\n", - " 'content': 'You are a helpful recipe assistant. You are to extract the generic ingredients from each of the recipes provided.'},\n", - " {'role': 'user',\n", - " 'content': 'Title: No-Bake Nut Cookies\\n\\nIngredients: [\"1 c. firmly packed brown sugar\", \"1/2 c. evaporated milk\", \"1/2 tsp. vanilla\", \"1/2 c. broken nuts (pecans)\", \"2 Tbsp. butter or margarine\", \"3 1/2 c. bite size shredded rice biscuits\"]\\n\\nDirections: [\"In a heavy 2-quart saucepan, mix brown sugar, nuts, evaporated milk and butter or margarine.\", \"Stir over medium heat until mixture bubbles all over top.\", \"Boil and stir 5 minutes more. Take off heat.\", \"Stir in vanilla and cereal; mix well.\", \"Using 2 teaspoons, drop and shape into 30 clusters on wax paper.\", \"Let stand until firm, about 30 minutes.\"]\\n\\nGeneric ingredients: '},\n", - " {'role': 'assistant',\n", - " 'content': '[\"brown sugar\", \"milk\", \"vanilla\", \"nuts\", \"butter\", \"bite size shredded rice biscuits\"]'}]\n", - "```\n", - "\n", - "During the training process this conversation will be split, with the final entry being the `completion` that the model will produce, and the remainder of the `messages` acting as the prompt. Consider this when building your training examples - if your model will act on multi-turn conversations, then please provide representative examples so it doesn't perform poorly when the conversation starts to expand.\n", - "\n", - "For fine-tuning with `ChatCompletion` you can begin with even 30-50 well-pruned examples. However, given we have a large representative dataset at hand, we'll take a larger set of 1500 training examples to start with. You should see performance continue to scale linearly as you increase the size of the training set.\n", - "\n", - "Please note that currently there is a 4096 token limit for each training example. Anything longer than this will be truncated at 4096 tokens." - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "9a8216b0-d1dc-472d-b07d-1be03acd70a5", - "metadata": {}, - "outputs": [], - "source": [ - "training_data = []\n", - "\n", - "# Take first 1500 records for training\n", - "for x, y in recipe_df.head(1500).iterrows():\n", - " training_prompt_message = []\n", - " training_prompt_message.append({\"role\":\"system\",\"content\":system_prompt})\n", - " \n", - " user_prompt = create_prompt(y)\n", - " training_prompt_message.append({\"role\":\"user\",\"content\": user_prompt})\n", - "\n", - " training_prompt_message.append({\"role\":\"assistant\",\"content\": y['NER']})\n", - " training_message_dict = {\"messages\": training_prompt_message}\n", - " training_data.append(training_message_dict)\n", - " " - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "5b853efa-dfea-4770-ab88-9b7e17794421", - "metadata": {}, - "outputs": [], - "source": [ - "validation_data = []\n", - "\n", - "for x, y in test_df.iterrows():\n", - " validation_prompt_message = []\n", - " validation_prompt_message.append({\"role\":\"system\",\"content\":system_prompt})\n", - " \n", - " user_prompt = create_prompt(y)\n", - " validation_prompt_message.append({\"role\":\"user\",\"content\": user_prompt})\n", - "\n", - " validation_prompt_message.append({\"role\":\"assistant\",\"content\": y['NER']})\n", - " validation_message_dict = {\"messages\": validation_prompt_message}\n", - " validation_data.append(validation_message_dict)\n" - ] - }, - { - "cell_type": "markdown", - "id": "1d5e7bfe-f6c8-4a23-a951-3df3f3791d7f", - "metadata": {}, - "source": [ - "We then need to export these as `.jsonl` files, with each row being one training example." - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "8d2eb207-2c2b-43f6-a613-64a7e92d494d", - "metadata": {}, - "outputs": [], - "source": [ - "def dicts_to_jsonl(data_list: list, filename: str) -> None:\n", - " \"\"\"\n", - " Method saves list of dicts into jsonl file.\n", - " :param data: (list) list of dicts to be stored,\n", - " :param filename: (str) path to the output file. If suffix .jsonl is not given then methods appends\n", - " .jsonl suffix into the file.\n", - " \"\"\"\n", - " sjsonl = '.jsonl'\n", - "\n", - " # Check filename\n", - " if not filename.endswith(sjsonl):\n", - " filename = filename + sjsonl\n", - " # Save data\n", - " \n", - " with open(filename, 'w') as out:\n", - " for ddict in data_list:\n", - " jout = json.dumps(ddict) + '\\n'\n", - " out.write(jout)" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "8b53e7a2-1cac-4c5f-8ba4-3292ba2a0770", - "metadata": {}, - "outputs": [], - "source": [ - "# Save training_data to JSONL\n", - "dicts_to_jsonl(training_data,'recipe_finetune_training')\n", - "\n", - "# Save validation_data to JSONL\n", - "dicts_to_jsonl(validation_data,'recipe_finetune_validation')" - ] - }, - { - "cell_type": "markdown", - "id": "0d149e2e-50dd-45c1-bd8d-1291975670b4", - "metadata": {}, - "source": [ - "### Upload files\n", - "\n", - "You can then upload the files to our `Files` endpoint to be used by the fine-tuned model." - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "id": "25302bbc-80b1-4d56-8d19-ab8bd41579ac", - "metadata": {}, - "outputs": [], - "source": [ - "def upload_chat_file(filename):\n", - " \n", - "\n", - " headers = {\n", - " 'Authorization': 'Bearer ' + os.getenv('OPENAI_API_KEY', ''),\n", - " }\n", - " \n", - " files = {\n", - " 'purpose': (None, 'fine-tune'),\n", - " #'format': (None, 'fine-tune-chat'),\n", - " 'file': open(filename, 'rb'),\n", - " }\n", - "\n", - " response = requests.post('https://api.openai.com/v1/files', headers=headers, files=files)\n", - " return response" - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "id": "69462d9e-e6bd-49b9-a064-9eae4ea5b7a8", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'object': 'file', 'id': 'file-5bQjePpOBzsxyjYE5hvd6WSs', 'purpose': 'fine-tune', 'filename': 'recipe_finetune_training.jsonl', 'bytes': 1347985, 'created_at': 1692727484, 'status': 'uploaded', 'status_details': None}\n", - "{'object': 'file', 'id': 'file-h7SRtdHu3Mr5gTnuIX4mmVED', 'purpose': 'fine-tune', 'filename': 'recipe_finetune_validation.jsonl', 'bytes': 269054, 'created_at': 1692727486, 'status': 'uploaded', 'status_details': None}\n", - "['file-5bQjePpOBzsxyjYE5hvd6WSs', 'file-h7SRtdHu3Mr5gTnuIX4mmVED']\n" - ] - } - ], - "source": [ - "file_ids = []\n", - "files = ['recipe_finetune_training.jsonl','recipe_finetune_validation.jsonl']\n", - "\n", - "for file in files:\n", - " response = upload_chat_file(file)\n", - " #print(response.json())\n", - " file_ids.append(response.json()['id'])\n", - " print(response.json())\n", - " \n", - "print(file_ids)" - ] - }, - { - "cell_type": "markdown", - "id": "d61cd381-63ad-4ed9-b0be-47a438891028", - "metadata": {}, - "source": [ - "### Create fine-tune job\n", - "\n", - "Now we can create our fine-tuned job - we do this using the ```https://api.openai.com/v1/alpha/fine-tunes``` endpoint. You must supply the following parameters:\n", - "- **training_file:** the name of the file to train on\n", - "- **validation_file:** the name of the file to validate on\n", - "- **model:** the model name to finetune\n", - "- **suffix:** up to 18 character suffix to customize output name\n", - "\n", - "The response will contain an `id` which you can use to retrieve updates on the job." - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "id": "3452cafc", - "metadata": {}, - "outputs": [], - "source": [ - "training_file_id = file_ids[0]\n", - "validation_file_id = file_ids[1]\n", - "suffix_name = 'recipe-cj'\n" - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "id": "d84ee75d-823c-4cfc-bb1f-7342b485ddd3", - "metadata": {}, - "outputs": [], - "source": [ - "def create_chatcompletion_finetune(training_file_id,validation_file_id,suffix):\n", - " \"\"\"This function creates a fine-tuned job given a training file, a validation file and a suffix\"\"\"\n", - " \n", - "\n", - " headers = {\n", - " 'Content-Type': 'application/json',\n", - " 'Authorization': 'Bearer ' + os.getenv('OPENAI_API_KEY', ''),\n", - " }\n", - "\n", - " json_data = {\n", - " 'training_file': training_file_id,\n", - " 'validation_file': validation_file_id,\n", - " 'model': 'gpt-3.5-turbo-0613-alpha',\n", - " 'suffix': suffix,\n", - " }\n", - " \n", - " response = requests.post('https://api.openai.com/v1/fine_tuning/jobs', headers=headers, json=json_data)\n", - " \n", - " return response\n" - ] - }, - { - "cell_type": "code", - "execution_count": 32, - "id": "05541ceb-5628-447e-962d-7e57c112439c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'error': {'message': 'Invalid URL (POST /v1/fine_tuning/jobs)', 'type': 'invalid_request_error', 'param': None, 'code': None}}\n", - "'id'\n" - ] - } - ], - "source": [ - "try:\n", - " response = create_chatcompletion_finetune(training_file_id,validation_file_id,suffix_name)\n", - " print(response.json())\n", - " job_id = response.json()['id']\n", - " print(response.json())\n", - "\n", - "except Exception as e:\n", - " print(e)" - ] - }, - { - "cell_type": "markdown", - "id": "1de3ed71-f2d4-4138-95a3-70da187a007e", - "metadata": {}, - "source": [ - "#### Check job status\n", - "\n", - "You can make a `GET` request to the `https://api.openai.com/v1/alpha/fine-tunes` endpoint to list your alpha fine-tune jobs. In this instance you'll want to check that the ID you got from the previous step ends up as `status: succeeded`.\n", - "\n", - "Once it is completed, you can use the `result_files` to sample the results from the validation set (if you uploaded one), and use the ID from the `fine_tuned_model` parameter to invoke your trained model." - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "id": "b6f65822-ed88-4667-bea5-4575f7df62a2", - "metadata": {}, - "outputs": [], - "source": [ - "def check_chatcompletion_finetunes(job_id):\n", - "\n", - " headers = {\n", - " 'Authorization': 'Bearer ' + os.getenv('OPENAI_API_KEY', ''),\n", - " }\n", - "\n", - " return requests.get(f'https://api.openai.com/v1/fine_tuning/jobs/{job_id}', headers=headers)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 31, - "id": "6df1faf1", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'object': 'fine_tuning.job', 'id': 'ft-laDr3mkgkv5N5FMlsKNzueBu', 'model': 'gpt-3.5-turbo-0613-alpha', 'created_at': 1692592059, 'fine_tuned_model': None, 'organization_id': 'org-p13k3klgno5cqxbf0q8hpgrk', 'result_files': [], 'status': 'running', 'validation_file': 'file-NLQBnaKFUSS8NYsnBRltq2Rd', 'training_file': 'file-lNnvY0Qon022gY4Fyh33EK6T', 'hyperparameters': {'n_epochs': 3, 'batch_size': 3, 'learning_rate_multiplier': 2}}\n" - ] - } - ], - "source": [ - "response = check_chatcompletion_finetunes(job_id)\n", - "\n", - "print(response.json())" - ] - }, - { - "cell_type": "markdown", - "id": "0025e392-84cd-4566-a384-ea31ca43e567", - "metadata": {}, - "source": [ - "## Evaluate fine-tuned model\n", - "\n", - "The last step is to use your fine-tuned model for inference. Similar to the classic `FineTuning` or Reserved Capacity, you simply call `ChatCompletions` with your new fine-tuned model name filling the `model` parameter.\n", - "\n", - "**NOTE:** If you uploaded your fine-tune job with a validation file, you can inspect the results immediately following training by using the `Files` endpoint." - ] - }, - { - "cell_type": "code", - "execution_count": 47, - "id": "85b109e5-f183-4537-9b5d-439a549be7c0", - "metadata": {}, - "outputs": [], - "source": [ - "# Input your FT_MODEL ID here - this will usually be something like ft:{MODEL_NAME}:{ORG_ID}:{SUFFIX_NAME}{STRING}\n", - "FT_MODEL = 'gpt-recipe-ft'" - ] - }, - { - "cell_type": "code", - "execution_count": 48, - "id": "1c7de631-b68f-4eff-9ae7-051641579c2b", - "metadata": {}, - "outputs": [], - "source": [ - "validation_results = []\n", - "for x,y in test_df.iterrows():\n", - " test_messages = messages.copy()\n", - " user_prompt = create_prompt(y)\n", - " test_messages.append({\"role\":\"user\",\"content\": user_prompt})\n", - "\n", - " try:\n", - " response = openai.ChatCompletion.create(\n", - " model=FT_MODEL,\n", - " messages=test_messages,\n", - " temperature=0,\n", - " max_tokens=500\n", - " )\n", - " validation_results.append((user_prompt,y['NER'],response['choices'][0]['message']['content']))\n", - "\n", - " except Exception as e:\n", - " print(e)" - ] - }, - { - "cell_type": "code", - "execution_count": 49, - "id": "8e3de5a9", - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
inputactualpredictionscore
0Title: Pretzel Candy\\n\\nIngredients: [\"1 lb. w...[\"white chocolate\", \"pretzel sticks\", \"peanuts\"][\"white chocolate\", \"pretzel sticks\", \"peanuts\"]1.000000
1Title: Salmon Party Ball\\n\\nIngredients: [\"8 o...[\"cream cheese\", \"salmon\", \"lemon juice\", \"hor...[\"cream cheese\", \"salmon\", \"lemon juice\", \"hor...1.000000
2Title: Fancy Fried Green Tomatoes\\n\\nIngredien...[\"sour cream\", \"green onion\", \"salt\", \"eggs\", ...[\"sour cream\", \"green onion\", \"salt\", \"eggs\", ...0.900000
3Title: Potluck Potatoes\\n\\nIngredients: [\"1 (2...[\"frozen hash brown potatoes\", \"onions\", \"salt...[\"frozen hash brown potatoes\", \"onions\", \"salt...1.000000
4Title: Old-Fashioned Sweet-Sour Cole Slaw\\n\\nI...[\"shredded green cabbage\", \"salt\", \"sugar\", \"c...[\"green cabbage\", \"salt\", \"sugar\", \"cider vine...0.800000
5Title: Peanut Brittle\\n\\nIngredients: [\"3 c. w...[\"white sugar\", \"water\", \"butter\", \"soda\", \"wh...[\"white sugar\", \"water\", \"butter\", \"soda\", \"wh...1.000000
6Title: Chicken Inspiration\\n\\nIngredients: [\"1...[\"chicken breast\", \"fresh mushrooms\", \"Provolo...[\"boneless\", \"fresh mushrooms\", \"Provolone che...0.636364
7Title: Down East Blueberry Cake\\n\\nIngredients...[\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue...[\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue...1.000000
8Title: Cranberry-Pecan Bars\\n\\nIngredients: [\"...[\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan...[\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan...1.000000
9Title: Stacked Twinkie Cake\\n\\nIngredients: [\"...[\"coconut\", \"Twinkies\", \"pineapple\", \"pecans\",...[\"coconut\", \"instant French vanilla pudding\", ...1.000000
\n", - "
" - ], - "text/plain": [ - " input \\\n", - "0 Title: Pretzel Candy\\n\\nIngredients: [\"1 lb. w... \n", - "1 Title: Salmon Party Ball\\n\\nIngredients: [\"8 o... \n", - "2 Title: Fancy Fried Green Tomatoes\\n\\nIngredien... \n", - "3 Title: Potluck Potatoes\\n\\nIngredients: [\"1 (2... \n", - "4 Title: Old-Fashioned Sweet-Sour Cole Slaw\\n\\nI... \n", - "5 Title: Peanut Brittle\\n\\nIngredients: [\"3 c. w... \n", - "6 Title: Chicken Inspiration\\n\\nIngredients: [\"1... \n", - "7 Title: Down East Blueberry Cake\\n\\nIngredients... \n", - "8 Title: Cranberry-Pecan Bars\\n\\nIngredients: [\"... \n", - "9 Title: Stacked Twinkie Cake\\n\\nIngredients: [\"... \n", - "\n", - " actual \\\n", - "0 [\"white chocolate\", \"pretzel sticks\", \"peanuts\"] \n", - "1 [\"cream cheese\", \"salmon\", \"lemon juice\", \"hor... \n", - "2 [\"sour cream\", \"green onion\", \"salt\", \"eggs\", ... \n", - "3 [\"frozen hash brown potatoes\", \"onions\", \"salt... \n", - "4 [\"shredded green cabbage\", \"salt\", \"sugar\", \"c... \n", - "5 [\"white sugar\", \"water\", \"butter\", \"soda\", \"wh... \n", - "6 [\"chicken breast\", \"fresh mushrooms\", \"Provolo... \n", - "7 [\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue... \n", - "8 [\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan... \n", - "9 [\"coconut\", \"Twinkies\", \"pineapple\", \"pecans\",... \n", - "\n", - " prediction score \n", - "0 [\"white chocolate\", \"pretzel sticks\", \"peanuts\"] 1.000000 \n", - "1 [\"cream cheese\", \"salmon\", \"lemon juice\", \"hor... 1.000000 \n", - "2 [\"sour cream\", \"green onion\", \"salt\", \"eggs\", ... 0.900000 \n", - "3 [\"frozen hash brown potatoes\", \"onions\", \"salt... 1.000000 \n", - "4 [\"green cabbage\", \"salt\", \"sugar\", \"cider vine... 0.800000 \n", - "5 [\"white sugar\", \"water\", \"butter\", \"soda\", \"wh... 1.000000 \n", - "6 [\"boneless\", \"fresh mushrooms\", \"Provolone che... 0.636364 \n", - "7 [\"butter\", \"sugar\", \"eggs\", \"sour milk\", \"blue... 1.000000 \n", - "8 [\"flour\", \"sugar\", \"salt\", \"margarine\", \"pecan... 1.000000 \n", - "9 [\"coconut\", \"instant French vanilla pudding\", ... 1.000000 " - ] - }, - "execution_count": 49, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "validation_results_df = pd.DataFrame(validation_results)\n", - "validation_results_df.columns = ['input','actual','prediction']\n", - "validation_results_df['score'] = validation_results_df.apply(lambda x: evaluate_ner(x),axis=1)\n", - "validation_results_df.head(10)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 50, - "id": "cb51b42f", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "0.923313319921679" - ] - }, - "execution_count": 50, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "validation_results_df['score'].mean()" - ] - }, - { - "cell_type": "code", - "execution_count": 51, - "id": "9f31acb1", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 51, - "metadata": {}, - "output_type": "execute_result" - }, - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjsAAAGdCAYAAAD0e7I1AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8pXeV/AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAo9klEQVR4nO3de3SU9YHG8WcCZAI0FwPkVkcCkduK3EsWRQWhQuIiCttyJ9gsqAXUpC6SKnKpa6gUiheEbReCHkEsa4oWFOUqsgaUS0qlGkm4REoSLUiGxGXI5d0/PM46TQJkMslMfnw/57zn8F7zzO8MZ57zm3dmbJZlWQIAADBUkL8DAAAANCbKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaC39HSAQVFdX68yZMwoNDZXNZvN3HAAAcBUsy9KFCxcUFxenoKC6528oO5LOnDkjh8Ph7xgAAMALX3zxha6//vo691N2JIWGhkr6drDCwsL8nAYAAFwNp9Mph8Phfh2vC2VHcr91FRYWRtkBAKCZudItKNygDAAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaJQdAABgNMoOAAAwGmUHAAAYjbIDAACMRtkBAABGo+wAAACjUXYAAIDRKDsAAMBolB0AAGC0lv4OAAAArl783C3+jlBvJxff7de/z8wOAAAwml/Lzp49ezRq1CjFxcXJZrNp06ZNHvttNluty5IlS9zHxMfH19i/ePHiJn4kAAAgUPm17JSXl6t3795asWJFrfuLioo8ljVr1shms2ns2LEexy1atMjjuNmzZzdFfAAA0Az49Z6dpKQkJSUl1bk/JibGY/3NN9/U0KFD1blzZ4/toaGhNY4FAACQmtE9OyUlJdqyZYtSU1Nr7Fu8eLHatWunvn37asmSJaqsrLzstVwul5xOp8cCAADM1Gw+jfXyyy8rNDRUY8aM8dj+8MMPq1+/foqMjNSHH36ojIwMFRUVadmyZXVeKzMzUwsXLmzsyAAAIAA0m7KzZs0aTZo0SSEhIR7b09PT3f/u1auXgoOD9cADDygzM1N2u73Wa2VkZHic53Q65XA4Gic4AADwq2ZRdj744APl5eXp9ddfv+KxiYmJqqys1MmTJ9WtW7daj7Hb7XUWIQAAYJZmcc/O6tWr1b9/f/Xu3fuKx+bm5iooKEhRUVFNkAwAAAQ6v87slJWVKT8/371+4sQJ5ebmKjIyUjfccIOkb99i2rhxo5YuXVrj/JycHO3fv19Dhw5VaGiocnJylJaWpsmTJ+u6665rsscBAAACl1/LzoEDBzR06FD3+nf30aSkpGjt2rWSpA0bNsiyLE2YMKHG+Xa7XRs2bNCCBQvkcrnUqVMnpaWledyPAwAArm02y7Isf4fwN6fTqfDwcJWWliosLMzfcQAAqBO/jfX/rvb1u1ncswMAAOAtyg4AADAaZQcAABiNsgMAAIxG2QEAAEaj7AAAAKNRdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaJQdAABgNMoOAAAwGmUHAAAYjbIDAACMRtkBAABGo+wAAACjUXYAAIDRKDsAAMBolB0AAGA0yg4AADAaZQcAABiNsgMAAIxG2QEAAEaj7AAAAKNRdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo/m17OzZs0ejRo1SXFycbDabNm3a5LF/2rRpstlsHsvIkSM9jjl37pwmTZqksLAwRUREKDU1VWVlZU34KAAAQCDza9kpLy9X7969tWLFijqPGTlypIqKitzLa6+95rF/0qRJOnr0qLZt26bNmzdrz549mjFjRmNHBwAAzURLf/7xpKQkJSUlXfYYu92umJiYWvd9+umn2rp1qz7++GMNGDBAkvTCCy8oOTlZv/nNbxQXF+fzzAAAoHkJ+Ht2du/eraioKHXr1k0PPfSQzp49696Xk5OjiIgId9GRpOHDhysoKEj79+/3R1wAABBg/DqzcyUjR47UmDFj1KlTJxUUFOiXv/ylkpKSlJOToxYtWqi4uFhRUVEe57Rs2VKRkZEqLi6u87oul0sul8u97nQ6G+0xAAAA/wrosjN+/Hj3v2+++Wb16tVLCQkJ2r17t4YNG+b1dTMzM7Vw4UJfRAQAAAEu4N/G+r7OnTurffv2ys/PlyTFxMToyy+/9DimsrJS586dq/M+H0nKyMhQaWmpe/niiy8aNTcAAPCfZlV2Tp8+rbNnzyo2NlaSNGjQIJ0/f14HDx50H7Nz505VV1crMTGxzuvY7XaFhYV5LAAAwEx+fRurrKzMPUsjSSdOnFBubq4iIyMVGRmphQsXauzYsYqJiVFBQYHmzJmjG2+8USNGjJAk9ejRQyNHjtT06dO1atUqVVRUaNasWRo/fjyfxAIAAJL8PLNz4MAB9e3bV3379pUkpaenq2/fvnrqqafUokULHTlyRPfcc4+6du2q1NRU9e/fXx988IHsdrv7GuvWrVP37t01bNgwJScna/Dgwfrd737nr4cEAAACjF9ndoYMGSLLsurc/+67717xGpGRkVq/fr0vYwEAAIM0q3t2AAAA6ouyAwAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaJQdAABgNMoOAAAwGmUHAAAYjbIDAACMRtkBAABGo+wAAACjUXYAAIDRKDsAAMBolB0AAGA0yg4AADAaZQcAABiNsgMAAIxG2QEAAEaj7AAAAKNRdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaJQdAABgNMoOAAAwGmUHAAAYjbIDAACMRtkBAABGo+wAAACjUXYAAIDRKDsAAMBolB0AAGA0v5adPXv2aNSoUYqLi5PNZtOmTZvc+yoqKvT444/r5ptvVtu2bRUXF6epU6fqzJkzHteIj4+XzWbzWBYvXtzEjwQAAAQqv5ad8vJy9e7dWytWrKix75tvvtGhQ4c0b948HTp0SNnZ2crLy9M999xT49hFixapqKjIvcyePbsp4gMAgGagpT//eFJSkpKSkmrdFx4erm3btnlse/HFFzVw4EAVFhbqhhtucG8PDQ1VTExMo2YFAADNU7O6Z6e0tFQ2m00REREe2xcvXqx27dqpb9++WrJkiSorKy97HZfLJafT6bEAAAAz+XVmpz4uXryoxx9/XBMmTFBYWJh7+8MPP6x+/fopMjJSH374oTIyMlRUVKRly5bVea3MzEwtXLiwKWIDAAA/axZlp6KiQj/96U9lWZZWrlzpsS89Pd397169eik4OFgPPPCAMjMzZbfba71eRkaGx3lOp1MOh6NxwgMAAL8K+LLzXdE5deqUdu7c6TGrU5vExERVVlbq5MmT6tatW63H2O32OosQAAAwS0CXne+KzrFjx7Rr1y61a9fuiufk5uYqKChIUVFRTZAQAAAEOr+WnbKyMuXn57vXT5w4odzcXEVGRio2Nlb/+q//qkOHDmnz5s2qqqpScXGxJCkyMlLBwcHKycnR/v37NXToUIWGhionJ0dpaWmaPHmyrrvuOn89LAAAEED8WnYOHDigoUOHute/u48mJSVFCxYs0FtvvSVJ6tOnj8d5u3bt0pAhQ2S327VhwwYtWLBALpdLnTp1Ulpamsf9OAAA4Nrm17IzZMgQWZZV5/7L7ZOkfv36ad++fb6OBQAADNKsvmcHAACgvig7AADAaJQdAABgNMoOAAAwGmUHAAAYjbIDAACMRtkBAABGo+wAAACjUXYAAIDRKDsAAMBolB0AAGA0yg4AADAaZQcAABiNsgMAAIxG2QEAAEaj7AAAAKNRdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARvOq7Bw/ftzXOQAAABqFV2Xnxhtv1NChQ/Xqq6/q4sWLvs4EAADgM16VnUOHDqlXr15KT09XTEyMHnjgAX300Ue+zgYAANBgXpWdPn366LnnntOZM2e0Zs0aFRUVafDgwerZs6eWLVumr776ytc5AQAAvNKgG5RbtmypMWPGaOPGjfr1r3+t/Px8PfbYY3I4HJo6daqKiop8lRMAAMArDSo7Bw4c0M9//nPFxsZq2bJleuyxx1RQUKBt27bpzJkzGj16tK9yAgAAeKWlNyctW7ZMWVlZysvLU3Jysl555RUlJycrKOjb7tSpUyetXbtW8fHxvswKAABQb16VnZUrV+pnP/uZpk2bptjY2FqPiYqK0urVqxsUDgAAoKG8KjvHjh274jHBwcFKSUnx5vIAAAA+49U9O1lZWdq4cWON7Rs3btTLL7/c4FAAAAC+4lXZyczMVPv27Wtsj4qK0jPPPNPgUAAAAL7iVdkpLCxUp06damzv2LGjCgsLGxwKAADAV7wqO1FRUTpy5EiN7X/+85/Vrl27BocCAADwFa/KzoQJE/Twww9r165dqqqqUlVVlXbu3KlHHnlE48eP93VGAAAAr3n1aaxf/epXOnnypIYNG6aWLb+9RHV1taZOnco9OwAAIKB4NbMTHBys119/XZ999pnWrVun7OxsFRQUaM2aNQoODr7q6+zZs0ejRo1SXFycbDabNm3a5LHfsiw99dRTio2NVevWrTV8+PAaH3s/d+6cJk2apLCwMEVERCg1NVVlZWXePCwAAGCgBv1cRNeuXfWTn/xE//Iv/6KOHTvW+/zy8nL17t1bK1asqHX/s88+q+eff16rVq3S/v371bZtW40YMUIXL150HzNp0iQdPXpU27Zt0+bNm7Vnzx7NmDHD68cEAADM4tXbWFVVVVq7dq127NihL7/8UtXV1R77d+7ceVXXSUpKUlJSUq37LMvS8uXL9eSTT7p/Y+uVV15RdHS0Nm3apPHjx+vTTz/V1q1b9fHHH2vAgAGSpBdeeEHJycn6zW9+o7i4OG8eHgAAMIhXMzuPPPKIHnnkEVVVValnz57q3bu3x+ILJ06cUHFxsYYPH+7eFh4ersTEROXk5EiScnJyFBER4S46kjR8+HAFBQVp//79PskBAACaN69mdjZs2KA//OEPSk5O9nUet+LiYklSdHS0x/bo6Gj3vuLiYkVFRXnsb9mypSIjI93H1MblcsnlcrnXnU6nr2IDAIAA4/UNyjfeeKOvszSZzMxMhYeHuxeHw+HvSAAAoJF4VXZ+8Ytf6LnnnpNlWb7O4xYTEyNJKikp8dheUlLi3hcTE6Mvv/zSY39lZaXOnTvnPqY2GRkZKi0tdS9ffPGFj9MDAIBA4dXbWHv37tWuXbv0zjvv6KabblKrVq089mdnZzc4WKdOnRQTE6MdO3aoT58+kr59u2n//v166KGHJEmDBg3S+fPndfDgQfXv31/StzdHV1dXKzExsc5r2+122e32BmcEAACBz6uyExERofvuu6/Bf7ysrEz5+fnu9RMnTig3N1eRkZG64YYb9Oijj+rpp59Wly5d1KlTJ82bN09xcXG69957JUk9evTQyJEjNX36dK1atUoVFRWaNWuWxo8fzyexAACAJC/LTlZWlk/++IEDBzR06FD3enp6uiQpJSVFa9eu1Zw5c1ReXq4ZM2bo/PnzGjx4sLZu3aqQkBD3OevWrdOsWbM0bNgwBQUFaezYsXr++ed9kg8AADR/NsvLG28qKyu1e/duFRQUaOLEiQoNDdWZM2cUFhamH/zgB77O2aicTqfCw8NVWlqqsLAwf8cBAKBO8XO3+DtCvZ1cfHejXPdqX7+9mtk5deqURo4cqcLCQrlcLv34xz9WaGiofv3rX8vlcmnVqlVeBwcAAPAlr79UcMCAAfr666/VunVr9/b77rtPO3bs8Fk4AACAhvJqZueDDz7Qhx9+WONHP+Pj4/W3v/3NJ8EAAAB8wauZnerqalVVVdXYfvr0aYWGhjY4FAAAgK94VXbuuusuLV++3L1us9lUVlam+fPnN+pPSAAAANSXV29jLV26VCNGjNA//dM/6eLFi5o4caKOHTum9u3b67XXXvN1RgAAAK95VXauv/56/fnPf9aGDRt05MgRlZWVKTU1VZMmTfK4YRkAAMDfvCo70re/Lj558mRfZgEAAPA5r8rOK6+8ctn9U6dO9SoMAACAr3lVdh555BGP9YqKCn3zzTcKDg5WmzZtKDsAACBgePVprK+//tpjKSsrU15engYPHswNygAAIKB4VXZq06VLFy1evLjGrA8AAIA/+azsSN/etHzmzBlfXhIAAKBBvLpn56233vJYtyxLRUVFevHFF3Xrrbf6JBgAAIAveFV27r33Xo91m82mDh066M4779TSpUt9kQsAAMAnvCo71dXVvs4BAADQKHx6zw4AAECg8WpmJz09/aqPXbZsmTd/AgAAwCe8KjuHDx/W4cOHVVFRoW7dukmSPv/8c7Vo0UL9+vVzH2ez2XyTEgAAwEtelZ1Ro0YpNDRUL7/8sq677jpJ337R4P3336/bbrtNv/jFL3waEgAAwFte3bOzdOlSZWZmuouOJF133XV6+umn+TQWAAAIKF6VHafTqa+++qrG9q+++koXLlxocCgAAABf8ars3Hfffbr//vuVnZ2t06dP6/Tp03rjjTeUmpqqMWPG+DojAACA17y6Z2fVqlV67LHHNHHiRFVUVHx7oZYtlZqaqiVLlvg0IAAAQEN4VXbatGmjl156SUuWLFFBQYEkKSEhQW3btvVpOAAAgIZq0JcKFhUVqaioSF26dFHbtm1lWZavcgEAAPiEV2Xn7NmzGjZsmLp27ark5GQVFRVJklJTU/nYOQAACChelZ20tDS1atVKhYWFatOmjXv7uHHjtHXrVp+FAwAAaCiv7tl577339O677+r666/32N6lSxedOnXKJ8EAAAB8wauZnfLyco8Zne+cO3dOdru9waEAAAB8xauyc9ttt+mVV15xr9tsNlVXV+vZZ5/V0KFDfRYOAACgobx6G+vZZ5/VsGHDdODAAV26dElz5szR0aNHde7cOf3P//yPrzMCAAB4zauZnZ49e+rzzz/X4MGDNXr0aJWXl2vMmDE6fPiwEhISfJ0RAADAa/We2amoqNDIkSO1atUqPfHEE42RCQAAwGfqPbPTqlUrHTlypDGyAAAA+JxXb2NNnjxZq1ev9nUWAAAAn/PqBuXKykqtWbNG27dvV//+/Wv8JtayZct8Eg4AAKCh6lV2jh8/rvj4eH3yySfq16+fJOnzzz/3OMZms/kuHQAAQAPVq+x06dJFRUVF2rVrl6Rvfx7i+eefV3R0dKOEAwAAaKh63bPzj79q/s4776i8vNyngf5RfHy8bDZbjWXmzJmSpCFDhtTY9+CDDzZqJgAA0Hx4dc/Od/6x/DSGjz/+WFVVVe71Tz75RD/+8Y/1k5/8xL1t+vTpWrRokXu9tp+yAAAA16Z6lZ3vZk7+cVtj6tChg8f64sWLlZCQoDvuuMO9rU2bNoqJiWnUHAAAoHmqV9mxLEvTpk1z/9jnxYsX9eCDD9b4NFZ2drbvEn7PpUuX9Oqrryo9Pd2jZK1bt06vvvqqYmJiNGrUKM2bN++yszsul0sul8u97nQ6GyUvAADwv3qVnZSUFI/1yZMn+zTMlWzatEnnz5/XtGnT3NsmTpyojh07Ki4uTkeOHNHjjz+uvLy8yxauzMxMLVy4sAkSAwAAf7NZTXHjjY+MGDFCwcHB+tOf/lTnMTt37tSwYcOUn59f5+901Taz43A4VFpaqrCwMJ/nBgDAV+LnbvF3hHo7ufjuRrmu0+lUeHj4FV+/G3SDclM6deqUtm/ffsW3yBITEyXpsmXHbre734oDAABm8+rnIvwhKytLUVFRuvvuy7fD3NxcSVJsbGwTpAIAAIGuWczsVFdXKysrSykpKWrZ8v8jFxQUaP369UpOTla7du105MgRpaWl6fbbb1evXr38mBgAAASKZlF2tm/frsLCQv3sZz/z2B4cHKzt27dr+fLlKi8vl8Ph0NixY/Xkk0/6KSkAAAg0zaLs3HXXXbV+gaHD4dD777/vh0QAAKC5aDb37AAAAHiDsgMAAIxG2QEAAEaj7AAAAKNRdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaJQdAABgNMoOAAAwGmUHAAAYjbIDAACMRtkBAABGo+wAAACjUXYAAIDRKDsAAMBolB0AAGA0yg4AADAaZQcAABiNsgMAAIxG2QEAAEaj7AAAAKNRdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaAFddhYsWCCbzeaxdO/e3b3/4sWLmjlzptq1a6cf/OAHGjt2rEpKSvyYGAAABJqALjuSdNNNN6moqMi97N27170vLS1Nf/rTn7Rx40a9//77OnPmjMaMGePHtAAAINC09HeAK2nZsqViYmJqbC8tLdXq1au1fv163XnnnZKkrKws9ejRQ/v27dM///M/N3VUAAAQgAJ+ZufYsWOKi4tT586dNWnSJBUWFkqSDh48qIqKCg0fPtx9bPfu3XXDDTcoJyfnstd0uVxyOp0eCwAAMFNAl53ExEStXbtWW7du1cqVK3XixAnddtttunDhgoqLixUcHKyIiAiPc6Kjo1VcXHzZ62ZmZio8PNy9OByORnwUAADAnwL6baykpCT3v3v16qXExER17NhRf/jDH9S6dWuvr5uRkaH09HT3utPppPAAAGCogJ7Z+UcRERHq2rWr8vPzFRMTo0uXLun8+fMex5SUlNR6j8/32e12hYWFeSwAAMBMzarslJWVqaCgQLGxserfv79atWqlHTt2uPfn5eWpsLBQgwYN8mNKAAAQSAL6bazHHntMo0aNUseOHXXmzBnNnz9fLVq00IQJExQeHq7U1FSlp6crMjJSYWFhmj17tgYNGsQnsQAAgFtAl53Tp09rwoQJOnv2rDp06KDBgwdr37596tChgyTpt7/9rYKCgjR27Fi5XC6NGDFCL730kp9TAwCAQGKzLMvydwh/czqdCg8PV2lpKffvAAACWvzcLf6OUG8nF9/dKNe92tfvZnXPDgAAQH0F9NtYAIDmgxkHBCpmdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaJQdAABgNMoOAAAwGmUHAAAYjbIDAACMRtkBAABGo+wAAACjUXYAAIDRKDsAAMBolB0AAGA0yg4AADAaZQcAABiNsgMAAIxG2QEAAEaj7AAAAKNRdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0Sg7AADAaJQdAABgNMoOAAAwGmUHAAAYLaDLTmZmpn70ox8pNDRUUVFRuvfee5WXl+dxzJAhQ2Sz2TyWBx980E+JAQBAoAnosvP+++9r5syZ2rdvn7Zt26aKigrdddddKi8v9zhu+vTpKioqci/PPvusnxIDAIBA09LfAS5n69atHutr165VVFSUDh48qNtvv929vU2bNoqJiWnqeACAZi5+7hZ/R0ATCOiZnX9UWloqSYqMjPTYvm7dOrVv3149e/ZURkaGvvnmm8tex+Vyyel0eiwAAMBMAT2z833V1dV69NFHdeutt6pnz57u7RMnTlTHjh0VFxenI0eO6PHHH1deXp6ys7PrvFZmZqYWLlzYFLEBAICf2SzLsvwd4mo89NBDeuedd7R3715df/31dR63c+dODRs2TPn5+UpISKj1GJfLJZfL5V53Op1yOBwqLS1VWFiYz7MDwLWAt4RQl5OL726U6zqdToWHh1/x9btZzOzMmjVLmzdv1p49ey5bdCQpMTFRki5bdux2u+x2u89zAgCAwBPQZceyLM2ePVt//OMftXv3bnXq1OmK5+Tm5kqSYmNjGzkdAABoDgK67MycOVPr16/Xm2++qdDQUBUXF0uSwsPD1bp1axUUFGj9+vVKTk5Wu3btdOTIEaWlpen2229Xr169/JweAAAEgoAuOytXrpT07RcHfl9WVpamTZum4OBgbd++XcuXL1d5ebkcDofGjh2rJ5980g9pAQBAIArosnOle6cdDofef//9JkoDAACao2b1PTsAAAD1RdkBAABGo+wAAACjUXYAAIDRKDsAAMBoAf1pLADwheb4MwaN9fX6wLWImR0AAGA0yg4AADAaZQcAABiNsgMAAIxG2QEAAEbj01gAEICa4yfIgEDFzA4AADAaZQcAABiNsgMAAIxG2QEAAEaj7AAAAKNRdgAAgNEoOwAAwGiUHQAAYDTKDgAAMBplBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0Vr6OwCA5iV+7hZ/RwCAemFmBwAAGI2yAwAAjEbZAQAARqPsAAAAo1F2AACA0fg0FuAnfKoJAJoGMzsAAMBolB0AAGA0yg4AADAaZQcAABiNsgMAAIxmTNlZsWKF4uPjFRISosTERH300Uf+jgQAAAKAEWXn9ddfV3p6uubPn69Dhw6pd+/eGjFihL788kt/RwMAAH5msyzL8neIhkpMTNSPfvQjvfjii5Kk6upqORwOzZ49W3Pnzr3i+U6nU+Hh4SotLVVYWJhPszXH71I5ufhuf0eot+Y4zgBwrWis15Wrff1u9l8qeOnSJR08eFAZGRnubUFBQRo+fLhycnJqPcflcsnlcrnXS0tLJX07aL5W7frG59dsbI0xDo2tOY4zAFwrGut15bvrXmneptmXnb///e+qqqpSdHS0x/bo6Gh99tlntZ6TmZmphQsX1tjucDgaJWNzE77c3wkAACZp7NeVCxcuKDw8vM79zb7seCMjI0Pp6enu9fPnz6tjx44qLCy87GChJqfTKYfDoS+++MLnbwGajHHzHmPnPcbOe4yd9xpz7CzL0oULFxQXF3fZ45p92Wnfvr1atGihkpISj+0lJSWKiYmp9Ry73S673V5je3h4OE9iL4WFhTF2XmDcvMfYeY+x8x5j573GGrurmaRo9p/GCg4OVv/+/bVjxw73turqau3YsUODBg3yYzIAABAImv3MjiSlp6crJSVFAwYM0MCBA7V8+XKVl5fr/vvv93c0AADgZ0aUnXHjxumrr77SU089peLiYvXp00dbt26tcdNyXex2u+bPn1/rW1u4PMbOO4yb9xg77zF23mPsvBcIY2fE9+wAAADUpdnfswMAAHA5lB0AAGA0yg4AADAaZQcAABjtmik7K1asUHx8vEJCQpSYmKiPPvroqs7bsGGDbDab7r333sYNGKDqM25r166VzWbzWEJCQpowbWCp73Pu/PnzmjlzpmJjY2W329W1a1e9/fbbTZQ2sNRn7IYMGVLjeWez2XT33c3vB219ob7Pu+XLl6tbt25q3bq1HA6H0tLSdPHixSZKG1jqM3YVFRVatGiREhISFBISot69e2vr1q1NmDZw7NmzR6NGjVJcXJxsNps2bdp0xXN2796tfv36yW6368Ybb9TatWsbN6R1DdiwYYMVHBxsrVmzxjp69Kg1ffp0KyIiwiopKbnseSdOnLB++MMfWrfddps1evTopgkbQOo7bllZWVZYWJhVVFTkXoqLi5s4dWCo79i5XC5rwIABVnJysrV3717rxIkT1u7du63c3NwmTu5/9R27s2fPejznPvnkE6tFixZWVlZW0wYPAPUdu3Xr1ll2u91at26ddeLECevdd9+1YmNjrbS0tCZO7n/1Hbs5c+ZYcXFx1pYtW6yCggLrpZdeskJCQqxDhw41cXL/e/vtt60nnnjCys7OtiRZf/zjHy97/PHjx602bdpY6enp1l//+lfrhRdesFq0aGFt3bq10TJeE2Vn4MCB1syZM93rVVVVVlxcnJWZmVnnOZWVldYtt9xi/dd//ZeVkpJyTZad+o5bVlaWFR4e3kTpAlt9x27lypVW586drUuXLjVVxIDlzf/X7/vtb39rhYaGWmVlZY0VMWDVd+xmzpxp3XnnnR7b0tPTrVtvvbVRcwai+o5dbGys9eKLL3psGzNmjDVp0qRGzRnorqbszJkzx7rppps8to0bN84aMWJEo+Uy/m2sS5cu6eDBgxo+fLh7W1BQkIYPH66cnJw6z1u0aJGioqKUmpraFDEDjrfjVlZWpo4dO8rhcGj06NE6evRoU8QNKN6M3VtvvaVBgwZp5syZio6OVs+ePfXMM8+oqqqqqWIHBG+fd9+3evVqjR8/Xm3btm2smAHJm7G75ZZbdPDgQffbNcePH9fbb7+t5OTkJskcKLwZO5fLVeNt+tatW2vv3r2NmtUEOTk5HmMtSSNGjLjq/+PeML7s/P3vf1dVVVWNb1OOjo5WcXFxrefs3btXq1ev1u9///umiBiQvBm3bt26ac2aNXrzzTf16quvqrq6WrfccotOnz7dFJEDhjdjd/z4cf33f/+3qqqq9Pbbb2vevHlaunSpnn766aaIHDC8Gbvv++ijj/TJJ5/o3/7t3xorYsDyZuwmTpyoRYsWafDgwWrVqpUSEhI0ZMgQ/fKXv2yKyAHDm7EbMWKEli1bpmPHjqm6ulrbtm1Tdna2ioqKmiJys1ZcXFzrWDudTv3v//5vo/xN48tOfV24cEFTpkzR73//e7Vv397fcZqVQYMGaerUqerTp4/uuOMOZWdnq0OHDvrP//xPf0cLeNXV1YqKitLvfvc79e/fX+PGjdMTTzyhVatW+Ttas7J69WrdfPPNGjhwoL+jNAu7d+/WM888o5deekmHDh1Sdna2tmzZol/96lf+jhbwnnvuOXXp0kXdu3dXcHCwZs2apfvvv19BQbysBiIjfhvrctq3b68WLVqopKTEY3tJSYliYmJqHF9QUKCTJ09q1KhR7m3V1dWSpJYtWyovL08JCQmNGzoA1HfcatOqVSv17dtX+fn5jRExYHkzdrGxsWrVqpVatGjh3tajRw8VFxfr0qVLCg4ObtTMgaIhz7vy8nJt2LBBixYtasyIAcubsZs3b56mTJningm7+eabVV5erhkzZuiJJ564Zl64vRm7Dh06aNOmTbp48aLOnj2ruLg4zZ07V507d26KyM1aTExMrWMdFham1q1bN8rfNP6ZHBwcrP79+2vHjh3ubdXV1dqxY4cGDRpU4/ju3bvrL3/5i3Jzc93LPffco6FDhyo3N1cOh6Mp4/tNfcetNlVVVfrLX/6i2NjYxooZkLwZu1tvvVX5+fnuYi1Jn3/+uWJjY6+ZoiM17Hm3ceNGuVwuTZ48ubFjBiRvxu6bb76pUWi+K9zWNfSziQ153oWEhOiHP/yhKisr9cYbb2j06NGNHbfZGzRokMdYS9K2bduu+rXFK41263MA2bBhg2W32621a9daf/3rX60ZM2ZYERER7o9FT5kyxZo7d26d51+rn8aq77gtXLjQevfdd62CggLr4MGD1vjx462QkBDr6NGj/noIflPfsSssLLRCQ0OtWbNmWXl5edbmzZutqKgo6+mnn/bXQ/Abb/+/Dh482Bo3blxTxw0o9R27+fPnW6GhodZrr71mHT9+3HrvvfeshIQE66c//am/HoLf1Hfs9u3bZ73xxhtWQUGBtWfPHuvOO++0OnXqZH399dd+egT+c+HCBevw4cPW4cOHLUnWsmXLrMOHD1unTp2yLMuy5s6da02ZMsV9/HcfPf/3f/9369NPP7VWrFjBR8995YUXXrBuuOEGKzg42Bo4cKC1b98+97477rjDSklJqfPca7XsWFb9xu3RRx91HxsdHW0lJydfk9858Z36Puc+/PBDKzEx0bLb7Vbnzp2t//iP/7AqKyubOHVgqO/YffbZZ5Yk67333mvipIGnPmNXUVFhLViwwEpISLBCQkIsh8Nh/fznP78mX7Atq35jt3v3bqtHjx6W3W632rVrZ02ZMsX629/+5ofU/rdr1y5LUo3lu/FKSUmx7rjjjhrn9OnTxwoODrY6d+7c6N+LZbOsa2iuEgAAXHOMv2cHAABc2yg7AADAaJQdAABgNMoOAAAwGmUHAAAYjbIDAACMRtkBAABGo+wAAACjUXYAAIDRKDsAAMBolB0AAGA0yg4AADDa/wFoCfxT2P/4LwAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "validation_results_df['score'].plot.hist(bins=10)" - ] - }, - { - "cell_type": "markdown", - "id": "8972f998", - "metadata": {}, - "source": [ - "Looks like a great improvement - **92%** of entities were extracted correctly, with around 60% completely correct, an improvement of roughly **10/11%**." - ] - }, - { - "cell_type": "markdown", - "id": "318b7175", - "metadata": {}, - "source": [ - "## Model comparison\n", - "\n", - "The last step is to summarize our results in a table for comparison." - ] - }, - { - "cell_type": "code", - "execution_count": 52, - "id": "c7c2a539", - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
modelresult
0gpt-3.5-turbo0.810288
1gpt-40.820128
2gpt-3.5-turbo-ft0.923313
\n", - "
" - ], - "text/plain": [ - " model result\n", - "0 gpt-3.5-turbo 0.810288\n", - "1 gpt-4 0.820128\n", - "2 gpt-3.5-turbo-ft 0.923313" - ] - }, - "execution_count": 52, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "gpt_35_score = results_df['score'].mean()\n", - "gpt_4_score = results_gpt4_df['score'].mean()\n", - "gpt_35_ft_score = validation_results_df['score'].mean()\n", - "\n", - "eval_results = pd.DataFrame({'model': ['gpt-3.5-turbo', 'gpt-4','gpt-3.5-turbo-ft'], 'result': [gpt_35_score,gpt_4_score,gpt_35_ft_score]})\n", - "eval_results.head()" - ] - }, - { - "cell_type": "markdown", - "id": "04ebcaa7", - "metadata": {}, - "source": [ - "Great! Our fine-tuned model is faster and cheaper than `gpt-4` for this task - mission accomplished." - ] - }, - { - "cell_type": "markdown", - "id": "07799909-3f2a-4274-b81e-dabc048be28f", - "metadata": {}, - "source": [ - "## Conclusion\n", - "\n", - "Congratulations, you are now ready to fine-tune your own models using the `ChatCompletion` format!\n", - "\n", - "Please reach out via Slack or email to your Account Director or Account Engineer with any successes or issues you experience, we're excited to hear what works and what doesn't. Look forward to seeing what you build!" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "chat_ft", - "language": "python", - "name": "chat_ft" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.11" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -}