openai-cookbook/examples/evaluation/How_to_evaluate_LLMs_for_SQL_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d5dd7679-5ed6-4243-8e0f-549f8118bff7",
   "metadata": {},
   "source": [
    "# How to test and evaluate LLMs for SQL generation\n",
    "\n",
    "LLMs are fundamentatlly non-deterministic in their responses, this attribute makes them wonderfully creative and dynamic in their responses. However, this trait poses significant challenges in achieving consistency, a crucial aspect for integrating LLMs into production environments.\n",
    "\n",
    "The key to harnessing the potential of LLMs in practical applications lies in consistent and systematic evaluation. This enables the identification and rectification of inconsistencies and helps in monitoring progress over time as the application evolves.\n",
    "\n",
    "## Scope of this notebook\n",
    "\n",
    "This notebook aims to demonstrate a framework for evaluating LLMs, particularly focusing on:\n",
    "\n",
    "* **Unit Testing:** Essential for assessing individual components of the application.\n",
    "* **Evaluation Metrics:** Methods to quantitatively measure the model's effectiveness.\n",
    "* **Runbook Documentation:** A record of historical evaluations to track progress and regression.\n",
    "\n",
    "This example focuses on a natural language to SQL use case - code generation use cases fit well with this approach when you combine **code validation** with **code execution**, so your application can test code for real as it is generated to ensure consistency.\n",
    "\n",
    "Although this notebook uses SQL generation usecase to demonstrate the concept, the approach is generic and can be applied to a wide variety of LLM driven applications.\n",
    "\n",
    "We will use two versions of a prompt to perform SQL generation.  We will then use the unit tests and evaluation functions to test the perforamance of the prompts.  Specifically, in this demonstration, we will evaluate:\n",
    "\n",
    "1. The consistency of JSON response.\n",
    "2. Syntactic correctness of SQL in response.\n",
    "\n",
    "\n",
    "## Table of contents\n",
    "\n",
    "1. **[Setup](#Setup):** Install required libraries, download data consisting of SQL queries and corresponding natural language translations.\n",
    "2. **[Test Development](#Test-development):** Create unit tests and define evaluation metrics for the SQL generation process.\n",
    "4. **[Evaluation](#Evaluation):** Conduct tests using different prompts to assess the impact on performance.\n",
    "5. **[Reporting](#Report):** Compile a report that succinctly presents the performance differences observed across various tests."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2913d615",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Import our libraries and the dataset we'll use, which is the natural language to SQL [b-mc2/sql-create-context](https://huggingface.co/datasets/b-mc2/sql-create-context) dataset from HuggingFace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "245fcedb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "from openai import OpenAI\n",
    "import pandas as pd\n",
    "import pydantic\n",
    "import os\n",
    "import sqlite3\n",
    "from sqlite3 import Error\n",
    "from pprint import pprint\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "# Loads key from local .env file to setup API KEY in env variables\n",
    "%reload_ext dotenv\n",
    "%dotenv\n",
    "    \n",
    "GPT_MODEL = 'gpt-3.5-turbo'\n",
    "dataset = load_dataset(\"b-mc2/sql-create-context\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04c7fde6-d7dc-4a0d-b9a0-32858f3bac25",
   "metadata": {},
   "source": [
    "### Looking at the dataset\n",
    "\n",
    "We use Huggingface datasets library to download SQL create context dataset.  This dataset consists of:\n",
    "\n",
    "1. Question, expressed in natural language\n",
    "2. Answer, expressed in SQL designed to answer the question in natural language.\n",
    "3. Context, expressed as a CREATE SQL statement, that describes the table that may be used to answer the question.\n",
    "\n",
    "In our demonstration today, we will use LLM to attempt to answer the question (in natural language).  The LLM will be expected to generate a CREATE SQL statement to create a context suitable to answer the user question and a coresponding SELECT SQL query designed to answer the user question completely.\n",
    "\n",
    "The dataset looks like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "f8027115",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>answer</th>\n",
       "      <th>context</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How many heads of the departments are older th...</td>\n",
       "      <td>SELECT COUNT(*) FROM head WHERE age &gt; 56</td>\n",
       "      <td>CREATE TABLE head (age INTEGER)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>List the name, born state and age of the heads...</td>\n",
       "      <td>SELECT name, born_state, age FROM head ORDER B...</td>\n",
       "      <td>CREATE TABLE head (name VARCHAR, born_state VA...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>List the creation year, name and budget of eac...</td>\n",
       "      <td>SELECT creation, name, budget_in_billions FROM...</td>\n",
       "      <td>CREATE TABLE department (creation VARCHAR, nam...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What are the maximum and minimum budget of the...</td>\n",
       "      <td>SELECT MAX(budget_in_billions), MIN(budget_in_...</td>\n",
       "      <td>CREATE TABLE department (budget_in_billions IN...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What is the average number of employees of the...</td>\n",
       "      <td>SELECT AVG(num_employees) FROM department WHER...</td>\n",
       "      <td>CREATE TABLE department (num_employees INTEGER...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  How many heads of the departments are older th...   \n",
       "1  List the name, born state and age of the heads...   \n",
       "2  List the creation year, name and budget of eac...   \n",
       "3  What are the maximum and minimum budget of the...   \n",
       "4  What is the average number of employees of the...   \n",
       "\n",
       "                                              answer  \\\n",
       "0           SELECT COUNT(*) FROM head WHERE age > 56   \n",
       "1  SELECT name, born_state, age FROM head ORDER B...   \n",
       "2  SELECT creation, name, budget_in_billions FROM...   \n",
       "3  SELECT MAX(budget_in_billions), MIN(budget_in_...   \n",
       "4  SELECT AVG(num_employees) FROM department WHER...   \n",
       "\n",
       "                                             context  \n",
       "0                    CREATE TABLE head (age INTEGER)  \n",
       "1  CREATE TABLE head (name VARCHAR, born_state VA...  \n",
       "2  CREATE TABLE department (creation VARCHAR, nam...  \n",
       "3  CREATE TABLE department (budget_in_billions IN...  \n",
       "4  CREATE TABLE department (num_employees INTEGER...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sql_df = dataset['train'].to_pandas()\n",
    "sql_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b04cb5eb",
   "metadata": {},
   "source": [
    "\n",
    "## Test development\n",
    "\n",
    "To test to output of the LLM generations, we'll develop two unit tests and an evaluation, which will combine to give us a basic evaluation framework to grade the quality of our LLM iterations.\n",
    "\n",
    "To re-iterate, our purpose is to measure the correctness and consistency of LLM output given our questions.\n",
    "\n",
    "### Unit tests\n",
    "\n",
    "Unit tests should test the most granular components of your LLM application.\n",
    "\n",
    "For this section we'll develop unit tests to test the following:\n",
    "- `test_valid_schema` will check that a parseable `create` and `select` statement are returned by the LLM.\n",
    "- `test_llm_sql` will execute both the `create` and `select` statements on a `sqlite` database to ensure they are syntactically correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "eb811101",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pydantic import BaseModel\n",
    "\n",
    "class LLMResponse(BaseModel):\n",
    "   \"\"\"This simple Class expects to receive a JSON string that can be parsed into a `create` and `select` statement.\"\"\"\n",
    "   create: str\n",
    "   select: str"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19fadf67-8b2f-4e17-95df-030a36aad90b",
   "metadata": {},
   "source": [
    "#### Prompt\n",
    "\n",
    "For this demonstration purposes, we use a fairly simple prompt requesting GPT to generate a pair of context CREATE SQL and a answering SELECT SQL query.  We supply the natural language question as part of the prompt.  We request the response to be in JSON format, so that it can be parsed easily."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c2be3ba4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Translate this natural language request into a JSON object containing two '\n",
      " 'SQL queries. \\n'\n",
      " 'The first query should be a CREATE statement for a table answering the '\n",
      " \"user's request, while the second should be a SELECT query answering their \"\n",
      " 'question. \\n'\n",
      " 'This should be returned as a JSON with keys \"create\" and \"select\". \\n'\n",
      " 'For example:\\n'\n",
      " '\\n'\n",
      " '{\"create\": \"\"\"CREATE_QUERY\"\"\",\"select\": \"\"\"SELECT_QUERY\"\"\"}\"')\n"
     ]
    }
   ],
   "source": [
    "system_prompt = '''Translate this natural language request into a JSON object containing two SQL queries. \n",
    "The first query should be a CREATE statement for a table answering the user's request, while the second should be a SELECT query answering their question. \n",
    "This should be returned as a JSON with keys \"create\" and \"select\". \n",
    "For example:\\n\\n{\"create\": \"\"\"CREATE_QUERY\"\"\",\"select\": \"\"\"SELECT_QUERY\"\"\"}\"'''\n",
    "\n",
    "pprint(system_prompt)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3a20d712",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'content': 'Translate this natural language request into a JSON object '\n",
      "             'containing two SQL queries. \\n'\n",
      "             'The first query should be a CREATE statement for a table '\n",
      "             \"answering the user's request, while the second should be a \"\n",
      "             'SELECT query answering their question. \\n'\n",
      "             'This should be returned as a JSON with keys \"create\" and '\n",
      "             '\"select\". \\n'\n",
      "             'For example:\\n'\n",
      "             '\\n'\n",
      "             '{\"create\": \"\"\"CREATE_QUERY\"\"\",\"select\": \"\"\"SELECT_QUERY\"\"\"}\"',\n",
      "  'role': 'system'},\n",
      " {'content': 'How many heads of the departments are older than 56 ?',\n",
      "  'role': 'user'}]\n"
     ]
    }
   ],
   "source": [
    "# Compiling the system prompt and user question into message array\n",
    "\n",
    "messages = []\n",
    "messages.append({\"role\": \"system\", \"content\": system_prompt})\n",
    "messages.append({\"role\":\"user\",\"content\": sql_df.iloc[0]['question']})\n",
    "pprint(messages)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "38b704b3-6f0e-4708-bc70-96723d69da6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sending the message array to GPT, requesting a response (ensure that you have API key loaded to Env for this step)\n",
    "\n",
    "client = OpenAI()\n",
    "completion = client.chat.completions.create(model = GPT_MODEL, messages = messages)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "901e3bb7",
   "metadata": {},
   "source": [
    "#### Check JSON formatting\n",
    "\n",
    "Our first simple unit test checks that the LLM response is parseable into the `LLMResponse` Pydantic class that we've defined.\n",
    "\n",
    "We'll test that our first response passes, then create a failing example to check that the check fails. This logic will be wrapped in a simple function `test_valid_schema`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2b057391-4f83-4b5a-8843-a9ee74bee871",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('{\"create\": \"CREATE TABLE departments (department_id INTEGER PRIMARY KEY, '\n",
      " 'department_name TEXT, head_name TEXT, age INTEGER)\", \"select\": \"SELECT '\n",
      " 'COUNT(*) FROM departments WHERE age > 56\"}')\n"
     ]
    }
   ],
   "source": [
    "# Viewing the output from  GPT\n",
    "\n",
    "content = completion.choices[0].message.content\n",
    "pprint(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b98bbb4-dd17-49bc-828c-e561abf5b481",
   "metadata": {},
   "source": [
    "#### Validating the output schema\n",
    "\n",
    "We expect GPT to respond with a valid SQL, we can validate this using LLMResponse base model.  `test_valid_schema` is designed to help us validate this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4c7133f1-74d6-43f1-9443-09a3f8308c35",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_valid_schema(content):\n",
    "    \"\"\"Tests whether the content provided can be parsed into our Pydantic model.\"\"\"\n",
    "    try:\n",
    "        LLMResponse.model_validate_json(content)\n",
    "        return True\n",
    "    # Catch pydantic's validation errors:\n",
    "    except pydantic.ValidationError as exc:\n",
    "        print(f\"ERROR: Invalid schema: {exc}\")\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6a9a9128",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_valid_schema(content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78f1af23-4dd0-4860-8a1a-88e5146be6ed",
   "metadata": {},
   "source": [
    "#### Testing negative scenario\n",
    "\n",
    "To simulate a scenario in which we get an invalid JSON response from GPT, we hardcode an invalid JSON as response.  We expect `test_valid_schema` function to throw an exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "a0a26690",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ERROR: Invalid schema: 1 validation error for LLMResponse\n",
      "  Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='CREATE departments, select * from departments', input_type=str]\n",
      "    For further information visit https://errors.pydantic.dev/2.5/v/json_invalid\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "failing_query = 'CREATE departments, select * from departments'\n",
    "test_valid_schema(failing_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5fdc420-94cc-4e47-80e1-82bf51e44f2a",
   "metadata": {},
   "source": [
    "As expected, we get an exception thrown from the `test_valid_schema` fucntion."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4e972cd-5734-43c0-a103-b9ceb41552fd",
   "metadata": {},
   "source": [
    "### Test SQL queries\n",
    "\n",
    "Next we'll validate the correctness of the SQL.  This test will be desined to validate:\n",
    "\n",
    "1. The CREATE SQL returned in GPT response is syntactically correct.\n",
    "2. The SELECT SQL returned in the GPT response is syntactically correct.\n",
    "\n",
    "To achieve this, we will use a sqlite instance. We will direct the retured SQL functions to a sqlite instance.  If the SQL statements are valid, sqlite instance will accept and execute the statements; otherwise we will expect an exception to be thrown.\n",
    "\n",
    "`create_connection` function below will setup a sqlite instance (in-memory by default) and create a connection to be used later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9cc95481",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up SQLite to act as our test database\n",
    "def create_connection(db_file=\":memory:\"):\n",
    "    \"\"\"create a database connection to a SQLite database\"\"\"\n",
    "    try:\n",
    "        conn = sqlite3.connect(db_file)\n",
    "        # print(sqlite3.version)\n",
    "    except Error as e:\n",
    "        print(e)\n",
    "        return None\n",
    "\n",
    "    return conn\n",
    "\n",
    "def close_connection(conn):\n",
    "    \"\"\"close a database connection\"\"\"\n",
    "    try:\n",
    "        conn.close()\n",
    "    except Error as e:\n",
    "        print(e)\n",
    "\n",
    "\n",
    "conn = create_connection()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa5c5cb8-1c81-403b-a3f2-f2784d8235fc",
   "metadata": {},
   "source": [
    "Next, we will create the following functions to carry out the syntactical correctness checks.\n",
    "\n",
    "\n",
    "- `test_create`: Function testing if the CREATE SQL statement succeeds.\n",
    "- `test_select`: Function testing if the SELECT SQL statement succeeds.\n",
    "- `test_llm_sql`: Wrapper function executing the two tests above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c6d2573d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def test_select(conn, cursor, select):\n",
    "    \"\"\"Tests that a SQLite select query can be executed successfully.\"\"\"\n",
    "    try:\n",
    "        print(f\"Testing select query: {select}\")\n",
    "        cursor.execute(select)\n",
    "        record = cursor.fetchall()\n",
    "        print(record)\n",
    "\n",
    "        return True\n",
    "\n",
    "    except sqlite3.Error as error:\n",
    "        print(\"Error while executing select query:\", error)\n",
    "\n",
    "        return False\n",
    "\n",
    "\n",
    "def test_create(conn, cursor, create):\n",
    "    \"\"\"Tests that a SQLite create query can be executed successfully\"\"\"\n",
    "    try:\n",
    "        print(f\"Testing create query: {create}\")\n",
    "        cursor.execute(create)\n",
    "        conn.commit()\n",
    "\n",
    "        return True\n",
    "\n",
    "    except sqlite3.Error as error:\n",
    "        print(\"Error while creating the SQLite table:\", error)\n",
    "\n",
    "        return False\n",
    "\n",
    "\n",
    "def test_llm_sql(LLMResponse):\n",
    "    \"\"\"Runs a suite of SQLite tests\"\"\"\n",
    "    try:\n",
    "        conn = create_connection()\n",
    "        cursor = conn.cursor()\n",
    "\n",
    "        create_response = test_create(conn, cursor, LLMResponse.create)\n",
    "\n",
    "        select_response = test_select(conn, cursor, LLMResponse.select)\n",
    "\n",
    "        if conn:\n",
    "            close_connection(conn)\n",
    "\n",
    "        if create_response is not True:\n",
    "            return False\n",
    "\n",
    "        elif select_response is not True:\n",
    "            return False\n",
    "\n",
    "        else:\n",
    "            return True\n",
    "\n",
    "    except sqlite3.Error as error:\n",
    "        print(\"Error while creating a sqlite table\", error)\n",
    "\n",
    "        return False"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a9266753-4646-4901-bc14-632d3bf47aaa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CREATE SQL is: CREATE TABLE departments (department_id INTEGER PRIMARY KEY, department_name TEXT, head_name TEXT, age INTEGER)\n",
      "SELECT SQL is: SELECT COUNT(*) FROM departments WHERE age > 56\n"
     ]
    }
   ],
   "source": [
    "# Viewing CREATE and SELECT sqls returned by GPT\n",
    "\n",
    "test_query = LLMResponse.model_validate_json(content)\n",
    "print(f\"CREATE SQL is: {test_query.create}\")\n",
    "print(f\"SELECT SQL is: {test_query.select}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "83bc1f1b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing create query: CREATE TABLE departments (department_id INTEGER PRIMARY KEY, department_name TEXT, head_name TEXT, age INTEGER)\n",
      "Testing select query: SELECT COUNT(*) FROM departments WHERE age > 56\n",
      "[(0,)]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Testing the CREATE and SELECT sqls are valid (we expect this to be succesful)\n",
    "\n",
    "test_llm_sql(test_query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "589c7cc7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing create query: CREATE TABLE departments (id INT, name VARCHAR(255), head_of_department VARCHAR(255))\n",
      "Testing select query: SELECT COUNT(*) FROM departments WHERE age > 56\n",
      "Error while executing select query: no such column: age\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Again we'll perform a negative test to confirm that a failing SELECT will return an error.\n",
    "\n",
    "test_failure_query = '{\"create\": \"CREATE TABLE departments (id INT, name VARCHAR(255), head_of_department VARCHAR(255))\", \"select\": \"SELECT COUNT(*) FROM departments WHERE age > 56\"}'\n",
    "test_failure_query = LLMResponse.model_validate_json(test_failure_query)\n",
    "test_llm_sql(test_failure_query)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8148f820",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "The last component is to **evaluate** whether the generate SQL actually answers the user's question. This test will be performed by `gpt-4`, and will assess how **relevant** the produced SQL query is when compared to the initial user request.\n",
    "\n",
    "This is a simple example which adapts an approach outlined in the [G-Eval paper](https://arxiv.org/abs/2303.16634), and tested in one of our other [cookbooks](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "029c8426",
   "metadata": {},
   "outputs": [],
   "source": [
    "EVALUATION_MODEL = \"gpt-4\"\n",
    "\n",
    "EVALUATION_PROMPT_TEMPLATE = \"\"\"\n",
    "You will be given one summary written for an article. Your task is to rate the summary on one metric.\n",
    "Please make sure you read and understand these instructions very carefully. \n",
    "Please keep this document open while reviewing, and refer to it as needed.\n",
    "\n",
    "Evaluation Criteria:\n",
    "\n",
    "{criteria}\n",
    "\n",
    "Evaluation Steps:\n",
    "\n",
    "{steps}\n",
    "\n",
    "Example:\n",
    "\n",
    "Request:\n",
    "\n",
    "{request}\n",
    "\n",
    "Queries:\n",
    "\n",
    "{queries}\n",
    "\n",
    "Evaluation Form (scores ONLY):\n",
    "\n",
    "- {metric_name}\n",
    "\"\"\"\n",
    "\n",
    "# Relevance\n",
    "\n",
    "RELEVANCY_SCORE_CRITERIA = \"\"\"\n",
    "Relevance(1-5) - review of how relevant the produced SQL queries are to the original question. \\\n",
    "The queries should contain all points highlighted in the user's request. \\\n",
    "Annotators were instructed to penalize queries which contained redundancies and excess information.\n",
    "\"\"\"\n",
    "\n",
    "RELEVANCY_SCORE_STEPS = \"\"\"\n",
    "1. Read the request and the queries carefully.\n",
    "2. Compare the queries to the request document and identify the main points of the request.\n",
    "3. Assess how well the queries cover the main points of the request, and how much irrelevant or redundant information it contains.\n",
    "4. Assign a relevance score from 1 to 5.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "85cfb78d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_geval_score(\n",
    "    criteria: str, steps: str, request: str, queries: str, metric_name: str\n",
    "):\n",
    "    \"\"\"Given evaluation criteria and an observation, this function uses EVALUATION GPT to evaluate the observation against those criteria.\n",
    "\"\"\"\n",
    "    prompt = EVALUATION_PROMPT_TEMPLATE.format(\n",
    "        criteria=criteria,\n",
    "        steps=steps,\n",
    "        request=request,\n",
    "        queries=queries,\n",
    "        metric_name=metric_name,\n",
    "    )\n",
    "    response = client.chat.completions.create(\n",
    "        model=EVALUATION_MODEL,\n",
    "        messages=[{\"role\": \"user\", \"content\": prompt}],\n",
    "        temperature=0,\n",
    "        max_tokens=5,\n",
    "        top_p=1,\n",
    "        frequency_penalty=0,\n",
    "        presence_penalty=0,\n",
    "    )\n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "607ee304",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test out evaluation on a few records\n",
    "\n",
    "evaluation_results = []\n",
    "\n",
    "for x,y in sql_df.head(3).iterrows():\n",
    "    \n",
    "    score = get_geval_score(RELEVANCY_SCORE_CRITERIA,RELEVANCY_SCORE_STEPS,y['question'],y['context'] + '\\n' + y['answer'],'relevancy')\n",
    "    \n",
    "    evaluation_results.append((y['question'],y['context'] + '\\n' + y['answer'],score))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "bd1002c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "User Question \t: How many heads of the departments are older than 56 ?\n",
      "CREATE SQL Returned \t: CREATE TABLE head (age INTEGER)\n",
      "SELECT SQL Returned \t: SELECT COUNT(*) FROM head WHERE age > 56\n",
      "5\n",
      "********************\n",
      "User Question \t: List the name, born state and age of the heads of departments ordered by age.\n",
      "CREATE SQL Returned \t: CREATE TABLE head (name VARCHAR, born_state VARCHAR, age VARCHAR)\n",
      "SELECT SQL Returned \t: SELECT name, born_state, age FROM head ORDER BY age\n",
      "5\n",
      "********************\n",
      "User Question \t: List the creation year, name and budget of each department.\n",
      "CREATE SQL Returned \t: CREATE TABLE department (creation VARCHAR, name VARCHAR, budget_in_billions VARCHAR)\n",
      "SELECT SQL Returned \t: SELECT creation, name, budget_in_billions FROM department\n",
      "5\n",
      "********************\n"
     ]
    }
   ],
   "source": [
    "for result in evaluation_results:\n",
    "    print(f\"User Question \\t: {result[0]}\")\n",
    "    print(f\"CREATE SQL Returned \\t: {result[1].splitlines()[0]}\")\n",
    "    print(f\"SELECT SQL Returned \\t: {result[1].splitlines()[1]}\")\n",
    "    print(f\"{result[2]}\")\n",
    "    print(\"*\" * 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afe98f7a-3e88-437f-a5cd-d105969d3020",
   "metadata": {},
   "source": [
    "## "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe04c6c7",
   "metadata": {},
   "source": [
    "## Putting it all together\n",
    "\n",
    "We'll now test these functions in combination including our unit test and evaluations to test out two system prompts.\n",
    "\n",
    "Each iteration of input/output and scores should be stored as a **run**. Optionally you can add GPT-4 annotation within your evaluations or as a separate step to review an entire run and highlight the reasons for errors.\n",
    "\n",
    "For this example, the second system prompt will include an extra line of clarification, so we can assess the impact of this for both SQL validity and quality of solution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b68e2a",
   "metadata": {},
   "source": [
    "### First run - System Prompt 1\n",
    "\n",
    "The system under test is the first system prompt as shown below.  This `run` will generate responses for this system prompt and evaluate the responses using the functions we've created so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "85c44a17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Translate this natural language request into a JSON object containing two SQL queries. \\n'\n",
      " \"The first query should be a CREATE statement for a table answering the user's request, while the second should be a \"\n",
      " 'SELECT query answering their question. \\n'\n",
      " 'This should be returned as a JSON with keys \"create\" and \"select\". \\n'\n",
      " 'For example:\\n'\n",
      " '\\n'\n",
      " '{\"create\": \"CREATE_QUERY\",\"select\": \"SELECT_QUERY\"}\" ')\n"
     ]
    }
   ],
   "source": [
    "# Set first system prompt\n",
    "system_prompt = \"\"\"Translate this natural language request into a JSON object containing two SQL queries. \n",
    "The first query should be a CREATE statement for a table answering the user's request, while the second should be a SELECT query answering their question. \n",
    "This should be returned as a JSON with keys \"create\" and \"select\". \n",
    "For example:\\n\\n{\"create\": \"CREATE_QUERY\",\"select\": \"SELECT_QUERY\"}\" \"\"\"\n",
    "\n",
    "pprint(system_prompt, width = 120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "1244c44e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_response(system_prompt,user_message,model=GPT_MODEL):\n",
    "    messages = []\n",
    "    messages.append({\"role\": \"system\", \"content\": system_prompt})\n",
    "    messages.append({\"role\":\"user\",\"content\": user_message})\n",
    "    \n",
    "    response = client.chat.completions.create(model=GPT_MODEL,messages=messages,temperature=0)\n",
    "    \n",
    "    return response.choices[0].message.content"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76c2723b-3060-400f-b6fe-c3c3c9d6907e",
   "metadata": {},
   "source": [
    "#### Run the tests and evaluations\n",
    "\n",
    "The functions below, run unit test and evaluate responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "a98afa30",
   "metadata": {},
   "outputs": [],
   "source": [
    "def execute_unit_tests(input_df,output_list,system_prompt):\n",
    "    \"\"\"Unit testing function that takes in a dataframe and appends test results to an output_list.\n",
    "    The system prompt is configurable to allow us to test a couple with this framework.\"\"\"\n",
    "\n",
    "    for x,y in input_df.iterrows():\n",
    "        model_response = get_response(system_prompt,y['question'])\n",
    "\n",
    "        format_valid = test_valid_schema(model_response)\n",
    "\n",
    "        try:\n",
    "            test_query = LLMResponse.model_validate_json(model_response)\n",
    "            sql_valid = test_llm_sql(test_query)\n",
    "\n",
    "        except:\n",
    "            sql_valid = False\n",
    "\n",
    "        output_list.append((y['question'],model_response,format_valid,sql_valid))\n",
    "        \n",
    "def evaluate_row(row):\n",
    "    \"\"\"Simple evaluation function to categorize unit testing results. \n",
    "    If the format or SQL are flagged it returns a label, otherwise it is correct\"\"\"\n",
    "    if row['format'] == False:\n",
    "        return 'Format incorrect'\n",
    "    \n",
    "    elif row['sql'] == False:\n",
    "        return 'SQL incorrect'\n",
    "    \n",
    "    else:\n",
    "        return 'SQL correct'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "898e5069",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Select 100 unseen queries to test this one\n",
    "test_df = sql_df.tail(50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2baec278",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing create query: CREATE TABLE partnerships (id INT, player1 VARCHAR(255), player2 VARCHAR(255), venue VARCHAR(255))\n",
      "Testing select query: SELECT venue FROM partnerships WHERE player1 = 'shoaib malik' AND player2 = 'misbah-ul-haq'\n",
      "[]\n",
      "Testing create query: CREATE TABLE partnerships (id INT, player1 VARCHAR(255), player2 VARCHAR(255), venue VARCHAR(255))\n",
      "Testing select query: SELECT venue FROM partnerships WHERE player1 = 'herschelle gibbs' AND player2 = 'justin kemp'\n",
      "[]\n",
      "Testing create query: CREATE TABLE Played (Number INT, Points INT)\n",
      "Testing select query: SELECT * FROM Played WHERE Points = 310\n",
      "[]\n",
      "Testing create query: CREATE TABLE losing_bonus (bonus_id INT, points_against INT)\n",
      "Testing select query: SELECT * FROM losing_bonus WHERE points_against = 588\n",
      "[]\n",
      "Testing create query: CREATE TABLE rugby_teams (team_name VARCHAR(255), tries_against INT, losing_bonus INT);\n",
      "Testing select query: SELECT * FROM rugby_teams WHERE tries_against = 7;\n",
      "[]\n",
      "Testing create query: CREATE TABLE try_bonus (id INT, points_against INT, try_bonus VARCHAR(50))\n",
      "Testing select query: SELECT try_bonus FROM try_bonus WHERE points_against = 488\n",
      "[]\n",
      "Testing create query: CREATE TABLE Points (Team VARCHAR(255), Try_Bonus INT)\n",
      "Testing select query: SELECT Team FROM Points WHERE Try_Bonus = 140\n",
      "[]\n",
      "Testing create query: CREATE TABLE Drawn (Team TEXT, Tries_against INT);\n",
      "Testing select query: SELECT * FROM Drawn WHERE Tries_against = 0;\n",
      "[]\n",
      "Testing create query: CREATE TABLE champion (id INT, name VARCHAR(255), reign_days INT, defenses INT)\n",
      "Testing select query: SELECT days_held FROM champion WHERE reign_days > 3 AND defenses = 1\n",
      "Error while executing select query: no such column: days_held\n",
      "Testing create query: CREATE TABLE champion (id INT, name VARCHAR(255), reign_days INT, defenses INT);\n",
      "Testing select query: SELECT days_held FROM champion WHERE reign_days > 3 AND defenses < 1;\n",
      "Error while executing select query: no such column: days_held\n",
      "Testing create query: CREATE TABLE champion_stats (champion_id INT, days_held INT, reign INT, defenses INT)\n",
      "Testing select query: SELECT AVG(defenses) FROM champion_stats WHERE days_held = 404 AND reign > 1\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE champions (name VARCHAR(255), days_held INT, defense INT)\n",
      "Testing select query: SELECT MIN(defense) FROM champions WHERE days_held = 345\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE records (date DATE, record VARCHAR(10))\n",
      "Testing select query: SELECT date FROM records WHERE record = '76-72'\n",
      "[]\n",
      "Testing create query: CREATE TABLE attendance (game_id INT, loss VARCHAR(20), attendance INT);\n",
      "Testing select query: SELECT attendance FROM attendance WHERE loss = 'Ponson (1-5)';\n",
      "[]\n",
      "Testing create query: CREATE TABLE records (day VARCHAR(10), record INT);\n",
      "Testing select query: SELECT day FROM records WHERE record >= 36 AND record <= 39;\n",
      "[]\n",
      "Testing create query: CREATE TABLE records (id INT, date DATE);\n",
      "Testing select query: SELECT date FROM records WHERE id = '30-31';\n",
      "[]\n",
      "Testing create query: CREATE TABLE games (game_id INT, opponent VARCHAR(255), result VARCHAR(255))\n",
      "Testing select query: SELECT opponent FROM games WHERE result = 'loss' AND game_id = (SELECT game_id FROM games WHERE opponent = 'Leonard' AND result = '7–8')\n",
      "[]\n",
      "Testing create query: CREATE TABLE game_scores (record VARCHAR(10), score INT);\n",
      "Testing select query: SELECT score FROM game_scores WHERE record = '18–43';\n",
      "[]\n",
      "Testing create query: CREATE TABLE game_scores (game_id INT, opponent VARCHAR(255), score INT)\n",
      "Testing select query: SELECT score FROM game_scores WHERE opponent = 'Royals' AND record = '24–52'\n",
      "Error while executing select query: no such column: record\n",
      "Testing create query: CREATE TABLE game_scores (record VARCHAR(10), score INT);\n",
      "Testing select query: SELECT score FROM game_scores WHERE record = '22–46';\n",
      "[]\n",
      "Testing create query: CREATE TABLE military_specialties (id INT, specialty_name VARCHAR(255), real_name VARCHAR(255))\n",
      "Testing select query: SELECT real_name FROM military_specialties WHERE specialty_name = 'shock paratrooper'\n",
      "[]\n",
      "Testing create query: CREATE TABLE IF NOT EXISTS people (id INT PRIMARY KEY, name VARCHAR(255), birthplace VARCHAR(255))\n",
      "Testing select query: SELECT birthplace FROM people WHERE name = 'Pete Sanderson'\n",
      "[]\n",
      "Testing create query: CREATE TABLE roles (id INT, name VARCHAR(255), role VARCHAR(255))\n",
      "Testing select query: SELECT role FROM roles WHERE name = 'Jean-Luc Bouvier'\n",
      "[]\n",
      "Testing create query: CREATE TABLE pilots (id INT, name VARCHAR(255), kayak VARCHAR(255))\n",
      "Testing select query: SELECT name FROM pilots WHERE kayak = 'silent attack kayak'\n",
      "[]\n",
      "Testing create query: CREATE TABLE persons (id INT, name VARCHAR(255), birthplace VARCHAR(255), code_name VARCHAR(255))\n",
      "Testing select query: SELECT code_name FROM persons WHERE birthplace = 'Liverpool'\n",
      "[]\n",
      "Testing create query: CREATE TABLE medalists (name VARCHAR(255), sport VARCHAR(255))\n",
      "Testing select query: SELECT name FROM medalists WHERE sport = 'canoeing'\n",
      "[]\n",
      "Testing create query: CREATE TABLE IF NOT EXISTS events (event_id INT, event_name VARCHAR(255), event_category VARCHAR(255), event_gender VARCHAR(255), event_weight VARCHAR(255))\n",
      "Testing select query: SELECT event_name FROM events WHERE event_category = 'Women' AND event_weight = 'Half Middleweight'\n",
      "[]\n",
      "Testing create query: CREATE TABLE medals (id INT, athlete_name VARCHAR(255), medal VARCHAR(255), year INT, city VARCHAR(255))\n",
      "Testing select query: SELECT athlete_name FROM medals WHERE medal = 'bronze' AND year = 2000 AND city = 'Sydney'\n",
      "[]\n",
      "Testing create query: CREATE TABLE attendance (id INT, opponent VARCHAR(255), attendees INT);\n",
      "Testing select query: SELECT COUNT(*) FROM attendance WHERE opponent = 'twins';\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE records (id INT, date DATE, record INT);\n",
      "Testing select query: SELECT date FROM records WHERE record BETWEEN 41 AND 46;\n",
      "[]\n",
      "Testing create query: CREATE TABLE scores (id INT, record VARCHAR(10), score INT);\n",
      "Testing select query: SELECT score FROM scores WHERE record = '48-55';\n",
      "[]\n",
      "Testing create query: CREATE TABLE scores (id INT, record VARCHAR(10), score INT);\n",
      "Testing select query: SELECT score FROM scores WHERE record = '44-49';\n",
      "[]\n",
      "Testing create query: CREATE TABLE scores (Opponent TEXT, Record TEXT, Score INTEGER);\n",
      "Testing select query: SELECT Score FROM scores WHERE Opponent = 'white sox' AND Record = '2-0';\n",
      "[]\n",
      "Testing create query: CREATE TABLE votes (candidate_name VARCHAR(255), vote_count INT);\n",
      "Testing select query: SELECT vote_count FROM votes WHERE candidate_name = 'candice sjostrom';\n",
      "[]\n",
      "Testing create query: CREATE TABLE percentages (name VARCHAR(255), percentage FLOAT);\n",
      "Testing select query: SELECT percentage FROM percentages WHERE name = 'chris wright';\n",
      "[]\n",
      "Testing create query: CREATE TABLE votes (year INT, candidate VARCHAR(255), vote_percentage FLOAT, office VARCHAR(255))\n",
      "Testing select query: SELECT COUNT(*) FROM votes WHERE year > 1992 AND vote_percentage = 1.59 AND office = 'us representative 4'\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE representatives (id INT, name VARCHAR(255), year_start INT, year_end INT)\n",
      "Testing select query: SELECT year_start, year_end FROM representatives WHERE name = 'J. Smith Young'\n",
      "[]\n",
      "Testing create query: CREATE TABLE politicians (id INT, name VARCHAR(255), party VARCHAR(255))\n",
      "Testing select query: SELECT party FROM politicians WHERE name = 'Thomas L. Young'\n",
      "[]\n",
      "Testing create query: CREATE TABLE medals (country TEXT, gold INTEGER, silver INTEGER, bronze INTEGER);\n",
      "Testing select query: SELECT MIN(gold + silver + bronze) AS lowest_total_medals FROM medals WHERE gold = 0 AND silver > 1 AND bronze > 2;\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE medals (country TEXT, rank INTEGER, gold INTEGER, silver INTEGER, bronze INTEGER, total INTEGER);\n",
      "Testing select query: SELECT SUM(silver) AS total_silver FROM medals WHERE rank = 14 AND total < 1;\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE player_stats (player_id INT, tackles INT, fumble_recoveries INT, forced_fumbles INT);\n",
      "Testing select query: SELECT COUNT(tackles) FROM player_stats WHERE fumble_recoveries > 0 AND forced_fumbles > 0;\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE forced_fumbles (player_name VARCHAR(255), solo_tackles INT, forced_fumbles INT);\n",
      "Testing select query: SELECT COUNT(forced_fumbles) FROM forced_fumbles WHERE player_name = 'jim laney' AND solo_tackles < 2;\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE player_stats (player_id INT, solo_tackles INT, total INT);\n",
      "Testing select query: SELECT MAX(total) AS high_total FROM player_stats WHERE solo_tackles > 15;\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE fumble_recoveries (player_name VARCHAR(255), forced_fumbles INT, sacks INT, solo_tackles INT, fumble_recoveries INT);\n",
      "Testing select query: SELECT COUNT(fumble_recoveries) FROM fumble_recoveries WHERE player_name = 'scott gajos' AND forced_fumbles = 0 AND sacks = 0 AND solo_tackles < 2;\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE matches (id INT, opponent VARCHAR(255), time VARCHAR(255), location VARCHAR(255))\n",
      "Testing select query: SELECT opponent FROM matches WHERE time = '20:00 GMT' AND location = 'Camp Nou'\n",
      "[]\n",
      "Testing create query: CREATE TABLE matches (id INT, match_time TIME, score VARCHAR(5))\n",
      "Testing select query: SELECT match_time FROM matches WHERE score = '3-2'\n",
      "[]\n",
      "Testing create query: CREATE TABLE grounds (id INT, name VARCHAR(255))\n",
      "Testing select query: SELECT name FROM grounds WHERE id IN (SELECT ground_id FROM team_grounds WHERE team_id = (SELECT id FROM teams WHERE name = 'Aston Villa'))\n",
      "Error while executing select query: no such table: team_grounds\n",
      "Testing create query: CREATE TABLE competition (id INT, name VARCHAR(255), type VARCHAR(255), time TIMESTAMP);\n",
      "Testing select query: SELECT type FROM competition WHERE time = '18:30 GMT' AND name = 'San Siro';\n",
      "[]\n",
      "Testing create query: CREATE TABLE decile (school_name VARCHAR(255), locality VARCHAR(255), decile INT)\n",
      "Testing select query: SELECT COUNT(decile) AS total_decile FROM decile WHERE locality = 'redwood school'\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE reports (id INT, report_name VARCHAR(255), circuit_name VARCHAR(255))\n",
      "Testing select query: SELECT report_name FROM reports WHERE circuit_name = 'Tripoli'\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "# Execute unit tests and capture results\n",
    "results = []\n",
    "\n",
    "execute_unit_tests(input_df=test_df,output_list=results,system_prompt=system_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a070a4bf-7435-4059-bf74-eb6129cbab2b",
   "metadata": {},
   "source": [
    "#### Run Evaluation\n",
    "\n",
    "Now that we have generated the SQL based on system prompt 1 (run 1), we can run evaluation against the results.  We use pandas `apply` functin to \"apply\" evaluation to each resulting generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "8fe18367",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df = pd.DataFrame(results)\n",
    "results_df.columns = ['question','response','format','sql']\n",
    "\n",
    "# Execute evaluation\n",
    "results_df['evaluation_score'] = results_df.apply(lambda x: get_geval_score(RELEVANCY_SCORE_CRITERIA,RELEVANCY_SCORE_STEPS,x['question'],x['response'],'relevancy'),axis=1)\n",
    "results_df['unit_test_evaluation'] = results_df.apply(lambda x: evaluate_row(x),axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3dd9b04-44e2-476c-86fd-c0a261b1cbdd",
   "metadata": {},
   "source": [
    "## Viewing unit test results and evaluations - Run 1\n",
    "\n",
    "We can now group the outcomes of the unit test (which test the structure of response) and evaluation (which checks if the SQL is syntatically correct)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "650e6159",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unit_test_evaluation\n",
       "correct          46\n",
       "SQL incorrect     4\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['unit_test_evaluation'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b3f98f81",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "evaluation_score\n",
       "5    46\n",
       "4     3\n",
       "3     1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_df['evaluation_score'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "019f3a1d",
   "metadata": {},
   "source": [
    "### Second run\n",
    "\n",
    "We now use a new system prompt to run same unit test and evaluation.  Please note that we are using the same functions for unit testing and evaluations; the only change is the system prompt (which is under the test)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "513a2da1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Translate this natural language request into a JSON object containing two SQL queries. \\n'\n",
      " \"The first query should be a CREATE statement for a table answering the user's request, while the second should be a \"\n",
      " 'SELECT query answering their question. \\n'\n",
      " 'This should be returned as a JSON with keys \"create\" and \"select\". \\n'\n",
      " 'For example:\\n'\n",
      " '\\n'\n",
      " '{\"create\": \"CREATE_QUERY\",\"select\": \"SELECT_QUERY\"}\" \\n'\n",
      " 'Ensure the SQL is always generated on one line, never use \\n'\n",
      " ' to separate rows.')\n"
     ]
    }
   ],
   "source": [
    "system_prompt_2 = \"\"\"Translate this natural language request into a JSON object containing two SQL queries. \n",
    "The first query should be a CREATE statement for a table answering the user's request, while the second should be a SELECT query answering their question. \n",
    "This should be returned as a JSON with keys \"create\" and \"select\". \n",
    "For example:\\n\\n{\"create\": \"CREATE_QUERY\",\"select\": \"SELECT_QUERY\"}\" \n",
    "Ensure the SQL is always generated on one line, never use \\n to separate rows.\"\"\"\n",
    "\n",
    "pprint(system_prompt_2, width=120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "70bd3e32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Testing create query: CREATE TABLE partnerships (id INT, player1 VARCHAR(255), player2 VARCHAR(255), venue VARCHAR(255))\n",
      "Testing select query: SELECT venue FROM partnerships WHERE player1 = 'shoaib malik' AND player2 = 'misbah-ul-haq'\n",
      "[]\n",
      "Testing create query: CREATE TABLE partnerships (partnership_id INT, player1 VARCHAR(255), player2 VARCHAR(255), venue VARCHAR(255));\n",
      "Testing select query: SELECT venue FROM partnerships WHERE player1 = 'herschelle gibbs' AND player2 = 'justin kemp';\n",
      "[]\n",
      "Testing create query: CREATE TABLE scores (NumberPlayed INT, Points INT);\n",
      "Testing select query: SELECT NumberPlayed FROM scores WHERE Points = 310;\n",
      "[]\n",
      "Testing create query: CREATE TABLE losing_bonus (bonus_id INT, points_against INT)\n",
      "Testing select query: SELECT * FROM losing_bonus WHERE points_against = 588\n",
      "[]\n",
      "Testing create query: CREATE TABLE rugby_teams (team_name VARCHAR(255), tries_against INT, losing_bonus INT);\n",
      "Testing select query: SELECT * FROM rugby_teams WHERE tries_against = 7;\n",
      "[]\n",
      "Testing create query: CREATE TABLE try_bonus (try_bonus_id INT, points_against INT)\n",
      "Testing select query: SELECT try_bonus FROM try_bonus WHERE points_against = 488\n",
      "Error while executing select query: no such column: try_bonus\n",
      "Testing create query: CREATE TABLE Points (Team VARCHAR(255), Try_Bonus INT)\n",
      "Testing select query: SELECT Team FROM Points WHERE Try_Bonus = 140\n",
      "[]\n",
      "Testing create query: CREATE TABLE Drawn (Team TEXT, Tries_against INT);\n",
      "Testing select query: SELECT * FROM Drawn WHERE Tries_against = 0;\n",
      "[]\n",
      "Testing create query: CREATE TABLE IF NOT EXISTS champions (id INT, name VARCHAR(255), reign_days INT, defenses INT);\n",
      "Testing select query: SELECT days_held FROM champions WHERE reign_days > 3 AND defenses = 1;\n",
      "Error while executing select query: no such column: days_held\n",
      "Testing create query: CREATE TABLE champion (name VARCHAR(255), reign_days INT, defenses INT);\n",
      "Testing select query: SELECT days_held FROM champion WHERE reign_days > 3 AND defenses < 1;\n",
      "Error while executing select query: no such column: days_held\n",
      "Testing create query: CREATE TABLE champion_stats (champion_name VARCHAR(255), days_held INT, reign INT, defenses INT);\n",
      "Testing select query: SELECT AVG(defenses) FROM champion_stats WHERE days_held = 404 AND reign > 1;\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE champions (name VARCHAR(255), days_held INT, defense INT);\n",
      "Testing select query: SELECT MIN(defense) FROM champions WHERE days_held = 345;\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE records (date DATE, record VARCHAR(10));\n",
      "Testing select query: SELECT date FROM records WHERE record = '76-72';\n",
      "[]\n",
      "Testing create query: CREATE TABLE attendance (game_id INT, loss VARCHAR(10), attendance INT);\n",
      "Testing select query: SELECT attendance FROM attendance WHERE loss = 'Ponson (1-5)';\n",
      "[]\n",
      "Testing create query: CREATE TABLE records (day VARCHAR(10), record INT);\n",
      "Testing select query: SELECT day FROM records WHERE record >= 36 AND record <= 39;\n",
      "[]\n",
      "Testing create query: CREATE TABLE records (id INT, date DATE);\n",
      "Testing select query: SELECT date FROM records WHERE record = '30-31';\n",
      "Error while executing select query: no such column: record\n",
      "Testing create query: CREATE TABLE games (game_id INT, opponent VARCHAR(255), result VARCHAR(255));\n",
      "Testing select query: SELECT opponent FROM games WHERE result = 'loss' AND game_id = (SELECT game_id FROM games WHERE opponent = 'Leonard' AND result = '7–8');\n",
      "[]\n",
      "Testing create query: CREATE TABLE game_scores (game_id INT, home_team_score INT, away_team_score INT);\n",
      "Testing select query: SELECT home_team_score, away_team_score FROM game_scores WHERE home_team_score = 18 AND away_team_score = 43;\n",
      "[]\n",
      "Testing create query: CREATE TABLE game_scores (game_id INT, opponent VARCHAR(50), score INT)\n",
      "Testing select query: SELECT score FROM game_scores WHERE opponent = 'Royals' AND record = '24–52'\n",
      "Error while executing select query: no such column: record\n",
      "Testing create query: CREATE TABLE game_scores (id INT, home_team_score INT, away_team_score INT);\n",
      "Testing select query: SELECT * FROM game_scores WHERE home_team_score = 22 AND away_team_score = 46;\n",
      "[]\n",
      "Testing create query: CREATE TABLE military_specialties (id INT, specialty_name VARCHAR(255), real_name VARCHAR(255));\n",
      "Testing select query: SELECT real_name FROM military_specialties WHERE specialty_name = 'shock paratrooper';\n",
      "[]\n",
      "Testing create query: CREATE TABLE birthplaces (name VARCHAR(255), birthplace VARCHAR(255));\n",
      "Testing select query: SELECT birthplace FROM birthplaces WHERE name = 'Pete Sanderson';\n",
      "[]\n",
      "Testing create query: CREATE TABLE roles (id INT, name VARCHAR(255), role VARCHAR(255))\n",
      "Testing select query: SELECT role FROM roles WHERE name = 'Jean-Luc Bouvier'\n",
      "[]\n",
      "Testing create query: CREATE TABLE pilots (id INT, name VARCHAR(255), kayak_type VARCHAR(255))\n",
      "Testing select query: SELECT name FROM pilots WHERE kayak_type = 'silent attack kayak'\n",
      "[]\n",
      "Testing create query: CREATE TABLE persons (id INT, name VARCHAR(255), birthplace VARCHAR(255), code_name VARCHAR(255));\n",
      "Testing select query: SELECT code_name FROM persons WHERE birthplace = 'Liverpool';\n",
      "[]\n",
      "Testing create query: CREATE TABLE medalists (name VARCHAR(255), sport VARCHAR(255))\n",
      "Testing select query: SELECT name FROM medalists WHERE sport = 'canoeing'\n",
      "[]\n",
      "Testing create query: CREATE TABLE IF NOT EXISTS events (event_id INT, event_name VARCHAR(255), event_category VARCHAR(255))\n",
      "Testing select query: SELECT event_name FROM events WHERE event_category = 'women's half middleweight'\n",
      "Error while executing select query: near \"s\": syntax error\n",
      "Testing create query: CREATE TABLE medals (id INT, athlete VARCHAR(255), event VARCHAR(255), medal VARCHAR(255), year INT);\n",
      "Testing select query: SELECT athlete FROM medals WHERE medal = 'bronze' AND year = 2000 AND event = 'Sydney games';\n",
      "[]\n",
      "Testing create query: CREATE TABLE attendance (opponent VARCHAR(50), people_attended INT);\n",
      "Testing select query: SELECT people_attended FROM attendance WHERE opponent = 'twins';\n",
      "[]\n",
      "Testing create query: CREATE TABLE records (date DATE, record INT);\n",
      "Testing select query: SELECT date FROM records WHERE record BETWEEN 41 AND 46;\n",
      "[]\n",
      "Testing create query: CREATE TABLE scores (id INT, record VARCHAR(10), score INT);\n",
      "Testing select query: SELECT score FROM scores WHERE record = '48-55';\n",
      "[]\n",
      "Testing create query: CREATE TABLE scores (id INT, record VARCHAR(10), score INT);\n",
      "Testing select query: SELECT score FROM scores WHERE record = '44-49';\n",
      "[]\n",
      "Testing create query: CREATE TABLE scores (Opponent VARCHAR(50), Record VARCHAR(10), Score INT);\n",
      "Testing select query: SELECT Score FROM scores WHERE Opponent = 'white sox' AND Record = '2-0';\n",
      "[]\n",
      "Testing create query: CREATE TABLE votes (candidate_name VARCHAR(255), vote_count INT);\n",
      "Testing select query: SELECT vote_count FROM votes WHERE candidate_name = 'candice sjostrom';\n",
      "[]\n",
      "Testing create query: CREATE TABLE percentages (name VARCHAR(255), percentage FLOAT);\n",
      "Testing select query: SELECT percentage FROM percentages WHERE name = 'chris wright';\n",
      "[]\n",
      "Testing create query: CREATE TABLE votes (year INT, candidate VARCHAR(255), vote_percentage FLOAT, office VARCHAR(255));\n",
      "Testing select query: SELECT COUNT(*) FROM votes WHERE year > 1992 AND vote_percentage = 1.59 AND office = 'us representative 4';\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE representatives (id INT, name VARCHAR(255), start_year INT, end_year INT)\n",
      "Testing select query: SELECT start_year, end_year FROM representatives WHERE name = 'J. Smith Young'\n",
      "[]\n",
      "Testing create query: CREATE TABLE politicians (id INT, name VARCHAR(255), party VARCHAR(255))\n",
      "Testing select query: SELECT party FROM politicians WHERE name = 'Thomas L. Young'\n",
      "[]\n",
      "Testing create query: CREATE TABLE medal_counts (country TEXT, gold INTEGER, silver INTEGER, bronze INTEGER);\n",
      "Testing select query: SELECT MIN(gold + silver + bronze) AS lowest_total_medals FROM medal_counts WHERE gold = 0 AND silver > 1 AND bronze > 2;\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE medals (country TEXT, rank INTEGER, gold INTEGER, silver INTEGER, bronze INTEGER, total INTEGER);\n",
      "Testing select query: SELECT SUM(silver) AS total_silver FROM medals WHERE rank = 14 AND total < 1;\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE player_stats (player_id INT, tackles INT, fumble_recoveries INT, forced_fumbles INT);\n",
      "Testing select query: SELECT COUNT(tackles) FROM player_stats WHERE fumble_recoveries > 0 AND forced_fumbles = 0;\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE forced_fumbles (player_name VARCHAR(255), solo_tackles INT, forced_fumbles INT);\n",
      "Testing select query: SELECT COUNT(forced_fumbles) FROM forced_fumbles WHERE player_name = 'Jim Laney' AND solo_tackles < 2;\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE player_stats (player_id INT, solo_tackles INT, total INT);\n",
      "Testing select query: SELECT MAX(total) AS high_total FROM player_stats WHERE solo_tackles > 15;\n",
      "[(None,)]\n",
      "Testing create query: CREATE TABLE fumble_recoveries (player_name VARCHAR(255), forced_fumbles INT, sacks INT, solo_tackles INT, fumble_recoveries INT);\n",
      "Testing select query: SELECT fumble_recoveries FROM fumble_recoveries WHERE player_name = 'scott gajos' AND forced_fumbles = 0 AND sacks = 0 AND solo_tackles < 2;\n",
      "[]\n",
      "Testing create query: CREATE TABLE matches (id INT, opponent VARCHAR(255), time VARCHAR(255), location VARCHAR(255))\n",
      "Testing select query: SELECT opponent FROM matches WHERE time = '20:00 GMT' AND location = 'Camp Nou'\n",
      "[]\n",
      "Testing create query: CREATE TABLE matches (id INT, match_time TIME, score VARCHAR(5))\n",
      "Testing select query: SELECT match_time FROM matches WHERE score = '3-2'\n",
      "[]\n",
      "Testing create query: CREATE TABLE grounds (id INT, name VARCHAR(255));\n",
      "Testing select query: SELECT name FROM grounds WHERE id IN (SELECT ground_id FROM team_grounds WHERE team_id = (SELECT id FROM teams WHERE name = 'Aston Villa'));\n",
      "Error while executing select query: no such table: team_grounds\n",
      "Testing create query: CREATE TABLE competition (id INT, type VARCHAR(255), location VARCHAR(255), time TIME)\n",
      "Testing select query: SELECT type FROM competition WHERE location = 'San Siro' AND time = '18:30:00'\n",
      "[]\n",
      "Testing create query: CREATE TABLE decile (school_name VARCHAR(255), locality VARCHAR(255), decile INT)\n",
      "Testing select query: SELECT COUNT(DISTINCT decile) AS total_decile FROM decile WHERE locality = 'redwood school'\n",
      "[(0,)]\n",
      "Testing create query: CREATE TABLE reports (id INT, name VARCHAR(255), circuit VARCHAR(255))\n",
      "Testing select query: SELECT * FROM reports WHERE circuit = 'Tripoli'\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "# Execute unit tests\n",
    "results_2 = []\n",
    "\n",
    "execute_unit_tests(input_df=test_df,output_list=results_2,system_prompt=system_prompt_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "04532d59",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_2_df = pd.DataFrame(results_2)\n",
    "results_2_df.columns = ['question','response','format','sql']\n",
    "\n",
    "# Execute evaluation\n",
    "results_2_df['evaluation_score'] = results_2_df.apply(lambda x: get_geval_score(RELEVANCY_SCORE_CRITERIA,RELEVANCY_SCORE_STEPS,x['question'],x['response'],'relevancy'),axis=1)\n",
    "results_2_df['unit_test_evaluation'] = results_2_df.apply(lambda x: evaluate_row(x),axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd95c3f9-f90d-451d-a32b-aeb066906779",
   "metadata": {},
   "source": [
    "## Viewing unit test results and evaluations - Run 2\n",
    "\n",
    "We can now group the outcomes of the unit test (which test the structure of response) and evaluation (which checks if the SQL is syntatically correct)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "cbaa4bdf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "unit_test_evaluation\n",
       "correct          43\n",
       "SQL incorrect     7\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_2_df['unit_test_evaluation'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "1ada474e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "evaluation_score\n",
       "5    46\n",
       "4     3\n",
       "3     1\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_2_df['evaluation_score'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1908c933",
   "metadata": {},
   "source": [
    "## Report\n",
    "\n",
    "We'll make a simple dataframe to store and display the run performance - this is where you can use tools like Weights & Biases Prompts or Gantry to store the results for analytics on your different iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "d277222d",
   "metadata": {},
   "outputs": [],
   "source": [
    "results_df['run'] = 1\n",
    "results_df['Evaluating Model'] = 'gpt-4'\n",
    "\n",
    "results_2_df['run'] = 2\n",
    "results_2_df['Evaluating Model'] = 'gpt-4'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6da35c99",
   "metadata": {},
   "outputs": [],
   "source": [
    "run_df = pd.concat([results_df,results_2_df])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "4116cb37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>response</th>\n",
       "      <th>format</th>\n",
       "      <th>sql</th>\n",
       "      <th>evaluation_score</th>\n",
       "      <th>unit_test_evaluation</th>\n",
       "      <th>run</th>\n",
       "      <th>Evaluating Model</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>What venue did the parntership of shoaib malik...</td>\n",
       "      <td>{\"create\": \"CREATE TABLE partnerships (id INT,...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What venue did the partnership of herschelle g...</td>\n",
       "      <td>{\"create\": \"CREATE TABLE partnerships (id INT,...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>What is the number Played that has 310 Points ...</td>\n",
       "      <td>{\"create\": \"CREATE TABLE Played (Number INT, P...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What Losing bonus has a Points against of 588?</td>\n",
       "      <td>{\"create\": \"CREATE TABLE losing_bonus (bonus_i...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>5</td>\n",
       "      <td>correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What Tries against has a Losing bonus of 7?</td>\n",
       "      <td>{\"create\": \"CREATE TABLE rugby_teams (team_nam...</td>\n",
       "      <td>True</td>\n",
       "      <td>True</td>\n",
       "      <td>3</td>\n",
       "      <td>correct</td>\n",
       "      <td>1</td>\n",
       "      <td>gpt-4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  What venue did the parntership of shoaib malik...   \n",
       "1  What venue did the partnership of herschelle g...   \n",
       "2  What is the number Played that has 310 Points ...   \n",
       "3     What Losing bonus has a Points against of 588?   \n",
       "4        What Tries against has a Losing bonus of 7?   \n",
       "\n",
       "                                            response  format   sql  \\\n",
       "0  {\"create\": \"CREATE TABLE partnerships (id INT,...    True  True   \n",
       "1  {\"create\": \"CREATE TABLE partnerships (id INT,...    True  True   \n",
       "2  {\"create\": \"CREATE TABLE Played (Number INT, P...    True  True   \n",
       "3  {\"create\": \"CREATE TABLE losing_bonus (bonus_i...    True  True   \n",
       "4  {\"create\": \"CREATE TABLE rugby_teams (team_nam...    True  True   \n",
       "\n",
       "  evaluation_score unit_test_evaluation  run Evaluating Model  \n",
       "0                5              correct    1            gpt-4  \n",
       "1                5              correct    1            gpt-4  \n",
       "2                5              correct    1            gpt-4  \n",
       "3                5              correct    1            gpt-4  \n",
       "4                3              correct    1            gpt-4  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "run_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "ed800f0c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Number of records</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run</th>\n",
       "      <th>unit_test_evaluation</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">1</th>\n",
       "      <th>SQL incorrect</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>correct</th>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"2\" valign=\"top\">2</th>\n",
       "      <th>SQL incorrect</th>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>correct</th>\n",
       "      <td>43</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          Number of records\n",
       "run unit_test_evaluation                   \n",
       "1   SQL incorrect                         4\n",
       "    correct                              46\n",
       "2   SQL incorrect                         7\n",
       "    correct                              43"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Unit test results\n",
    "unittest_df_pivot = pd.pivot_table(run_df,  values='format',index=['run','unit_test_evaluation'], #columns='position',\n",
    "                          aggfunc='count')\n",
    "unittest_df_pivot.columns = ['Number of records']\n",
    "unittest_df_pivot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0162a009-fc43-484c-90f6-d59a8e52f365",
   "metadata": {},
   "source": [
    "#### Plotting the results\n",
    "\n",
    "We can create a simple bar chart to visualise the results of unit tests for both runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e2b4aa03-42f5-4c30-a610-e553937bf160",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAAIjCAYAAADWYVDIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABY/ElEQVR4nO3dZ3RUVf/28WsS0khIQgkJSEmkhc4NihQFhEhTpEm7UQggKL2IBW+l2ChKsYGIUsRGFcVCEem9g0gnIEIgSEkMgQTIfl74ZP5nSIAMJJkEvp+1Zq3MPmX/5kzLNeecfWzGGCMAAAAAgCTJzdUFAAAAAEB2QkgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAKyscjISIWGhrq6jLtKvXr1VK9ePZf1b7PZNHz4cJf1j7TVq1dPFSpUcHUZ6TZz5kyFh4fLw8NDgYGBri4nS2Xk52J8fLyeffZZhYSEyGazacCAARmy3uxk+vTpstls2rJli6tLAXIUQhJwh4YPHy6bzaa///47zekVKlTIsH/KExISNHz4cK1YseKW84aGhspms93yNn369Ayp7Z133tGCBQvSNe/Ro0dvWtOoUaMypCZX+fnnnwlCaUh5Tfbt2zfVtBUrVshms2nu3LkuqCxn2bdvnyIjI1WiRAlNmTJFn3766Q3nTfl8Srl5eHgoNDRU/fr104ULF7Ku6GzqnXfe0fTp09WzZ0/NnDlTzzzzTKb2d7PP5caNG2dq37eL1xDuVblcXQCAG5syZYqSk5Pt9xMSEjRixAhJumXwmjBhguLj4+33f/75Z33zzTcaP368ChQoYG+vVatWhtT6zjvv6KmnnlKLFi3SvUyHDh3UtGnTVO3/+c9/MqQmV/n555/18ccfpxmULl26pFy57u2P3ilTpmjIkCEqXLiwq0vJkVasWKHk5GS9//77KlmyZLqWmTRpkvz8/HTx4kUtW7ZMH374obZt26Y1a9ZkcrXZ22+//aYaNWpo2LBhWdZnlSpV9MILL6Rqz+7vB15DuNfc29/UQDbn4eFx28teH1ZOnTqlb775Ri1atMg2h/BVrVpVTz/9tKvLyFLe3t6uLsGlypcvr/3792vUqFH64IMPXF1OlkpOTlZSUtIdvwZiYmIkyanD7J566in7jyPPPfec2rdvr1mzZmnTpk2qXr36HdWT0S5fvixPT0+5uWX+wS4xMTEqV65chq3v6tWrSk5Olqen5w3nue+++3Lk515Oeg0BGYHD7YAslnJY0ezZs/X222+rSJEi8vb2VoMGDXTo0CGHea3H3h89elRBQUGSpBEjRtgPfbjTw7q+/PJLVatWTT4+PsqXL5/at2+v48ePO8xz8OBBtW7dWiEhIfL29laRIkXUvn17xcbGSvr3PJuLFy9qxowZ9roiIyPvqC5JeuKJJ3T//fenOa1mzZp64IEH7PenTZum+vXrq2DBgvLy8lK5cuU0adKkW/aRcrz+0aNHHdpTnifroY2rV69WmzZtVKxYMXl5ealo0aIaOHCgLl26ZJ8nMjJSH3/8sSQ5HKKSIq3nbPv27WrSpIn8/f3l5+enBg0aaMOGDWnWuXbtWg0aNEhBQUHy9fVVy5YtdebMGYd5t2zZokaNGqlAgQLy8fFRWFiYunbtetPt4My2Xrp0qR5++GEFBgbKz89PZcqU0auvvnrT9acIDQ1Vp06dNGXKFJ08efKm897o3JOUw3+sbDab+vTpozlz5qhcuXLy8fFRzZo1tXv3bknS5MmTVbJkSXl7e6tevXqpnu8UW7duVa1atezb7ZNPPkk1T2JiooYNG6aSJUvaXwcvvfSSEhMT06zpq6++Uvny5eXl5aVFixbd9DFPnDjRPm/hwoXVu3dvh0OaQkND7Xs9goKCbvsz4JFHHpEkHT582KF948aNaty4sQICApQ7d27VrVtXa9euTbX8iRMn1K1bNxUuXFheXl4KCwtTz549lZSUZJ/nyJEjatOmjfLly6fcuXOrRo0a+umnnxzWk/I++/bbb/Xaa6/pvvvuU+7cuRUXFydJWrBggSpUqCBvb29VqFBB3333XZqP59tvv1W1atWUJ08e+fv7q2LFinr//fdv+PhT+o2KitJPP/1kf5+mvC5iYmLUrVs3BQcHy9vbW5UrV9aMGTMc1pFy2PB7772nCRMmqESJEvLy8tIff/xxw37Ta9euXYqMjNT9998vb29vhYSEqGvXrjp79myqedPzXEj/vm5v9dnhjLReQ6GhoWl+9l9/Lqgz34OAq7AnCXCRUaNGyc3NTYMHD1ZsbKzGjBmjjh07auPGjWnOHxQUpEmTJqlnz55q2bKlWrVqJUmqVKnSbdfw9ttv6/XXX1fbtm317LPP6syZM/rwww9Vp04dbd++XYGBgUpKSlKjRo2UmJiovn37KiQkRCdOnNCPP/6oCxcuKCAgQDNnztSzzz6r6tWrq0ePHpKkEiVK3LL/hISENM/lCgwMVK5cudSuXTt16tRJmzdv1oMPPmiffuzYMW3YsEHvvvuuvW3SpEkqX768nnzySeXKlUsLFy5Ur169lJycrN69e9/2NrKaM2eOEhIS1LNnT+XPn1+bNm3Shx9+qL/++ktz5syR9O8vrCdPntTSpUs1c+bMW65zz549euSRR+Tv76+XXnpJHh4emjx5surVq6eVK1fqoYcecpi/b9++yps3r4YNG6ajR49qwoQJ6tOnj2bNmiXp33/uGjZsqKCgIL3yyisKDAzU0aNHNX/+/JvWkd5tvWfPHj3xxBOqVKmS3njjDXl5eenQoUNp/iN9I//73//0xRdfZPjepNWrV+uHH36wP98jR47UE088oZdeekkTJ05Ur169dP78eY0ZM0Zdu3bVb7/95rD8+fPn1bRpU7Vt21YdOnTQ7Nmz1bNnT3l6etpDZnJysp588kmtWbNGPXr0UNmyZbV7926NHz9eBw4cSHVe3m+//abZs2erT58+KlCgwE334g4fPlwjRoxQRESEevbsqf3792vSpEnavHmz1q5dKw8PD02YMEFffPGFvvvuO/vhT7fzGZASBvLmzetQa5MmTVStWjUNGzZMbm5u9h8fVq9ebd9bcPLkSVWvXl0XLlxQjx49FB4erhMnTmju3LlKSEiQp6enTp8+rVq1aikhIUH9+vVT/vz5NWPGDD355JOaO3euWrZs6VDPm2++KU9PTw0ePFiJiYny9PTUkiVL1Lp1a5UrV04jR47U2bNn1aVLFxUpUsRh2aVLl6pDhw5q0KCBRo8eLUnau3ev1q5dq/79+6f5+MuWLauZM2dq4MCBKlKkiP3wt6CgIF26dEn16tXToUOH1KdPH4WFhWnOnDmKjIzUhQsXUq1z2rRpunz5snr06CEvLy/ly5fvptv+ypUraX7u+fr6ysfHx/6Yjhw5oi5duigkJER79uzRp59+qj179mjDhg32HwnS81ykuNVnh7PSeg05y9nvQSBLGQB3ZNiwYUaSOXPmTJrTy5cvb+rWrWu/v3z5ciPJlC1b1iQmJtrb33//fSPJ7N69297WuXNnU7x4cfv9M2fOGElm2LBhTtf57rvvGkkmKirKGGPM0aNHjbu7u3n77bcd5tu9e7fJlSuXvX379u1GkpkzZ85N1+/r62s6d+6crlqioqKMpBve1q9fb4wxJjY21nh5eZkXXnjBYfkxY8YYm81mjh07Zm9LSEhI1U+jRo3M/fff79BWt25dh+dj2rRpDtslRcrztHz58pv2MXLkyFS19O7d29zo4/X6569FixbG09PTHD582N528uRJkydPHlOnTp1UdUZERJjk5GR7+8CBA427u7u5cOGCMcaY7777zkgymzdvTrP/G0nvth4/fvxNX+83U7x4cfP4448bY4zp0qWL8fb2NidPnjTG/N/2tr7Orn/9p0h5z1lJMl5eXg7P4+TJk40kExISYuLi4uztQ4YMSfWc161b10gyY8eOtbclJiaaKlWqmIIFC5qkpCRjjDEzZ840bm5uZvXq1Q79f/LJJ0aSWbt2rUNNbm5uZs+ePbfcNjExMcbT09M0bNjQXLt2zd7+0UcfGUlm6tSpqR5/ep6DlHn3799vzpw5Y44ePWqmTp1qfHx8TFBQkLl48aIxxpjk5GRTqlQp06hRI4fXV0JCggkLCzOPPfaYva1Tp07Gzc0tzddYyrIDBgwwkhy20z///GPCwsJMaGio/TGmPO/3339/qvdXlSpVTKFCheyvbWOMWbJkiZHk8Lro37+/8ff3N1evXr3l9rie9TWZYsKECUaS+fLLL+1tSUlJpmbNmsbPz8/+Wkr5HPP39zcxMTHp7u9Gn3sjR460z5fWZ80333xjJJlVq1bZ29LzXKT3s+NG0vsaSnl8aX0PXP+568z3IOAqHG4HuEiXLl0cfuVLOXThyJEjWdL//PnzlZycrLZt2+rvv/+230JCQlSqVCktX75ckhQQECBJWrx4sRISEjK0hh49emjp0qWpbinnCPj7+6tJkyaaPXu2jDH25WbNmqUaNWqoWLFi9raUX2AlKTY2Vn///bfq1q2rI0eO2A8LvFPWPi5evKi///5btWrVkjFG27dvd3p9165d05IlS9SiRQuHQ90KFSqk//73v1qzZo39sKMUPXr0cDjU7JFHHtG1a9d07NgxSf93nsqPP/6oK1eupLuW9G7rlPV///33DoOKOOu1117T1atXM3QkwwYNGjjsqUnZC9e6dWvlyZMnVfv177VcuXLpueees9/39PTUc889p5iYGG3dulXSv3sTy5Ytq/DwcIf3Tf369SXJ/r5JUbdu3XSd8/Lrr78qKSlJAwYMcDgXp3v37vL39091mJqzypQpo6CgIIWGhqpr164qWbKkfvnlF+XOnVuStGPHDh08eFD//e9/dfbsWfvjunjxoho0aKBVq1YpOTlZycnJWrBggZo1a+ZwCGaKlNfmzz//rOrVq+vhhx+2T/Pz81OPHj109OjRVIekde7c2eH9FR0drR07dqhz5872zyBJeuyxx1Jtz8DAQF28eFFLly69o22U4ueff1ZISIg6dOhgb/Pw8FC/fv0UHx+vlStXOszfunVr+6HQ6fHQQw+l+bln7c+6LS5fvqy///5bNWrUkCRt27ZNktL9XKS41WfHrdzqNXQ7XP09CNwMIQnIAtd/WUly+Adf+r9DFs6fP58lNR08eFDGGJUqVUpBQUEOt71799pPDg8LC9OgQYP02WefqUCBAmrUqJE+/vjjDAkepUqVUkRERKqbv7+/fZ527drp+PHjWr9+vaR/j3/funWr2rVr57CutWvXKiIiQr6+vgoMDFRQUJD9PJmMCkl//vmnIiMjlS9fPvn5+SkoKEh169a97T7OnDmjhIQElSlTJtW0smXLKjk5OdX5Ybd63dStW1etW7fWiBEjVKBAATVv3lzTpk1Ldb5MWtKzrdu1a6fatWvr2WefVXBwsNq3b6/Zs2c7HZjuv/9+PfPMM/r0008VHR3t1LI3cv22SfnnumjRomm2X/9eK1y4sHx9fR3aSpcuLen/Di06ePCg9uzZk+o9kzJfyvsmRVhYWLpqT/lH9frXgqenp+6///50/yN7I/PmzdPSpUv19ddfq0aNGoqJiXH4R/zgwYOS/g0r1z+2zz77TImJiYqNjdWZM2cUFxd3y2tKHTt27Iava+vjTXH9dkqZXqpUqVTruH69vXr1UunSpdWkSRMVKVJEXbt2veW5X7eqvVSpUqkGjkhv7bdSoECBND/3ihcvbp/n3Llz6t+/v4KDg+Xj46OgoCB7PymfNel9LlLc6XfOrV5Dt8PV34PAzXBOEnCHUkaqsp68b5WQkJDmaFbu7u5pzm/9FT8zJScny2az6ZdffkmzFj8/P/vfY8eOVWRkpL7//nstWbJE/fr108iRI7Vhw4ZU5wdktGbNmil37tyaPXu2atWqpdmzZ8vNzU1t2rSxz3P48GE1aNBA4eHhGjdunIoWLSpPT0/9/PPPGj9+/E3/gU8rwEr/7uW5/v5jjz2mc+fO6eWXX1Z4eLh8fX114sQJRUZG3tFeFWfc6nWTcq2hDRs2aOHChVq8eLG6du2qsWPHasOGDQ7P6/XSs619fHy0atUqLV++XD/99JMWLVqkWbNmqX79+lqyZMkN60vL//73P82cOVOjR49Oc+j49D43KW7Ud0a+15KTk1WxYkWNGzcuzenXB7I7/Scyo9SpU8c+MlmzZs1UsWJFdezYUVu3bpWbm5v99fvuu++qSpUqaa7Dz89P586dy5T67mQ7FSxYUDt27NDixYv1yy+/6JdfftG0adPUqVOnVIMtZIbMeI7btm2rdevW6cUXX1SVKlXk5+en5ORkNW7c+LY/a+70fXCr15B08/dsWv27+nsQuBlCEnCHUn79279/f6p/kBISEnT8+HE1bNgwQ/q60RfQ7ShRooSMMQoLC7P/Cn4zFStWVMWKFfXaa69p3bp1ql27tj755BO99dZbGV6bla+vr5544gnNmTNH48aN06xZs/TII484XFNk4cKFSkxM1A8//ODwy+T1hz6lJeWXy+svinj9r8W7d+/WgQMHNGPGDHXq1MnentYhPundFkFBQcqdO7f279+fatq+ffvk5uaW6jWVXjVq1FCNGjX09ttv6+uvv1bHjh317bff6tlnn73hMunZ1pLk5uamBg0aqEGDBho3bpzeeecd/e9//9Py5csVERGR7hpLlCihp59+WpMnT041QIX073OT1sUq73Svyo2cPHlSFy9edNibdODAAUmyH8ZXokQJ7dy5Uw0aNMjQ17z1c8R66GVSUpKioqKc2q634ufnp2HDhqlLly6aPXu22rdvbx9oxd/f/6Z9BQUFyd/fX7///vtN+yhevPgNX9cp02+1vPR/e7is0lqvp6enmjVrpmbNmik5OVm9evXS5MmT9frrr6f7WlLWvnft2qXk5GSHvUnprf1OnT9/XsuWLdOIESM0dOhQe/v12yK9z0VmSOs1JN38PXuj0TOB7IrD7YA71KBBA3l6emrSpEmpfuH79NNPdfXqVTVp0iRD+ko59jsjrnLeqlUrubu7a8SIEal+tTPG2IeajYuL09WrVx2mV6xYUW5ubg6HcPn6+mba1dfbtWunkydP6rPPPtPOnTtTHWqX8muk9XHExsZq2rRpt1x3yj+Hq1atsrddu3ZNn3766S37MMakOcxwyj/Zt9oe7u7uatiwob7//nuHIalPnz6tr7/+Wg8//LDDoYfpcf78+VTPZ8qegfQecnezbZ3WngRn1n+91157TVeuXNGYMWNSTStRooRiY2O1a9cue1t0dPQNh4G+U1evXtXkyZPt95OSkjR58mQFBQWpWrVqkv79hf/EiROaMmVKquUvXbqkixcv3lbfERER8vT01AcffODw/H3++eeKjY3V448/flvrvZGOHTuqSJEi9tHgqlWrphIlSui9995zuAh1ipShot3c3NSiRQstXLhQW7ZsSTVfSu1NmzbVpk2b7IduSv+ex/fpp58qNDT0ludpFSpUSFWqVNGMGTMcDmVdunRpqvOZrh8W283NzT7i3+28Jps2bapTp045jPp29epVffjhh/Lz87MfYptZ0vqskf69QLhVep+LzHL9a0j69z27YcMGh+HHf/zxx1SHDQM5AXuSgDtUsGBBDR06VK+99prq1KmjJ598Urlz59a6dev0zTffqGHDhmrWrFmG9OXj46Ny5cpp1qxZKl26tPLly6cKFSqk+5h0qxIlSuitt97SkCFDdPToUbVo0UJ58uRRVFSUvvvuO/Xo0UODBw/Wb7/9pj59+qhNmzYqXbq0rl69qpkzZ8rd3V2tW7e2r69atWr69ddfNW7cOBUuXFhhYWFp7h2w2rZtm7788ss0a6tZs6b9ftOmTZUnTx4NHjw4Vb+S1LBhQ/svyc8995zi4+M1ZcoUFSxY8Jbnu5QvX141atTQkCFDdO7cOeXLl0/ffvttqmAYHh6uEiVKaPDgwTpx4oT8/f01b968NI+dT/mHul+/fmrUqJHc3d3tv7Re76233rJfd6hXr17KlSuXJk+erMTExDSDw63MmDFDEydOVMuWLVWiRAn9888/mjJlivz9/dW0adNbLn+rbf3GG29o1apVevzxx1W8eHHFxMRo4sSJKlKkiMNJ+umVsjcprcOi2rdvr5dfflktW7ZUv379lJCQoEmTJql06dL2k9czUuHChTV69GgdPXpUpUuX1qxZs7Rjxw59+umn9gs7P/PMM5o9e7aef/55LV++XLVr19a1a9e0b98+zZ49W4sXL07zJPpbCQoK0pAhQzRixAg1btxYTz75pPbv36+JEyfqwQcfzPCLj3p4eKh///568cUXtWjRIjVu3FifffaZmjRpovLly6tLly667777dOLECS1fvlz+/v5auHChJOmdd97RkiVLVLduXfsw6NHR0ZozZ47WrFmjwMBAvfLKK/rmm2/UpEkT9evXT/ny5dOMGTMUFRWlefPmpetCsSNHjtTjjz+uhx9+WF27dtW5c+f04Ycfqnz58g5B7tlnn9W5c+dUv359FSlSRMeOHdOHH36oKlWq2M8jckaPHj00efJkRUZGauvWrQoNDdXcuXO1du1aTZgwwWEQkNtx4sSJND/3/Pz81KJFC/n7+6tOnToaM2aMrly5ovvuu09LlixRVFRUqmXS81xklrReQ88++6zmzp2rxo0bq23btjp8+LC+/PLLdF0SAsh2snYwPeDu9eWXX5oaNWoYX19f4+XlZcLDw82IESPM5cuXHeZLa6hjY/5vONlp06bZ29IaAnndunWmWrVqxtPT06nhwK8fAjzFvHnzzMMPP2x8fX2Nr6+vCQ8PN7179zb79+83xhhz5MgR07VrV1OiRAnj7e1t8uXLZx599FHz66+/Oqxn3759pk6dOsbHx8dIuulw4LcaAjytZTt27GgfxjYtP/zwg6lUqZLx9vY2oaGhZvTo0Wbq1KlpDvVsHYrWGGMOHz5sIiIijJeXlwkODjavvvqqWbp0aaohwP/44w8TERFh/Pz8TIECBUz37t3Nzp07Uz1vV69eNX379jVBQUHGZrM5DFed1nO2bds206hRI+Pn52dy585tHn30UbNu3TqHeVKG8b1+qN/rhyrftm2b6dChgylWrJjx8vIyBQsWNE888YTZsmVLmtstLTfb1suWLTPNmzc3hQsXNp6enqZw4cKmQ4cO5sCBA7dcb1rDLRtjzMGDB427u3ua74slS5aYChUqGE9PT1OmTBnz5Zdf3nAI8N69ezu0pbzO3n33XYf2tN6DdevWNeXLlzdbtmwxNWvWNN7e3qZ48eLmo48+SlVvUlKSGT16tClfvrzx8vIyefPmNdWqVTMjRowwsbGxN63pVj766CMTHh5uPDw8THBwsOnZs6c5f/68wzy3MwR4WvPGxsaagIAAh/fD9u3bTatWrUz+/PmNl5eXKV68uGnbtq1ZtmyZw7LHjh0znTp1MkFBQcbLy8vcf//9pnfv3g7DOR8+fNg89dRTJjAw0Hh7e5vq1aubH3/80WE9N/o8TDFv3jxTtmxZ4+XlZcqVK2fmz5+f6nNx7ty5pmHDhqZgwYLG09PTFCtWzDz33HMmOjr6ltvnRq/J06dPmy5dupgCBQoYT09PU7FiRYf3uDE3fn3dqr8bfe5ZH9Nff/1lWrZsaQIDA01AQIBp06aNOXnyZJqfH7d6LtL72XEjzr6Gxo4da+677z7j5eVlateubbZs2XLDIcDT8z0IuIrNGM6OAwAAAIAUnJMEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAACLu/5issnJyTp58qTy5Mkjm83m6nIAAAAAuIgxRv/8848KFy580wtb3/Uh6eTJkypatKirywAAAACQTRw/flxFihS54fS7PiTlyZNH0r8bwt/f38XVAAAAAHCVuLg4FS1a1J4RbuSuD0kph9j5+/sTkgAAAADc8jQcBm4AAAAAAAtCEgAAAABYEJIAAAAAwOKuPycpPYwxunr1qq5du+bqUu467u7uypUrF8OvAwAAIMe450NSUlKSoqOjlZCQ4OpS7lq5c+dWoUKF5Onp6epSAAAAgFu6p0NScnKyoqKi5O7ursKFC8vT05M9HhnIGKOkpCSdOXNGUVFRKlWq1E0v2gUAAABkB/d0SEpKSlJycrKKFi2q3Llzu7qcu5KPj488PDx07NgxJSUlydvb29UlAQAAADfFz/oSezcyGdsXAAAAOQn/vQIAAACABSEJAAAAACzu6XOSbsS2a2GW9mcqNcvS/gAAAADcGHuScqDIyEjZbDbZbDZ5eHgoLCxML730ki5fvpzpfc+fP18NGzZU/vz5ZbPZtGPHjkzvEwAAAMhKhKQcqnHjxoqOjtaRI0c0fvx4TZ48WcOGDcv0fi9evKiHH35Yo0ePzvS+AAAAAFcgJOVQXl5eCgkJUdGiRdWiRQtFRERo6dKl9umhoaGaMGGCwzJVqlTR8OHD7fdtNps+++wztWzZUrlz51apUqX0ww8/3LTfZ555RkOHDlVERERGPhwAAAAg2yAk3QV+//13rVu3Tp6enk4vO2LECLVt21a7du1S06ZN1bFjR507dy4TqgQAAAByBkJSDvXjjz/Kz89P3t7eqlixomJiYvTiiy86vZ7IyEh16NBBJUuW1DvvvKP4+Hht2rQpEyoGAAAAcgZGt8uhHn30UU2aNEkXL17U+PHjlStXLrVu3drp9VSqVMn+t6+vr/z9/RUTE5ORpQIAAAA5CnuScihfX1+VLFlSlStX1tSpU7Vx40Z9/vnn9ulubm4yxjgsc+XKlVTr8fDwcLhvs9mUnJycOUUDAAAAOQAh6S7g5uamV199Va+99pouXbokSQoKClJ0dLR9nri4OEVFRbmqRAAAACDH4HC7u0SbNm304osv6uOPP9bgwYNVv359TZ8+Xc2aNVNgYKCGDh0qd3f3O+7n3Llz+vPPP3Xy5ElJ0v79+yVJISEhCgkJueP1AwAyXlZfJB13Hy58j3sNISkNOfGDIFeuXOrTp4/GjBmjnj17asiQIYqKitITTzyhgIAAvfnmmxmyJ+mHH35Qly5d7Pfbt28vSRo2bJjD8OIAAABATmUz15+4cpeJi4tTQECAYmNj5e/v7zDt8uXLioqKUlhYmLy9vV1U4d2P7QwArsWeJNypnPgDMpCWm2UDK85JAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAItcri4gO7LZbFnanzEmS/sDAAAAcGPsScqBIiMjZbPZZLPZ5OHhobCwML300ku6fPlypvZ75coVvfzyy6pYsaJ8fX1VuHBhderUSSdPnszUfgEAAICsREjKoRo3bqzo6GgdOXJE48eP1+TJkzVs2LBM7TMhIUHbtm3T66+/rm3btmn+/Pnav3+/nnzyyUztFwAAAMhKhKQcysvLSyEhISpatKhatGihiIgILV261D49NDRUEyZMcFimSpUqGj58uP2+zWbTZ599ppYtWyp37twqVaqUfvjhhxv2GRAQoKVLl6pt27YqU6aMatSooY8++khbt27Vn3/+mdEPEQAAAHAJQtJd4Pfff9e6devk6enp9LIjRoxQ27ZttWvXLjVt2lQdO3bUuXPn0r18bGysbDabAgMDne4bAAAAyI4ISTnUjz/+KD8/P3l7e6tixYqKiYnRiy++6PR6IiMj1aFDB5UsWVLvvPOO4uPjtWnTpnQte/nyZb388svq0KGD/P39ne4bAAAAyI4Y3S6HevTRRzVp0iRdvHhR48ePV65cudS6dWun11OpUiX7376+vvL391dMTMwtl7ty5Yratm0rY4wmTZrkdL8AAABAdsWepBzK19dXJUuWVOXKlTV16lRt3LhRn3/+uX26m5tbqqHFr1y5kmo9Hh4eDvdtNpuSk5Nv2ndKQDp27JiWLl3KXiQAAADcVQhJdwE3Nze9+uqreu2113Tp0iVJUlBQkKKjo+3zxMXFKSoq6o77SglIBw8e1K+//qr8+fPf8ToBAACA7ISQdJdo06aN3N3d9fHHH0uS6tevr5kzZ2r16tXavXu3OnfuLHd39zvq48qVK3rqqae0ZcsWffXVV7p27ZpOnTqlU6dOKSkpKSMeBgAAAOBynJOUhusPU8sJcuXKpT59+mjMmDHq2bOnhgwZoqioKD3xxBMKCAjQm2++ecd7kk6cOGEfIrxKlSoO05YvX6569erd0foBAACA7MBmcmIicEJcXJwCAgIUGxub6tyZy5cvKyoqSmFhYfL29nZRhXc/tjMAuJZt10JXl4AczlRq5uoSgAxxs2xgxeF2AAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgtHtlDNHs8tJ2L4AAORsNpvN1SUgh8tp/w/e03uSPDw8JEkJCQkuruTulrJ9U7Y3AAAAkJ3d03uS3N3dFRgYqJiYGElS7ty5+aUkAxljlJCQoJiYGAUGBt7xxWwBAACArHBPhyRJCgkJkSR7UELGCwwMtG9nAAAAILu750OSzWZToUKFVLBgQV25csXV5dx1PDw82IMEAACAHOWeD0kp3N3d+WceAAAAwL09cAMAAAAAXI+QBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALDINiFp1KhRstlsGjBggL3t8uXL6t27t/Lnzy8/Pz+1bt1ap0+fdl2RAAAAAO562SIkbd68WZMnT1alSpUc2gcOHKiFCxdqzpw5WrlypU6ePKlWrVq5qEoAAAAA9wKXh6T4+Hh17NhRU6ZMUd68ee3tsbGx+vzzzzVu3DjVr19f1apV07Rp07Ru3Tpt2LDBhRUDAAAAuJu5PCT17t1bjz/+uCIiIhzat27dqitXrji0h4eHq1ixYlq/fv0N15eYmKi4uDiHGwAAAACkVy5Xdv7tt99q27Zt2rx5c6ppp06dkqenpwIDAx3ag4ODderUqRuuc+TIkRoxYkRGlwoAAADgHuGyPUnHjx9X//799dVXX8nb2zvD1jtkyBDFxsbab8ePH8+wdQMAAAC4+7ksJG3dulUxMTGqWrWqcuXKpVy5cmnlypX64IMPlCtXLgUHByspKUkXLlxwWO706dMKCQm54Xq9vLzk7+/vcAMAAACA9HLZ4XYNGjTQ7t27Hdq6dOmi8PBwvfzyyypatKg8PDy0bNkytW7dWpK0f/9+/fnnn6pZs6YrSgYAAABwD3BZSMqTJ48qVKjg0Obr66v8+fPb27t166ZBgwYpX7588vf3V9++fVWzZk3VqFHDFSUDAAAAuAe4dOCGWxk/frzc3NzUunVrJSYmqlGjRpo4caKrywIAAABwF7MZY4yri8hMcXFxCggIUGxsLOcnAQDuSbZdC11dAnK6yk+6ugLkcNklcqQ3G7j8OkkAAAAAkJ0QkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALp0PSpUuXlJCQYL9/7NgxTZgwQUuWLMnQwgAAAADAFZwOSc2bN9cXX3whSbpw4YIeeughjR07Vs2bN9ekSZMyvEAAAAAAyEpOh6Rt27bpkUcekSTNnTtXwcHBOnbsmL744gt98MEHGV4gAAAAAGQlp0NSQkKC8uTJI0lasmSJWrVqJTc3N9WoUUPHjh3L8AIBAAAAICs5HZJKliypBQsW6Pjx41q8eLEaNmwoSYqJiZG/v3+GFwgAAAAAWcnpkDR06FANHjxYoaGheuihh1SzZk1J/+5V+s9//uPUuiZNmqRKlSrJ399f/v7+qlmzpn755Rf79MuXL6t3797Knz+//Pz81Lp1a50+fdrZkgEAAAAg3WzGGOPsQqdOnVJ0dLQqV64sN7d/c9amTZvk7++v8PDwdK9n4cKFcnd3V6lSpWSM0YwZM/Tuu+9q+/btKl++vHr27KmffvpJ06dPV0BAgPr06SM3NzetXbs23X3ExcUpICBAsbGx7OkCANyTbLsWuroE5HSVn3R1BcjhbiNyZIr0ZoPbCkmZKV++fHr33Xf11FNPKSgoSF9//bWeeuopSdK+fftUtmxZrV+/XjVq1EjX+ghJAIB7HSEJd4yQhDuUXSJHerNBrvSsrFWrVunueP78+eme1+ratWuaM2eOLl68qJo1a2rr1q26cuWKIiIi7POEh4erWLFiNw1JiYmJSkxMtN+Pi4u7rXoAAAAA3JvSdU5SQECA/ebv769ly5Zpy5Yt9ulbt27VsmXLFBAQ4HQBu3fvlp+fn7y8vPT888/ru+++U7ly5XTq1Cl5enoqMDDQYf7g4GCdOnXqhusbOXKkQ71FixZ1uiYAAAAA96507UmaNm2a/e+XX35Zbdu21SeffCJ3d3dJ/+4F6tWr120dzlamTBnt2LFDsbGxmjt3rjp37qyVK1c6vZ4UQ4YM0aBBg+z34+LiCEoAAAAA0i1dIclq6tSpWrNmjT0gSZK7u7sGDRqkWrVq6d1333VqfZ6enipZsqQkqVq1atq8ebPef/99tWvXTklJSbpw4YLD3qTTp08rJCTkhuvz8vKSl5eXcw8KAAAAAP4/p4cAv3r1qvbt25eqfd++fUpOTr7jgpKTk5WYmKhq1arJw8NDy5Yts0/bv3+//vzzT/uw4wAAAACQ0Zzek9SlSxd169ZNhw8fVvXq1SVJGzdu1KhRo9SlSxen1jVkyBA1adJExYoV0z///KOvv/5aK1as0OLFixUQEKBu3bpp0KBBypcvn/z9/dW3b1/VrFkz3SPbAQAAAICznA5J7733nkJCQjR27FhFR0dLkgoVKqQXX3xRL7zwglPriomJUadOnRQdHa2AgABVqlRJixcv1mOPPSZJGj9+vNzc3NS6dWslJiaqUaNGmjhxorMlAwAAAEC6OXWdpKtXr+rrr79Wo0aNFBwcbB9eOztff4jrJAEA7nVcJwl3jOsk4Q7ltOskOXVOUq5cufT888/r8uXLkv4NRwQPAAAAAHcTpwduqF69urZv354ZtQAAAACAyzl9TlKvXr30wgsv6K+//lK1atXk6+vrML1SpUoZVhwAAAAAZDWnQ1L79u0lSf369bO32Ww2GWNks9l07dq1jKsOAAAAALKY0yEpKioqM+oAAAAAgGzB6ZBUvHjxzKgDAAAAALIFp0OSJB0+fFgTJkzQ3r17JUnlypVT//79VaJEiQwtDgAAAACymtOj2y1evFjlypXTpk2bVKlSJVWqVEkbN25U+fLltXTp0syoEQAAAACyjNN7kl555RUNHDhQo0aNStX+8ssv67HHHsuw4gAAAAAgqzm9J2nv3r3q1q1bqvauXbvqjz/+yJCiAAAAAMBVnA5JQUFB2rFjR6r2HTt2qGDBghlREwAAAAC4jNOH23Xv3l09evTQkSNHVKtWLUnS2rVrNXr0aA0aNCjDCwQAAACArOR0SHr99deVJ08ejR07VkOGDJEkFS5cWMOHD3e4wCwAAAAA5EQ2Y4y53YX/+ecfSVKePHkyrKCMFhcXp4CAAMXGxsrf39/V5QAAkOVsuxa6ugTkdJWfdHUFyOHuIHJkqPRmA6f3JEVFRenq1asqVaqUQzg6ePCgPDw8FBoaelsFAwAAAEB24PTADZGRkVq3bl2q9o0bNyoyMjIjagIAAAAAl3E6JG3fvl21a9dO1V6jRo00R70DAAAAgJzE6ZBks9ns5yJZxcbG6tq1axlSFAAAAAC4itMhqU6dOho5cqRDILp27ZpGjhyphx9+OEOLAwAAAICs5vTADaNHj1adOnVUpkwZPfLII5Kk1atXKy4uTr/99luGFwgAAAAAWcnpPUnlypXTrl271LZtW8XExOiff/5Rp06dtG/fPlWoUCEzagQAAACALOP0niTp34vHvvPOOxldCwAAAAC4nNN7kqR/D697+umnVatWLZ04cUKSNHPmTK1ZsyZDiwMAAACArOZ0SJo3b54aNWokHx8fbdu2TYmJiZL+Hd2OvUsAAAAAcjqnQ9Jbb72lTz75RFOmTJGHh4e9vXbt2tq2bVuGFgcAAAAAWc3pkLR//37VqVMnVXtAQIAuXLiQETUBAAAAgMs4HZJCQkJ06NChVO1r1qzR/fffnyFFAQAAAICrOB2Sunfvrv79+2vjxo2y2Ww6efKkvvrqKw0ePFg9e/bMjBoBAAAAIMs4PQT4K6+8ouTkZDVo0EAJCQmqU6eOvLy8NHjwYPXt2zczagQAAACALGMzxpjbWTApKUmHDh1SfHy8ypUrJz8/P126dEk+Pj4ZXeMdiYuLU0BAgGJjY+Xv7+/qcgAAyHK2XQtdXQJyuspPuroC5HC3GTkyXHqzwW1dJ0mSPD09Va5cOVWvXl0eHh4aN26cwsLCbnd1AAAAAJAtpDskJSYmasiQIXrggQdUq1YtLViwQJI0bdo0hYWFafz48Ro4cGBm1QkAAAAAWSLd5yQNHTpUkydPVkREhNatW6c2bdqoS5cu2rBhg8aNG6c2bdrI3d09M2sFAAAAgEyX7pA0Z84cffHFF3ryySf1+++/q1KlSrp69ap27twpm82WmTUCAAAAQJZJ9+F2f/31l6pVqyZJqlChgry8vDRw4EACEgAAAIC7SrpD0rVr1+Tp6Wm/nytXLvn5+WVKUQAAAADgKuk+3M4Yo8jISHl5eUmSLl++rOeff16+vr4O882fPz9jKwQAAACALJTukNS5c2eH+08//XSGFwMAAAAArpbukDRt2rTMrAMAAAAAsoXbvpgsAAAAANyNCEkAAAAAYEFIAgAAAAALQhIAAAAAWKQrJFWtWlXnz5+XJL3xxhtKSEjI1KIAAAAAwFXSFZL27t2rixcvSpJGjBih+Pj4TC0KAAAAAFwlXUOAV6lSRV26dNHDDz8sY4zee+89+fn5pTnv0KFDM7RAAAAAAMhK6QpJ06dP17Bhw/Tjjz/KZrPpl19+Ua5cqRe12WyEJAAAAAA5WrpCUpkyZfTtt99Kktzc3LRs2TIVLFgwUwsDAAAAAFdIV0iySk5Ozow6AAAAACBbcDokSdLhw4c1YcIE7d27V5JUrlw59e/fXyVKlMjQ4gAAAAAgqzl9naTFixerXLly2rRpkypVqqRKlSpp48aNKl++vJYuXZoZNQIAAABAlnF6T9Irr7yigQMHatSoUanaX375ZT322GMZVhwAAAAAZDWn9yTt3btX3bp1S9XetWtX/fHHHxlSFAAAAAC4itMhKSgoSDt27EjVvmPHDka8AwAAAJDjOX24Xffu3dWjRw8dOXJEtWrVkiStXbtWo0eP1qBBgzK8QAAAAADISk6HpNdff1158uTR2LFjNWTIEElS4cKFNXz4cPXr1y/DCwQAAACArGQzxpjbXfiff/6RJOXJkyfDCspocXFxCggIUGxsrPz9/V1dDgAAWc62a6GrS0BOV/lJV1eAHO4OIkeGSm82uK3rJKXIzuEIAAAAAG6H0wM3AAAAAMDdjJAEAAAAABaEJAAAAACwcCokXblyRQ0aNNDBgwczqx4AAAAAcCmnQpKHh4d27dqVWbUAAAAAgMs5fbjd008/rc8//zwzagEAAAAAl3N6CPCrV69q6tSp+vXXX1WtWjX5+vo6TB83blyGFQcAAAAAWc3pkPT777+ratWqkqQDBw44TLPZbBlTFQAAAAC4iNMhafny5ZlRBwAAAABkC7c9BPihQ4e0ePFiXbp0SZJkjMmwogAAAADAVZwOSWfPnlWDBg1UunRpNW3aVNHR0ZKkbt266YUXXsjwAgEAAAAgKzkdkgYOHCgPDw/9+eefyp07t729Xbt2WrRoUYYWBwAAAABZzelzkpYsWaLFixerSJEiDu2lSpXSsWPHMqwwAAAAAHAFp/ckXbx40WEPUopz587Jy8srQ4oCAAAAAFdxOiQ98sgj+uKLL+z3bTabkpOTNWbMGD366KMZWhwAAAAAZDWnD7cbM2aMGjRooC1btigpKUkvvfSS9uzZo3Pnzmnt2rWZUSMAAAAAZBmn9yRVqFBBBw4c0MMPP6zmzZvr4sWLatWqlbZv364SJUpkRo0AAAAAkGWc3pMkSQEBAfrf//6X0bUAAAAAgMvdVkg6f/68Pv/8c+3du1eSVK5cOXXp0kX58uXL0OIAAAAAIKs5fbjdqlWrFBoaqg8++EDnz5/X+fPn9cEHHygsLEyrVq3KjBoBAAAAIMs4vSepd+/eateunSZNmiR3d3dJ0rVr19SrVy/17t1bu3fvzvAiAQAAACCrOL0n6dChQ3rhhRfsAUmS3N3dNWjQIB06dChDiwMAAACArOZ0SKpatar9XCSrvXv3qnLlyhlSFAAAAAC4SroOt9u1a5f97379+ql///46dOiQatSoIUnasGGDPv74Y40aNSpzqgQAAACALGIzxphbzeTm5iabzaZbzWqz2XTt2rUMKy4jxMXFKSAgQLGxsfL393d1OQAAZDnbroWuLgE5XeUnXV0Bcrh0RI4skd5skK49SVFRURlWGAAAAABkZ+kKScWLF8/sOgAAAAAgW7iti8mePHlSa9asUUxMjJKTkx2m9evXL93rGTlypObPn699+/bJx8dHtWrV0ujRo1WmTBn7PJcvX9YLL7ygb7/9VomJiWrUqJEmTpyo4ODg2ykdAAAAAG7K6ZA0ffp0Pffcc/L09FT+/Plls9ns02w2m1MhaeXKlerdu7cefPBBXb16Va+++qoaNmyoP/74Q76+vpKkgQMH6qefftKcOXMUEBCgPn36qFWrVlq7dq2zpQMAAADALaVr4AarokWL6vnnn9eQIUPk5ub0COI3debMGRUsWFArV65UnTp1FBsbq6CgIH399dd66qmnJEn79u1T2bJltX79evvoelaJiYlKTEy034+Li1PRokUZuAEAcM9i4AbcMQZuwB3KaQM3OJ1yEhIS1L59+wwPSJIUGxsrScqXL58kaevWrbpy5YoiIiLs84SHh6tYsWJav359musYOXKkAgIC7LeiRYtmeJ0AAAAA7l5OJ51u3bppzpw5GV5IcnKyBgwYoNq1a6tChQqSpFOnTsnT01OBgYEO8wYHB+vUqVNprmfIkCGKjY21344fP57htQIAAAC4ezl9TtLIkSP1xBNPaNGiRapYsaI8PDwcpo8bN+62Cundu7d+//13rVmz5raWT+Hl5SUvL687WgcAAACAe9dthaTFixfbR6C7fuCG29GnTx/9+OOPWrVqlYoUKWJvDwkJUVJSki5cuOCwN+n06dMKCQm5rb4AAAAA4GacDkljx47V1KlTFRkZecedG2PUt29ffffdd1qxYoXCwsIcplerVk0eHh5atmyZWrduLUnav3+//vzzT9WsWfOO+wcAAACA6zkdkry8vFS7du0M6bx37976+uuv9f333ytPnjz284wCAgLk4+OjgIAAdevWTYMGDVK+fPnk7++vvn37qmbNmmmObAcAAAAAd8rpgRv69++vDz/8MEM6nzRpkmJjY1WvXj0VKlTIfps1a5Z9nvHjx+uJJ55Q69atVadOHYWEhGj+/PkZ0j8AAAAAXM/p6yS1bNlSv/32m/Lnz6/y5cunGrghuwWY9I6FDgDA3YrrJOGOcZ0k3KGcdp0kpw+3CwwMVKtWre6oOAAAAADIrpwOSdOmTcuMOgAAAAAgW3D6nCQAAAAAuJs5vScpLCzsptdDOnLkyB0VBAAAAACu5HRIGjBggMP9K1euaPv27Vq0aJFefPHFjKoLAAAAAFzC6ZDUv3//NNs//vhjbdmy5Y4LAgAAAABXyrBzkpo0aaJ58+Zl1OoAAAAAwCUyLCTNnTtX+fLly6jVAQAAAIBLOH243X/+8x+HgRuMMTp16pTOnDmjiRMnZmhxAAAAAJDVnA5JLVq0cLjv5uamoKAg1atXT+Hh4RlVFwAAAAC4hNMhadiwYZlRBwAAAABkC1xMFgAAAAAs0r0nyc3N7aYXkZUkm82mq1ev3nFRAAAAAOAq6Q5J33333Q2nrV+/Xh988IGSk5MzpCgAAAAAcJV0h6TmzZunatu/f79eeeUVLVy4UB07dtQbb7yRocUBAAAAQFa7rXOSTp48qe7du6tixYq6evWqduzYoRkzZqh48eIZXR8AAAAAZCmnQlJsbKxefvlllSxZUnv27NGyZcu0cOFCVahQIbPqAwAAAIAsle7D7caMGaPRo0crJCRE33zzTZqH3wEAAABATmczxpj0zOjm5iYfHx9FRETI3d39hvPNnz8/w4rLCHFxcQoICFBsbKz8/f1dXQ4AAFnOtmuhq0tATlf5SVdXgBwunZEj06U3G6R7T1KnTp1uOQQ4AAAAAOR06Q5J06dPz8QyAAAAACB7uK3R7QAAAADgbkVIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAICFS0PSqlWr1KxZMxUuXFg2m00LFixwmG6M0dChQ1WoUCH5+PgoIiJCBw8edE2xAAAAAO4JLg1JFy9eVOXKlfXxxx+nOX3MmDH64IMP9Mknn2jjxo3y9fVVo0aNdPny5SyuFAAAAMC9IpcrO2/SpImaNGmS5jRjjCZMmKDXXntNzZs3lyR98cUXCg4O1oIFC9S+ffusLBUAAADAPSLbnpMUFRWlU6dOKSIiwt4WEBCghx56SOvXr7/hcomJiYqLi3O4AQAAAEB6ZduQdOrUKUlScHCwQ3twcLB9WlpGjhypgIAA+61o0aKZWicAAACAu0u2DUm3a8iQIYqNjbXfjh8/7uqSAAAAAOQg2TYkhYSESJJOnz7t0H769Gn7tLR4eXnJ39/f4QYAAAAA6ZVtQ1JYWJhCQkK0bNkye1tcXJw2btyomjVrurAyAAAAAHczl45uFx8fr0OHDtnvR0VFaceOHcqXL5+KFSumAQMG6K233lKpUqUUFham119/XYULF1aLFi1cVzQAAACAu5pLQ9KWLVv06KOP2u8PGjRIktS5c2dNnz5dL730ki5evKgePXrowoULevjhh7Vo0SJ5e3u7qmQAAAAAdzmbMca4uojMFBcXp4CAAMXGxnJ+EgDgnmTbtdDVJSCnq/ykqytADpddIkd6s0G2PScJAAAAAFyBkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYJHL1QUAcI7NZnN1CcjhjDGuLgEAgGyNPUkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWORydQH3Gtuuha4uAQAAAMBNsCcJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGCRI0LSxx9/rNDQUHl7e+uhhx7Spk2bXF0SAAAAgLtUtg9Js2bN0qBBgzRs2DBt27ZNlStXVqNGjRQTE+Pq0gAAAADchbJ9SBo3bpy6d++uLl26qFy5cvrkk0+UO3duTZ061dWlAQAAALgL5XJ1ATeTlJSkrVu3asiQIfY2Nzc3RUREaP369Wkuk5iYqMTERPv92NhYSVJcXFzmFpte8QmurgDAPS7bfB4i6/DdA8DFsst3T0odxpibzpetQ9Lff/+ta9euKTg42KE9ODhY+/btS3OZkSNHasSIEanaixYtmik1AkBOExAQ4OoSAAD3mOz23fPPP//ctKZsHZJux5AhQzRo0CD7/eTkZJ07d0758+eXzWZzYWXAnYuLi1PRokV1/Phx+fv7u7ocAMA9gO8e3E2MMfrnn39UuHDhm86XrUNSgQIF5O7urtOnTzu0nz59WiEhIWku4+XlJS8vL4e2wMDAzCoRcAl/f3++qAAAWYrvHtwt0rNXK1sP3ODp6alq1app2bJl9rbk5GQtW7ZMNWvWdGFlAAAAAO5W2XpPkiQNGjRInTt31gMPPKDq1atrwoQJunjxorp06eLq0gAAAADchbJ9SGrXrp3OnDmjoUOH6tSpU6pSpYoWLVqUajAH4F7g5eWlYcOGpTqkFACAzMJ3D+5FNnOr8e8AAAAA4B6Src9JAgAAAICsRkgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBLiQzWbTggULXF0GAABZZvjw4apSpYqrywBuipCEe9qZM2fUs2dPFStWTF5eXgoJCVGjRo20du1ah/nWrVunpk2bKm/evPL29lbFihU1btw4Xbt2zWE+Z0NPdHS0mjRpkhEPJds5evSobDabduzY4epSAABOSkpKSrP9ypUrWVwJ4BqEJNzTWrdure3bt2vGjBk6cOCAfvjhB9WrV09nz561z/Pdd9+pbt26KlKkiJYvX659+/apf//+euutt9S+fXvdySj6ISEhLr3uhDFGV69eTdV+oy9HAED2lZycrDFjxqhkyZLy8vJSsWLF9Pbbb0uSdu/erfr168vHx0f58+dXjx49FB8fb182MjJSLVq00Ntvv63ChQurTJky9h+7Zs2apbp168rb21tfffWVJOmzzz5T2bJl5e3trfDwcE2cONGhlr/++ksdOnRQvnz55OvrqwceeEAbN27U9OnTNWLECO3cuVM2m002m03Tp0/Psm0EpJsB7lHnz583ksyKFStuOE98fLzJnz+/adWqVappP/zwg5Fkvv32W3ubJPPdd9+luwbr/FFRUUaSmTdvnqlXr57x8fExlSpVMuvWrXNYZs2aNaZu3brGx8fHBAYGmoYNG5pz584ZY4y5fPmy6du3rwkKCjJeXl6mdu3aZtOmTfZlly9fbiSZn3/+2VStWtV4eHiY5cuXm7p165revXub/v37m/z585t69eoZY4zZvXu3ady4sfH19TUFCxY0Tz/9tDlz5ox9fdeuXTOjR482JUqUMJ6enqZo0aLmrbfesj82661u3brp3i4AAOe99NJLJm/evGb69Onm0KFDZvXq1WbKlCkmPj7eFCpUyLRq1crs3r3bLFu2zISFhZnOnTvbl+3cubPx8/MzzzzzjPn999/N77//bv9eCg0NNfPmzTNHjhwxJ0+eNF9++aUpVKiQvW3evHkmX758Zvr06cYYY/755x9z//33m0ceecSsXr3aHDx40MyaNcusW7fOJCQkmBdeeMGUL1/eREdHm+joaJOQkOCiLQbcGCEJ96wrV64YPz8/M2DAAHP58uU055k/f76RlCqopChdurRp3ry5/X5GhKTw8HDz448/mv3795unnnrKFC9e3Fy5csUYY8z27duNl5eX6dmzp9mxY4f5/fffzYcffmgPLv369TOFCxc2P//8s9mzZ4/p3LmzyZs3rzl79qwx5v9CUqVKlcySJUvMoUOHzNmzZ03dunWNn5+fefHFF82+ffvMvn37zPnz501QUJAZMmSI2bt3r9m2bZt57LHHzKOPPmqv/0ZfyMYYs2nTJiPJ/PrrryY6OtpeAwAg48XFxRkvLy/7Z7DVp59+avLmzWvi4+PtbT/99JNxc3Mzp06dMsb8G5KCg4NNYmKifZ6U76UJEyY4rK9EiRLm66+/dmh78803Tc2aNY0xxkyePNnkyZPnhp/7w4YNM5UrV76txwlkFUIS7mlz5841efPmNd7e3qZWrVpmyJAhZufOnfbpo0aNMpLM+fPn01z+ySefNGXLlrXfz4iQ9Nlnn9mn79mzx0gye/fuNcYY06FDB1O7du001xUfH288PDzMV199ZW9LSkoyhQsXNmPGjDHG/F9IWrBggcOydevWNf/5z38c2t58803TsGFDh7bjx48bSWb//v03/UK2Pp7t27ffekMAAO7Ixo0bjSRz5MiRVNMGDhxoP0IgxYULF4wks3LlSmPMvyEpIiLCYZ6Uz/E1a9bY2+Lj440k4+PjY3x9fe03Ly8vU7BgQWOMMT179jR16tS5Ya2EJOQEubLikD4gu2rdurUef/xxrV69Whs2bNAvv/yiMWPG6LPPPlNkZKR9PnOT8448PT0ztKZKlSrZ/y5UqJAkKSYmRuHh4dqxY4fatGmT5nKHDx/WlStXVLt2bXubh4eHqlevrr179zrM+8ADD6Ravlq1ag73d+7cqeXLl8vPzy/Nvi5cuKDExEQ1aNAg/Q8OAJApfHx87ngdvr6+t2xPOY9pypQpeuihhxzmc3d3z7BaAFdj4Abc87y9vfXYY4/p9ddf17p16xQZGalhw4ZJkkqVKiVJqUJGir1796p06dIZWo+Hh4f9b5vNJunfk3GljPviSeuL8Pq2+Ph4NWvWTDt27HC4HTx4UHXq1OFLEACykVKlSsnHx0fLli1LNa1s2bLauXOnLl68aG9bu3at3NzcVKZMGaf6CQ4OVuHChXXkyBGVLFnS4RYWFibp3x/7duzYoXPnzqW5Dk9Pz1SjwwLZDSEJuE65cuXsXySNGjVSvnz5NHbs2FTz/fDDDzp48KDDHqfMVqlSpTS/ACWpRIkS8vT0dBi+/MqVK9q8ebPKlSvndF9Vq1bVnj17FBoamuqL0NfX96ZfyNL/7WHjixAAMp+3t7defvllvfTSS/riiy90+PBhbdiwQZ9//rk6duwob29vde7cWb///ruWL1+uvn376plnnlFwcLDTfY0YMUIjR47UBx98oAMHDmj37t2aNm2axo0bJ0nq0KGDQkJC1KJFC61du1ZHjhzRvHnztH79eklSaGiooqKitGPHDv39999KTEzM0G0BZARCEu5ZZ8+eVf369fXll19q165dioqK0pw5czRmzBg1b95c0r97VyZPnqzvv/9ePXr00K5du3T06FF9/vnnioyMVPfu3dW0aVOH9aZ88Ftv1l/v7sSQIUO0efNm9erVS7t27dK+ffs0adIk/f333/L19VXPnj314osvatGiRfrjjz/UvXt3JSQkqFu3bk731bt3b507d04dOnTQ5s2bdfjwYS1evFhdunTRtWvXbvqFLEkFCxaUj4+PFi1apNOnTys2NjZDtgEAIG2vv/66XnjhBQ0dOlRly5ZVu3btFBMTo9y5c2vx4sU6d+6cHnzwQT311FNq0KCBPvroo9vq59lnn9Vnn32madOmqWLFiqpbt66mT59u35Pk6empJUuWqGDBgmratKkqVqyoUaNG2Q/Ha926tRo3bqxHH31UQUFB+uabbzJsGwAZxtUnRQGucvnyZfPKK6+YqlWrmoCAAJM7d25TpkwZ89prr6UajnTVqlWmUaNGxt/f3z6k9ejRo1OtU9cNe51yW716dZo1KI2BG6wDHaQMU758+XJ724oVK0ytWrWMl5eXCQwMNI0aNbIPLHHp0iXTt29fU6BAgZsOAX79QBR169Y1/fv3T1XfgQMHTMuWLU1gYKDx8fEx4eHhZsCAASY5OdkY8+8Q4G+99ZYpXry48fDwMMWKFTPvvPOOffkpU6aYokWLGjc3N4YABwAAOYbNmDu4EiZwD7p8+bKaN2+u48ePa+XKlQoKCnJ1SQAAAMhAhCTgNly+fFkTJkxQqVKl1Lp1a1eXAwAAgAxESAIAAAAACwZuAAAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEA0iU0NFQTJkxwdRnZUlZtm6NHj8pms2nHjh2Z3hcA3MsISQBwF6tXr54GDBiQqn369OkKDAx0al2bN29Wjx497PdtNpsWLFhww/mnT58um81209vRo0edqsHZ+m9Ug7e39231m5UiIyPVokULh7aiRYsqOjpaFSpUcE1RAHCPyOXqAgAAOUNQUJBT87dr106NGze232/VqpUqVKigN95447bXeTv8/f21f/9+hzabzZbp/WYGd3d3hYSEuLoMALjrsScJAGDfa/Hee++pUKFCyp8/v3r37q0rV67Y57EeUhYaGipJatmypWw2m/2+lY+Pj0JCQuw3T09P5c6d237f29tbzz33nIKCguTv76/69etr586d9uV37typRx99VHny5JG/v7+qVaumLVu2aMWKFerSpYtiY2Pte4aGDx9+w8dms9kc6ggJCVFwcLAk6dNPP1XhwoWVnJzssEzz5s3VtWtXSdLhw4fVvHlzBQcHy8/PTw8++KB+/fXXG/aX1iFxFy5ckM1m04oVKyRJ165dU7du3RQWFiYfHx+VKVNG77//vn3+4cOHa8aMGfr+++/tj3HFihVprnvlypWqXr26vLy8VKhQIb3yyiu6evWqfXq9evXUr18/vfTSS8qXL59CQkJuur0AAIQkAMD/t3z5ch0+fFjLly/XjBkzNH36dE2fPj3NeTdv3ixJmjZtmqKjo+33ndGmTRvFxMTol19+0datW1W1alU1aNBA586dkyR17NhRRYoU0ebNm7V161a98sor8vDwUK1atTRhwgT5+/srOjpa0dHRGjx48G095jZt2ujs2bNavny5ve3cuXNatGiROnbsKEmKj49X06ZNtWzZMm3fvl2NGzdWs2bN9Oeff95Wn5KUnJysIkWKaM6cOfrjjz80dOhQvfrqq5o9e7YkafDgwWrbtq0aN25sf4y1atVKtZ4TJ06oadOmevDBB7Vz505NmjRJn3/+ud566y2H+WbMmCFfX19t3LhRY8aM0RtvvKGlS5fedv0AcLfjcDsAgCQpb968+uijj+Tu7q7w8HA9/vjjWrZsmbp3755q3pTD5AIDA2/r8K81a9Zo06ZNiomJkZeXlyTpvffe04IFCzR37lz16NFDf/75p1588UWFh4dLkkqVKmVfPiAgwL6H6FZiY2Pl5+fn0PbII4/ol19+Ud68edWkSRN9/fXXatCggSRp7ty5KlCggB599FFJUuXKlVW5cmX7sm+++aa+++47/fDDD+rTp4/Tj12SPDw8NGLECPv9sLAwrV+/XrNnz1bbtm3l5+cnHx8fJSYm3vQxTpw4UUWLFtVHH30km82m8PBwnTx5Ui+//LKGDh0qN7d/fwutVKmShg0bJunf7fjRRx9p2bJleuyxx26rfgC42xGSAACSpPLly8vd3d1+v1ChQtq9e3em9LVz507Fx8crf/78Du2XLl3S4cOHJUmDBg3Ss88+q5kzZyoiIkJt2rRRiRIlnO4rT5482rZtm0Obj4+P/e+OHTuqe/fumjhxory8vPTVV1+pffv29oARHx+v4cOH66efflJ0dLSuXr2qS5cu3dGeJEn6+OOPNXXqVP3555+6dOmSkpKSVKVKFafWsXfvXtWsWdPhHKvatWsrPj5ef/31l4oVKybp35BkVahQIcXExNxR/QBwNyMkAcBdzN/fX7GxsanaL1y4oICAAIc2Dw8Ph/s2my3VuToZJT4+XoUKFbKfo2OVMmrd8OHD9d///lc//fSTfvnlFw0bNkzffvutWrZs6VRfbm5uKlmy5A2nN2vWTMYY/fTTT3rwwQe1evVqjR8/3j598ODBWrp0qd577z2VLFlSPj4+euqpp5SUlHTD/iTJGGNvs57bJUnffvutBg8erLFjx6pmzZrKkyeP3n33XW3cuNGpx5ZeWfncAsDdgJAEAHexMmXKaMmSJanat23bptKlS9/Ruj08PHTt2rXbWrZq1ao6deqUcuXKleagDylKly6t0qVLa+DAgerQoYOmTZumli1bytPT87b7vp63t7datWqlr776SocOHVKZMmVUtWpV+/S1a9cqMjLSHs7i4+NvOnR5yqGI0dHR+s9//iNJqa5rtHbtWtWqVUu9evWyt6XsQUuRnsdYtmxZzZs3T8YY+96ktWvXKk+ePCpSpMjNHzgA4IYYuAEA7mI9e/bUgQMH1K9fP+3atUv79+/XuHHj9M033+iFF164o3WHhoZq2bJlOnXqlM6fP+/UshEREapZs6ZatGihJUuW6OjRo1q3bp3+97//acuWLbp06ZL69OmjFStW6NixY1q7dq02b96ssmXL2vuOj4/XsmXL9PfffyshIeGGfRljdOrUqVQ3656Ujh076qefftLUqVPtAzakKFWqlObPn68dO3Zo586d+u9//3vTvTA+Pj6qUaOGRo0apb1792rlypV67bXXUq1zy5YtWrx4sQ4cOKDXX3891eAXoaGh9ufs77//TrU3SpJ69eql48ePq2/fvtq3b5++//57DRs2TIMGDbLv0QIAOI9PUAC4i91///1atWqV9u3bp4iICD300EOaPXu25syZ43ANo9sxduxYLV26VEWLFrXvMUkvm82mn3/+WXXq1FGXLl1UunRptW/fXseOHVNwcLDc3d119uxZderUSaVLl1bbtm3VpEkT+2AHtWrV0vPPP6927dopKChIY8aMuWFfcXFxKlSoUKqb9Zyc+vXrK1++fNq/f7/++9//Oiw/btw45c2bV7Vq1VKzZs3UqFEjhz1NaZk6daquXr2qatWqacCAAalGm3vuuefUqlUrtWvXTg899JDOnj3rsFdJkrp3764yZcrogQceUFBQkNauXZuqn/vuu08///yzNm3apMqVK+v5559Xt27dUoUyAIBzbMZ60DQAAAAA3OPYkwQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIDF/wPzUn4Q+IkF2gAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "unittest_df_pivot.reset_index(inplace=True)\n",
    "\n",
    "# Plotting\n",
    "plt.figure(figsize=(10, 6))\n",
    "\n",
    "# Set the width of each bar\n",
    "bar_width = 0.35\n",
    "\n",
    "# OpenAI brand colors\n",
    "openai_colors = ['#00D1B2', '#000000']  # Green and Black\n",
    "\n",
    "# Get unique runs and unit test evaluations\n",
    "unique_runs = unittest_df_pivot['run'].unique()\n",
    "unique_unit_test_evaluations = unittest_df_pivot['unit_test_evaluation'].unique()\n",
    "\n",
    "# Ensure we have enough colors (repeating the pattern if necessary)\n",
    "colors = openai_colors * (len(unique_runs) // len(openai_colors) + 1)\n",
    "\n",
    "# Iterate over each run to plot\n",
    "for i, run in enumerate(unique_runs):\n",
    "    run_data = unittest_df_pivot[unittest_df_pivot['run'] == run]\n",
    "\n",
    "    # Position of bars for this run\n",
    "    positions = np.arange(len(unique_unit_test_evaluations)) + i * bar_width\n",
    "\n",
    "    plt.bar(positions, run_data['Number of records'], width=bar_width, label=f'Run {run}', color=colors[i])\n",
    "\n",
    "# Setting the x-axis labels to be the unit test evaluations, centered under the groups\n",
    "plt.xticks(np.arange(len(unique_unit_test_evaluations)) + bar_width / 2, unique_unit_test_evaluations)\n",
    "\n",
    "plt.xlabel('Unit Test Evaluation')\n",
    "plt.ylabel('Number of Records')\n",
    "plt.title('Unit Test Evaluations vs Number of Records for Each Run')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "7228eac7-e0a9-473d-9432-e558bbc91841",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>Number of records</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run</th>\n",
       "      <th>evaluation_score</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">1</th>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"3\" valign=\"top\">2</th>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>46</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      Number of records\n",
       "run evaluation_score                   \n",
       "1   3                                 1\n",
       "    4                                 3\n",
       "    5                                46\n",
       "2   3                                 1\n",
       "    4                                 3\n",
       "    5                                46"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Unit test results\n",
    "evaluation_df_pivot = pd.pivot_table(run_df,  values='format',index=['run','evaluation_score'], #columns='position',\n",
    "                          aggfunc='count')\n",
    "evaluation_df_pivot.columns = ['Number of records']\n",
    "evaluation_df_pivot"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "786515fa-6841-4820-98f9-aa29ae76cf76",
   "metadata": {},
   "source": [
    "#### Plotting the results\n",
    "\n",
    "We can create a simple bar chart to visualise the results of unit tests for both runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b2a18a78-55ec-43f6-9d62-929707a94364",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA0kAAAIjCAYAAADWYVDIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAABRtUlEQVR4nO3dd3gU5f7+8XvTNiFlQ2gBibQgIUBA4dCrgBTpVUQpcrChdIVYaAcpCoKFZgEE9YgiIHhERFRURAQFsQACRooQgpSEIklInt8f/rLfWZJAFpJsIO/Xde11Mc/MPvPZ3dkh987MMzZjjBEAAAAAQJLk5ekCAAAAAKAgISQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAGFjM1m04QJEzyy7i+++EI2m01ffPGFR9aPwiFjO1u+fLmnS8mRY8eOqUePHipWrJhsNptmz57t6ZLyzR9//CGbzabFixfnSn8ff/yxatWqJX9/f9lsNp0+fTpX+i1Iypcvrw4dOni6DOCGR0gCPGDx4sWy2WzZPr799ltPl3hN5s6dm2t/9OSW9PR0LVmyRPXq1VNYWJiCg4N1yy23qF+/ftf9++0JGduwv7+//vzzz0zzmzdvrurVq3ugsuvPiBEjtG7dOsXGxmrp0qVq27Zttsteuq8ICQlRs2bN9L///S8fKy6YTpw4oV69eikgIEBz5szR0qVLFRgYmGfru17342xDQM74eLoAoDCbNGmSKlSokKk9MjLSA9Xknrlz56p48eIaMGCAS3vTpk31999/y8/PL99rGjp0qObMmaPOnTurb9++8vHx0Z49e7R27VpVrFhR9evXz/eabgTJycmaNm2aXnrpJU+Xct367LPP1LlzZ40ePTpHy7du3Vr9+vWTMUYHDhzQvHnz1LFjR61du1Zt2rTJ42oLrq1bt+rMmTP6z3/+o1atWuXbeq/H/TjbEHBlhCTAg9q1a6c6dep4uox84+XlJX9//3xf77FjxzR37lwNHjxYr7zyisu82bNn6/jx4/lWy8WLF5Wenu6RoJgXatWqpVdffVWxsbEqU6aMp8vJV+fOncuVIxUJCQkKDQ3N8fK33HKL7rnnHud09+7dFR0drRdeeKFA/oF7/vx5FSlSJM/Xk5CQIEluvZdXkpPP+Hrcj19v2xDgCZxuBxRQqampCgsL08CBAzPNS0pKkr+/v/OX55SUFI0bN061a9eWw+FQYGCgmjRpos8///yK6xkwYIDKly+fqX3ChAmy2WwubYsWLdLtt9+ukiVLym63Kzo6WvPmzXNZpnz58vrll1+0ceNG5+kczZs3l5T9NUnvvfeeateurYCAABUvXlz33HNPplO4BgwYoKCgIP3555/q0qWLgoKCVKJECY0ePVppaWmXfY1xcXEyxqhRo0aZ5tlsNpUsWdKl7fTp0xoxYoTKly8vu92usmXLql+/fvrrr7+cyyQkJGjQoEEqVaqU/P39VbNmTb3xxhsu/WRcbzFjxgzNnj1blSpVkt1u16+//ipJ2r17t3r06KGwsDD5+/urTp06Wr16tUsfqampmjhxoipXrix/f38VK1ZMjRs31vr167N9vdu2bZPNZstUjyStW7dONptNH374oSTpzJkzGj58uPO1lixZUq1bt9YPP/xw2fc0wxNPPKG0tDRNmzbtsstd7tqTS6+Ty9j2fvvtN91zzz1yOBwqUaKEnn76aRljdOjQIXXu3FkhISEKDw/XzJkzs1xnWlqannjiCYWHhyswMFCdOnXSoUOHMi23ZcsWtW3bVg6HQ0WKFFGzZs20adMml2Uyavr111919913q2jRomrcuPFlX/Pvv/+unj17KiwsTEWKFFH9+vVdTmnKOF3LGKM5c+Y4vy/uqlq1qooXL679+/e7tCcnJ2v8+PGKjIyU3W5XRESEHn/8cSUnJ2fq480331TdunVVpEgRFS1aVE2bNtUnn3zisszcuXNVrVo12e12lSlTRkOGDMl0zU/GaZbff/+9mjZtqiJFiuiJJ56Q9M/3asCAAXI4HAoNDVX//v2zvGYoPj5eAwcOVNmyZWW321W6dGl17txZf/zxR7bvQfPmzdW/f39J0r/+9S/ZbDaXI9nu7GP279+v9u3bKzg4WH379s12ne6YMWOGGjZsqGLFiikgIEC1a9fO9pq5nHwWkvT111+rbt268vf3V8WKFbVkyZKrri+rbShj+7z0fc9qP57xuf/6669q0aKFihQpoptuuknPPvvsVdcEeBohCfCgxMRE/fXXXy6PEydOSJJ8fX3VtWtXrVq1SikpKS7PW7VqlZKTk3XXXXdJ+ic0vfbaa2revLmmT5+uCRMm6Pjx42rTpo127NiRa/XOmzdP5cqV0xNPPKGZM2cqIiJCDz/8sObMmeNcZvbs2SpbtqyioqK0dOlSLV26VE8++WS2fS5evFi9evWSt7e3pk6dqsGDB2vFihVq3Lhxpj+g0tLS1KZNGxUrVkwzZsxQs2bNNHPmzExHhy5Vrlw5Sf/8oXT+/PnLLnv27Fk1adJEL730ku644w698MILevDBB7V7924dPnxYkvT333+refPmWrp0qfr27avnnntODodDAwYM0AsvvJCpz0WLFumll17S/fffr5kzZyosLEy//PKL6tevr127dmns2LGaOXOmAgMD1aVLF61cudL53AkTJmjixIlq0aKFXn75ZT355JO6+eabLxti6tSpo4oVK+rdd9/NNG/ZsmUqWrSo89fiBx98UPPmzVP37t01d+5cjR49WgEBAdq1a9dl36cMFSpUUL9+/fTqq6/qyJEjOXpOTvXu3Vvp6emaNm2a6tWrp8mTJ2v27Nlq3bq1brrpJk2fPl2RkZEaPXq0vvzyy0zPf+aZZ/S///1PY8aM0dChQ7V+/Xq1atVKf//9t3OZzz77TE2bNlVSUpLGjx+vKVOm6PTp07r99tv13XffZeqzZ8+eOn/+vKZMmaLBgwdnW/uxY8fUsGFDrVu3Tg8//LCeeeYZXbhwQZ06dXJ+vk2bNtXSpUsl/XP6U8b3xV2JiYk6deqUihYt6mxLT09Xp06dNGPGDHXs2FEvvfSSunTpolmzZql3794uz584caLuvfde+fr6atKkSZo4caIiIiL02WefOZeZMGGChgwZojJlymjmzJnq3r27FixYoDvuuEOpqaku/Z04cULt2rVTrVq1NHv2bLVo0ULGGHXu3FlLly7VPffco8mTJ+vw4cPOYGPVvXt3rVy5UgMHDtTcuXM1dOhQnTlzRgcPHsz2PXjyySd1//33S/rn9LelS5fqgQcekOTePubixYtq06aNSpYsqRkzZqh79+45ev+z249neOGFF3Trrbdq0qRJmjJlinx8fNSzZ89M1wHl5LOQpH379qlHjx5q3bq1Zs6cqaJFi2rAgAH65Zdfrlhvdq/h0m3IXadOnVLbtm1Vs2ZNzZw5U1FRURozZozWrl171X0CHmUA5LtFixYZSVk+7Ha7c7l169YZSWbNmjUuz2/fvr2pWLGic/rixYsmOTnZZZlTp06ZUqVKmfvuu8+lXZIZP368c7p///6mXLlymWocP368uXQXcf78+UzLtWnTxqUWY4ypVq2aadasWaZlP//8cyPJfP7558YYY1JSUkzJkiVN9erVzd9//+1c7sMPPzSSzLhx41zqlGQmTZrk0uett95qateunWldl+rXr5+RZIoWLWq6du1qZsyYYXbt2pVpuXHjxhlJZsWKFZnmpaenG2OMmT17tpFk3nzzTee8lJQU06BBAxMUFGSSkpKMMcbExcUZSSYkJMQkJCS49NWyZUtTo0YNc+HCBZf+GzZsaCpXruxsq1mzprnzzjuv+PouFRsba3x9fc3JkyedbcnJySY0NNRlm3A4HGbIkCFu95+xDW/dutXs37/f+Pj4mKFDhzrnN2vWzFSrVs05nfFeLFq0KFNfl26TGdve/fff72y7ePGiKVu2rLHZbGbatGnO9lOnTpmAgADTv39/Z1vGdnbTTTc5PwtjjHn33XeNJPPCCy8YY/55vytXrmzatGnj/GyN+Wc7r1ChgmndunWmmvr06ZOj92f48OFGkvnqq6+cbWfOnDEVKlQw5cuXN2lpaS6vP6efgSQzaNAgc/z4cZOQkGC2bdtm2rZtaySZ5557zrnc0qVLjZeXl8v6jTFm/vz5RpLZtGmTMcaYvXv3Gi8vL9O1a1eXmoz5v+09ISHB+Pn5mTvuuMNlmZdfftlIMgsXLnS2NWvWzEgy8+fPd+lr1apVRpJ59tlnnW0XL140TZo0cdkuTp06lem15JR1m8xwNfuYsWPHurW+K+3Hjcm870xJSTHVq1c3t99+u7MtJ5+FMcaUK1fOSDJffvmlsy0hIcHY7XYzatSoK9ad020o4/XFxcW5PP/S/bgx//e5L1myxNmWnJxswsPDTffu3a9YE1AQcSQJ8KA5c+Zo/fr1Lg/rr2633367ihcvrmXLljnbTp06pfXr17v8Guzt7e28xiU9PV0nT57UxYsXVadOnRyfNpUTAQEBzn9n/HrarFkz/f7770pMTHS7v23btikhIUEPP/ywy7VKd955p6KiorIcbenBBx90mW7SpIl+//33K65r0aJFevnll1WhQgWtXLlSo0ePVtWqVdWyZUuX027ef/991axZU127ds3UR8apUB999JHCw8PVp08f5zxfX18NHTpUZ8+e1caNG12e1717d5UoUcI5ffLkSX322Wfq1auXzpw54/Lrc5s2bbR3715nTaGhofrll1+0d+/eK75Gq969eys1NVUrVqxwtn3yySc6ffq0y7YTGhqqLVu2XNNRoIoVK+ree+/VK6+8oqNHj151P5f697//7fy3t7e36tSpI2OMBg0a5GwPDQ1VlSpVstwG+vXrp+DgYOd0jx49VLp0aX300UeSpB07dmjv3r26++67deLECefncO7cObVs2VJffvml0tPTXfq8dPvLzkcffaS6deu6nJIXFBSk+++/X3/88YfzlMur8frrr6tEiRIqWbKk6tSpow0bNujxxx/XyJEjncu89957qlq1qqKiolyOcNx+++2S5DwVd9WqVUpPT9e4cePk5eX6J0HG9v7pp58qJSVFw4cPd1lm8ODBCgkJyfQ9tdvtmU4T/uijj+Tj46OHHnrI2ebt7a1HH33UZbmAgAD5+fnpiy++0KlTp672LXK6mn2MtcacuNJ+XHLdd546dUqJiYlq0qSJy/45J59FhujoaDVp0sQ5XaJEiWy/B1nJyTbkrqCgIJfrnPz8/FS3bt0c1wQUNAzcAHhQ3bp1L3vBr4+Pj7p37663335bycnJstvtWrFihVJTUzOdMvPGG29o5syZ2r17t8vpL1mNunS1Nm3apPHjx2vz5s2ZTltLTEyUw+Fwq78DBw5IkqpUqZJpXlRUlL7++muXNn9/f5ewIUlFixbN0R9TXl5eGjJkiIYMGaITJ05o06ZNmj9/vtauXau77rpLX331lSRp//79VzzF5sCBA6pcuXKmP2SqVq3q8royXPoZ7Nu3T8YYPf3003r66aezXEdCQoJuuukmTZo0SZ07d9Ytt9yi6tWrq23btrr33nsVExNz2Rpr1qypqKgoLVu2zBkqli1bpuLFizv/UJakZ599Vv3791dERIRq166t9u3bq1+/fqpYseJl+7/UU089paVLl2ratGlZnnJ4NW6++WaXaYfDIX9/fxUvXjxT+6WnN0lS5cqVXaZtNpsiIyOd11hkBM+sTvnKkJiY6HIKUk6/TwcOHFC9evUytVu3kasdIr1z58565JFHlJKSoq1bt2rKlCk6f/68y/a4d+9e7dq1K9P3JUPGIAf79++Xl5eXoqOjL/tapMzfUz8/P1WsWDHT9n7TTTdlGpjkwIEDKl26tIKCglzaL+3Tbrdr+vTpGjVqlEqVKqX69eurQ4cO6tevn8LDw7Ot0d3apaz3MT4+Pipbtqxb67jSflySPvzwQ02ePFk7duxwuSbMGn5y8llkuPS7IeV8XyjlbBtyV9myZTOFuaJFi2rnzp1X3SfgSYQkoIC76667tGDBAq1du1ZdunTRu+++q6ioKNWsWdO5zJtvvqkBAwaoS5cueuyxx1SyZEnn+feXXsx9qewuFL90MIT9+/erZcuWioqK0vPPP6+IiAj5+fnpo48+0qxZszL94p4XvL29c6WfYsWKqVOnTurUqZOaN2+ujRs36sCBA85rl3Kb9VdkSc73avTo0dmOJJUxfHDTpk21f/9+ffDBB/rkk0/02muvadasWZo/f77LkZas9O7dW88884z++usvBQcHa/Xq1erTp498fP5v19+rVy81adJEK1eu1CeffKLnnntO06dP14oVK9SuXbscv8aKFSvqnnvu0SuvvKKxY8dmmp/T7cwqq887u23AGJPDSv9Pxufw3HPPqVatWlkuc+kf9Zd+lp5QtmxZ5xDX7du3V/HixfXII4+oRYsW6tatm6R/XluNGjX0/PPPZ9lHREREntV3re/R8OHD1bFjR61atUrr1q3T008/ralTp+qzzz7TrbfemktVZs1ut19TUMjKV199pU6dOqlp06aaO3euSpcuLV9fXy1atEhvv/32VfV5rd+DnGxD7n5nc/O7CRQEhCSggGvatKlKly6tZcuWqXHjxvrss88yDYSwfPlyVaxYUStWrHD5j238+PFX7L9o0aJZjjB16a/Da9asUXJyslavXu3yK2ZWI+jldISujFCyZ88el6MbGW15FVqs6tSpo40bN+ro0aMqV66cKlWqpJ9//vmyzylXrpx27typ9PR0lz+odu/e7Zx/ORlHaXx9fXN0P5eMUQ4HDhyos2fPqmnTppowYUKOQtLEiRP1/vvvq1SpUkpKSnIO9mFVunRpPfzww3r44YeVkJCg2267Tc8884xbIUn652jSm2++qenTp2eal3E05tJt7dLtLDddeoqiMUb79u1zHoWrVKmSJCkkJCTX76tTrlw57dmzJ1N7TrcRdzzwwAOaNWuWnnrqKXXt2lU2m02VKlXSjz/+qJYtW172+1ipUiWlp6fr119/zTYoWr+n1iOMKSkpiouLy9F7V65cOW3YsEFnz551CZ5ZvUcZdY0aNUqjRo3S3r17VatWLc2cOVNvvvnmFdeVXe2e2se8//778vf317p162S3253tixYtclkuJ59FXslqG/LEdxYoSLgmCSjgvLy81KNHD61Zs0ZLly7VxYsXM51ql/ELnvUXuy1btmjz5s1X7L9SpUpKTEx0OSXi6NGjLiOsZbeOxMTETP/RS1JgYGCWwetSderUUcmSJTV//nyXU1DWrl2rXbt26c4777xiHzkRHx+f5TUgKSkp2rBhg7y8vJxHbrp3764ff/wx0+uX/u+1t2/fXvHx8S7Xil28eFEvvfSSgoKC1KxZs8vWU7JkSTVv3lwLFizI8hoe632bLj2NLCgoSJGRkVkO43ypqlWrqkaNGlq2bJmWLVum0qVLq2nTps75aWlpma4lK1mypMqUKZOj/i9VqVIl3XPPPVqwYIHi4+Nd5oWEhKh48eKZRqGbO3eu2+vJqSVLlujMmTPO6eXLl+vo0aPO8Fe7dm1VqlRJM2bM0NmzZzM9/1run9W+fXt99913Lt/Bc+fO6ZVXXlH58uVzdEpVTvn4+GjUqFHatWuXPvjgA0n/HCH8888/9eqrr2Za/u+//9a5c+ckSV26dJGXl5cmTZqU6WhwxvbeqlUr+fn56cUXX3T5/r/++utKTEzM0fe0ffv2unjxosstA9LS0jLdhPj8+fO6cOGCS1ulSpUUHBx8Vdtkfu1jLsfb21s2m83lCMwff/yhVatWuSyXk88ir2S1DWX8iGD9zqalpV1xNFHgRsGRJMCD1q5d6/xl2aphw4Yuv9j27t1bL730ksaPH68aNWo4r2vI0KFDB61YsUJdu3bVnXfeqbi4OM2fP1/R0dFZ/vFnddddd2nMmDHq2rWrhg4dqvPnz2vevHm65ZZbXC4qvuOOO+Tn56eOHTvqgQce0NmzZ/Xqq6+qZMmSmf7Qr127tubNm6fJkycrMjJSJUuWzPQrrvTPkZTp06dr4MCBatasmfr06aNjx47phRdeUPny5TVixIgcvY9XcvjwYdWtW1e33367WrZsqfDwcCUkJOi///2vfvzxRw0fPtx5nctjjz2m5cuXq2fPnrrvvvtUu3ZtnTx5UqtXr9b8+fNVs2ZN3X///VqwYIEGDBig77//XuXLl9fy5cu1adMmzZ4922WwgOzMmTNHjRs3Vo0aNTR48GBVrFhRx44d0+bNm3X48GH9+OOPkv65QLt58+aqXbu2wsLCtG3bNi1fvlyPPPJIjl577969NW7cOPn7+2vQoEEuR77OnDmjsmXLqkePHqpZs6aCgoL06aefauvWrdnee+hKnnzySS1dulR79uxRtWrVXOb9+9//1rRp0/Tvf/9bderU0ZdffqnffvvtqtaTE2FhYWrcuLEGDhyoY8eOafbs2YqMjHQO3e3l5aXXXntN7dq1U7Vq1TRw4EDddNNN+vPPP/X5558rJCREa9asuap1jx07Vv/973/Vrl07DR06VGFhYXrjjTcUFxen999/P9dP6RowYIDGjRun6dOnq0uXLrr33nv17rvv6sEHH9Tnn3+uRo0aKS0tTbt379a7776rdevWqU6dOoqMjNSTTz6p//znP2rSpIm6desmu92urVu3qkyZMpo6dapKlCih2NhYTZw4UW3btlWnTp20Z88ezZ07V//6179cLtbPTseOHdWoUSONHTtWf/zxh6Kjo7VixYpMIf23335Ty5Yt1atXL0VHR8vHx0crV67UsWPHsjwKeiX5sY+50n78zjvv1PPPP6+2bdvq7rvvVkJCgubMmaPIyEiXH6dy8lnkpUu3oWrVqql+/fqKjY3VyZMnFRYWpnfeeUcXL17M0zqAAsMzg+oBhdvlho5VFsMkp6enm4iICCPJTJ48OVN/6enpZsqUKaZcuXLGbrebW2+91Xz44YdZDu+tS4ZbNsaYTz75xFSvXt34+fmZKlWqmDfffDPLIcBXr15tYmJijL+/vylfvryZPn26WbhwYaZhYuPj482dd95pgoODjSTncOBZDR1rjDHLli0zt956q7Hb7SYsLMz07dvXHD582GWZ/v37m8DAwEyvPas6L5WUlGReeOEF06ZNG1O2bFnj6+trgoODTYMGDcyrr77qMryuMcacOHHCPPLII+amm24yfn5+pmzZsqZ///7mr7/+ci5z7NgxM3DgQFO8eHHj5+dnatSokelzyxj2OrvhjPfv32/69etnwsPDja+vr7nppptMhw4dzPLly53LTJ482dStW9eEhoaagIAAExUVZZ555hmTkpJy2decYe/evc7t6uuvv3aZl5ycbB577DFTs2ZNExwcbAIDA03NmjXN3Llzr9hvVsMtZ8gYStk6BLgx/wyDPGjQIONwOExwcLDp1auXSUhIyHYI8OPHj2fqN6tt4NLhxjO2s//+978mNjbWlCxZ0gQEBJg777zTHDhwINPzt2/fbrp162aKFStm7Ha7KVeunOnVq5fZsGHDFWu6nP3795sePXqY0NBQ4+/vb+rWrWs+/PDDTMvJzSHAs1t2woQJmYbYnz59uqlWrZqx2+2maNGipnbt2mbixIkmMTHR5bkLFy50fgeLFi1qmjVrZtavX++yzMsvv2yioqKMr6+vKVWqlHnooYfMqVOnXJa59LOwOnHihLn33ntNSEiIcTgc5t577zXbt2932ef99ddfZsiQISYqKsoEBgYah8Nh6tWrZ959990rvjeX2yavZR9zpfXlZD/++uuvm8qVKxu73W6ioqLMokWLst13XemzKFeuXJa3BWjWrFmWt164lDvb0P79+02rVq2M3W43pUqVMk888YRZv359lkOAZ/W5Z3eLCeB6YDOGK+oAAAAAIAPXJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwOKGv5lsenq6jhw5ouDgYNlsNk+XAwAAAMBDjDE6c+aMypQpc9kbe9/wIenIkSOKiIjwdBkAAAAACohDhw6pbNmy2c6/4UNScHCwpH/eiJCQEA9XAwAAAMBTkpKSFBER4cwI2bnhQ1LGKXYhISGEJAAAAABXvAyHgRsAAAAAwIKQBAAAAAAWhCQAAAAAsLjhr0nKCWOMLl68qLS0NE+XcsPx9vaWj48Pw68DAADgulHoQ1JKSoqOHj2q8+fPe7qUG1aRIkVUunRp+fn5eboUAAAA4IoKdUhKT09XXFycvL29VaZMGfn5+XHEIxcZY5SSkqLjx48rLi5OlStXvuxNuwAAAICCoFCHpJSUFKWnpysiIkJFihTxdDk3pICAAPn6+urAgQNKSUmRv7+/p0sCAAAALouf9SWObuQx3l8AAABcT/jrFQAAAAAsCEkAAAAAYFGor0nKjm3nmnxdn4npmK/rAwAAAJA9jiRdhwYMGCCbzSabzSZfX19VqFBBjz/+uC5cuJDn616xYoXuuOMOFStWTDabTTt27MjzdQIAAAD5iZB0nWrbtq2OHj2q33//XbNmzdKCBQs0fvz4PF/vuXPn1LhxY02fPj3P1wUAAAB4AiHpOmW32xUeHq6IiAh16dJFrVq10vr1653zy5cvr9mzZ7s8p1atWpowYYJz2maz6bXXXlPXrl1VpEgRVa5cWatXr77seu+9916NGzdOrVq1ys2XAwAAABQYhKQbwM8//6xvvvlGfn5+bj934sSJ6tWrl3bu3Kn27durb9++OnnyZB5UCQAAAFwfCEnXqQ8//FBBQUHy9/dXjRo1lJCQoMcee8ztfgYMGKA+ffooMjJSU6ZM0dmzZ/Xdd9/lQcUAAADA9YHR7a5TLVq00Lx583Tu3DnNmjVLPj4+6t69u9v9xMTEOP8dGBiokJAQJSQk5GapAAAAwHWFI0nXqcDAQEVGRqpmzZpauHChtmzZotdff90538vLS8YYl+ekpqZm6sfX19dl2mazKT09PW+KBgAAAK4DhKQbgJeXl5544gk99dRT+vvvvyVJJUqU0NGjR53LJCUlKS4uzlMlAgAAANcNTre7QfTs2VOPPfaY5syZo9GjR+v222/X4sWL1bFjR4WGhmrcuHHy9va+5vWcPHlSBw8e1JEjRyRJe/bskSSFh4crPDz8mvsHAKAwy+8b2hdqNTt5uoJC5dIznAo6QlIWTExHT5fgNh8fHz3yyCN69tln9dBDDyk2NlZxcXHq0KGDHA6H/vOf/+TKkaTVq1dr4MCBzum77rpLkjR+/HiX4cUBAACA65XNXG+xzk1JSUlyOBxKTExUSEiIy7wLFy4oLi5OFSpUkL+/v4cqvPHxPgMAkDMcScpHHEnKVwUlclwuG1hxTRIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABY+Hi6gILIZrPl6/oKyh2IAQAAAHAk6bo0YMAA2Ww22Ww2+fr6qkKFCnr88cd14cKFPF1vamqqxowZoxo1aigwMFBlypRRv379dOTIkTxdLwAAAJCfCEnXqbZt2+ro0aP6/fffNWvWLC1YsEDjx4/P03WeP39eP/zwg55++mn98MMPWrFihfbs2aNOnTrl6XoBAACA/ERIuk7Z7XaFh4crIiJCXbp0UatWrbR+/Xrn/PLly2v27Nkuz6lVq5YmTJjgnLbZbHrttdfUtWtXFSlSRJUrV9bq1auzXafD4dD69evVq1cvValSRfXr19fLL7+s77//XgcPHsztlwgAAAB4BCHpBvDzzz/rm2++kZ+fn9vPnThxonr16qWdO3eqffv26tu3r06ePJnj5ycmJspmsyk0NNTtdQMAAAAFESHpOvXhhx8qKChI/v7+qlGjhhISEvTYY4+53c+AAQPUp08fRUZGasqUKTp79qy+++67HD33woULGjNmjPr06aOQkBC31w0AAAAURIxud51q0aKF5s2bp3PnzmnWrFny8fFR9+7d3e4nJibG+e/AwECFhIQoISHhis9LTU1Vr169ZIzRvHnz3F4vAAAAUFBxJOk6FRgYqMjISNWsWVMLFy7Uli1b9Prrrzvne3l5ZRpaPDU1NVM/vr6+LtM2m03p6emXXXdGQDpw4IDWr1/PUSQAAADcUAhJNwAvLy898cQTeuqpp/T3339LkkqUKKGjR486l0lKSlJcXNw1rysjIO3du1effvqpihUrds19AgAAAAUJIekG0bNnT3l7e2vOnDmSpNtvv11Lly7VV199pZ9++kn9+/eXt7f3Na0jNTVVPXr00LZt2/TWW28pLS1N8fHxio+PV0pKSm68DAAAAMDjuCYpC5eepnY98PHx0SOPPKJnn31WDz30kGJjYxUXF6cOHTrI4XDoP//5zzUfSfrzzz+dQ4TXqlXLZd7nn3+u5s2bX1P/AAAAQEFgM9djInBDUlKSHA6HEhMTM107c+HCBcXFxalChQry9/f3UIU3Pt5nAAByxrZzjadLKDxqdvJ0BYVKQYkcl8sGVpxuBwAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJKjgXkt2oeH8BAABwPSnUIcnX11eSdP78eQ9XcmPLeH8z3m8AAACgICvU90ny9vZWaGioEhISJElFihSRzWbzcFU3DmOMzp8/r4SEBIWGhl7zzWwBAACA/FCoQ5IkhYeHS5IzKCH3hYaGOt9nAAAAoKAr9CHJZrOpdOnSKlmypFJTUz1dzg3H19eXI0gAAAC4rhT6kJTB29ubP+YBAAAAFO6BGwAAAADgUoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgEWBCUnTpk2TzWbT8OHDnW0XLlzQkCFDVKxYMQUFBal79+46duyY54oEAAAAcMMrECFp69atWrBggWJiYlzaR4wYoTVr1ui9997Txo0bdeTIEXXr1s1DVQIAAAAoDDweks6ePau+ffvq1VdfVdGiRZ3tiYmJev311/X888/r9ttvV+3atbVo0SJ98803+vbbbz1YMQAAAIAbmcdD0pAhQ3TnnXeqVatWLu3ff/+9UlNTXdqjoqJ08803a/Pmzdn2l5ycrKSkJJcHAAAAAOSUjydX/s477+iHH37Q1q1bM82Lj4+Xn5+fQkNDXdpLlSql+Pj4bPucOnWqJk6cmNulAgAAACgkPHYk6dChQxo2bJjeeust+fv751q/sbGxSkxMdD4OHTqUa30DAAAAuPF5LCR9//33SkhI0G233SYfHx/5+Pho48aNevHFF+Xj46NSpUopJSVFp0+fdnnesWPHFB4enm2/drtdISEhLg8AAAAAyCmPnW7XsmVL/fTTTy5tAwcOVFRUlMaMGaOIiAj5+vpqw4YN6t69uyRpz549OnjwoBo0aOCJkgEAAAAUAh4LScHBwapevbpLW2BgoIoVK+ZsHzRokEaOHKmwsDCFhITo0UcfVYMGDVS/fn1PlAwAAACgEPDowA1XMmvWLHl5eal79+5KTk5WmzZtNHfuXE+XBQAAAOAGZjPGGE8XkZeSkpLkcDiUmJjI9UkAAKBAs+1c4+kSCo+anTxdQaFSUCJHTrOBx++TBAAAAAAFCSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsHA7JP399986f/68c/rAgQOaPXu2Pvnkk1wtDAAAAAA8we2Q1LlzZy1ZskSSdPr0adWrV08zZ85U586dNW/evFwvEAAAAADyk9sh6YcfflCTJk0kScuXL1epUqV04MABLVmyRC+++GKuFwgAAAAA+cntkHT+/HkFBwdLkj755BN169ZNXl5eql+/vg4cOJDrBQIAAABAfnI7JEVGRmrVqlU6dOiQ1q1bpzvuuEOSlJCQoJCQkFwvEAAAAADyk9shady4cRo9erTKly+vevXqqUGDBpL+Oap06623utXXvHnzFBMTo5CQEIWEhKhBgwZau3atc/6FCxc0ZMgQFStWTEFBQerevbuOHTvmbskAAAAAkGM2Y4xx90nx8fE6evSoatasKS+vf3LWd999p5CQEEVFReW4nzVr1sjb21uVK1eWMUZvvPGGnnvuOW3fvl3VqlXTQw89pP/9739avHixHA6HHnnkEXl5eWnTpk05XkdSUpIcDocSExM50gUAAAo02841ni6h8KjZydMVFCpXETnyRE6zwVWFpLwUFham5557Tj169FCJEiX09ttvq0ePHpKk3bt3q2rVqtq8ebPq16+fo/4ISQAA4HpBSMpHhKR8VVAiR06zgU9OOuvWrVuOV7xixYocL2uVlpam9957T+fOnVODBg30/fffKzU1Va1atXIuExUVpZtvvvmyISk5OVnJycnO6aSkpKuqBwAAAEDhlKNrkhwOh/MREhKiDRs2aNu2bc7533//vTZs2CCHw+F2AT/99JOCgoJkt9v14IMPauXKlYqOjlZ8fLz8/PwUGhrqsnypUqUUHx+fbX9Tp051qTciIsLtmgAAAAAUXjk6krRo0SLnv8eMGaNevXpp/vz58vb2lvTPUaCHH374qk5nq1Klinbs2KHExEQtX75c/fv318aNG93uJ0NsbKxGjhzpnE5KSiIoAQAAAMixHIUkq4ULF+rrr792BiRJ8vb21siRI9WwYUM999xzbvXn5+enyMhISVLt2rW1detWvfDCC+rdu7dSUlJ0+vRpl6NJx44dU3h4eLb92e122e12914UAAAAAPx/bg8BfvHiRe3evTtT++7du5Wenn7NBaWnpys5OVm1a9eWr6+vNmzY4Jy3Z88eHTx40DnsOAAAAADkNrePJA0cOFCDBg3S/v37VbduXUnSli1bNG3aNA0cONCtvmJjY9WuXTvdfPPNOnPmjN5++2198cUXWrdunRwOhwYNGqSRI0cqLCxMISEhevTRR9WgQYMcj2wHAAAAAO5yOyTNmDFD4eHhmjlzpo4ePSpJKl26tB577DGNGjXKrb4SEhLUr18/HT16VA6HQzExMVq3bp1at24tSZo1a5a8vLzUvXt3JScnq02bNpo7d667JQMAAABAjrl1n6SLFy/q7bffVps2bVSqVCnn8NoF+f5D3CcJAABcL7hPUj7iPkn56nq7T5Jb1yT5+PjowQcf1IULFyT9E44IHgAAAABuJG4P3FC3bl1t3749L2oBAAAAAI9z+5qkhx9+WKNGjdLhw4dVu3ZtBQYGusyPiYnJteIAAAAAIL+5HZLuuusuSdLQoUOdbTabTcYY2Ww2paWl5V51AAAAAJDP3A5JcXFxeVEHAAAAABQIboekcuXK5UUdAAAAAFAguB2SJGn//v2aPXu2du3aJUmKjo7WsGHDVKlSpVwtDgAAAADym9uj261bt07R0dH67rvvFBMTo5iYGG3ZskXVqlXT+vXr86JGAAAAAMg3bh9JGjt2rEaMGKFp06Zlah8zZoxat26da8UBAAAAQH5z+0jSrl27NGjQoEzt9913n3799ddcKQoAAAAAPMXtkFSiRAnt2LEjU/uOHTtUsmTJ3KgJAAAAADzG7dPtBg8erPvvv1+///67GjZsKEnatGmTpk+frpEjR+Z6gQAAAACQn9wOSU8//bSCg4M1c+ZMxcbGSpLKlCmjCRMmuNxgFgAAAACuRzZjjLnaJ585c0aSFBwcnGsF5bakpCQ5HA4lJiYqJCTE0+UAAABky7ZzjadLKDxqdvJ0BYXKNUSOXJXTbOD2kaS4uDhdvHhRlStXdglHe/fula+vr8qXL39VBQMAAABAQeD2wA0DBgzQN998k6l9y5YtGjBgQG7UBAAAAAAe43ZI2r59uxo1apSpvX79+lmOegcAAAAA1xO3Q5LNZnNei2SVmJiotLS0XCkKAAAAADzF7ZDUtGlTTZ061SUQpaWlaerUqWrcuHGuFgcAAAAA+c3tgRumT5+upk2bqkqVKmrSpIkk6auvvlJSUpI+++yzXC8QAAAAAPKT20eSoqOjtXPnTvXq1UsJCQk6c+aM+vXrp927d6t69ep5USMAAAAA5Bu3jyRJ/9w8dsqUKbldCwAAAAB4nNtHkqR/Tq+755571LBhQ/3555+SpKVLl+rrr7/O1eIAAAAAIL+5HZLef/99tWnTRgEBAfrhhx+UnJws6Z/R7Ti6BAAAAOB653ZImjx5subPn69XX31Vvr6+zvZGjRrphx9+yNXiAAAAACC/uR2S9uzZo6ZNm2ZqdzgcOn36dG7UBAAAAAAe43ZICg8P1759+zK1f/3116pYsWKuFAUAAAAAnuJ2SBo8eLCGDRumLVu2yGaz6ciRI3rrrbc0evRoPfTQQ3lRIwAAAADkG7eHAB87dqzS09PVsmVLnT9/Xk2bNpXdbtfo0aP16KOP5kWNAAAAAJBvbMYYczVPTElJ0b59+3T27FlFR0crKChIf//9twICAnK7xmuSlJQkh8OhxMREhYSEeLocAACAbNl2rvF0CYVHzU6erqBQucrIketymg2u6j5JkuTn56fo6GjVrVtXvr6+ev7551WhQoWr7Q4AAAAACoQch6Tk5GTFxsaqTp06atiwoVatWiVJWrRokSpUqKBZs2ZpxIgReVUnAAAAAOSLHF+TNG7cOC1YsECtWrXSN998o549e2rgwIH69ttv9fzzz6tnz57y9vbOy1oBAAAAIM/lOCS99957WrJkiTp16qSff/5ZMTExunjxon788UfZbLa8rBEAAAAA8k2OT7c7fPiwateuLUmqXr267Ha7RowYQUACAAAAcEPJcUhKS0uTn5+fc9rHx0dBQUF5UhQAAAAAeEqOT7czxmjAgAGy2+2SpAsXLujBBx9UYGCgy3IrVqzI3QoBAAAAIB/lOCT179/fZfqee+7J9WIAAAAAwNNyHJIWLVqUl3UAAAAAQIFw1TeTBQAAAIAbESEJAAAAACwISQAAAABgQUgCAAAAAIschaTbbrtNp06dkiRNmjRJ58+fz9OiAAAAAMBTchSSdu3apXPnzkmSJk6cqLNnz+ZpUQAAAADgKTkaArxWrVoaOHCgGjduLGOMZsyYoaCgoCyXHTduXK4WCAAAAAD5KUchafHixRo/frw+/PBD2Ww2rV27Vj4+mZ9qs9kISQAAAACuazkKSVWqVNE777wjSfLy8tKGDRtUsmTJPC0MAAAAADwhRyHJKj09PS/qAAAAAIACwe2QJEn79+/X7NmztWvXLklSdHS0hg0bpkqVKuVqcQAAAACQ39y+T9K6desUHR2t7777TjExMYqJidGWLVtUrVo1rV+/Pi9qBAAAAIB84/aRpLFjx2rEiBGaNm1apvYxY8aodevWuVYcAAAAAOQ3t48k7dq1S4MGDcrUft999+nXX3/NlaIAAAAAwFPcDkklSpTQjh07MrXv2LGDEe8AAAAAXPfcPt1u8ODBuv/++/X777+rYcOGkqRNmzZp+vTpGjlyZK4XCAAAAAD5ye2Q9PTTTys4OFgzZ85UbGysJKlMmTKaMGGChg4dmusFAgAAAEB+shljzNU++cyZM5Kk4ODgXCsotyUlJcnhcCgxMVEhISGeLgcAACBbtp1rPF1C4VGzk6crKFSuIXLkqpxmg6u6T1KGghyOAAAAAOBquD1wAwAAAADcyAhJAAAAAGBBSAIAAAAAC7dCUmpqqlq2bKm9e/fmVT0AAAAA4FFuhSRfX1/t3Lkzr2oBAAAAAI9z+3S7e+65R6+//npe1AIAAAAAHuf2EOAXL17UwoUL9emnn6p27doKDAx0mf/888/nWnEAAAAAkN/cDkk///yzbrvtNknSb7/95jLPZrPlTlUAAAAA4CFuh6TPP/88L+oAAAAAgALhqocA37dvn9atW6e///5bkmSMybWiAAAAAMBT3A5JJ06cUMuWLXXLLbeoffv2Onr0qCRp0KBBGjVqVK4XCAAAAAD5ye2QNGLECPn6+urgwYMqUqSIs7137976+OOPc7U4AAAAAMhvbl+T9Mknn2jdunUqW7asS3vlypV14MCBXCsMAAAAADzB7SNJ586dczmClOHkyZOy2+25UhQAAAAAeIrbIalJkyZasmSJc9pmsyk9PV3PPvusWrRokavFAQAAAEB+c/t0u2effVYtW7bUtm3blJKSoscff1y//PKLTp48qU2bNuVFjQAAAACQb9w+klS9enX99ttvaty4sTp37qxz586pW7du2r59uypVqpQXNQIAAABAvnH7SJIkORwOPfnkk7ldCwAAAAB43FWFpFOnTun111/Xrl27JEnR0dEaOHCgwsLCcrU4AAAAAMhvbp9u9+WXX6p8+fJ68cUXderUKZ06dUovvviiKlSooC+//DIvagQAAACAfOP2kaQhQ4aod+/emjdvnry9vSVJaWlpevjhhzVkyBD99NNPuV4kAAAAAOQXt48k7du3T6NGjXIGJEny9vbWyJEjtW/fvlwtDgAAAADym9sh6bbbbnNei2S1a9cu1axZM1eKAgAAAABPydHpdjt37nT+e+jQoRo2bJj27dun+vXrS5K+/fZbzZkzR9OmTcubKgEAAAAgn9iMMeZKC3l5eclms+lKi9psNqWlpeVacbkhKSlJDodDiYmJCgkJ8XQ5AAAA2bLtXOPpEgqPmp08XUGhkoPIkS9ymg1ydCQpLi4u1woDAAAAgIIsRyGpXLlyeV0HAAAAABQIV3Uz2SNHjujrr79WQkKC0tPTXeYNHTo0x/1MnTpVK1as0O7duxUQEKCGDRtq+vTpqlKlinOZCxcuaNSoUXrnnXeUnJysNm3aaO7cuSpVqtTVlA4AAAAAl+V2SFq8eLEeeOAB+fn5qVixYrLZbM55NpvNrZC0ceNGDRkyRP/617908eJFPfHEE7rjjjv066+/KjAwUJI0YsQI/e9//9N7770nh8OhRx55RN26ddOmTZvcLR0AAAAArihHAzdYRURE6MEHH1RsbKy8vNweQfyyjh8/rpIlS2rjxo1q2rSpEhMTVaJECb399tvq0aOHJGn37t2qWrWqNm/e7Bxdzyo5OVnJycnO6aSkJEVERDBwAwAAKPAYuCEfMXBDvrreBm5wO+WcP39ed911V64HJElKTEyUJIWFhUmSvv/+e6WmpqpVq1bOZaKionTzzTdr8+bNWfYxdepUORwO5yMiIiLX6wQAAABw43I76QwaNEjvvfderheSnp6u4cOHq1GjRqpevbokKT4+Xn5+fgoNDXVZtlSpUoqPj8+yn9jYWCUmJjofhw4dyvVaAQAAANy43L4maerUqerQoYM+/vhj1ahRQ76+vi7zn3/++asqZMiQIfr555/19ddfX9XzM9jtdtnt9mvqAwAAAEDhdVUhad26dc4R6C4duOFqPPLII/rwww/15ZdfqmzZss728PBwpaSk6PTp0y5Hk44dO6bw8PCrWhcAAAAAXI7bIWnmzJlauHChBgwYcM0rN8bo0Ucf1cqVK/XFF1+oQoUKLvNr164tX19fbdiwQd27d5ck7dmzRwcPHlSDBg2uef0AAAAAcCm3Q5LdblejRo1yZeVDhgzR22+/rQ8++EDBwcHO64wcDocCAgLkcDg0aNAgjRw5UmFhYQoJCdGjjz6qBg0aZDmyHQAAAABcK7cHbhg2bJheeumlXFn5vHnzlJiYqObNm6t06dLOx7Jly5zLzJo1Sx06dFD37t3VtGlThYeHa8WKFbmyfgAAAAC4lNv3Seratas+++wzFStWTNWqVcs0cENBCzA5HQsdAADA07hPUj7iPkn56nq7T5Lbp9uFhoaqW7du11QcAAAAABRUboekRYsW5UUdAAAAAFAguH1NEgAAAADcyNw+klShQoXL3g/p999/v6aCAAAAAMCT3A5Jw4cPd5lOTU3V9u3b9fHHH+uxxx7LrboAAAAAwCPcDknDhg3Lsn3OnDnatm3bNRcEAAAAAJ6Ua9cktWvXTu+//35udQcAAAAAHpFrIWn58uUKCwvLre4AAAAAwCPcPt3u1ltvdRm4wRij+Ph4HT9+XHPnzs3V4gAAAAAgv7kdkrp06eIy7eXlpRIlSqh58+aKiorKrboAAAAAwCPcDknjx4/PizoAAAAAoEDgZrIAAAAAYJHjI0leXl6XvYmsJNlsNl28ePGaiwIAAAAAT8lxSFq5cmW28zZv3qwXX3xR6enpuVIUAAAAAHhKjkNS586dM7Xt2bNHY8eO1Zo1a9S3b19NmjQpV4sDAAAAgPx2VdckHTlyRIMHD1aNGjV08eJF7dixQ2+88YbKlSuX2/UBAAAAQL5yKyQlJiZqzJgxioyM1C+//KINGzZozZo1ql69el7VBwAAAAD5Ksen2z377LOaPn26wsPD9d///jfL0+8AAAAA4HpnM8aYnCzo5eWlgIAAtWrVSt7e3tkut2LFilwrLjckJSXJ4XAoMTFRISEhni4HAAAgW7adazxdQuFRs5OnKyhUchg58lxOs0GOjyT169fvikOAAwAAAMD1LschafHixXlYBgAAAAAUDFc1uh0AAAAA3KgISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACw8GhI+vLLL9WxY0eVKVNGNptNq1atcplvjNG4ceNUunRpBQQEqFWrVtq7d69nigUAAABQKHg0JJ07d041a9bUnDlzspz/7LPP6sUXX9T8+fO1ZcsWBQYGqk2bNrpw4UI+VwoAAACgsPDx5MrbtWundu3aZTnPGKPZs2frqaeeUufOnSVJS5YsUalSpbRq1Srddddd+VkqAAAAgEKiwF6TFBcXp/j4eLVq1crZ5nA4VK9ePW3evDnb5yUnJyspKcnlAQAAAAA5VWBDUnx8vCSpVKlSLu2lSpVyzsvK1KlT5XA4nI+IiIg8rRMAAADAjaXAhqSrFRsbq8TEROfj0KFDni4JAAAAwHWkwIak8PBwSdKxY8dc2o8dO+aclxW73a6QkBCXBwAAAADkVIENSRUqVFB4eLg2bNjgbEtKStKWLVvUoEEDD1YGAAAA4Ebm0dHtzp49q3379jmn4+LitGPHDoWFhenmm2/W8OHDNXnyZFWuXFkVKlTQ008/rTJlyqhLly6eKxoAAADADc2jIWnbtm1q0aKFc3rkyJGSpP79+2vx4sV6/PHHde7cOd1///06ffq0GjdurI8//lj+/v6eKhkAAADADc5mjDGeLiIvJSUlyeFwKDExkeuTAABAgWbbucbTJRQeNTt5uoJCpaBEjpxmgwJ7TRIAAAAAeAIhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAsCEkAAAAAYEFIAgAAAAALQhIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAsfTxcAAIWZbecaT5dQeNTs5OkKChVjjKdLAICrxpEkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAAAABYEJIAAAAAwIKQBAAAAAAWhCQAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgIWPpwsobGw713i6hMKjZidPV1CoGGM8XQIAAECu4EgSAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgMV1EZLmzJmj8uXLy9/fX/Xq1dN3333n6ZIAAAAA3KAKfEhatmyZRo4cqfHjx+uHH35QzZo11aZNGyUkJHi6NAAAAAA3oAIfkp5//nkNHjxYAwcOVHR0tObPn68iRYpo4cKFni4NAAAAwA2oQN9MNiUlRd9//71iY2OdbV5eXmrVqpU2b96c5XOSk5OVnJzsnE5MTJQkJSUl5W2xOXX2vKcrAPJEgfmOXW/YJ+AGxT7hKrFPwA2qoOwTMuowxlx2uQIdkv766y+lpaWpVKlSLu2lSpXS7t27s3zO1KlTNXHixEztEREReVIjgH84HA5PlwCgAGGfAMCqoO0Tzpw5c9maCnRIuhqxsbEaOXKkczo9PV0nT55UsWLFZLPZPFgZ8lNSUpIiIiJ06NAhhYSEeLocAB7GPgGAFfuEwssYozNnzqhMmTKXXa5Ah6TixYvL29tbx44dc2k/duyYwsPDs3yO3W6X3W53aQsNDc2rElHAhYSEsPMD4MQ+AYAV+4TCKSdHtQr0wA1+fn6qXbu2NmzY4GxLT0/Xhg0b1KBBAw9WBgAAAOBGVaCPJEnSyJEj1b9/f9WpU0d169bV7Nmzde7cOQ0cONDTpQEAAAC4ARX4kNS7d28dP35c48aNU3x8vGrVqqWPP/4402AOgJXdbtf48eMznXoJoHBinwDAin0CrsRmrjT+HQAAAAAUIgX6miQAAAAAyG+EJAAAAACwICQBAAAAgAUhCQAAAAAsCEm4ocybN08xMTHOm8M1aNBAa9eu9XRZAAqIadOmyWazafjw4Z4uBYAHTJgwQTabzeURFRXl6bJQABX4IcABd5QtW1bTpk1T5cqVZYzRG2+8oc6dO2v79u2qVq2ap8sD4EFbt27VggULFBMT4+lSAHhQtWrV9OmnnzqnfXz4cxiZcSQJN5SOHTuqffv2qly5sm655RY988wzCgoK0rfffuvp0gB40NmzZ9W3b1+9+uqrKlq0qKfLAeBBPj4+Cg8Pdz6KFy/u6ZJQABGScMNKS0vTO++8o3PnzqlBgwaeLgeABw0ZMkR33nmnWrVq5elSAHjY3r17VaZMGVWsWFF9+/bVwYMHPV0SCiCOL+KG89NPP6lBgwa6cOGCgoKCtHLlSkVHR3u6LAAe8s477+iHH37Q1q1bPV0KAA+rV6+eFi9erCpVqujo0aOaOHGimjRpop9//lnBwcGeLg8FCCEJN5wqVapox44dSkxM1PLly9W/f39t3LiRoAQUQocOHdKwYcO0fv16+fv7e7ocAB7Wrl07579jYmJUr149lStXTu+++64GDRrkwcpQ0NiMMcbTRQB5qVWrVqpUqZIWLFjg6VIA5LNVq1apa9eu8vb2dralpaXJZrPJy8tLycnJLvMAFD7/+te/1KpVK02dOtXTpaAA4UgSbnjp6elKTk72dBkAPKBly5b66aefXNoGDhyoqKgojRkzhoAEFHJnz57V/v37de+993q6FBQwhCTcUGJjY9WuXTvdfPPNOnPmjN5++2198cUXWrdunadLA+ABwcHBql69uktbYGCgihUrlqkdwI1v9OjR6tixo8qVK6cjR45o/Pjx8vb2Vp8+fTxdGgoYQhJuKAkJCerXr5+OHj0qh8OhmJgYrVu3Tq1bt/Z0aQAAwMMOHz6sPn366MSJEypRooQaN26sb7/9ViVKlPB0aShguCYJAAAAACy4TxIAAAAAWBCSAAAAAMCCkAQAAAAAFoQkAAAAALAgJAEAAACABSEJAAAAACwISQAAAABgQUgCAAAAAAtCEgAgX/zxxx+y2WzasWNHnq9r8eLFCg0NzfP1AABuTIQkAIAGDBggm82W6dG2bVtPl3ZF5cuX1+zZs13aevfurd9++y3P1x0XF6e7775bZcqUkb+/v8qWLavOnTtr9+7deb5uAEDe8fF0AQCAgqFt27ZatGiRS5vdbvdQNdcmICBAAQEBebqO1NRUtW7dWlWqVNGKFStUunRpHT58WGvXrtXp06fzdL2+vr551j8AgCNJAID/z263Kzw83OVRtGhRSdLdd9+t3r17uyyfmpqq4sWLa8mSJZKkjz/+WI0bN1ZoaKiKFSumDh06aP/+/dmuL6tT4latWiWbzeac3r9/vzp37qxSpUopKChI//rXv/Tpp5865zdv3lwHDhzQiBEjnEe/sut73rx5qlSpkvz8/FSlShUtXbrUZb7NZtNrr72mrl27qkiRIqpcubJWr16dbf2//PKL9u/fr7lz56p+/foqV66cGjVqpMmTJ6t+/frO5Q4fPqw+ffooLCxMgYGBqlOnjrZs2eJWXfPmzVOnTp0UGBioZ555RpL0wQcf6LbbbpO/v78qVqyoiRMn6uLFi9nWCwDIOUISAOCK+vbtqzVr1ujs2bPOtnXr1un8+fPq2rWrJOncuXMaOXKktm3bpg0bNsjLy0tdu3ZVenr6Va/37Nmzat++vTZs2KDt27erbdu26tixow4ePChJWrFihcqWLatJkybp6NGjOnr0aJb9rFy5UsOGDdOoUaP0888/64EHHtDAgQP1+eefuyw3ceJE9erVSzt37lT79u3Vt29fnTx5Mss+S5QoIS8vLy1fvlxpaWnZ1t+sWTP9+eefWr16tX788Uc9/vjjzvckp3VNmDBBXbt21U8//aT77rtPX331lfr166dhw4bp119/1YIFC7R48WJngAIAXCMDACj0+vfvb7y9vU1gYKDL45lnnjHGGJOammqKFy9ulixZ4nxOnz59TO/evbPt8/jx40aS+emnn4wxxsTFxRlJZvv27cYYYxYtWmQcDofLc1auXGmu9F9TtWrVzEsvveScLleunJk1a5bLMpf23bBhQzN48GCXZXr27Gnat2/vnJZknnrqKef02bNnjSSzdu3abGt5+eWXTZEiRUxwcLBp0aKFmTRpktm/f79z/oIFC0xwcLA5ceJEls/PaV3Dhw93WaZly5ZmypQpLm1Lly41pUuXzrZWAEDOcSQJACBJatGihXbs2OHyePDBByVJPj4+6tWrl9566y1J/xw1+uCDD9S3b1/n8/fu3as+ffqoYsWKCgkJUfny5SXJedTnapw9e1ajR49W1apVFRoaqqCgIO3atcvtPnft2qVGjRq5tDVq1Ei7du1yaYuJiXH+OzAwUCEhIUpISMi23yFDhig+Pl5vvfWWGjRooPfee0/VqlXT+vXrJUk7duzQrbfeqrCwsGuqq06dOi7TP/74oyZNmqSgoCDnY/DgwTp69KjOnz+fbb0AgJxh4AYAgKR/QkFkZGS28/v27atmzZopISFB69evV0BAgMvodx07dlS5cuX06quvqkyZMkpPT1f16tWVkpKSZX9eXl4yxri0paamukyPHj1a69ev14wZMxQZGamAgAD16NEj2z6v1aUDIthstiueLhgcHKyOHTuqY8eOmjx5stq0aaPJkyerdevWuTZ4RGBgoMv02bNnNXHiRHXr1i3Tsv7+/rmyTgAozDiSBADIkYYNGyoiIkLLli3TW2+9pZ49ezpDxYkTJ7Rnzx499dRTatmypapWrapTp05dtr8SJUrozJkzOnfunLPt0nsobdq0SQMGDFDXrl1Vo0YNhYeH648//nBZxs/PL9trgjJUrVpVmzZtytR3dHT0FV61e2w2m6KiopyvKSYmRjt27Mj2uqarreu2227Tnj17FBkZmenh5cV/7QBwrTiSBACQJCUnJys+Pt6lzcfHR8WLF3dO33333Zo/f75+++03l8EFihYtqmLFiumVV15R6dKldfDgQY0dO/ay66tXr56KFCmiJ554QkOHDtWWLVu0ePFil2UqV66sFStWqGPHjrLZbHr66aczHdkpX768vvzyS911112y2+0u9WZ47LHH1KtXL916661q1aqV1qxZoxUrVriMlOeuHTt2aPz48br33nsVHR0tPz8/bdy4UQsXLtSYMWMkSX369NGUKVPUpUsXTZ06VaVLl9b27dtVpkwZNWjQ4KrrGjdunDp06KCbb75ZPXr0kJeXl3788Uf9/PPPmjx58lW/JgDA/+fpi6IAAJ7Xv39/IynTo0qVKi7L/frrr0aSKVeunElPT3eZt379elO1alVjt9tNTEyM+eKLL4wks3LlSmNM5oEbjPlnoIbIyEgTEBBgOnToYF555RWXgRvi4uJMixYtTEBAgImIiDAvv/yyadasmRk2bJhzmc2bN5uYmBhjt9udz81qUIi5c+eaihUrGl9fX3PLLbe4DEJhjHGpNYPD4TCLFi3K8j07fvy4GTp0qKlevboJCgoywcHBpkaNGmbGjBkmLS3Nudwff/xhunfvbkJCQkyRIkVMnTp1zJYtW66pLmOM+fjjj03Dhg1NQECACQkJMXXr1jWvvPJKlrUCANxjM+aSE8IBAAAAoBDjxGUAAAAAsCAkAQAAAIAFIQkAAAAALAhJAAAAAGBBSAIAAAAAC0ISAAAAAFgQkgAAAADAgpAEAAAAABaEJAAAAACwICQBAAAAgAUhCQAAAAAs/h9at1uYvWSJwwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluation_df_pivot.reset_index(inplace=True)\n",
    "\n",
    "# Plotting\n",
    "plt.figure(figsize=(10, 6))\n",
    "\n",
    "# Set the width of each bar\n",
    "bar_width = 0.35\n",
    "\n",
    "# OpenAI brand colors\n",
    "openai_colors = ['#00D1B2', '#000000']  # Green and Black\n",
    "\n",
    "# Get unique runs and evaluation scores\n",
    "unique_runs = evaluation_df_pivot['run'].unique()\n",
    "unique_evaluation_scores = evaluation_df_pivot['evaluation_score'].unique()\n",
    "\n",
    "# Ensure we have enough colors (repeating the pattern if necessary)\n",
    "colors = openai_colors * (len(unique_runs) // len(openai_colors) + 1)\n",
    "\n",
    "# Iterate over each run to plot\n",
    "for i, run in enumerate(unique_runs):\n",
    "    run_data = evaluation_df_pivot[evaluation_df_pivot['run'] == run]\n",
    "\n",
    "    # Position of bars for this run\n",
    "    positions = np.arange(len(unique_evaluation_scores)) + i * bar_width\n",
    "\n",
    "    plt.bar(positions, run_data['Number of records'], width=bar_width, label=f'Run {run}', color=colors[i])\n",
    "\n",
    "# Setting the x-axis labels to be the evaluation scores, centered under the groups\n",
    "plt.xticks(np.arange(len(unique_evaluation_scores)) + bar_width / 2, unique_evaluation_scores)\n",
    "\n",
    "plt.xlabel('Evaluation Score')\n",
    "plt.ylabel('Number of Records')\n",
    "plt.title('Evaluation Scores vs Number of Records for Each Run')\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "992f9aa0",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Now you have a framework to test SQL generation using LLMs, and with some tweaks this approach can be extended to many other code generation use cases. With GPT-4 and engaged human labellers you can aim to automate the evaluation of these test cases, making an iterative loop where new examples are added to the test set and this structure detects any performance regressions. \n",
    "\n",
    "We hope you find this useful, and please supply any feedback."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai_test",
   "language": "python",
   "name": "openai_test"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}