openai-cookbook/examples/Zero-shot_classification_with_embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Zero-shot classification with embeddings\n",
    "\n",
    "In this notebook we will classify the sentiment of reviews using embeddings and zero labeled data! The dataset is created in the [Obtain_dataset Notebook](Obtain_dataset.ipynb).\n",
    "\n",
    "We'll define positive sentiment to be 4 and 5-star reviews, and negative sentiment to be 1 and 2-star reviews. 3-star reviews are considered neutral and we won't use them for this example.\n",
    "\n",
    "We will perform zero-shot classification by embedding descriptions of each class and then comparing new samples to those class embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "datafile_path = \"https://cdn.openai.com/API/examples/data/fine_food_reviews_with_embeddings_1k.csv\"  # for your convenience, we precomputed the embeddings\n",
    "df = pd.read_csv(datafile_path)\n",
    "df['babbage_similarity'] = df.babbage_similarity.apply(eval).apply(np.array)\n",
    "df['babbage_search'] = df.babbage_search.apply(eval).apply(np.array)\n",
    "\n",
    "df= df[df.Score!=3]\n",
    "df['sentiment'] = df.Score.replace({1:'negative', 2:'negative', 4:'positive', 5:'positive'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Zero-Shot Classification\n",
    "To perform zero shot classification, we want to predict labels for our samples without any training. To do this, we can simply embed short descriptions of each label, such as positive and negative, and then compare the cosine distance between embeddings of samples and label descriptions. \n",
    "\n",
    "The highest similarity label to the sample input is the predicted label. We can also define a prediction score to be the difference between the cosine distance to the positive and to the negative label. This score can be used for plotting a precision-recall curve, which can be used to select a different tradeoff between precision and recall, by selecting a different threshold."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative       0.67      0.88      0.76       136\n",
      "    positive       0.98      0.93      0.95       789\n",
      "\n",
      "    accuracy                           0.92       925\n",
      "   macro avg       0.82      0.90      0.86       925\n",
      "weighted avg       0.93      0.92      0.92       925\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAyIElEQVR4nO3dd5xV1b3//9ebJhZsgIoggqgxYAQVNaJGYlewm9iV2Fv0F6/eq9dIIolRY8kvxthyNWLDQoxdCSIoGk0oAoIoLaiMqAjSRNrM5/vH3oOH4czMBubMmfJ+Ph7zmLP7Z52B8zlrrb3XUkRgZmZWUZNiB2BmZnWTE4SZmeXlBGFmZnk5QZiZWV5OEGZmlpcThJmZ5eUEYTVKUj9JbxU7jpok6XRJ/8iw372Srq+NmGqDpJmSDklf/1rSo8WOyWqXE4QhaQNJD0j6WNIiSeMkHVnsuLJIP8S+lbRY0heSHpK0SU1eIyIei4jDMux3UUT8piavXU5SSPomLWeJpDskNS3EtczKOUEYQDPgU+BAYDPgl8BTkjoVM6i1cHREbALsAfQkiX81kprVelQ1r3tazgOBk4FzihxPjWogf6MGxQnCiIhvIuLXETEzIsoi4kXgP8CelR0jaTtJz0iaI2mupLsq2e+Pkj6VtFDSGEkH5GzbW9LodNsXku5I17eU9Gh63vmSRknaOkM5SoBXgF3T84SkSyVNBaam6/qmNaT5kv4pabfqypTbbKbEHyR9mcb9vqTy6z0k6bc55ztf0jRJ8yQ9L2nbnG0h6SJJU9NY/ixJ1ZUxLec04G2gR8751qVcXSS9nq77StJjkjbPEkNFko5Nr79Q0nRJR6TrVzVTpcurmqokdUrfh3MlfQK8LukVSZdVOPd4SSekr3eRNDR9Tz+S9NN1ideycYKwNaQfxjsDkyrZ3hR4EfgY6AS0B56o5HSjSD7ItgQeB56W1DLd9kfgjxGxKdAFeCpdfzZJTWY7oDVwEfBthri3A44C3stZfRywD9BV0u7Ag8CF6XnvA55Pm9iylukw4Eck789mwE+BuXliOQi4Kd3eLj1vxfP1BfYCdkv3O7y6Mqbn3gU4AJiWLq9ruZTGuC3wfZL3+9dZYqgQz97Aw8DVwOYk78/MtTjFgen1DwcGAafmnLsrsD3wkqSNgaEk/462Ak4B7k73sQJwgrDVSGoOPAYMjIgPK9ltb5IPlavT2sfSiMjbMR0Rj0bE3IhYGRG3AxsA30s3rwB2lNQmIhZHxLs561sDO0ZEaUSMiYiFVYT9rKT5wFvAG8DvcrbdFBHzIuJb4ALgvoj4V3regcAy4IdrUaYVQCtgF0ARMTkiZufZ73TgwYgYGxHLgGuBfSs0290cEfMj4hNgODk1gkqMlfQNMBkYAdydrl+nckXEtIgYGhHLImIOcAfJh/XaOjct69C0BlpSxb+dfH6dxvYt8Hegh6Tt022nA8+k72FfYGZE/DX99/Qe8DfgJ+sQs2XgBGGrSGoCPAIsBy7LWf+Kks7RxZJOJ/mm+XFErMxwzqskTZa0IP0Q3wxok24+l+Sb+IdpM1LfdP0jwBDgCUmfSfp9mrgqc1xEbB4R20fEJekHTblPc15vD/xX2gwzP41nO5IP0ExliojXgbuAPwNfSrpf0qZ5dt2W5Ft7+XGLSWoa7XP2+Tzn9RJgEwBJk3Le7wNy9tkj3edkklrRxutTLklbS3pCSaf3QuBRvvvbrI3tgOnrcFy5VX+jiFgEvERSO4CkNvFY+np7YJ8K5Twd2GY9rm1VcIIwIGlbBx4AtgZOjIgV5dsi4siI2CT9eYzkP3RHVdOpmH64/TdJ88kWEbE5sICkaYOImBoRp5I0F9wCDJa0cUSsiIgbIqIr0Ivkm+NZ61i03OGKPwVuTJNJ+c9GETEoa5nSuO+MiD2BriQJ7uo8u31G8oEGQNo80hooyXD+bjnv98gK2yIingLeAfqvZ7l+R/L+/CBt5juD9G+zlj4laSLM5xtgo5zlfB/mFYeUHgScKmlfoCVJ7ar8Om9UKOcmEXHxOsRsGThBWLl7SNqBj67wDTyffwOzgZslbaykU3m/PPu1AlYCc4BmkvoDq75tSzpDUtuIKAPmp6vLJP1Y0g/S9vOFJM06ZetTuNRfgIsk7aPExpL6SGqVtUyS9kqPb07y4be0ktgGAT+T1EPSBiQfxv+KiJk1UA6Am4HzJW2zHuVqBSwGFkhqT/5El8UDJGU9WFITSe3TfhKAccApkppL6gmclOF8L5Mk1wHAk+m/D0j6UnaWdGZ6vubp3+P76xi3VcMJwkjbey8kaQP/vEJz0hoiohQ4GtgR+ASYRdLsUdEQ4FVgCklzy1JWb/I5ApgkaTFJh/UpaXLaBhhMkhwmk/QrPLKexSQiRgPnkzQRfU3SydtvLcu0KckH8tdpmeYCt+a51mvA9SRt5LNJvmGfUnG/9SjL+8CbJH0L61quG0iarRaQNOs8s46x/Bv4GfCH9Fxv8F3t6XqSsn+dXu/xDOdblsZySO7+afPTYSTv42ckTXS3kPRrWQHIEwaZmVk+rkGYmVleThBmZpaXE4SZmeXlBGFmZnk1mMGx2rRpE506dSp2GGZm9cqYMWO+ioi2+bY1mATRqVMnRo8eXewwzMzqFUkfV7bNTUxmZpaXE4SZmeXlBGFmZnk5QZiZWV5OEGZmllfBEoSkB5VMyzixku2SdKeSKRknSNojZ9vZSqZinCrp7ELFaGZmlStkDeIhktE6K3MksFP6cwHJcNNI2hL4FcmEKHsDv5K0RQHjNDOzPAr2HEREvFlhesWKjgUejmQ42XclbS6pHdAbGBoR8wAkDSVJNIMKEeeS5Su5d8T6TIZlZnVVyxZN6derExu1aDCPfNWqYr5r7Vl9boBZ6brK1q9B0gUktQ86duy4TkF8u7yUPw2ftk7HmlndVT6TwS7btOKgXbYubjD1VL1OqxFxP3A/QM+ePddpYovWm2zAf27qU6NxmVnxTSxZQN8/vUVpTcxF2EgVM0GUkEx2Xq5Duq6EpJkpd/2IWovKzBqE8hrEKxNns2xlKaVlQVkEpWWwT+ct2W7Ljao+gRU1QTwPXCbpCZIO6QURMVvSEOB3OR3ThwHXFitIM6ufgiRDPDO2hGfGlqy27Yhu23DvmXsWI6x6pWAJQtIgkppAG0mzSO5Mag4QEfeSTEx+FMn8uUtI5rQlIuZJ+g0wKj3VgPIOazOzrHbrsDl/v6QXLZs3pWkT0USiaRNx4SOjWe52p0wKeRfTqdVsD+DSSrY9CDxYiLjMrPHYveOad8jPX7KC6XPm8NoHXzD3m2W0bN6UFaXBitIyVpaW8dmCpWzasvmq5RVlwafzlrDlxi1W229FWbBR86bccGy3BnuXVMMslZlZJb5ctAyA8x7ONj1Ai6ZNkGDZyjLabLIBLZqKZk2bsHxlGZ8vXMope3dkz+0b5qNaThBm1qiMuKo3H32xiHabtaRZkyY0aQIbNW9G82aiedMmNG/ShBbNmtC8adIkJSnved6YMoezH/x3LUdfu5wgzKxR6dRmYzq12bjYYdQLHqzPzMzycoIwM7O8nCDMzCwvJwgzM8vLCcLMzPJygjAzs7ycIMzMLC8nCDMzyyvTg3LpyKrbAt8CMyPCI12ZmTVwlSYISZuRDKZ3KtACmAO0BLaW9C5wd0QMr5Uozcys1lVVgxgMPAwcEBHzczdI2hM4U9IOEfFAAeMzM7MiqTRBRMShVWwbA4wpSERmZlYnVNtJrcQZkvqnyx0l7V340MzMrJiydFLfDZQBBwEDgEXA34C9ChiXmVmdtmJlcq/OdX9/n45bbkTTJmLSZwtZ8O0KIoLlpWXccEw3Tt6rY5EjXXdZEsQ+EbGHpPcAIuJrSS0KHJeZWZ22siyZ8/rDzxfx4eeLaNG0Cd/bphVbbNScTTdszjvT5zL1i8VFjnL9ZEkQKyQ1hWQGcEltSWoUZmaN1hG7bsP7vz6MDZo1pUWzNVvru/V/tQhR1awsD8rdCfwd2ErSjcBbwO8KGpWZWT3QqmXzvMmhoai2BhERj0kaAxwMCDguIiYXPDIzMyuqahOEpDuBJyLiz7UQj5mZ1RFZ6kZjgF9Kmi7pNkk9Cx2UmZkVX7UJIiIGRsRRJLe1fgTcImlqwSMzM7OiWpvelR2BXYDtgQ8LE46ZWcPwzfJS/u+t/9DrpmH811Pjix3OOsnyJPXv0xrDAGAi0DMiji54ZGZmDcBnC5Yyaua8YoexTrI8BzEd2Dcivip0MGZmDcXMm/sA8IsnxzHm46+LHM26qWq4710i4kNgFNBR0mrPi0fE2EIHZ2ZmxVNVDeJK4ALg9jzbgmRsJjMza6CqGu77gvTlkRGxNHebpJYFjcrMzIouy11M/8y4zszMGpCq+iC2AdoDG0ranWSYDYBNgY1qITYzMyuiqmoQhwO3AR2AO0j6Im4n6Zv43ywnl3SEpI8kTZN0TZ7t20saJmmCpBGSOuRs+72kSZImS7pTkioeb2ZW170w/jM+mbeE0//vXcrSIcLri0oTRPoE9Y+BfhHx45yfYyLimepOnA4R/mfgSKArcKqkrhV2uw14OCJ2I3nO4qb02F7AfsBuwK4kT3EfuPbFMzMrrp/0TL73vj1t7qo5JOqLqpqYzoiIR4FOkq6suD0i7qjm3HsD0yJiRnq+J4BjgQ9y9ulKUiMBGA48W356oCXQgqRpqznwRXWFMTOra246YTc6bLERtw75qNihrLWqmpg2Tn9vArTK81Od9sCnOcuz0nW5xgMnpK+PB1pJah0R75AkjNnpz5B8Q4xLukDSaEmj58yZkyEkMzPLqqrbXO9Lf99QwOtfBdwlqR/wJlAClEraEfg+Sf8HwFBJB0TEyAox3g/cD9CzZ8/6VXczM6vjso7FtKmk5mmH8hxJZ2Q4dwmwXc5yh3TdKhHxWUScEBG7A9el6+aT1CbejYjFEbEYeAXYN1uRzMysJmR5DuKwiFgI9AVmkozqenWG40YBO0nqLKkFcArwfO4OktpIKo/hWuDB9PUnwIGSmklqTtJB7VnszMxqUZYEUd4M1Qd4OiIWZDlxRKwELgOGkHy4PxURkyQNkHRMultv4CNJU4CtgRvT9YNJBgl8n6SfYnxEvJDlumZmVjOyjOb6oqQPgW+BiyW1BZZWcwwAEfEy8HKFdf1zXg8mSQYVjysFLsxyDTMzK4wsM8pdA/QimQdiBfANye2qZmbWgFVbg0j7AM4AfpQ+zPwGcG+B4zIzsyLL0sR0D8mDaneny2em684rVFBmZlZ8WRLEXhHRPWf5dUn1c4JVMzPLLEuCKJXUJSKmA0jaASgtbFhmZo3PitIy5i5ezorSMlaWBRtv0JStWhVv+p0sCeJqYLikGSTjIm0P/KygUZmZNRLLV5YxYdZ83p42lz+8NmW1bU2biH//78G03mSDosRWbYKIiGGSdgK+l676KCKWFTYsM7OGZ8oXi5j25WKGTv6CIRM/B1hjhNddtmnFuft35r1P5/P4vz5h8bKVdTdBpNOLXgLsTzLK6khJ91achtTMzPIbNjkZjLrvn95abf1RP9iGLm03QRL7dWlNj46bs0GzpkBSe3j8X5/Ueqy5sjQxPQwsAv6ULp8GPAL8pFBBmZk1JP3268zYT97j6sO/R4ctNmSPjluw3ZZ1f2LOLAli14jInehnuKQPKt3bzMxWc0z3bTmm+7bFDmOtZRmLaaykH5YvSNoHGF24kMzMrC7IUoPYE/inpPLGsI4kA+y9D0Q6XaiZmTUwWRLEEQWPwszM6pwst7l+XBuBmJlZ3ZKlD8LMzBohJwgzM8vLCcLMzPLKlCAk3V/VspmZNTxZaxD3VbNsZmYNTKYEERFjqlo2M7OGp9LbXCW9QDI4X14RcUxBIjIzszqhqucgbqu1KMzMrM6pNEFExBvlryVtCHSMiI9qJSozMyu6avsgJB0NjANeTZd7SHq+wHGZmVmRZemk/jWwNzAfICLGAZ0LFpGZmdUJWRLEiohYUGFdpZ3XZmbWMGQZzXWSpNOApunc1JcD/yxsWGZmVmxZahA/B7oBy4BBwELg/ytgTGZmVgdkGe57CXCdpFuSxVhU+LDMzKzYstzFtFc6e9wE4H1J4yXtWfjQzMysmLL0QTwAXBIRIwEk7Q/8FfBUo2ZmDViWPojS8uQAEBFvASsLF5KZmdUFlSYISXtI2gN4Q9J9knpLOlDS3cCILCeXdISkjyRNk3RNnu3bSxomaYKkEZI65GzrKOkfkiZL+kBSp7UvnplZ/VTy9bcA3DNietFiqKqJ6fYKy7/KeV3tcxCSmgJ/Bg4FZgGjJD0fER/k7HYb8HBEDJR0EHATcGa67WHgxogYKmkToKy6a5qZNRSbbdQcgCdGfcrNJxanRb+qsZh+vJ7n3huYFhEzACQ9ARwL5CaIrsCV6evhwLPpvl2BZhExNI1l8XrGYmZWr5y1byc++nwRQyZ9XrQYsnRSI6kPybMQLcvXRcSAag5rD3yaszwL2KfCPuOBE4A/AscDrSS1BnYG5kt6hmRYj9eAayKitEJcFwAXAHTs2DFLUczMLKMst7neC5xM8sCcgJ8A29fQ9a8CDpT0HnAgUAKUkiSuA9LtewE7AP0qHhwR90dEz4jo2bZt2xoKyczMINtdTL0i4izg64i4AdiX5Bt+dUqA7XKWO6TrVomIzyLihIjYHbguXTefpLYxLiJmRMRKkqanPTJc08zMakiWBPFt+nuJpG2BFUC7DMeNAnaS1FlSC+AUYLVhwiW1kVQew7XAgznHbi6pvFpwEKv3XZiZWYFlSRAvStocuBUYC8wkGZOpSuk3/8uAIcBk4KmImCRpgKTy6Up7Ax9JmgJsDdyYHltK0rw0LH2KW8BfshfLzMzWV5axmH6TvvybpBeBlnmG/67s2JeBlyus65/zejAwuJJjh+Kntc3MiqbSBCHphCq2ERHPFCYkMzOrC6qqQRxdxbYAnCDMzBqwqh6U+1ltBmJmZnVLlk5qMzMrgulzFvPV4uUsWV6c8VGdIMzM6qh3Z8wD4F//mVeU6ztBmJnVUc9duh8A4z6ZX5TrZxlqYyNJ10v6S7q8k6S+hQ/NzKxxm7dkOQB/HDa1KM1MWWoQfwWWkQyxAclwGb8tWERmZgZA752/G2NuxcpqZ1mocVkSRJeI+D3JEBtExBKSJ5vNzKyAJNG/b9eiXT9LglguaUPSSYIkdSGpUZiZWQOWZT6IXwOvAttJegzYjzxDb5uZWcOSZSymf0gaA/yQpGnpioj4quCRmZlZUVWbICS9ADwOPB8R3xQ+JDMzqwuy9EHcRjK72weSBks6SVLL6g4yM7P6LUsT0xvAG5Kakkzccz7JxD6bFjg2MzNLjZjyJUuWl3Jsj23ZqEWW7uP1l+kq6V1MR5PMTb0HMLCQQZmZWWJiSTL9zhVPjAOgZfMmHL97h1q5dpYnqZ8imRHuIOAukucifl7owMzMDHp03Hy15fvemMGcRbXzpEGWPogHSJLCRRExPCLKCh2UmZklztq3EzNv7sPwq3oD8OHni7jvjem1cu2qZpQ7KCJeBzYGjpVWf3jaM8qZmdWezm025oCd2jBy6ld8U0vjMlVVgzgw/X10nh8P1mdmVsseOXcfAAb9+9NauV5VM8r9Kn05ICL+k7tNUueCRmVmZkWXpQ/ib3nWDa7pQMzMrHpn77s9m2/UvFauVVUfxC5AN2AzSSfkbNoU8INyZmYNXFXPQXyPpK9hc5J+h3KLSB6WMzOzWvbOjLnMX7KCFyd8Rt/dti3otarqg3gOeE7SvhHxTkGjMDOzTJatTJ40ePa9kuIlCEn/nU4UdJqkUytuj4jLCxqZmZmt4Y2rf0yna17itclfFvxaVTUxTU5/jy54FGZmttbmL1nO5hu1KNj5q2pieiH9vWrcJUlNgE0iYmHBIjIzsypdd9T3ufHlyZSWFXae6ixjMT0uaVNJGwMTSYb9vrqgUZmZWaU2aJ7lCYX1l+UqXdMaw3HAK0Bn4MxCBmVmZtUr77AulCwJormk5iQJ4vmIWAEUtl5jZmaVGjk1mfX5mLveKuh1siSI+4CZJIP2vSlpe8B9EGZmRXL+ATsAsHhZYQftqzZBRMSdEdE+Io6KxMfAj7OcXNIRkj6SNE3SNXm2by9pmKQJkkZI6lBh+6aSZkm6K3OJzMwauL07bwnA0hVllMz/tmDXydJJvZmkOySNTn9uJ6lNVHdcU+DPwJFAV+BUSV0r7HYb8HBE7AYMAG6qsP03wJsZymFm1qjs2j6Z9bnk6yImCJL5pxcBP01/FgJ/zXDc3sC0iJgREcuBJ4BjK+zTFXg9fT08d7ukPYGtgX9kuJaZWaNy7ZHfL/g1siSILhHxq/SDfkZE3ADskOG49kDuoOWz0nW5xgPlAwEeD7SS1Dp93uJ24KqqLiDpgvKazZw5czKEZGZmWWVJEN9K2r98QdJ+QE3Vaa4CDpT0HskERSVAKXAJ8HJEzKrq4Ii4PyJ6RkTPtm3b1lBIZmYGVQ+1Ue4i4GFJm6XLXwNnZziuBNguZ7lDum6ViPiMtAYhaRPgxIiYL2lf4ABJlwCbAC0kLY6INTq6zcysMKpMEJJ6ADsCp5B+uK/FMBujgJ3S2edK0nOcVuH8bYB5EVEGXEvS30FEnJ6zTz+gp5ODmVntqrSJSVJ/4CngROAl4OS1GYMpIlYClwFDSAb+eyoiJkkaIOmYdLfewEeSppB0SN+4TqUwM7MaV1UN4mSgR0QskdQaeBX4y9qcPCJeBl6usK5/zuvBVDN9aUQ8BDy0Ntc1M7P1V1Un9bKIWAIQEXOr2dfMzBqYqmoQO0h6Pn0toEvOMhFxTP7DzMysIagqQVR8qO22QgZiZmZ1S1UTBr1Rm4GYmVndUtVdTC9IOjod6rvith3Su5HOKWx4ZmZWLFU1MZ0PXAn8/5LmAXOAlkAnYDpwV0Q8V/AIzcysKKpqYvoc+G/gvyV1AtqRDLExpfzuJjMza7iyDLVBRMwkmTTIzMwaCT/bYGZmeTlBmJlZXk4QZmaWV5YpR/eTNFTSFEkzJP1H0ozaCM7MzPJbvrIMgGlfLi7YNbLUIB4A7gD2B/YCeqa/zcysSJYsLwVg4D9nFuwaWRLEgoh4JSK+jIi55T8Fi8jMzKrVZ7d2AOzafrNq9lx3WW5zHS7pVuAZYFn5yogYW7CozMysWu0337Cg58+SIPZJf/fMWRfAQTUfjpmZZbVk+UoWfLuiYOevNkFExI8LdnUzM1tnXy9ZwWuTv6CsLGjSRDV+/ix3MW0m6Q5Jo9Of2yUVrtHLzMzWShTovFk6qR8EFgE/TX8WAn8tUDxmZpbRlYfuXNDzZ+mD6BIRJ+Ys3yBpXIHiMTOzOiJLDeJbSfuXL0jaj2RUVzMza8Cy1CAuBgam/Q4C5gH9ChmUmZkVX5a7mMYB3SVtmi4vLHRQZmZWfJUmCElnRMSjkq6ssB6AiLijwLGZmVkRVVWD2Dj93ao2AjEzs7qlqilH70t/31B74ZiZWV2R5UG530vaVFJzScMkzZF0Rm0EZ2ZmxZPlNtfD0o7pviTzUu8IXF3IoMzMrPiyJIjyZqg+wNMRsaCA8ZiZWR2R5TmIFyV9SPJw3MWS2gJLCxuWmZkVW7U1iIi4BugF9IyIFcA3wLGFDszMzIqrqucgDoqI1yWdkLMud5dnChmYmZkVV1U1iAPT30fn+emb5eSSjpD0kaRpkq7Js3379M6oCZJGSOqQru8h6R1Jk9JtJ69VqczMbL1V9RzEr9LfP1uXE0tqCvwZOBSYBYyS9HxEfJCz223AwxExUNJBwE3AmcAS4KyImCppW2CMpCERMX9dYjEzs7WX5TmI30naPGd5C0m/zXDuvYFpETEjIpYDT7Bm30VX4PX09fDy7RExJSKmpq8/A74E2ma4ppmZ1ZAst7kemfvNPSK+Bo7KcFx74NOc5VnpulzjgfI+juOBVpJa5+4gaW+gBTC94gUkXVA+092cOXMyhGRmZlllSRBNJW1QviBpQ2CDKvZfG1cBB0p6j6TPowQozblWO+AR4GcRUVbx4Ii4PyJ6RkTPtm1dwTAzq0lZnoN4DBgmqXya0Z8BAzMcVwJsl7PcIV23Stp8dAKApE2AE8trK+nw4i8B10XEuxmuZ2ZmNSjLfBC3SBoPHJKu+k1EDMlw7lHATpI6kySGU4DTcneQ1AaYl9YOriWZ/xpJLYC/k3RgD85aGDMzqzlZahAAk4GVEfGapI0ktYqIRVUdEBErJV0GDAGaAg9GxCRJA4DREfE80Bu4SVIAbwKXpof/FPgR0FpSv3Rdv3TyIjMzqwXVJghJ5wMXAFsCXUg6mu8FDq7u2Ih4GXi5wrr+Oa8HA2vUECLiUeDR6s5vZmaFk6WT+lJgP2AhQHr76VaFDMrMzIovS4JYlj7HAICkZkAULiQzM6sLsiSINyT9L7ChpEOBp4EXChuWmZkVW5YE8T/AHOB94EKSPoVfFjIoMzMrvio7qdPxlCZFxC7AX2onJDMzqwuqrEFERCnwkaSOtRSPmZnVEVmeg9gCmCTp3ySTBQEQEccULCozMyu6LAni+oJHYWZmdU5VM8q1BC4CdiTpoH4gIlbWVmBmZlZcVfVBDAR6kiSHI4HbayUiMzOrE6pqYuoaET8AkPQA8O/aCcnMzOqCqmoQK8pfuGnJzKzxqaoG0V3SwvS1SJ6kXpi+jojYtODRmZlZ0VSaICKiaW0GYmZmdUuWoTbMzKwRyjphUL20YsUKZs2axdKlS4sditUjLVu2pEOHDjRv3rzYoZgVVYNOELNmzaJVq1Z06tQJScUOx+qBiGDu3LnMmjWLzp07FzscsyqVfP0tAHO/WcZWrVrW+PkbdBPT0qVLad26tZODZSaJ1q1bu9Zp9cI7M+YCMGzylwU5f4NOEICTg601/5ux+uLRc/cp6PkbfIIwM2uoWjQr7Ee4E0SBff7555xyyil06dKFPffck6OOOoopU6Ywc+ZMdt111xq7Tv/+/XnttdcAGDlyJN26daNHjx6UlJRw0kknrde5I4KDDjqIhQsXrlr37LPPIokPP/xw1bqZM2ey4YYb0qNHD7p27cpFF11EWVnZel37zTffZI899qBZs2YMHjy40v3GjBnDD37wA3bccUcuv/xyIpJZcefNm8ehhx7KTjvtxKGHHsrXX38NwIsvvkj//v3XKzazhs4JooAiguOPP57evXszffp0xowZw0033cQXX3xR49caMGAAhxxyCACPPfYY1157LePGjaN9+/ZVfrBWtHLlmg/Nv/zyy3Tv3p1NN/3u2chBgwax//77M2jQoNX27dKlC+PGjWPChAl88MEHPPvss+tWoFTHjh156KGHOO2006rc7+KLL+Yvf/kLU6dOZerUqbz66qsA3HzzzRx88MFMnTqVgw8+mJtvvhmAPn368MILL7BkyZL1is+sIWvQdzHluuGFSXzw2cLqd1wLXbfdlF8d3a3S7cOHD6d58+ZcdNFFq9Z1794dSL5tl5s5cyZnnnkm33yTTLdx11130atXL2bPns3JJ5/MwoULWblyJffccw+9evXi3HPPZfTo0UjinHPO4Re/+AX9+vWjb9++zJ8/n6eeeoohQ4bwyiuvcOONN9K3b18mTpxIaWkp11xzDSNGjGDZsmVceumlXHjhhYwYMYLrr7+eLbbYgg8//JApU6asVo7HHnuMCy64YNXy4sWLeeuttxg+fDhHH300N9xwwxplb9asGb169WLatGnr9N6W69SpEwBNmlT+XWb27NksXLiQH/7whwCcddZZPPvssxx55JE899xzjBgxAoCzzz6b3r17c8sttyCJ3r178+KLL/LTn/50vWI0a6gaTYIohokTJ7LnnntWu99WW23F0KFDadmyJVOnTuXUU09l9OjRPP744xx++OFcd911lJaWsmTJEsaNG0dJSQkTJ04EYP78+aud67zzzuOtt96ib9++nHTSSaslogceeIDNNtuMUaNGsWzZMvbbbz8OO+wwAMaOHcvEiRPz3tr59ttvc999961afu655zjiiCPYeeedad26NWPGjFmjnEuWLGHYsGEMGDBgjfMdcMABLFq0aI31t91226pa0NooKSmhQ4cOq5Y7dOhASUkJAF988QXt2rUDYJtttlmt9tazZ09GjhzpBGFWiUaTIKr6pl9sK1as4LLLLmPcuHE0bdp01Tf4vfbai3POOYcVK1Zw3HHH0aNHD3bYYQdmzJjBz3/+c/r06bPqAz6Lf/zjH0yYMGFVk9OCBQuYOnUqLVq0YO+99670vv958+bRqlWrVcuDBg3iiiuuAOCUU05h0KBBqxLE9OnT6dGjB5I49thjOfLII9c438iRIzPHXJMkrXaH0lZbbcVnn31WlFjM6oNGkyCKoVu3bpna///whz+w9dZbM378eMrKymjZMnng5Uc/+hFvvvkmL730Ev369ePKK6/krLPOYvz48QwZMoR7772Xp556igcffDBTPBHBn/70Jw4//PDV1o8YMYKNN9640uOaNWtGWVkZTZo0Yd68ebz++uu8//77SKK0tBRJ3HrrrcB3fRBVqekaRPv27Zk1a9aq5VmzZtG+fXsAtt56a2bPnk27du2YPXs2W2211ar9li5dyoYbbrjW1zNrLNxJXUAHHXQQy5Yt4/7771+1bsKECWt8g16wYAHt2rWjSZMmPPLII5SWlgLw8ccfs/XWW3P++edz3nnnMXbsWL766ivKyso48cQT+e1vf8vYsWMzx3P44Ydzzz33sGJFMpL7lClTVvV7VOV73/seM2bMAGDw4MGceeaZfPzxx8ycOZNPP/2Uzp07r1WtYOTIkYwbN26Nn3VJDgDt2rVj00035d133yUiePjhhzn22GMBOOaYYxg4cCAAAwcOXLUekvLX5J1kZsXyzbLCzMjgBFFAkvj73//Oa6+9RpcuXejWrRvXXnst22yzzWr7XXLJJQwcOJDu3bvz4Ycfrvo2P2LECLp3787uu+/Ok08+yRVXXEFJSQm9e/emR48enHHGGdx0002Z4znvvPPo2rUre+yxB7vuuisXXnhh3ruWKurTp8+qjt5BgwZx/PHHr7b9xBNPXONuppoyatQoOnTowNNPP82FF15It27fNRX26NFj1eu7776b8847jx133JEuXbqsatq65pprGDp0KDvttBOvvfYa11xzzapjhg8fTp8+fQoSt1ltWLI8+f/725cmF+T8Kr9fvL7r2bNnjB49erV1kydP5vvf/36RImo4Zs+ezVlnncXQoUOLHUqN+eKLLzjttNMYNmxY3u3+t2P1QVlZsMP/vsx5+3fml327rtM5JI2JiJ75trkPwqrVrl07zj//fBYuXLjasxD12SeffMLtt3uadavfmjQRM28uXC3YCcIyaWi3gu61117FDsGszmvwfRANpQnNao//zZglCpogJB0h6SNJ0yRdk2f79pKGSZogaYSkDjnbzpY0Nf05e12u37JlS+bOnev/8JZZ+XwQ5bcamzVmBWtiktQU+DNwKDALGCXp+Yj4IGe324CHI2KgpIOAm4AzJW0J/AroCQQwJj3267WJoUOHDsyaNYs5c+bURJGskSifUc6ssStkH8TewLSImAEg6QngWCA3QXQFrkxfDweeTV8fDgyNiHnpsUOBI4C1upeyefPmnhXMzGwdFbKJqT3wac7yrHRdrvHACenr44FWklpnPBZJF0gaLWm0awlmZjWr2J3UVwEHSnoPOBAoAUqzHhwR90dEz4jo2bZt20LFaGbWKBWyiakE2C5nuUO6bpWI+Iy0BiFpE+DEiJgvqQToXeHYEQWM1czMKijYk9SSmgFTgINJEsMo4LSImJSzTxtgXkSUSboRKI2I/mkn9Rhgj3TXscCe5X0SlVxvDvDxeoTcBvhqPY6vjxpbmRtbecFlbizWp8zbR0TeJpiC1SAiYqWky4AhQFPgwYiYJGkAMDoiniepJdwkKYA3gUvTY+dJ+g1JUgEYUFVySI9ZrzYmSaMre9y8oWpsZW5s5QWXubEoVJkL+iR1RLwMvFxhXf+c14OBvONhR8SDQLZxrM3MrMYVu5PazMzqKCeI79xf/S4NTmMrc2MrL7jMjUVBytxghvs2M7Oa5RqEmZnl5QRhZmZ5NaoEkWF02Q0kPZlu/5ekTkUIs0ZlKPOVkj5IR9QdJmn7YsRZk6orc85+J0oKSfX+lsgsZZb00/RvPUnS47UdY03L8G+7o6Thkt5L/30fVYw4a4qkByV9KWliJdsl6c70/ZggaY98+62ViGgUPyTPYkwHdgBakIwD1bXCPpcA96avTwGeLHbctVDmHwMbpa8vbgxlTvdrRfLszbtAz2LHXQt/552A94At0uWtih13LZT5fuDi9HVXYGax417PMv+I5OHhiZVsPwp4BRDwQ+Bf63vNxlSDWDW6bEQsB8pHl811LDAwfT0YOFiSajHGmlZtmSNieEQsSRffJRnWpD7L8ncG+A1wC7C0NoMrkCxlPh/4c6RD5kfEl7UcY03LUuYAyufI3Qz4rBbjq3ER8SZQ1QPDx5JMnxAR8S6wuaR263PNxpQgsowQu2qfiFgJLABa10p0hZFpVNwc55J8A6nPqi1zWvXeLiJeqs3ACijL33lnYGdJb0t6V9IRtRZdYWQp86+BMyTNInlg9+e1E1rRrO3/92p5TmoDQNIZJBM0HVjsWApJUhPgDqBfkUOpbc1Impl6k9QS35T0g4iYX8ygCuxU4KGIuF3SvsAjknaNiLJiB1ZfNKYaRLWjy+bukw42uBkwt1aiK4wsZUbSIcB1wDERsayWYiuU6srcCtgVGCFpJklb7fP1vKM6y995FvB8RKyIiP+QDKS5Uy3FVwhZynwu8BRARLwDtCQZ1K6hyvT/fW00pgQxCthJUmdJLUg6oZ+vsM/zQPn81ycBr0fa+1NPVVtmSbsD95Ekh/reLg3VlDkiFkREm4joFBGdSPpdjomI0cUJt0Zk+bf9LOkQ+ukoyjsDM2oxxpqWpcyfkIwmjaTvkySIhjyz2PPAWendTD8EFkTE7PU5YaNpYopso8s+QFINnUbSGXRK8SJefxnLfCuwCfB02h//SUQcU7Sg11PGMjcoGcs8BDhM0gckk3JdHRH1tnacscz/BfxF0i9IOqz71ecvfJIGkST5Nmm/yq+A5gARcS9JP8tRwDRgCfCz9b5mPX6/zMysgBpTE5OZma0FJwgzM8vLCcLMzPJygjAzs7ycIMzMLC8nCFtnkkoljZM0UdILkjav4fPPTO/ZR9LiSvbZUNIbkppK6iTp2zSmDyTdmz45vTbX7CnpzvR1b0m9crZdJOms9SlTep5fS7qqmn0eknTSWpyzU2WjfNY0SceUj54q6ThJXXO2DUgfvFyX8z4hqT4/vNfgOEHY+vg2InpExK4kz41cWoQYzgGeiYjSdHl6RPQAdiMZwfO4tTlZRIyOiMvTxd5Ar5xt90bEw+sbcH0XEc9HxM3p4nEk73P5tv4R8do6nvoe4L/XMzyrQU4QVlPeIR0YTFIXSa9KGiNppKRd0vVbS/q7pPHpT690/bPpvpMkXbCW1z0deK7iynSwxX8CO6bfrl/Xd3NedEyv+5O09jNe0pvput6SXlQyF8hFwC/SGskB5d/8Je0i6d/l10rP/376es+0RjNG0hBVM5qmpPMljUpj+JukjXI2HyJptKQpkvqm+zeVdGt6zARJF67NmyVpsaQ/pO/1MElt0/U9lAziNyH9G22Rrr9c380X8kS6rp+ku9K/3zHArel71KW85qNkroanc67bW9KL6evDJL0jaaykpyVtku42Mi1zo3mAt65zgrD1JqkpyZAG5U8p3w/8PCL2BK4C7k7X3wm8ERHdSca1n5SuPyfdtydwuaRMI+gqGWJhh4iYmWfbRmlM7wN/AgZGxG7AY2kcAP2Bw9N4Vnt6PD3nvcAf0lrSyJxtHwItJHVOV50MPCmpeXqtk9LyPAjcWE0xnomIvdIYJpOMH1SuE8mw1n2AeyW1TLcviIi9gL2A83PiKC/7tpJeruR6G5M8adwNeIPkaVyAh4H/Sd+j93PWXwPsnq6/qMJ79E+Sv/nV6Xs0PWfza8A+kjZOl08GnlDSZPhL4JCI2AMYDVyZnq+M5Cng7pW/XVabnCBsfWwoaRzwObA1MDT9NtiLZOiOcSTjPJV/iz6IpBmBiCiNiAXp+ssljScZF2k7sg8i1waYX2Fdl/S6bwMvRcQrwL5A+QxqjwD7p6/fBh6SdD7JcA1r4ymSDz3S308C3yMZCHBoGsMvqX5+jV3TWtb7JLWhbrnXiIiyiJhKMm7SLsBhJOPtjAP+RTIc/WrvV0R8FhGVzZ5WlsYK8Ciwv6TNgM0j4o10/UCSyWkAJgCPKRntd2U1ZcmNYSXwKnB0WiPoQ1LT+yFJk9TbaRnOBnJnMfwS2DbrdaywXJWz9fFtRPRIv60PIemDeAiYn/YDVEtSb+AQYN+IWCJpBMmgapmun2ff6VmvHREXSdqH5MNrjKQ9M14Xkg/ZpyU9k5wqpkr6ATApIvZdi/M8BBwXEeMl9SMdUK88xIohk8wW9vOIGJK7Qes+PW51Y+30IUkWRwPXpWXM6gngMpL+qdERsUiSgKERcWolx7Qk+btaHeAahK23dEa6y0kGR1sC/EfST2DVPLnlTQbDSKY1LW9L34xkSPWv0+SwC8k3zKzX/Rpomja9VOWffDfw4ukkbd1I6hIR/4qI/iSjfG5X4bhFJMOD57v2dJJB767nu2/kHwFtlcw9gKTmkrrlOz5HK2B22jx1eoVtP5HURFIXkqk1PyJJxBen+yNp55xmnCyakIxUDHAa8FZak/ta0gHp+jOBN5TcAbZdRAwH/ofkb7VJhfNV+h6RNGHtQTKb3RPpuneB/STtmMa/saSdc47ZGaiVu7Gsek4QViMi4j2S5ohTST7ozk2bjSbx3VSQVwA/TptTxpA0NbwKNJM0GbiZ5ANkbfyD75qMKvNz4GeSJpB8+F2Rrr9V0vtKbg/9J8m8xrleAI5PO2APYE1PAmfw3ZwDy0k+fG9Jyz6OnLugKnE9SVPR28CHFbZ9AvybZJa/iyJiKfB/wAfA2DTu+6jQElBNH8Q3wN7psQcBA9L1Z5O8HxOAHun6psCj6d/rPeDOPBMMPQFcLem9NJGtkt5Z9iJwZPqbiJhDMlnToPRa75A0nSFpa5Ja6eeVxG61zKO5Wr2mZPrQX0TEmcWOpT6QtDgiKtYC6gQlw3IvjIgHih2LJVyDsHotIsYCw9M7qax+m0/SQW51hGsQZmaWl2sQZmaWlxOEmZnl5QRhZmZ5OUGYmVleThBmZpbX/wPov24If76fLgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from openai.embeddings_utils import cosine_similarity, get_embedding\n",
    "from sklearn.metrics import PrecisionRecallDisplay\n",
    "\n",
    "def evaluate_embeddings_approach(\n",
    "    labels = ['negative', 'positive'], \n",
    "    engine = 'babbage-similarity',\n",
    "):\n",
    "    label_embeddings = [get_embedding(label, engine=engine) for label in labels]\n",
    "\n",
    "    def label_score(review_embedding, label_embeddings):\n",
    "        return cosine_similarity(review_embedding, label_embeddings[1]) - cosine_similarity(review_embedding, label_embeddings[0])\n",
    "\n",
    "    engine_col_name = engine.replace('-','_').replace('_query','')\n",
    "    probas = df[engine_col_name].apply(lambda x: label_score(x, label_embeddings))\n",
    "    preds = probas.apply(lambda x: 'positive' if x>0 else 'negative')\n",
    "\n",
    "    report = classification_report(df.sentiment, preds)\n",
    "    print(report)\n",
    "\n",
    "    display = PrecisionRecallDisplay.from_predictions(df.sentiment, probas, pos_label='positive')\n",
    "    _ = display.ax_.set_title(\"2-class Precision-Recall curve\")\n",
    "\n",
    "evaluate_embeddings_approach(labels=['negative', 'positive'], engine='babbage-similarity')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this classifier already performs extremely well. We used similarity embeddings, and the simplest possible label name. Let's try to improve on this by using more descriptive label names, and search embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative       0.65      0.93      0.76       136\n",
      "    positive       0.99      0.91      0.95       789\n",
      "\n",
      "    accuracy                           0.92       925\n",
      "   macro avg       0.82      0.92      0.86       925\n",
      "weighted avg       0.94      0.92      0.92       925\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAwoUlEQVR4nO3deZwU1bn/8c+XAQRRwCAigiyiRkEFFY2iROJucNeoGBc04pKo+cWruXqNRsmiRk3uNSYuuXpFY3Ahxl0JKihuUUBAcAMMIoiKsokswszz+6NqxmbsmSmY6emZ4ft+veY1Vaeqq57TA/10nVN1jiICMzOzypoVOwAzM2uYnCDMzCwvJwgzM8vLCcLMzPJygjAzs7ycIMzMLC8nCKtTkoZKerHYcdQlST+U9M8M+90q6Yr6iKk+SJot6cB0+SpJfy12TFa/nCAMSRtJukPSB5K+kDRZ0mHFjiuL9ENshaRlkj6RdJekTeryHBFxb0QcnGG/cyPiV3V57nKSQtKXaT3nSfq9pJJCnMusnBOEATQHPgT2A9oBvwAekNSjmEGtgyMiYhNgN6A/SfxrkdS83qOqe33Teu4HnAicWeR46lQT+Rs1KU4QRkR8GRFXRcTsiCiLiMeBfwO7V/UaSVtLekjSAkmfS7q5iv3+R9KHkpZKmihpYM62PSVNSLd9Iun3aXkrSX9Nj7tY0uuSOmWoxzzgKWCn9Dgh6SeSZgAz0rLD0yukxZJelrRLTXXKbTZT4g+SPk3jflNS+fnukvTrnOMNkzRT0kJJj0raKmdbSDpX0ow0lj9JUk11TOs5E3gJ6JdzvPWpVy9Jz6Vln0m6V1L7LDFUJumo9PxLJc2SdGhaXtFMla5XNFVJ6pG+Dz+SNAd4TtJTks6vdOwpko5Nl3eQNCZ9T9+VdML6xGvZOEHYN6QfxtsD06vYXgI8DnwA9AC6APdVcbjXST7IvgX8DXhQUqt02/8A/xMRbYFewANp+ekkVzJbAx2Ac4EVGeLeGvg+8EZO8dHAd4DeknYF7gTOSY97G/Bo2sSWtU4HA98leX/aAScAn+eJZX/gmnR75/S4lY93OLAHsEu63yE11TE99g7AQGBmur6+9VIa41bAjiTv91VZYqgUz57A3cAlQHuS92f2Ohxiv/T8hwAjgSE5x+4NdAeekNQGGEPy72gL4CTgz+k+VgBOELYWSS2Ae4EREfFOFbvtSfKhckl69bEyIvJ2TEfEXyPi84hYExE3AhsB3043rwa2lbR5RCyLiFdzyjsA20ZEaURMjIil1YT9sKTFwIvA88Bvc7ZdExELI2IFcDZwW0T8Kz3uCGAVsNc61Gk1sCmwA6CIeDsi5ufZ74fAnRExKSJWAZcBe1dqtrs2IhZHxBxgLDlXBFWYJOlL4G1gHPDntHy96hURMyNiTESsiogFwO9JPqzX1Y/Suo5Jr0DnVfNvJ5+r0thWAP8A+knqnm77IfBQ+h4eDsyOiP9L/z29Afwd+MF6xGwZOEFYBUnNgHuAr4Dzc8qfUtI5ukzSD0m+aX4QEWsyHPNiSW9LWpJ+iLcDNk83/4jkm/g7aTPS4Wn5PcBo4D5JH0n6XZq4qnJ0RLSPiO4R8eP0g6bchznL3YH/SJthFqfxbE3yAZqpThHxHHAz8CfgU0m3S2qbZ9etSL61l79uGcmVRpecfT7OWV4ObAIgaXrO+z0wZ5/d0n1OJLkqalObeknqJOk+JZ3eS4G/8vXfZl1sDcxaj9eVq/gbRcQXwBMkVweQXE3cmy53B75TqZ4/BLasxbmtGk4QBiRt68AdQCfguIhYXb4tIg6LiE3Sn3tJ/kN3Uw2diumH289Jmk82i4j2wBKSpg0iYkZEDCFpLrgOGCWpTUSsjoirI6I3MIDkm+Np61m13OGKPwR+kyaT8p+NI2Jk1jqlcd8UEbsDvUkS3CV5dvuI5AMNgLR5pAMwL8Px++S83+MrbYuIeAB4BbiylvX6Lcn7s3PazHcK6d9mHX1I0kSYz5fAxjnr+T7MKw8pPRIYImlvoBXJ1VX5eZ6vVM9NIuK89YjZMnCCsHK3kLQDH1HpG3g+rwHzgWsltVHSqbxPnv02BdYAC4Dmkq4EKr5tSzpFUseIKAMWp8Vlkr4naee0/XwpSbNOWW0ql/oLcK6k7yjRRtJgSZtmrZOkPdLXtyD58FtZRWwjgTMk9ZO0EcmH8b8iYnYd1APgWmCYpC1rUa9NgWXAEkldyJ/osriDpK4HSGomqUvaTwIwGThJUgtJ/YHjMxzvSZLkOhy4P/33AUlfyvaSTk2P1yL9e+y4nnFbDZwgjLS99xySNvCPKzUnfUNElAJHANsCc4C5JM0elY0GngbeI2luWcnaTT6HAtMlLSPpsD4pTU5bAqNIksPbJP0K99SymkTEBGAYSRPRIpJO3qHrWKe2JB/Ii9I6fQ5cn+dczwBXkLSRzyf5hn1S5f1qUZc3gRdI+hbWt15XkzRbLSFp1nloPWN5DTgD+EN6rOf5+urpCpK6L0rP97cMx1uVxnJg7v5p89PBJO/jRyRNdNeR9GtZAcgTBpmZWT6+gjAzs7ycIMzMLC8nCDMzy8sJwszM8moyg2Ntvvnm0aNHj2KHYWbWqEycOPGziOiYb1uTSRA9evRgwoQJxQ7DzKxRkfRBVdvcxGRmZnk5QZiZWV5OEGZmlpcThJmZ5eUEYWZmeRUsQUi6U8m0jNOq2C5JNymZknGqpN1ytp2uZCrGGZJOL1SMZmZWtUJeQdxFMlpnVQ4Dtkt/ziYZbhpJ3wJ+STIhyp7ALyVtVsA4zcwsj4I9BxERL1SaXrGyo4C7IxlO9lVJ7SV1BgYBYyJiIYCkMSSJZmQh4lz+1RpuHVebybDMrKnYeKPmDB3Qg1YtSoodSoNQzAflurD23ABz07Kqyr9B0tkkVx9069ZtvYJY8VUpfxw7c71ea2ZNR/nMB327tmfvXh2KG0wD0aifpI6I24HbAfr3779eE1t02GQj/n3N4DqNy8wan9f+vZATbnuFMs+RU6GYCWIeyWTn5bqmZfNImplyy8fVW1RmtkEqTwyPT53PZ8tWURZBWRmURrBqdSnLVpXSZqMSSsuCsoCysqA0IllPy0ojaNOyhDP26UnL5o3/JtFiJohHgfMl3UfSIb0kIuZLGg38Nqdj+mDgsmIFaWYbhtWlydTXI1+bw8jX5tTqWP17fIvduzf+e2sKliAkjSS5Ethc0lySO5NaAETErSQTk3+fZP7c5SRz2hIRCyX9Cng9PdTw8g5rM7NCGbhdRx75yT60bllCM4mSZqJEQoKSZsnvFiXNKJFo1kw0S8tz931p1mecesdrTaaZqpB3MQ2pYXsAP6li253AnYWIy8ysKn23bl+r1wvVTSANRONvJDMzs4JwgjAzs7ycIMzMLC8nCDMzy8sJwszM8nKCMDOzvJwgzMwsLycIMzPLywnCzMzycoIwM7O8nCDMzCyvTGMxpSOrbgWsAGZHRFlBozIzs6KrMkFIakcymN4QoCWwAGgFdJL0KvDniBhbL1GamVm9q+4KYhRwNzAwIhbnbpC0O3CqpG0i4o4CxmdmZkVSZYKIiIOq2TYRmFiQiMzMGqnPv1wFwA9ufYWn/99AdtiybZEjqp0aO6mVOEXSlel6N0l7Fj40M7PGZZeu7SuW35izuGhx1JUsdzH9GdibpC8C4AvgTwWLyMyskeq5eRteveyAYodRZ7LcxfSdiNhN0hsAEbFIUssCx2VmZkWW5QpitaQSIAAkdQR8m6uZWROXJUHcBPwD2ELSb4AXgd8WNCozMyu6GpuYIuJeSROBAwABR0fE2wWPzMzMiqrGBCHpJuC+iHDHtJnZBiRLJ/VE4BeSvk3S1HRfREwobFhmZo3bG3MW0aKkGV+tKaOkGRy7W1dalDSu4e+yNDGNAEZI+hZwHHCdpG4RsV3BozMza2TKIgB4YMJcHpgwt6K8V8dN6N/jW8UKa71kGqwvtS2wA9AdcB+EmVkeW7VvzdH9tqL3Vm3ZdotN+HzZV1wyaiqrS6PYoa2zLH0QvwOOAWYB9wO/qjw2k5mZfe2/T9q1YvmVWZ8XMZLayXIFMQvYOyI+K3QwZmbWcFQ33PcOEfEO8DrQTVK33O0RManQwZmZWfFUdwVxEXA2cGOebQHsX5CIzMyakPIRXof85VW2ateKxy8cyLfaNI7Riqob7vvsdPGwiFiZu01Sq4JGZWbWRPTbuj0AJc3ER0tW8snSlY0mQWS5KffljGVmZlZJ1802Zva1g/nTybvWvHMDU10fxJZAF6C1pF1JhtkAaAtsXA+xmZlZEVXXB3EIMBToCvw+p/wL4L8KGJOZ2QZjTWkZzRvoE9bV9UGUP0F9XET8vR5jMjNrcr5KH5Q74dZX6LJZa5Z/VcqchcsB+MXgHTlr4DbFDC+vKtOWpFPSxR6SLqr8k+Xgkg6V9K6kmZIuzbO9u6RnJU2VNE5S15xtv5M0XdLbkm6SpMqvNzNrLD77Irmb6YtVa3jn4y/ou3V7jt21C0BFomhoqmtiapP+3mR9DpxOMvQn4CBgLvC6pEcj4q2c3W4A7o6IEZL2B64BTpU0ANgH2CXd70VgP2Dc+sRiZlZsZ+7bk127tWfHzm1p1aKkonzsu58WMarqVdfEdFv6++r1PPaewMyIeB9A0n3AUUBuguhN8rwFwFjg4fLTA62AliSd4y2AT9YzDjOzBmHXbpsVO4R1UmPPSNrU01ZSi7Q5aEFO81N1ugAf5qzPTctyTQGOTZePATaV1CEiXiFJGPPTn9H5JimSdLakCZImLFiwIENIZmaWVZau84MjYilwODCbZFTXS+ro/BcD+0l6g6QJaR5QKmlbYEeSO6i6APtLGlj5xRFxe0T0j4j+HTt2rKOQzMzqz6Llq7n7lQ94+I15xQ7lG7IkiPJmqMHAgxGxJOOx5wFb56x3TcsqRMRHEXFsROwKXJ6WLSa5mng1IpZFxDLgKWDvjOc1M2t07n/9w5p3qmdZRnN9XNI7wArgPEkdgZU1vAaSQf62k9STJDGcBJycu4OkzYGFEVEGXAbcmW6aAwyTdA1JH8R+wH9nOKeZWaMy+9rBHH/Ly8xcsIxT7/gXAKVlwZrS4Ih+W3HqXt2LFluNVxARcSkwAOgfEauBL0k6m2t63RrgfGA0yQRDD0TEdEnDJR2Z7jYIeFfSe0An4Ddp+SiSYcbfJOmnmBIRj61LxczMGosJHyxiwRerGD/jM17790JWl5bx9sdLeWzKR0WNK8uEQS2AU4Dvpo8iPA/cmuXgEfEk8GSlsitzlkeRJIPKrysFzslyDjOzxu4Xg3fkuXc+5dS9unNIny1p1kycdPsrlBV5ErosTUy3kNxm+ud0/dS07KxCBWVmtiE5a+A2DfJJ6iwJYo+I6Juz/pykKYUKyMzMGoYsdzGVSupVviJpG6C0cCGZmVlDkOUK4hJgrKT3Se4o6g6cUdCozMys6GpMEBHxrKTtgG+nRe9GxKrChmVmZsWW5S6mVsCPgX1JxkgaL+nWytOQmplZ05KlielukkmC/piunwzcA/ygUEGZmVnxZUkQO0VE75z1sZLeqnJvMzNrErLcxTRJ0l7lK5K+A0woXEhmZtYQZLmC2B14WdKcdL0byfAYbwIREbtU/VIzM2ussiSIQwsehZmZrWXq3CUs/6qUiKBYMy5nuc31g/oIxMzMvrb8q+R55HmLV9B1s42LEkOWPggzM6tnvz8hGeGotIgj9jlBmJk1QNM/WgrAE2/OL1oMThBmZg3Q8bt3BWDJ8tVFiyFTgpB0e3XrZmZWt3bs3JYWJaKkWXE6qCH7FcRtNaybmVkTkylBRMTE6tbNzKzpqfI2V0mPkQzOl1dEHFnVNjMza/yqew7ihnqLwszMGpwqE0REPF++LKk10C0i3q2XqMzMjNWlwVdryop2/hr7ICQdAUwGnk7X+0l6tMBxmZkZ8L8v/rto587SSX0VsCewGCAiJgM9CxaRmZmtJSKY+ekXPDrlI2Z+uqzezptlsL7VEbGk0mBRxXv228xsA9F1s9bMXbSCnpc9WVE2oFcH/jZsr2peVXeyXEFMl3QyUCJpO0l/BF4ucFxmZhu8FiXJR/TgnTtzwf7bArBydWm9nT/LFcQFwOXAKmAkMBr4VSGDMjMzGHvxoLXWJ81ZxKrV9ddpnWW47+XA5ZKuS1bji8KHZWZmxZblLqY90tnjpgJvSpoiaffCh2ZmZsWUpYnpDuDHETEeQNK+wP8BnmrUzKwJy9JJXVqeHAAi4kVgTeFCMjOzhqC6sZh2Sxefl3QbSQd1ACcC4wofmpmZ5Xpp5udAcidTqxYlBT9fdU1MN1Za/2XOsp+DMDMrknmLV9Cr4yYFP091YzF9r+BnNzOzzG78QV/+48EpvPbvhcVNELkkDQb6AK3KyyJieKGCMjOzb3rv0+Qpg8seepMhe3Yr+Pmy3OZ6K0m/wwWAgB8A3bMcXNKhkt6VNFPSpXm2d5f0rKSpksZJ6pqzrZukf0p6W9JbknpkrZSZWVP080N2oF3rFrRpWfj+B8h2F9OAiDgNWBQRVwN7A9vX9CJJJcCfgMOA3sAQSb0r7XYDcHdE7AIMB67J2XY3cH1E7EgyWOCnGWI1M2uySpqJo/ttRYvmWWeLrp0sZ1mR/l4uaStgNdA5w+v2BGZGxPsR8RVwH3BUpX16A8+ly2PLt6eJpHlEjAGIiGXpE91mZlZPsiSIxyW1B64HJgGzSW55rUkX4MOc9blpWa4pwLHp8jHAppI6kFyhLJb0kKQ3JF2fXpGsRdLZkiZImrBgwYIMIZmZWVY1JoiI+FVELI6Iv5P0PewQEVfU0fkvBvaT9AawHzAPKCXpPB+Ybt8D2AYYmie22yOif0T079ixYx2FZGZmUP2DcsdWs42IeKiGY88Dts5Z75qWVYiIj0ivICRtAhwXEYslzQUmR8T76baHgb1Ihv0wM7N6UN1trkdUsy2AmhLE68B2knqSJIaTgJNzd5C0ObAwIsqAy4A7c17bXlLHiFgA7A9MqOF8ZmZWh6p7UO6M2hw4ItZIOp9k/ogS4M6ImC5pODAhIh4FBgHXSArgBeAn6WtLJV0MPKtkKruJwF9qE4+Zma2bTA/Kra+IeBJ4slLZlTnLo4BRVbx2DB4x1sxsLZ8sXcXi5atZtaaUjZoX9nmI+rmZ1szM6sTT0z8G4OVZnxf8XE4QZmaNyH1n7wXA6jWFn3o0y1AbG0u6QtJf0vXtJB1e8MjMzOwbNtmooD0Da8lyBfF/wCqSITYguSPp1wWLyMzMGoQsCaJXRPyOZIgN0iEvVNCozMwsr7JIpuOZ9tHSgp8rS4L4SlJr0kmCJPUiuaIwM7N61kzJ9/P/Hf9+wc+VpTHrKuBpYGtJ9wL7kGfYCzMzK7ydurRj62+1pnO71gU/V40JIiL+KWkiyVAXAn4aEZ8VPDIzM8vrw4Ur+HDhClZ8VUrrAs4NkeUupseAg4FxEfG4k4OZWXF1+9bGACz/ak1Bz5OlD+IGkpFV35I0StLxklrV9CIzMyuMswb2rJfzZGlieh54Pp2PYX9gGMmgem0LHJuZmRVRpicu0ruYjiCZm3o3YEQhgzIzs+KrMUFIeoBk+tCngZuB59Phuc3MrAnLcgVxBzAkIkoLHYyZmTUc1c0ot39EPAe0AY6S1n54OsOMcmZmVgDT5yVPUU+as5iDencq2Hmqu4tpv/T3EXl+PFifmVmR7Ny1HQDTP1pS0PNUN6PcL9PF4RHx79xt6TSiZmZWBMfv3pVfPDyNls0LO2NDlqP/PU9Z3lngzMys6aiuD2IHoA/QTtKxOZvaAn5QzsysiavuLqZvk/Q1tCfpdyj3BcnDcmZm1oRV1wfxCPCIpL0j4pV6jMnMzBqA6pqYfp5OFHSypCGVt0fEhQWNzMzMiqq6Jqa3098T6iMQMzNrWKprYnos/V0x7pKkZsAmEVH4ue7MzKxaI1+bw/G7dWWLtoW5byjLfBB/k9RWUhtgGsmw35cUJBozM6tROi01Hy5cwVWPTS/YebI8B9E7vWI4GngK6AmcWrCIzMysWq1blrBHj80AaNWiiDPKAS0ktSBJEI9GxGogChaRmZnV6MFzB9ClfWuEat55PWVJELcBs0kG7XtBUnfAfRBmZk1clhnlbgJuyin6QNL3CheSmZk1BFk6qdtJ+r2kCenPjSRXE2Zm1oRlaWK6k2R4jRPSn6XA/xUyKDMzK74sM8r1iojjctavljS5QPGYmVkDkeUKYoWkfctXJO0DrChcSGZm1hBkuYI4F7hbUrt0fRFweuFCMjOzhqDaKwhJ/YDtgJOAXYBdImLXiJia5eCSDpX0rqSZki7Ns727pGclTZU0TlLXStvbSpor6ebMNTIzszpRZYKQdCXwAHAc8ARw4rqMwSSpBPgTcBjQGxgiqXel3W4A7o6IXYDhwDWVtv8KeCHrOc3MrO5UdwVxItAvIoYAewBnr+Ox9wRmRsT7EfEVcB9wVKV9egPPpctjc7dL2h3oBPxzHc9rZmZ1oLoEsSoilgNExOc17JtPF+DDnPW5aVmuKUD5dKbHAJtK6pCOGnsjcHF1J5B0dvnzGQsWLFjH8MzMrDrVdVJvI+nRdFlAr5x1IuLIOjj/xcDNkoaSNCXNA0qBHwNPRsRcqepxRiLiduB2gP79+3t8KDOzOlRdgqjcHHTDOh57HrB1znrXtKxCRHxEegUhaRPguIhYLGlvYKCkHwObAC0lLYuIb3R0m5lZYVQ3YdDztTz268B2knqSJIaTgJNzd5C0ObAwIsqAy0ie2iYifpizz1Cgv5ODmVn9qu4upsckHZEO9V152zaShks6s6rXR8Qa4HxgNMn0pQ9ExPT0deXNU4OAdyW9R9Ih/Zta1MXMzOpQdU1Mw4CLgP+WtBBYALQCegCzgJsj4pHqDh4RTwJPViq7Mmd5FDCqhmPcBdxV3T5mZlb3qmti+hj4OfBzST2AziRDbLxXfneTmZk1XVmG2iAiZpNMGmRmZhuIdX22wczMGoh5i1fw90lziSjMXf5OEGZmjVxZgZ4Cc4IwM2ukLjpo+4Iev8Y+iHT+h6uA7un+AiIitiloZGZmVlRZOqnvAH4GTCQZBsPMzDYAWRLEkoh4quCRmJlZg5IlQYyVdD3wELCqvDAiJhUsKjMzK7osCeI76e/+OWUB7F/34ZiZWUNRY4KIiO/VRyBmZtaw1Hibq6R2kn5fPjGPpBsltauP4MzMrHiyPAdxJ/AFcEL6sxT4v0IGZWZmxZelD6JXRByXs361pMkFisfMzBqILFcQKyTtW76SPji3onAhmZlZQ5DlCuI8YETa7yBgITC0kEGZmVnxZbmLaTLQV1LbdH1poYMyM7PiqzJBSDolIv4q6aJK5QBExO8LHJuZmRVRdVcQbdLfm9ZHIGZm1rBUN+Xobenvq+svHDMzayiyPCj3O0ltJbWQ9KykBZJOqY/gzMyseLLc5npw2jF9OMm81NsClxQyKDMzK74sCaK8GWow8GBELClgPGZm1kBkeQ7icUnvkDwcd56kjsDKwoZlZmbFVuMVRERcCgwA+kfEauBL4KhCB2ZmZsVV3XMQ+0fEc5KOzSnL3eWhQgZmZmbFVV0T037Ac8ARebYFThBmZk1adc9B/DL9fUb9hWNmZg1Flucgfiupfc76ZpJ+XdCozMys6LLc5npYRCwuX4mIRcD3CxaRmZk1CFkSRImkjcpXJLUGNqpmfzMzawKyPAdxL/CspPJpRs8ARhQuJDMzawiyzAdxnaQpwIFp0a8iYnRhwzIzs2LLcgUB8DawJiKekbSxpE0j4otCBmZmZsWV5S6mYcAo4La0qAvwcJaDSzpU0ruSZkq6NM/27ukIsVMljZPUNS3vJ+kVSdPTbSdmrpGZmdWJLJ3UPwH2AZYCRMQMYIuaXiSpBPgTcBjQGxgiqXel3W4A7o6IXYDhwDVp+XLgtIjoAxwK/HfurbZmZlZ4WRLEqoj4qnxFUnOSJ6lrsicwMyLeT19/H98cw6k3ydPaAGPLt0fEe2kiIiI+Aj4FOmY4p5mZ1ZEsCeJ5Sf8FtJZ0EPAg8FiG13UBPsxZn5uW5ZoClI/1dAywqaQOuTtI2hNoCcyqfAJJZ0uaIGnCggULMoRkZmZZZUkQ/wksAN4EzgGeBH5RR+e/GNhP0hskYz/NA0rLN0rqDNwDnBERZZVfHBG3R0T/iOjfsaMvMMzM6lK1dzGl/QjTI2IH4C/reOx5wNY5613Tsgpp89Gx6bk2AY4rf2pbUlvgCeDyiHh1Hc9tZma1VO0VRESUAu9K6rYex34d2E5ST0ktgZOAR3N3kLS5pPIYLgPuTMtbAv8g6cAetR7nNjOzWsryHMRmwHRJr5FMFgRARBxZ3YsiYo2k84HRQAlwZ0RMlzQcmBARjwKDgGskBfACyR1TACcA3wU6SBqalg2NiMlZK2ZmZrWTJUFcsb4Hj4gnSfoscsuuzFkeRfKMReXX/RX46/qe18zMaq+6GeVaAecC25J0UN8REWvqKzAzMyuu6vogRgD9SZLDYcCN9RKRmZk1CNU1MfWOiJ0BJN0BvFY/IZmZWUNQ3RXE6vIFNy2ZmW14qruC6Ctpaboskiepl6bLERFtCx6dmZkVTZUJIiJK6jMQMzNrWLIMtWFmZhugrBMGNUqrV69m7ty5rFy5stihWCPSqlUrunbtSosWLYodillRNekEMXfuXDbddFN69OiBpGKHY41ARPD5558zd+5cevbsWexwzIqqSTcxrVy5kg4dOjg5WGaS6NChg686zWjiCQJwcrB15n8zZokmnyDMzGz9OEEU2Mcff8xJJ51Er1692H333fn+97/Pe++9x+zZs9lpp53q7DxXXnklzzzzDADjx4+nT58+9OvXj3nz5nH88cfX6tgRwf7778/SpUsryh5++GEk8c4771SUzZ49m9atW9OvXz969+7NueeeS1nZN+Z5WicvvPACu+22G82bN2fUqKpHfp84cSI777wz2267LRdeeCERyay4Cxcu5KCDDmK77bbjoIMOYtGiRQA8/vjjXHnllVUez8ycIAoqIjjmmGMYNGgQs2bNYuLEiVxzzTV88skndX6u4cOHc+CBBwJw7733ctlllzF58mS6dOlS7QdrZWvWfPOh+SeffJK+ffvStu3Xz0aOHDmSfffdl5EjR661b69evZg8eTJTp07lrbfe4uGHH16/CqW6devGXXfdxcknn1ztfueddx5/+ctfmDFjBjNmzODpp58G4Nprr+WAAw5gxowZHHDAAVx77bUADB48mMcee4zly5fXKj6zpqxJ38WU6+rHpvPWR0tr3nEd9N6qLb88ok+V28eOHUuLFi0499xzK8r69u0LJN+2y82ePZtTTz2VL79Mptu4+eabGTBgAPPnz+fEE09k6dKlrFmzhltuuYUBAwbwox/9iAkTJiCJM888k5/97GcMHTqUww8/nMWLF/PAAw8wevRonnrqKX7zm99w+OGHM23aNEpLS7n00ksZN24cq1at4ic/+QnnnHMO48aN44orrmCzzTbjnXfe4b333lurHvfeey9nn312xfqyZct48cUXGTt2LEcccQRXX331N+revHlzBgwYwMyZM9frvS3Xo0cPAJo1q/q7zPz581m6dCl77bUXAKeddhoPP/wwhx12GI888gjjxo0D4PTTT2fQoEFcd911SGLQoEE8/vjjnHDCCbWK0ayp2mASRDFMmzaN3Xffvcb9tthiC8aMGUOrVq2YMWMGQ4YMYcKECfztb3/jkEMO4fLLL6e0tJTly5czefJk5s2bx7Rp0wBYvHjxWsc666yzePHFFzn88MM5/vjj10pEd9xxB+3ateP1119n1apV7LPPPhx88MEATJo0iWnTpuW9tfOll17itttuq1h/5JFHOPTQQ9l+++3p0KEDEydO/EY9ly9fzrPPPsvw4cO/cbyBAwfyxRdffKP8hhtuqLgKWhfz5s2ja9euFetdu3Zl3rxkdttPPvmEzp07A7DllluudfXWv39/xo8f7wRhVoUNJkFU902/2FavXs3555/P5MmTKSkpqfgGv8cee3DmmWeyevVqjj76aPr168c222zD+++/zwUXXMDgwYMrPuCz+Oc//8nUqVMrmpyWLFnCjBkzaNmyJXvuuWeV9/0vXLiQTTfdtGJ95MiR/PSnPwXgpJNOYuTIkRUJYtasWfTr1w9JHHXUURx22GHfON748eMzx1yXJK11h9IWW2zBRx99VJRYzBqDDSZBFEOfPn0ytf//4Q9/oFOnTkyZMoWysjJatWoFwHe/+11eeOEFnnjiCYYOHcpFF13EaaedxpQpUxg9ejS33norDzzwAHfeeWemeCKCP/7xjxxyyCFrlY8bN442bdpU+brmzZtTVlZGs2bNWLhwIc899xxvvvkmkigtLUUS119/PfB1H0R16voKokuXLsydO7dife7cuXTp0gWATp06MX/+fDp37sz8+fPZYostKvZbuXIlrVu3XufzmTUUf/vXHADGvvMpB/buVOfHdyd1Ae2///6sWrWK22+/vaJs6tSp3/gGvWTJEjp37kyzZs245557KC0tBeCDDz6gU6dODBs2jLPOOotJkybx2WefUVZWxnHHHcevf/1rJk2alDmeQw45hFtuuYXVq5OR3N97772Kfo/qfPvb3+b9998HYNSoUZx66ql88MEHzJ49mw8//JCePXuu01XB+PHjmTx58jd+1ic5AHTu3Jm2bdvy6quvEhHcfffdHHXUUQAceeSRjBgxAoARI0ZUlENS/7q8k8ysvp2//7YAzP685v/H68MJooAk8Y9//INnnnmGXr160adPHy677DK23HLLtfb78Y9/zIgRI+jbty/vvPNOxbf5cePG0bdvX3bddVfuv/9+fvrTnzJv3jwGDRpEv379OOWUU7jmmmsyx3PWWWfRu3dvdtttN3baaSfOOeecvHctVTZ48OCKjt6RI0dyzDHHrLX9uOOO+8bdTHXl9ddfp2vXrjz44IOcc8459OnzdVNhv379Kpb//Oc/c9ZZZ7HtttvSq1eviqatSy+9lDFjxrDddtvxzDPPcOmll1a8ZuzYsQwePLggcZvVhwG9OgDw6yfeLsjxVX6/eGPXv3//mDBhwlplb7/9NjvuuGORImo65s+fz2mnncaYMWOKHUqd+eSTTzj55JN59tln8273vx1rDMrKgm3+60mGDezJ5YN7r9cxJE2MiP75trkPwmrUuXNnhg0bxtKlS9d6FqIxmzNnDjfe6GnWrXFr1kzMvrZwV8FOEJZJU7sVdI899ih2CGYNXpPvg2gqTWhWf/xvxizRpBNEq1at+Pzzz/0f3jIrnw+i/FZjsw1Zk25i6tq1K3PnzmXBggXFDsUakfIZ5cw2dE06QbRo0cKzgpmZracm3cRkZmbrzwnCzMzycoIwM7O8msyT1JIWAB/U4hCbA5/VUTiNxYZW5w2tvuA6byhqU+fuEdEx34YmkyBqS9KEqh43b6o2tDpvaPUF13lDUag6u4nJzMzycoIwM7O8nCC+dnvNuzQ5G1qdN7T6guu8oShInd0HYWZmefkKwszM8nKCMDOzvDaoBCHpUEnvSpop6dI82zeSdH+6/V+SehQhzDqVoc4XSXpL0lRJz0rqXow461JNdc7Z7zhJIanR3xKZpc6STkj/1tMl/a2+Y6xrGf5td5M0VtIb6b/v7xcjzroi6U5Jn0qaVsV2SbopfT+mStqt1ieNiA3iBygBZgHbAC2BKUDvSvv8GLg1XT4JuL/YcddDnb8HbJwun7ch1Dndb1PgBeBVoH+x466Hv/N2wBvAZun6FsWOux7qfDtwXrrcG5hd7LhrWefvArsB06rY/n3gKUDAXsC/anvODekKYk9gZkS8HxFfAfcBR1Xa5yhgRLo8CjhAkuoxxrpWY50jYmxELE9XXwUa+zjXWf7OAL8CrgNW1mdwBZKlzsOAP0XEIoCI+LSeY6xrWeocQPkcue2Aj+oxvjoXES8AC6vZ5Sjg7ki8CrSX1Lk259yQEkQX4MOc9blpWd59ImINsAToUC/RFUaWOuf6Eck3kMasxjqnl95bR8QT9RlYAWX5O28PbC/pJUmvSjq03qIrjCx1vgo4RdJc4EnggvoJrWjW9f97jZr0fBCWnaRTgP7AfsWOpZAkNQN+Dwwtcij1rTlJM9MgkqvEFyTtHBGLixlUgQ0B7oqIGyXtDdwjaaeIKCt2YI3FhnQFMQ/YOme9a1qWdx9JzUkuSz+vl+gKI0udkXQgcDlwZESsqqfYCqWmOm8K7ASMkzSbpK320UbeUZ3l7zwXeDQiVkfEv4H3SBJGY5Wlzj8CHgCIiFeAViSD2jVVmf6/r4sNKUG8DmwnqaekliSd0I9W2udR4PR0+XjguUh7fxqpGussaVfgNpLk0NjbpaGGOkfEkojYPCJ6REQPkn6XIyNiQnHCrRNZ/m0/THL1gKTNSZqc3q/HGOtaljrPAQ4AkLQjSYJoyvMPPwqclt7NtBewJCLm1+aAG0wTU0SskXQ+MJrkDog7I2K6pOHAhIh4FLiD5DJ0Jkln0EnFi7j2Mtb5emAT4MG0P35ORBxZtKBrKWOdm5SMdR4NHCzpLaAUuCQiGu3VccY6/wfwF0k/I+mwHtqYv/BJGkmS5DdP+1V+CbQAiIhbSfpZvg/MBJYDZ9T6nI34/TIzswLakJqYzMxsHThBmJlZXk4QZmaWlxOEmZnl5QRhZmZ5OUHYepNUKmmypGmSHpPUvo6PPzu9Zx9Jy6rYp7Wk5yWVSOohaUUa01uSbk2fnF6Xc/aXdFO6PEjSgJxt50o6rTZ1So9zlaSLa9jnLknHr8Mxe1Q1ymddk3Rk+eipko6W1Dtn2/D0wcv1Oe59khrzw3tNjhOE1caKiOgXETuRPDfykyLEcCbwUESUpuuzIqIfsAvJCJ5Hr8vBImJCRFyYrg4CBuRsuzUi7q5twI1dRDwaEdemq0eTvM/l266MiGfW89C3AD+vZXhWh5wgrK68QjowmKRekp6WNFHSeEk7pOWdJP1D0pT0Z0Ba/nC673RJZ6/jeX8IPFK5MB1s8WVg2/Tb9XP6es6Lbul5f5Be/UyR9EJaNkjS40rmAjkX+Fl6RTKw/Ju/pB0kvVZ+rvT4b6bLu6dXNBMljVYNo2lKGibp9TSGv0vaOGfzgZImSHpP0uHp/iWSrk9fM1XSOevyZklaJukP6Xv9rKSOaXk/JYP4TU3/Rpul5Rfq6/lC7kvLhkq6Of37HQlcn75HvcqvfJTM1fBgznkHSXo8XT5Y0iuSJkl6UNIm6W7j0zpvMA/wNnROEFZrkkpIhjQof0r5duCCiNgduBj4c1p+E/B8RPQlGdd+elp+Zrpvf+BCSZlG0FUyxMI2ETE7z7aN05jeBP4IjIiIXYB70zgArgQOSeNZ6+nx9Ji3An9Ir5LG52x7B2gpqWdadCJwv6QW6bmOT+tzJ/CbGqrxUETskcbwNsn4QeV6kAxrPRi4VVKrdPuSiNgD2AMYlhNHed23kvRkFedrQ/KkcR/geZKncQHuBv4zfY/ezCm/FNg1LT+30nv0Msnf/JL0PZqVs/kZ4DuS2qTrJwL3KWky/AVwYETsBkwALkqPV0byFHDfqt8uq09OEFYbrSVNBj4GOgFj0m+DA0iG7phMMs5T+bfo/UmaEYiI0ohYkpZfKGkKybhIW5N9ELnNgcWVynql530JeCIingL2BspnULsH2Dddfgm4S9IwkuEa1sUDJB96pL/vB75NMhDgmDSGX1Dz/Bo7pVdZb5JcDfXJPUdElEXEDJJxk3YADiYZb2cy8C+S4ejXer8i4qOIqGr2tLI0VoC/AvtKage0j4jn0/IRJJPTAEwF7lUy2u+aGuqSG8Ma4GngiPSKYDDJld5eJE1SL6V1OB3IncXwU2CrrOexwvKlnNXGiojol35bH03SB3EXsDjtB6iRpEHAgcDeEbFc0jiSQdUynT/PvrOynjsizpX0HZIPr4mSds94Xkg+ZB+U9FByqJghaWdgekTsvQ7HuQs4OiKmSBpKOqBeeYiVQyaZLeyCiBidu0HrPz1uTWPtDCZJFkcAl6d1zOo+4HyS/qkJEfGFJAFjImJIFa9pRfJ3tQbAVxBWa+mMdBeSDI62HPi3pB9AxTy55U0Gz5JMa1relt6OZEj1RWly2IHkG2bW8y4CStKml+q8zNcDL/6QpK0bSb0i4l8RcSXJKJ9bV3rdFyTDg+c79yySQe+u4Otv5O8CHZXMPYCkFpL65Ht9jk2B+Wnz1A8rbfuBpGaSepFMrfkuSSI+L90fSdvnNONk0YxkpGKAk4EX0yu5RZIGpuWnAs8ruQNs64gYC/wnyd9qk0rHq/I9ImnC2o1kNrv70rJXgX0kbZvG30bS9jmv2R6ol7uxrGZOEFYnIuINkuaIISQfdD9Km42m8/VUkD8Fvpc2p0wkaWp4Gmgu6W3gWpIPkHXxT75uMqrKBcAZkqaSfPj9NC2/XtKbSm4PfZlkXuNcjwHHpB2wA/mm+4FT+HrOga9IPnyvS+s+mZy7oKpwBUlT0UvAO5W2zQFeI5nl79yIWAn8L/AWMCmN+zYqtQTU0AfxJbBn+tr9geFp+ekk78dUoF9aXgL8Nf17vQHclGeCofuASyS9kSayCumdZY8Dh6W/iYgFJJM1jUzP9QpJ0xmSOpFclX5cRexWzzyaqzVqSqYP/VlEnFrsWBoDScsiovJVQIOgZFjupRFxR7FjsYSvIKxRi4hJwNj0Tipr3BaTdJBbA+ErCDMzy8tXEGZmlpcThJmZ5eUEYWZmeTlBmJlZXk4QZmaW1/8HwFhzzjh8/JsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate_embeddings_approach(labels=['An Amazon review with a negative sentiment.', 'An Amazon review with a positive sentiment.'], engine='babbage-similarity')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the search embeddings and descriptive names leads to an additional improvement in performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    negative       0.65      0.93      0.76       136\n",
      "    positive       0.99      0.91      0.95       789\n",
      "\n",
      "    accuracy                           0.92       925\n",
      "   macro avg       0.82      0.92      0.86       925\n",
      "weighted avg       0.94      0.92      0.92       925\n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAwoUlEQVR4nO3deZwU1bn/8c+XAQRRwCAigiyiRkEFFY2iROJucNeoGBc04pKo+cWruXqNRsmiRk3uNSYuuXpFY3Ahxl0JKihuUUBAcAMMIoiKsokswszz+6NqxmbsmSmY6emZ4ft+veY1Vaeqq57TA/10nVN1jiICMzOzypoVOwAzM2uYnCDMzCwvJwgzM8vLCcLMzPJygjAzs7ycIMzMLC8nCKtTkoZKerHYcdQlST+U9M8M+90q6Yr6iKk+SJot6cB0+SpJfy12TFa/nCAMSRtJukPSB5K+kDRZ0mHFjiuL9ENshaRlkj6RdJekTeryHBFxb0QcnGG/cyPiV3V57nKSQtKXaT3nSfq9pJJCnMusnBOEATQHPgT2A9oBvwAekNSjmEGtgyMiYhNgN6A/SfxrkdS83qOqe33Teu4HnAicWeR46lQT+Rs1KU4QRkR8GRFXRcTsiCiLiMeBfwO7V/UaSVtLekjSAkmfS7q5iv3+R9KHkpZKmihpYM62PSVNSLd9Iun3aXkrSX9Nj7tY0uuSOmWoxzzgKWCn9Dgh6SeSZgAz0rLD0yukxZJelrRLTXXKbTZT4g+SPk3jflNS+fnukvTrnOMNkzRT0kJJj0raKmdbSDpX0ow0lj9JUk11TOs5E3gJ6JdzvPWpVy9Jz6Vln0m6V1L7LDFUJumo9PxLJc2SdGhaXtFMla5XNFVJ6pG+Dz+SNAd4TtJTks6vdOwpko5Nl3eQNCZ9T9+VdML6xGvZOEHYN6QfxtsD06vYXgI8DnwA9AC6APdVcbjXST7IvgX8DXhQUqt02/8A/xMRbYFewANp+ekkVzJbAx2Ac4EVGeLeGvg+8EZO8dHAd4DeknYF7gTOSY97G/Bo2sSWtU4HA98leX/aAScAn+eJZX/gmnR75/S4lY93OLAHsEu63yE11TE99g7AQGBmur6+9VIa41bAjiTv91VZYqgUz57A3cAlQHuS92f2Ohxiv/T8hwAjgSE5x+4NdAeekNQGGEPy72gL4CTgz+k+VgBOELYWSS2Ae4EREfFOFbvtSfKhckl69bEyIvJ2TEfEXyPi84hYExE3AhsB3043rwa2lbR5RCyLiFdzyjsA20ZEaURMjIil1YT9sKTFwIvA88Bvc7ZdExELI2IFcDZwW0T8Kz3uCGAVsNc61Gk1sCmwA6CIeDsi5ufZ74fAnRExKSJWAZcBe1dqtrs2IhZHxBxgLDlXBFWYJOlL4G1gHPDntHy96hURMyNiTESsiogFwO9JPqzX1Y/Suo5Jr0DnVfNvJ5+r0thWAP8A+knqnm77IfBQ+h4eDsyOiP9L/z29Afwd+MF6xGwZOEFYBUnNgHuAr4Dzc8qfUtI5ukzSD0m+aX4QEWsyHPNiSW9LWpJ+iLcDNk83/4jkm/g7aTPS4Wn5PcBo4D5JH0n6XZq4qnJ0RLSPiO4R8eP0g6bchznL3YH/SJthFqfxbE3yAZqpThHxHHAz8CfgU0m3S2qbZ9etSL61l79uGcmVRpecfT7OWV4ObAIgaXrO+z0wZ5/d0n1OJLkqalObeknqJOk+JZ3eS4G/8vXfZl1sDcxaj9eVq/gbRcQXwBMkVweQXE3cmy53B75TqZ4/BLasxbmtGk4QBiRt68AdQCfguIhYXb4tIg6LiE3Sn3tJ/kN3Uw2diumH289Jmk82i4j2wBKSpg0iYkZEDCFpLrgOGCWpTUSsjoirI6I3MIDkm+Np61m13OGKPwR+kyaT8p+NI2Jk1jqlcd8UEbsDvUkS3CV5dvuI5AMNgLR5pAMwL8Px++S83+MrbYuIeAB4BbiylvX6Lcn7s3PazHcK6d9mHX1I0kSYz5fAxjnr+T7MKw8pPRIYImlvoBXJ1VX5eZ6vVM9NIuK89YjZMnCCsHK3kLQDH1HpG3g+rwHzgWsltVHSqbxPnv02BdYAC4Dmkq4EKr5tSzpFUseIKAMWp8Vlkr4naee0/XwpSbNOWW0ql/oLcK6k7yjRRtJgSZtmrZOkPdLXtyD58FtZRWwjgTMk9ZO0EcmH8b8iYnYd1APgWmCYpC1rUa9NgWXAEkldyJ/osriDpK4HSGomqUvaTwIwGThJUgtJ/YHjMxzvSZLkOhy4P/33AUlfyvaSTk2P1yL9e+y4nnFbDZwgjLS99xySNvCPKzUnfUNElAJHANsCc4C5JM0elY0GngbeI2luWcnaTT6HAtMlLSPpsD4pTU5bAqNIksPbJP0K99SymkTEBGAYSRPRIpJO3qHrWKe2JB/Ii9I6fQ5cn+dczwBXkLSRzyf5hn1S5f1qUZc3gRdI+hbWt15XkzRbLSFp1nloPWN5DTgD+EN6rOf5+urpCpK6L0rP97cMx1uVxnJg7v5p89PBJO/jRyRNdNeR9GtZAcgTBpmZWT6+gjAzs7ycIMzMLC8nCDMzy8sJwszM8moyg2Ntvvnm0aNHj2KHYWbWqEycOPGziOiYb1uTSRA9evRgwoQJxQ7DzKxRkfRBVdvcxGRmZnk5QZiZWV5OEGZmlpcThJmZ5eUEYWZmeRUsQUi6U8m0jNOq2C5JNymZknGqpN1ytp2uZCrGGZJOL1SMZmZWtUJeQdxFMlpnVQ4Dtkt/ziYZbhpJ3wJ+STIhyp7ALyVtVsA4zcwsj4I9BxERL1SaXrGyo4C7IxlO9lVJ7SV1BgYBYyJiIYCkMSSJZmQh4lz+1RpuHVebybDMrKnYeKPmDB3Qg1YtSoodSoNQzAflurD23ABz07Kqyr9B0tkkVx9069ZtvYJY8VUpfxw7c71ea2ZNR/nMB327tmfvXh2KG0wD0aifpI6I24HbAfr3779eE1t02GQj/n3N4DqNy8wan9f+vZATbnuFMs+RU6GYCWIeyWTn5bqmZfNImplyy8fVW1RmtkEqTwyPT53PZ8tWURZBWRmURrBqdSnLVpXSZqMSSsuCsoCysqA0IllPy0ojaNOyhDP26UnL5o3/JtFiJohHgfMl3UfSIb0kIuZLGg38Nqdj+mDgsmIFaWYbhtWlydTXI1+bw8jX5tTqWP17fIvduzf+e2sKliAkjSS5Ethc0lySO5NaAETErSQTk3+fZP7c5SRz2hIRCyX9Cng9PdTw8g5rM7NCGbhdRx75yT60bllCM4mSZqJEQoKSZsnvFiXNKJFo1kw0S8tz931p1mecesdrTaaZqpB3MQ2pYXsAP6li253AnYWIy8ysKn23bl+r1wvVTSANRONvJDMzs4JwgjAzs7ycIMzMLC8nCDMzy8sJwszM8nKCMDOzvJwgzMwsLycIMzPLywnCzMzycoIwM7O8nCDMzCyvTGMxpSOrbgWsAGZHRFlBozIzs6KrMkFIakcymN4QoCWwAGgFdJL0KvDniBhbL1GamVm9q+4KYhRwNzAwIhbnbpC0O3CqpG0i4o4CxmdmZkVSZYKIiIOq2TYRmFiQiMzMGqnPv1wFwA9ufYWn/99AdtiybZEjqp0aO6mVOEXSlel6N0l7Fj40M7PGZZeu7SuW35izuGhx1JUsdzH9GdibpC8C4AvgTwWLyMyskeq5eRteveyAYodRZ7LcxfSdiNhN0hsAEbFIUssCx2VmZkWW5QpitaQSIAAkdQR8m6uZWROXJUHcBPwD2ELSb4AXgd8WNCozMyu6GpuYIuJeSROBAwABR0fE2wWPzMzMiqrGBCHpJuC+iHDHtJnZBiRLJ/VE4BeSvk3S1HRfREwobFhmZo3bG3MW0aKkGV+tKaOkGRy7W1dalDSu4e+yNDGNAEZI+hZwHHCdpG4RsV3BozMza2TKIgB4YMJcHpgwt6K8V8dN6N/jW8UKa71kGqwvtS2wA9AdcB+EmVkeW7VvzdH9tqL3Vm3ZdotN+HzZV1wyaiqrS6PYoa2zLH0QvwOOAWYB9wO/qjw2k5mZfe2/T9q1YvmVWZ8XMZLayXIFMQvYOyI+K3QwZmbWcFQ33PcOEfEO8DrQTVK33O0RManQwZmZWfFUdwVxEXA2cGOebQHsX5CIzMyakPIRXof85VW2ateKxy8cyLfaNI7Riqob7vvsdPGwiFiZu01Sq4JGZWbWRPTbuj0AJc3ER0tW8snSlY0mQWS5KffljGVmZlZJ1802Zva1g/nTybvWvHMDU10fxJZAF6C1pF1JhtkAaAtsXA+xmZlZEVXXB3EIMBToCvw+p/wL4L8KGJOZ2QZjTWkZzRvoE9bV9UGUP0F9XET8vR5jMjNrcr5KH5Q74dZX6LJZa5Z/VcqchcsB+MXgHTlr4DbFDC+vKtOWpFPSxR6SLqr8k+Xgkg6V9K6kmZIuzbO9u6RnJU2VNE5S15xtv5M0XdLbkm6SpMqvNzNrLD77Irmb6YtVa3jn4y/ou3V7jt21C0BFomhoqmtiapP+3mR9DpxOMvQn4CBgLvC6pEcj4q2c3W4A7o6IEZL2B64BTpU0ANgH2CXd70VgP2Dc+sRiZlZsZ+7bk127tWfHzm1p1aKkonzsu58WMarqVdfEdFv6++r1PPaewMyIeB9A0n3AUUBuguhN8rwFwFjg4fLTA62AliSd4y2AT9YzDjOzBmHXbpsVO4R1UmPPSNrU01ZSi7Q5aEFO81N1ugAf5qzPTctyTQGOTZePATaV1CEiXiFJGPPTn9H5JimSdLakCZImLFiwIENIZmaWVZau84MjYilwODCbZFTXS+ro/BcD+0l6g6QJaR5QKmlbYEeSO6i6APtLGlj5xRFxe0T0j4j+HTt2rKOQzMzqz6Llq7n7lQ94+I15xQ7lG7IkiPJmqMHAgxGxJOOx5wFb56x3TcsqRMRHEXFsROwKXJ6WLSa5mng1IpZFxDLgKWDvjOc1M2t07n/9w5p3qmdZRnN9XNI7wArgPEkdgZU1vAaSQf62k9STJDGcBJycu4OkzYGFEVEGXAbcmW6aAwyTdA1JH8R+wH9nOKeZWaMy+9rBHH/Ly8xcsIxT7/gXAKVlwZrS4Ih+W3HqXt2LFluNVxARcSkwAOgfEauBL0k6m2t63RrgfGA0yQRDD0TEdEnDJR2Z7jYIeFfSe0An4Ddp+SiSYcbfJOmnmBIRj61LxczMGosJHyxiwRerGD/jM17790JWl5bx9sdLeWzKR0WNK8uEQS2AU4Dvpo8iPA/cmuXgEfEk8GSlsitzlkeRJIPKrysFzslyDjOzxu4Xg3fkuXc+5dS9unNIny1p1kycdPsrlBV5ErosTUy3kNxm+ud0/dS07KxCBWVmtiE5a+A2DfJJ6iwJYo+I6Juz/pykKYUKyMzMGoYsdzGVSupVviJpG6C0cCGZmVlDkOUK4hJgrKT3Se4o6g6cUdCozMys6GpMEBHxrKTtgG+nRe9GxKrChmVmZsWW5S6mVsCPgX1JxkgaL+nWytOQmplZ05KlielukkmC/piunwzcA/ygUEGZmVnxZUkQO0VE75z1sZLeqnJvMzNrErLcxTRJ0l7lK5K+A0woXEhmZtYQZLmC2B14WdKcdL0byfAYbwIREbtU/VIzM2ussiSIQwsehZmZrWXq3CUs/6qUiKBYMy5nuc31g/oIxMzMvrb8q+R55HmLV9B1s42LEkOWPggzM6tnvz8hGeGotIgj9jlBmJk1QNM/WgrAE2/OL1oMThBmZg3Q8bt3BWDJ8tVFiyFTgpB0e3XrZmZWt3bs3JYWJaKkWXE6qCH7FcRtNaybmVkTkylBRMTE6tbNzKzpqfI2V0mPkQzOl1dEHFnVNjMza/yqew7ihnqLwszMGpwqE0REPF++LKk10C0i3q2XqMzMjNWlwVdryop2/hr7ICQdAUwGnk7X+0l6tMBxmZkZ8L8v/rto587SSX0VsCewGCAiJgM9CxaRmZmtJSKY+ekXPDrlI2Z+uqzezptlsL7VEbGk0mBRxXv228xsA9F1s9bMXbSCnpc9WVE2oFcH/jZsr2peVXeyXEFMl3QyUCJpO0l/BF4ucFxmZhu8FiXJR/TgnTtzwf7bArBydWm9nT/LFcQFwOXAKmAkMBr4VSGDMjMzGHvxoLXWJ81ZxKrV9ddpnWW47+XA5ZKuS1bji8KHZWZmxZblLqY90tnjpgJvSpoiaffCh2ZmZsWUpYnpDuDHETEeQNK+wP8BnmrUzKwJy9JJXVqeHAAi4kVgTeFCMjOzhqC6sZh2Sxefl3QbSQd1ACcC4wofmpmZ5Xpp5udAcidTqxYlBT9fdU1MN1Za/2XOsp+DMDMrknmLV9Cr4yYFP091YzF9r+BnNzOzzG78QV/+48EpvPbvhcVNELkkDQb6AK3KyyJieKGCMjOzb3rv0+Qpg8seepMhe3Yr+Pmy3OZ6K0m/wwWAgB8A3bMcXNKhkt6VNFPSpXm2d5f0rKSpksZJ6pqzrZukf0p6W9JbknpkrZSZWVP080N2oF3rFrRpWfj+B8h2F9OAiDgNWBQRVwN7A9vX9CJJJcCfgMOA3sAQSb0r7XYDcHdE7AIMB67J2XY3cH1E7EgyWOCnGWI1M2uySpqJo/ttRYvmWWeLrp0sZ1mR/l4uaStgNdA5w+v2BGZGxPsR8RVwH3BUpX16A8+ly2PLt6eJpHlEjAGIiGXpE91mZlZPsiSIxyW1B64HJgGzSW55rUkX4MOc9blpWa4pwLHp8jHAppI6kFyhLJb0kKQ3JF2fXpGsRdLZkiZImrBgwYIMIZmZWVY1JoiI+FVELI6Iv5P0PewQEVfU0fkvBvaT9AawHzAPKCXpPB+Ybt8D2AYYmie22yOif0T079ixYx2FZGZmUP2DcsdWs42IeKiGY88Dts5Z75qWVYiIj0ivICRtAhwXEYslzQUmR8T76baHgb1Ihv0wM7N6UN1trkdUsy2AmhLE68B2knqSJIaTgJNzd5C0ObAwIsqAy4A7c17bXlLHiFgA7A9MqOF8ZmZWh6p7UO6M2hw4ItZIOp9k/ogS4M6ImC5pODAhIh4FBgHXSArgBeAn6WtLJV0MPKtkKruJwF9qE4+Zma2bTA/Kra+IeBJ4slLZlTnLo4BRVbx2DB4x1sxsLZ8sXcXi5atZtaaUjZoX9nmI+rmZ1szM6sTT0z8G4OVZnxf8XE4QZmaNyH1n7wXA6jWFn3o0y1AbG0u6QtJf0vXtJB1e8MjMzOwbNtmooD0Da8lyBfF/wCqSITYguSPp1wWLyMzMGoQsCaJXRPyOZIgN0iEvVNCozMwsr7JIpuOZ9tHSgp8rS4L4SlJr0kmCJPUiuaIwM7N61kzJ9/P/Hf9+wc+VpTHrKuBpYGtJ9wL7kGfYCzMzK7ydurRj62+1pnO71gU/V40JIiL+KWkiyVAXAn4aEZ8VPDIzM8vrw4Ur+HDhClZ8VUrrAs4NkeUupseAg4FxEfG4k4OZWXF1+9bGACz/ak1Bz5OlD+IGkpFV35I0StLxklrV9CIzMyuMswb2rJfzZGlieh54Pp2PYX9gGMmgem0LHJuZmRVRpicu0ruYjiCZm3o3YEQhgzIzs+KrMUFIeoBk+tCngZuB59Phuc3MrAnLcgVxBzAkIkoLHYyZmTUc1c0ot39EPAe0AY6S1n54OsOMcmZmVgDT5yVPUU+as5iDencq2Hmqu4tpv/T3EXl+PFifmVmR7Ny1HQDTP1pS0PNUN6PcL9PF4RHx79xt6TSiZmZWBMfv3pVfPDyNls0LO2NDlqP/PU9Z3lngzMys6aiuD2IHoA/QTtKxOZvaAn5QzsysiavuLqZvk/Q1tCfpdyj3BcnDcmZm1oRV1wfxCPCIpL0j4pV6jMnMzBqA6pqYfp5OFHSypCGVt0fEhQWNzMzMiqq6Jqa3098T6iMQMzNrWKprYnos/V0x7pKkZsAmEVH4ue7MzKxaI1+bw/G7dWWLtoW5byjLfBB/k9RWUhtgGsmw35cUJBozM6tROi01Hy5cwVWPTS/YebI8B9E7vWI4GngK6AmcWrCIzMysWq1blrBHj80AaNWiiDPKAS0ktSBJEI9GxGogChaRmZnV6MFzB9ClfWuEat55PWVJELcBs0kG7XtBUnfAfRBmZk1clhnlbgJuyin6QNL3CheSmZk1BFk6qdtJ+r2kCenPjSRXE2Zm1oRlaWK6k2R4jRPSn6XA/xUyKDMzK74sM8r1iojjctavljS5QPGYmVkDkeUKYoWkfctXJO0DrChcSGZm1hBkuYI4F7hbUrt0fRFweuFCMjOzhqDaKwhJ/YDtgJOAXYBdImLXiJia5eCSDpX0rqSZki7Ns727pGclTZU0TlLXStvbSpor6ebMNTIzszpRZYKQdCXwAHAc8ARw4rqMwSSpBPgTcBjQGxgiqXel3W4A7o6IXYDhwDWVtv8KeCHrOc3MrO5UdwVxItAvIoYAewBnr+Ox9wRmRsT7EfEVcB9wVKV9egPPpctjc7dL2h3oBPxzHc9rZmZ1oLoEsSoilgNExOc17JtPF+DDnPW5aVmuKUD5dKbHAJtK6pCOGnsjcHF1J5B0dvnzGQsWLFjH8MzMrDrVdVJvI+nRdFlAr5x1IuLIOjj/xcDNkoaSNCXNA0qBHwNPRsRcqepxRiLiduB2gP79+3t8KDOzOlRdgqjcHHTDOh57HrB1znrXtKxCRHxEegUhaRPguIhYLGlvYKCkHwObAC0lLYuIb3R0m5lZYVQ3YdDztTz268B2knqSJIaTgJNzd5C0ObAwIsqAy0ie2iYifpizz1Cgv5ODmVn9qu4upsckHZEO9V152zaShks6s6rXR8Qa4HxgNMn0pQ9ExPT0deXNU4OAdyW9R9Ih/Zta1MXMzOpQdU1Mw4CLgP+WtBBYALQCegCzgJsj4pHqDh4RTwJPViq7Mmd5FDCqhmPcBdxV3T5mZlb3qmti+hj4OfBzST2AziRDbLxXfneTmZk1XVmG2iAiZpNMGmRmZhuIdX22wczMGoh5i1fw90lziSjMXf5OEGZmjVxZgZ4Cc4IwM2ukLjpo+4Iev8Y+iHT+h6uA7un+AiIitiloZGZmVlRZOqnvAH4GTCQZBsPMzDYAWRLEkoh4quCRmJlZg5IlQYyVdD3wELCqvDAiJhUsKjMzK7osCeI76e/+OWUB7F/34ZiZWUNRY4KIiO/VRyBmZtaw1Hibq6R2kn5fPjGPpBsltauP4MzMrHiyPAdxJ/AFcEL6sxT4v0IGZWZmxZelD6JXRByXs361pMkFisfMzBqILFcQKyTtW76SPji3onAhmZlZQ5DlCuI8YETa7yBgITC0kEGZmVnxZbmLaTLQV1LbdH1poYMyM7PiqzJBSDolIv4q6aJK5QBExO8LHJuZmRVRdVcQbdLfm9ZHIGZm1rBUN+Xobenvq+svHDMzayiyPCj3O0ltJbWQ9KykBZJOqY/gzMyseLLc5npw2jF9OMm81NsClxQyKDMzK74sCaK8GWow8GBELClgPGZm1kBkeQ7icUnvkDwcd56kjsDKwoZlZmbFVuMVRERcCgwA+kfEauBL4KhCB2ZmZsVV3XMQ+0fEc5KOzSnL3eWhQgZmZmbFVV0T037Ac8ARebYFThBmZk1adc9B/DL9fUb9hWNmZg1Flucgfiupfc76ZpJ+XdCozMys6LLc5npYRCwuX4mIRcD3CxaRmZk1CFkSRImkjcpXJLUGNqpmfzMzawKyPAdxL/CspPJpRs8ARhQuJDMzawiyzAdxnaQpwIFp0a8iYnRhwzIzs2LLcgUB8DawJiKekbSxpE0j4otCBmZmZsWV5S6mYcAo4La0qAvwcJaDSzpU0ruSZkq6NM/27ukIsVMljZPUNS3vJ+kVSdPTbSdmrpGZmdWJLJ3UPwH2AZYCRMQMYIuaXiSpBPgTcBjQGxgiqXel3W4A7o6IXYDhwDVp+XLgtIjoAxwK/HfurbZmZlZ4WRLEqoj4qnxFUnOSJ6lrsicwMyLeT19/H98cw6k3ydPaAGPLt0fEe2kiIiI+Aj4FOmY4p5mZ1ZEsCeJ5Sf8FtJZ0EPAg8FiG13UBPsxZn5uW5ZoClI/1dAywqaQOuTtI2hNoCcyqfAJJZ0uaIGnCggULMoRkZmZZZUkQ/wksAN4EzgGeBH5RR+e/GNhP0hskYz/NA0rLN0rqDNwDnBERZZVfHBG3R0T/iOjfsaMvMMzM6lK1dzGl/QjTI2IH4C/reOx5wNY5613Tsgpp89Gx6bk2AY4rf2pbUlvgCeDyiHh1Hc9tZma1VO0VRESUAu9K6rYex34d2E5ST0ktgZOAR3N3kLS5pPIYLgPuTMtbAv8g6cAetR7nNjOzWsryHMRmwHRJr5FMFgRARBxZ3YsiYo2k84HRQAlwZ0RMlzQcmBARjwKDgGskBfACyR1TACcA3wU6SBqalg2NiMlZK2ZmZrWTJUFcsb4Hj4gnSfoscsuuzFkeRfKMReXX/RX46/qe18zMaq+6GeVaAecC25J0UN8REWvqKzAzMyuu6vogRgD9SZLDYcCN9RKRmZk1CNU1MfWOiJ0BJN0BvFY/IZmZWUNQ3RXE6vIFNy2ZmW14qruC6Ctpaboskiepl6bLERFtCx6dmZkVTZUJIiJK6jMQMzNrWLIMtWFmZhugrBMGNUqrV69m7ty5rFy5stihWCPSqlUrunbtSosWLYodillRNekEMXfuXDbddFN69OiBpGKHY41ARPD5558zd+5cevbsWexwzIqqSTcxrVy5kg4dOjg5WGaS6NChg686zWjiCQJwcrB15n8zZokmnyDMzGz9OEEU2Mcff8xJJ51Er1692H333fn+97/Pe++9x+zZs9lpp53q7DxXXnklzzzzDADjx4+nT58+9OvXj3nz5nH88cfX6tgRwf7778/SpUsryh5++GEk8c4771SUzZ49m9atW9OvXz969+7NueeeS1nZN+Z5WicvvPACu+22G82bN2fUqKpHfp84cSI777wz2267LRdeeCERyay4Cxcu5KCDDmK77bbjoIMOYtGiRQA8/vjjXHnllVUez8ycIAoqIjjmmGMYNGgQs2bNYuLEiVxzzTV88skndX6u4cOHc+CBBwJw7733ctlllzF58mS6dOlS7QdrZWvWfPOh+SeffJK+ffvStu3Xz0aOHDmSfffdl5EjR661b69evZg8eTJTp07lrbfe4uGHH16/CqW6devGXXfdxcknn1ztfueddx5/+ctfmDFjBjNmzODpp58G4Nprr+WAAw5gxowZHHDAAVx77bUADB48mMcee4zly5fXKj6zpqxJ38WU6+rHpvPWR0tr3nEd9N6qLb88ok+V28eOHUuLFi0499xzK8r69u0LJN+2y82ePZtTTz2VL79Mptu4+eabGTBgAPPnz+fEE09k6dKlrFmzhltuuYUBAwbwox/9iAkTJiCJM888k5/97GcMHTqUww8/nMWLF/PAAw8wevRonnrqKX7zm99w+OGHM23aNEpLS7n00ksZN24cq1at4ic/+QnnnHMO48aN44orrmCzzTbjnXfe4b333lurHvfeey9nn312xfqyZct48cUXGTt2LEcccQRXX331N+revHlzBgwYwMyZM9frvS3Xo0cPAJo1q/q7zPz581m6dCl77bUXAKeddhoPP/wwhx12GI888gjjxo0D4PTTT2fQoEFcd911SGLQoEE8/vjjnHDCCbWK0ayp2mASRDFMmzaN3Xffvcb9tthiC8aMGUOrVq2YMWMGQ4YMYcKECfztb3/jkEMO4fLLL6e0tJTly5czefJk5s2bx7Rp0wBYvHjxWsc666yzePHFFzn88MM5/vjj10pEd9xxB+3ateP1119n1apV7LPPPhx88MEATJo0iWnTpuW9tfOll17itttuq1h/5JFHOPTQQ9l+++3p0KEDEydO/EY9ly9fzrPPPsvw4cO/cbyBAwfyxRdffKP8hhtuqLgKWhfz5s2ja9euFetdu3Zl3rxkdttPPvmEzp07A7DllluudfXWv39/xo8f7wRhVoUNJkFU902/2FavXs3555/P5MmTKSkpqfgGv8cee3DmmWeyevVqjj76aPr168c222zD+++/zwUXXMDgwYMrPuCz+Oc//8nUqVMrmpyWLFnCjBkzaNmyJXvuuWeV9/0vXLiQTTfdtGJ95MiR/PSnPwXgpJNOYuTIkRUJYtasWfTr1w9JHHXUURx22GHfON748eMzx1yXJK11h9IWW2zBRx99VJRYzBqDDSZBFEOfPn0ytf//4Q9/oFOnTkyZMoWysjJatWoFwHe/+11eeOEFnnjiCYYOHcpFF13EaaedxpQpUxg9ejS33norDzzwAHfeeWemeCKCP/7xjxxyyCFrlY8bN442bdpU+brmzZtTVlZGs2bNWLhwIc899xxvvvkmkigtLUUS119/PfB1H0R16voKokuXLsydO7dife7cuXTp0gWATp06MX/+fDp37sz8+fPZYostKvZbuXIlrVu3XufzmTUUf/vXHADGvvMpB/buVOfHdyd1Ae2///6sWrWK22+/vaJs6tSp3/gGvWTJEjp37kyzZs245557KC0tBeCDDz6gU6dODBs2jLPOOotJkybx2WefUVZWxnHHHcevf/1rJk2alDmeQw45hFtuuYXVq5OR3N97772Kfo/qfPvb3+b9998HYNSoUZx66ql88MEHzJ49mw8//JCePXuu01XB+PHjmTx58jd+1ic5AHTu3Jm2bdvy6quvEhHcfffdHHXUUQAceeSRjBgxAoARI0ZUlENS/7q8k8ysvp2//7YAzP685v/H68MJooAk8Y9//INnnnmGXr160adPHy677DK23HLLtfb78Y9/zIgRI+jbty/vvPNOxbf5cePG0bdvX3bddVfuv/9+fvrTnzJv3jwGDRpEv379OOWUU7jmmmsyx3PWWWfRu3dvdtttN3baaSfOOeecvHctVTZ48OCKjt6RI0dyzDHHrLX9uOOO+8bdTHXl9ddfp2vXrjz44IOcc8459OnzdVNhv379Kpb//Oc/c9ZZZ7HtttvSq1eviqatSy+9lDFjxrDddtvxzDPPcOmll1a8ZuzYsQwePLggcZvVhwG9OgDw6yfeLsjxVX6/eGPXv3//mDBhwlplb7/9NjvuuGORImo65s+fz2mnncaYMWOKHUqd+eSTTzj55JN59tln8273vx1rDMrKgm3+60mGDezJ5YN7r9cxJE2MiP75trkPwmrUuXNnhg0bxtKlS9d6FqIxmzNnDjfe6GnWrXFr1kzMvrZwV8FOEJZJU7sVdI899ih2CGYNXpPvg2gqTWhWf/xvxizRpBNEq1at+Pzzz/0f3jIrnw+i/FZjsw1Zk25i6tq1K3PnzmXBggXFDsUakfIZ5cw2dE06QbRo0cKzgpmZracm3cRkZmbrzwnCzMzycoIwM7O8msyT1JIWAB/U4hCbA5/VUTiNxYZW5w2tvuA6byhqU+fuEdEx34YmkyBqS9KEqh43b6o2tDpvaPUF13lDUag6u4nJzMzycoIwM7O8nCC+dnvNuzQ5G1qdN7T6guu8oShInd0HYWZmefkKwszM8nKCMDOzvDaoBCHpUEnvSpop6dI82zeSdH+6/V+SehQhzDqVoc4XSXpL0lRJz0rqXow461JNdc7Z7zhJIanR3xKZpc6STkj/1tMl/a2+Y6xrGf5td5M0VtIb6b/v7xcjzroi6U5Jn0qaVsV2SbopfT+mStqt1ieNiA3iBygBZgHbAC2BKUDvSvv8GLg1XT4JuL/YcddDnb8HbJwun7ch1Dndb1PgBeBVoH+x466Hv/N2wBvAZun6FsWOux7qfDtwXrrcG5hd7LhrWefvArsB06rY/n3gKUDAXsC/anvODekKYk9gZkS8HxFfAfcBR1Xa5yhgRLo8CjhAkuoxxrpWY50jYmxELE9XXwUa+zjXWf7OAL8CrgNW1mdwBZKlzsOAP0XEIoCI+LSeY6xrWeocQPkcue2Aj+oxvjoXES8AC6vZ5Sjg7ki8CrSX1Lk259yQEkQX4MOc9blpWd59ImINsAToUC/RFUaWOuf6Eck3kMasxjqnl95bR8QT9RlYAWX5O28PbC/pJUmvSjq03qIrjCx1vgo4RdJc4EnggvoJrWjW9f97jZr0fBCWnaRTgP7AfsWOpZAkNQN+Dwwtcij1rTlJM9MgkqvEFyTtHBGLixlUgQ0B7oqIGyXtDdwjaaeIKCt2YI3FhnQFMQ/YOme9a1qWdx9JzUkuSz+vl+gKI0udkXQgcDlwZESsqqfYCqWmOm8K7ASMkzSbpK320UbeUZ3l7zwXeDQiVkfEv4H3SBJGY5Wlzj8CHgCIiFeAViSD2jVVmf6/r4sNKUG8DmwnqaekliSd0I9W2udR4PR0+XjguUh7fxqpGussaVfgNpLk0NjbpaGGOkfEkojYPCJ6REQPkn6XIyNiQnHCrRNZ/m0/THL1gKTNSZqc3q/HGOtaljrPAQ4AkLQjSYJoyvMPPwqclt7NtBewJCLm1+aAG0wTU0SskXQ+MJrkDog7I2K6pOHAhIh4FLiD5DJ0Jkln0EnFi7j2Mtb5emAT4MG0P35ORBxZtKBrKWOdm5SMdR4NHCzpLaAUuCQiGu3VccY6/wfwF0k/I+mwHtqYv/BJGkmS5DdP+1V+CbQAiIhbSfpZvg/MBJYDZ9T6nI34/TIzswLakJqYzMxsHThBmJlZXk4QZmaWlxOEmZnl5QRhZmZ5OUHYepNUKmmypGmSHpPUvo6PPzu9Zx9Jy6rYp7Wk5yWVSOohaUUa01uSbk2fnF6Xc/aXdFO6PEjSgJxt50o6rTZ1So9zlaSLa9jnLknHr8Mxe1Q1ymddk3Rk+eipko6W1Dtn2/D0wcv1Oe59khrzw3tNjhOE1caKiOgXETuRPDfykyLEcCbwUESUpuuzIqIfsAvJCJ5Hr8vBImJCRFyYrg4CBuRsuzUi7q5twI1dRDwaEdemq0eTvM/l266MiGfW89C3AD+vZXhWh5wgrK68QjowmKRekp6WNFHSeEk7pOWdJP1D0pT0Z0Ba/nC673RJZ6/jeX8IPFK5MB1s8WVg2/Tb9XP6es6Lbul5f5Be/UyR9EJaNkjS40rmAjkX+Fl6RTKw/Ju/pB0kvVZ+rvT4b6bLu6dXNBMljVYNo2lKGibp9TSGv0vaOGfzgZImSHpP0uHp/iWSrk9fM1XSOevyZklaJukP6Xv9rKSOaXk/JYP4TU3/Rpul5Rfq6/lC7kvLhkq6Of37HQlcn75HvcqvfJTM1fBgznkHSXo8XT5Y0iuSJkl6UNIm6W7j0zpvMA/wNnROEFZrkkpIhjQof0r5duCCiNgduBj4c1p+E/B8RPQlGdd+elp+Zrpvf+BCSZlG0FUyxMI2ETE7z7aN05jeBP4IjIiIXYB70zgArgQOSeNZ6+nx9Ji3An9Ir5LG52x7B2gpqWdadCJwv6QW6bmOT+tzJ/CbGqrxUETskcbwNsn4QeV6kAxrPRi4VVKrdPuSiNgD2AMYlhNHed23kvRkFedrQ/KkcR/geZKncQHuBv4zfY/ezCm/FNg1LT+30nv0Msnf/JL0PZqVs/kZ4DuS2qTrJwL3KWky/AVwYETsBkwALkqPV0byFHDfqt8uq09OEFYbrSVNBj4GOgFj0m+DA0iG7phMMs5T+bfo/UmaEYiI0ohYkpZfKGkKybhIW5N9ELnNgcWVynql530JeCIingL2BspnULsH2Dddfgm4S9IwkuEa1sUDJB96pL/vB75NMhDgmDSGX1Dz/Bo7pVdZb5JcDfXJPUdElEXEDJJxk3YADiYZb2cy8C+S4ejXer8i4qOIqGr2tLI0VoC/AvtKage0j4jn0/IRJJPTAEwF7lUy2u+aGuqSG8Ma4GngiPSKYDDJld5eJE1SL6V1OB3IncXwU2CrrOexwvKlnNXGiojol35bH03SB3EXsDjtB6iRpEHAgcDeEbFc0jiSQdUynT/PvrOynjsizpX0HZIPr4mSds94Xkg+ZB+U9FByqJghaWdgekTsvQ7HuQs4OiKmSBpKOqBeeYiVQyaZLeyCiBidu0HrPz1uTWPtDCZJFkcAl6d1zOo+4HyS/qkJEfGFJAFjImJIFa9pRfJ3tQbAVxBWa+mMdBeSDI62HPi3pB9AxTy55U0Gz5JMa1relt6OZEj1RWly2IHkG2bW8y4CStKml+q8zNcDL/6QpK0bSb0i4l8RcSXJKJ9bV3rdFyTDg+c79yySQe+u4Otv5O8CHZXMPYCkFpL65Ht9jk2B+Wnz1A8rbfuBpGaSepFMrfkuSSI+L90fSdvnNONk0YxkpGKAk4EX0yu5RZIGpuWnAs8ruQNs64gYC/wnyd9qk0rHq/I9ImnC2o1kNrv70rJXgX0kbZvG30bS9jmv2R6ol7uxrGZOEFYnIuINkuaIISQfdD9Km42m8/VUkD8Fvpc2p0wkaWp4Gmgu6W3gWpIPkHXxT75uMqrKBcAZkqaSfPj9NC2/XtKbSm4PfZlkXuNcjwHHpB2wA/mm+4FT+HrOga9IPnyvS+s+mZy7oKpwBUlT0UvAO5W2zQFeI5nl79yIWAn8L/AWMCmN+zYqtQTU0AfxJbBn+tr9geFp+ekk78dUoF9aXgL8Nf17vQHclGeCofuASyS9kSayCumdZY8Dh6W/iYgFJJM1jUzP9QpJ0xmSOpFclX5cRexWzzyaqzVqSqYP/VlEnFrsWBoDScsiovJVQIOgZFjupRFxR7FjsYSvIKxRi4hJwNj0Tipr3BaTdJBbA+ErCDMzy8tXEGZmlpcThJmZ5eUEYWZmeTlBmJlZXk4QZmaW1/8HwFhzzjh8/JsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "evaluate_embeddings_approach(labels=['An Amazon review with a negative sentiment.', 'An Amazon review with a positive sentiment.'], engine='babbage-similarity')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As shown above, zero-shot classification with embeddings can lead to great results, especially when the labels are more descriptive than just simple words."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.9 ('openai')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}