openai-cookbook/examples/User_and_product_embeddings.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## User and product embeddings\n",
    "\n",
    "We calculate user and product embeddings based on the training set, and evaluate the results on the unseen test set. We will evaluate the results by plotting the user and product similarity versus the review score. The dataset is created in the [Get_embeddings_from_dataset Notebook](Get_embeddings_from_dataset.ipynb)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Calculate user and product embeddings\n",
    "\n",
    "We calculate these embeddings simply by averaging all the reviews about the same product or written by the same user within the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ProductId</th>\n",
       "      <th>UserId</th>\n",
       "      <th>Score</th>\n",
       "      <th>Summary</th>\n",
       "      <th>Text</th>\n",
       "      <th>combined</th>\n",
       "      <th>n_tokens</th>\n",
       "      <th>embedding</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>B003XPF9BO</td>\n",
       "      <td>A3R7JR3FMEBXQB</td>\n",
       "      <td>5</td>\n",
       "      <td>where does one  start...and stop... with a tre...</td>\n",
       "      <td>Wanted to save some to bring to my Chicago fam...</td>\n",
       "      <td>Title: where does one  start...and stop... wit...</td>\n",
       "      <td>52</td>\n",
       "      <td>[0.03599238395690918, -0.02116263099014759, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>297</th>\n",
       "      <td>B003VXHGPK</td>\n",
       "      <td>A21VWSCGW7UUAR</td>\n",
       "      <td>4</td>\n",
       "      <td>Good, but not Wolfgang Puck good</td>\n",
       "      <td>Honestly, I have to admit that I expected a li...</td>\n",
       "      <td>Title: Good, but not Wolfgang Puck good; Conte...</td>\n",
       "      <td>178</td>\n",
       "      <td>[-0.07042013108730316, -0.03175969794392586, -...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      ProductId          UserId  Score  \\\n",
       "0    B003XPF9BO  A3R7JR3FMEBXQB      5   \n",
       "297  B003VXHGPK  A21VWSCGW7UUAR      4   \n",
       "\n",
       "                                               Summary  \\\n",
       "0    where does one  start...and stop... with a tre...   \n",
       "297                   Good, but not Wolfgang Puck good   \n",
       "\n",
       "                                                  Text  \\\n",
       "0    Wanted to save some to bring to my Chicago fam...   \n",
       "297  Honestly, I have to admit that I expected a li...   \n",
       "\n",
       "                                              combined  n_tokens  \\\n",
       "0    Title: where does one  start...and stop... wit...        52   \n",
       "297  Title: Good, but not Wolfgang Puck good; Conte...       178   \n",
       "\n",
       "                                             embedding  \n",
       "0    [0.03599238395690918, -0.02116263099014759, -0...  \n",
       "297  [-0.07042013108730316, -0.03175969794392586, -...  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from ast import literal_eval\n",
    "\n",
    "df = pd.read_csv('data/fine_food_reviews_with_embeddings_1k.csv', index_col=0)  # note that you will need to generate this file to run the code below\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(577, 706)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['babbage_similarity'] = df[\"embedding\"].apply(literal_eval).apply(np.array)\n",
    "X_train, X_test, y_train, y_test = train_test_split(df, df.Score, test_size = 0.2, random_state=42)\n",
    "\n",
    "user_embeddings = X_train.groupby('UserId').babbage_similarity.apply(np.mean)\n",
    "prod_embeddings = X_train.groupby('ProductId').babbage_similarity.apply(np.mean)\n",
    "len(user_embeddings), len(prod_embeddings)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that most of the users and products appear within the 50k examples only once."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Evaluate the embeddings\n",
    "\n",
    "To evaluate the recommendations, we look at the similarity of the user and product embeddings amongst the reviews in the unseen test set. We calculate the cosine distance between the user and product embeddings, which gives us a similarity score between 0 and 1. We then normalize the scores to be evenly split between 0 and 1, by calculating the percentile of the similarity score amongst all predicted scores."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from utils.embeddings_utils import cosine_similarity\n",
    "\n",
    "# evaluate embeddings as recommendations on X_test\n",
    "def evaluate_single_match(row):\n",
    "    user_id = row.UserId\n",
    "    product_id = row.ProductId\n",
    "    try:\n",
    "        user_embedding = user_embeddings[user_id]\n",
    "        product_embedding = prod_embeddings[product_id]\n",
    "        similarity = cosine_similarity(user_embedding, product_embedding)\n",
    "        return similarity\n",
    "    except Exception as e:\n",
    "        return np.nan\n",
    "\n",
    "X_test['cosine_similarity'] = X_test.apply(evaluate_single_match, axis=1)\n",
    "X_test['percentile_cosine_similarity'] = X_test.cosine_similarity.rank(pct=True)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1 Visualize cosine similarity by review score\n",
    "\n",
    "We group the cosine similarity scores by the review score, and plot the distribution of cosine similarity scores for each review score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correlation between user & vector similarity percentile metric and review number of stars (score): 29.56%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAHNCAYAAADrIvo2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8g+/7EAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4kElEQVR4nO3de1jUZf7/8RfHQUA8pIIonlM8m7gqq6auCmlp7WVlaaKuWbsr64HaynYVD7tpZ9uyrHYNN7eyw5buWipZVJb+KlxLUzTN1DwgWIqA4gD37w9jvhGoDDXM3Ph8XJcX19zzuT/3+zM3Mq/r87lnPn7GGCMAAABL+Xu7AAAAgJ+CMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGqEGQAAYDXCDAAAsBphBgAAWI0wA1jAz89Pc+fO9XYZtV5GRob8/PyUkZFxwe3mzp0rPz8/5ebm1kxhAC6IMINLWlpamvz8/Mr9a9KkiQYPHqy33nrL2+X9ZDt27NDcuXP19ddfe7sUVENpaan++c9/qk+fPmrYsKHq1q2r9u3bKykpSZs3b/Z2eYDPCPR2AYAvmD9/vlq3bi1jjLKzs5WWlqYRI0boP//5j6655hpvl1dtO3bs0Lx58zRo0CC1atXK2+XATdOmTdOSJUt07bXXaty4cQoMDNSuXbv01ltvqU2bNurbt6+3SwR8AmEGkDR8+HD16tXL9Xjy5MmKjIzUiy++aHWYqUnFxcUqLS1VcHCwt0upFbKzs/Xkk09qypQpeuaZZ8o9t3jxYuXk5NRYLcwtfB2XmYBK1K9fX3Xq1FFgYPm8X1BQoDvuuEMxMTFyOBzq0KGDHnroIZXdfP706dOKjY1VbGysTp8+7er37bffqmnTpvrlL3+pkpISSdLEiRMVHh6ur776SomJiQoLC1N0dLTmz5+vqtzM/n//+5+GDx+uiIgIhYeHa8iQIeUuPaSlpemGG26QJA0ePNh1Ge1i60FeeeUVderUSSEhIerSpYtef/11TZw4sdyZna+//lp+fn566KGHtHjxYrVt21YOh0M7duyQJL3zzjsaMGCAwsLCVL9+fV177bXauXNnuXF+vM8yZetRfsjPz0/Jycn617/+pQ4dOigkJERxcXF6//33K/Q/dOiQfvOb3ygyMlIOh0OdO3fWsmXLKmz3zTff6LrrrlNYWJiaNGmimTNnqqio6IKvzY/l5ubqxhtvVEREhC677DJNnz5dZ86ccT0/cOBAde/evdK+HTp0UGJi4nn3vW/fPhlj1K9fvwrPlV0O/aETJ05o5syZatWqlRwOh5o3b66kpKRy63qOHTvmCuohISHq3r27li9fXm4/F5vbrKwsXX/99WrYsKFCQkLUq1cvrV69+uIvFuBBnJkBJJ08eVK5ubkyxujYsWN6/PHHlZ+fr1tuucW1jTFGo0aN0rvvvqvJkyerR48eWrdunf74xz/q0KFDevTRR1WnTh0tX75c/fr105/+9Cc98sgjkqSpU6fq5MmTSktLU0BAgGufJSUluuqqq9S3b1898MADWrt2rVJTU1VcXKz58+eft94vvvhCAwYMUEREhO666y4FBQXp6aef1qBBg/Tee++pT58+uvLKKzVt2jT97W9/07333quOHTtKkutnZdasWaMxY8aoa9euWrhwob777jtNnjxZzZo1q3T75557TmfOnNFtt90mh8Ohhg0b6u2339bw4cPVpk0bzZ07V6dPn9bjjz+ufv36acuWLdW+3PXee+9p5cqVmjZtmhwOh5588kldddVV+vjjj9WlSxdJ585m9O3b1xV+GjdurLfeekuTJ09WXl6eZsyYIelc6BwyZIgOHDigadOmKTo6Ws8//7zeeecdt2q68cYb1apVKy1cuFCbN2/W3/72N3333Xf65z//KUkaP368pkyZou3bt7tqlKRPPvlEu3fv1p///Ofz7rtly5aSzoXLG264QaGhoefdNj8/XwMGDNDOnTv1m9/8Rj179lRubq5Wr16tb775Ro0aNdLp06c1aNAg7dmzR8nJyWrdurVeeeUVTZw4USdOnND06dPL7bOyuf3iiy/Ur18/NWvWTPfcc4/CwsL08ssv67rrrtNrr72mX//61269fsDPxgCXsOeee85IqvDP4XCYtLS0ctu+8cYbRpL5y1/+Uq79+uuvN35+fmbPnj2utlmzZhl/f3/z/vvvm1deecVIMosXLy7Xb8KECUaS+cMf/uBqKy0tNVdffbUJDg42OTk5rnZJJjU11fX4uuuuM8HBwWbv3r2utsOHD5u6deuaK6+80tVWNva7775bpdeja9eupnnz5ubUqVOutoyMDCPJtGzZ0tW2b98+I8lERESYY8eOldtHjx49TJMmTczx48ddbZ999pnx9/c3SUlJ5Y7/h/ssk5qaan78p6lsXj799FNX2/79+01ISIj59a9/7WqbPHmyadq0qcnNzS3X/6abbjL16tUzhYWFxhhjFi9ebCSZl19+2bVNQUGBadeuXZVer7IaR40aVa7997//vZFkPvvsM2OMMSdOnDAhISHm7rvvLrfdtGnTTFhYmMnPz7/gOElJSUaSadCggfn1r39tHnroIbNz584K282ZM8dIMv/+978rPFdaWlrumFesWOF67uzZsyY+Pt6Eh4ebvLw8Y8yF53bIkCGma9eu5syZM+X2/8tf/tJcfvnlFzwWwJMIM7iklYWZJUuWmPT0dJOenm5WrFhhrrrqKhMYGGhee+0117a33XabCQgIcP3RL7Np0yYjyTz++OOutqKiItO1a1fTunVr07hxYzNw4EDXm0qZsjCza9eucu1vvfWWkWRefPFFV9sPw0xxcbEJDQ01N954Y4Xjuf32242/v785efKkMca9MHPo0CEjydx7770VnuvatWulYWbSpEnltjt8+LCRZO66664K+0hMTDSNGjUqd/zuhJn4+PgK244ZM8aEhoaa4uJiU1paaurXr29uu+02k5OTU+5f2Txv3LjRGGNMQkKCadq0aYU5eeCBB9wKM+vWrSvXvnPnTiPJLFy4sFyNLVq0cI1VXFxsIiMjzbhx4y44hjHGlJSUmCeeeML07NmzXNj+1a9+Zb755hvXdp07dzbdu3e/4L4SEhJMVFSUKSkpKdf+4osvGknmP//5jzHm/HN7/Phx4+fnZxYsWFDh9Z03b56RVK4moCaxZgaQ1Lt3bw0dOlRDhw7VuHHjtGbNGnXq1EnJyck6e/asJGn//v2Kjo5W3bp1y/Utu2yzf/9+V1twcLCWLVumffv26dSpU3ruuecqrAORJH9/f7Vp06ZcW/v27SXpvB+nzsnJUWFhoTp06FDhuY4dO6q0tFQHDx6s+sF/r6z+du3aVXiusjZJat26daX7OF9tubm5KigocLs2Sbr88ssrtLVv316FhYXKyclRTk6OTpw4oWeeeUaNGzcu92/SpEmSzq0ZKauzXbt2Feaksrrdqalt27by9/cvN3dJSUk6cOCAPvjgA0nS22+/rezsbI0fP/6i+/f399fUqVOVmZmp3NxcrVq1SsOHD9c777yjm266ybXd3r17y13Gqsz+/ft1+eWXy9+//J/9yn5/pYpzu2fPHhljNHv27Aqvb2pqqqT/e32BmsaaGaAS/v7+Gjx4sB577DF9+eWX6ty5s9v7WLdunSTpzJkz+vLLLyu8OdQGderUqXbfysKdJNcCaXeVlpZKkm655RZNmDCh0m26detWrX1XVWXHlJiYqMjISK1YsUJXXnmlVqxYoaioKA0dOtStfV922WUaNWqURo0a5VobtX//ftfamp/bj+e27PW98847z7tw+XyhF/A0wgxwHsXFxZLOLa6Uzi3IfPvtt3Xq1KlyZ2eysrJcz5f5/PPPNX/+fE2aNElbt27Vrbfeqm3btqlevXrlxigtLdVXX33lOhsjSbt375ak8y6Ubdy4sUJDQ7Vr164Kz2VlZcnf318xMTGSzh8YKlNW/549eyo8V1nbhfZxvtoaNWqksLAwSVKDBg104sSJCtv9+AxBmS+//LJC2+7duxUaGqrGjRtLkurWrauSkpKLBoWWLVtq+/btMsaUe40qq/tCfhxS9+zZo9LS0nJzFxAQoLFjxyotLU3333+/3njjDU2ZMqXcQnB39erVS++9956OHDmili1bqm3bttq+ffsF+7Rs2VKff/65SktLy52dqez3tzJlZxCDgoLcDmKAp3GZCaiE0+nU+vXrFRwc7DoNP2LECJWUlOiJJ54ot+2jjz4qPz8/DR8+3NV34sSJio6O1mOPPaa0tDRlZ2dr5syZlY71w/0ZY/TEE08oKChIQ4YMqXT7gIAAJSQkaNWqVeUuZ2RnZ+uFF15Q//79FRERIUmu4FBZaPix6OhodenSRf/85z9dAU469ymibdu2XbS/JDVt2lQ9evTQ8uXLy425fft2rV+/XiNGjHC1tW3bVidPntTnn3/uajty5Ihef/31Sve9adMmbdmyxfX44MGDWrVqlRISEhQQEKCAgACNHj1ar732WqVv7D/8XpYRI0bo8OHDevXVV11thYWFFb7P5WKWLFlS7vHjjz8uSa7fhTLjx4/Xd999p9tvv73Cp+TO5+jRo66PQ//Q2bNntWHDBvn7+7vOhIwePVqfffZZpa+d+f5j/iNGjNDRo0e1cuVK13PFxcV6/PHHFR4eroEDB16wniZNmmjQoEF6+umndeTIkQrP1+T33gAVeHfJDuBdZQtD58+fb55//nnz/PPPm4cfftjExcUZSeaee+5xbVtSUmIGDx5s/Pz8zG233WaWLFlirr32WiPJzJgxw7XdnDlzjJ+fn3nnnXdcbX/5y1+MJLNmzRpX24QJE0xISIi5/PLLTVJSklmyZIm55pprKl2Eqx99mmn79u0mLCzMNGvWzPz1r381999/v2nTpo1xOBxm8+bNru2OHDliAgICTN++fU1aWpp58cUXTXZ29nlfj9WrVxs/Pz/TrVs38+ijj5o5c+aYhg0bmi5duphWrVq5titbJPrggw9W2Ed6eroJDAw0sbGx5sEHHzTz5883jRs3Ng0aNDBfffWVa7vc3FwTFhZm2rRpYxYvXmzuu+8+ExMT41rs+uPj79Kli2nUqJGZP3++uf/++03Lli1NSEiI65NDxhhz9OhR07JlSxMaGmqmT59unn76abNw4UJzww03mAYNGri2K/vkUtknjRYvXmzi4uJMt27d3FoA3LVrVzNy5EizZMkSc8sttxhJZuzYsZX26dKli5FkOnbseMF9l/nkk0+Mn5+fGTJkiLnvvvvMsmXLzKJFi0z37t0r/M6dOnXKdOrUyQQEBJgpU6aYpUuXmvvuu8/07dvXbN261RhjTGFhoenYsaMJDg42d9xxh3n88cfNwIEDK3zS7kJz+8UXX5gGDRqYyy67zNxzzz3mmWeeMQsWLDAjRoww3bp1q9JxAZ5AmMElrbKPZoeEhJgePXqYp556qsKnXU6dOmVmzpxpoqOjTVBQkLn88svNgw8+6NouMzPTBAYGlvu4tTHnPsHyi1/8wkRHR5vvvvvOGHMuzISFhZm9e/eahIQEExoaaiIjI01qamqFT5z8OMwYY8yWLVtMYmKiCQ8PN6GhoWbw4MHmo48+qnCMzz77rGnTpo0JCAio0hv1Sy+9ZGJjY43D4TBdunQxq1evNqNHjzaxsbGubS70hmeMMW+//bbp16+fqVOnjomIiDAjR440O3bsqLDd+vXrTZcuXUxwcLDp0KGDWbFixXk/zTR16lSzYsUKc/nllxuHw2GuuOKKSo8lOzvbTJ061cTExJigoCATFRVlhgwZYp555ply2+3fv9+MGjXKhIaGmkaNGpnp06ebtWvXuhVmduzYYa6//npTt25d06BBA5OcnGxOnz5daZ+yT0rdd999F9x3mby8PPPYY4+ZxMRE07x5cxMUFGTq1q1r4uPjzbPPPlvhd/P48eMmOTnZNGvWzAQHB5vmzZubCRMmlPuYenZ2tpk0aZJp1KiRCQ4ONl27djXPPfdcuf1cbG737t1rkpKSTFRUlAkKCjLNmjUz11xzjXn11VerdFyAJ/gZU4WvGgXws5s4caJeffXVcpd0fFWPHj3UuHFjpaene2V8Pz8/TZ06tcIlPps89thjmjlzpr7++mu1aNHC2+UAtQprZgC4OJ1O18LnMhkZGfrss880aNAg7xRVCxhj9I9//EMDBw4kyAAewKeZALgcOnRIQ4cO1S233KLo6GhlZWVp6dKlioqK0m9/+1tvl2edgoICrV69Wu+++662bdumVatWebskoFYizABwadCggeLi4vT3v/9dOTk5CgsL09VXX61Fixbpsssu83Z51snJydHYsWNVv3593XvvvRo1apS3SwJqJdbMAAAAq7FmBgAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKwW6O0CqqK0tFSHDx9W3bp15efn5+1yAABADTDG6NSpU4qOjpa///nPv1gRZg4fPqyYmBhvlwEAALzg4MGDat68+XmftyLM1K1bV9K5g4mIiPByNT+N0+nU+vXrlZCQoKCgIG+Xg+8xL76LufFNzIvvqk1zk5eXp5iYGFcOOB8rwkzZpaWIiIhaEWZCQ0MVERFh/S9ZbcK8+C7mxjcxL76rNs7NxZaYsAAYAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFjN7TDz/vvva+TIkYqOjpafn5/eeOONi/bJyMhQz5495XA41K5dO6WlpVWjVAAAgIrcDjMFBQXq3r27lixZUqXt9+3bp6uvvlqDBw/W1q1bNWPGDN16661at26d28UCAAD8mNv3Zho+fLiGDx9e5e2XLl2q1q1b6+GHH5YkdezYURs3btSjjz6qxMREd4cHAAAox+M3mty0aZOGDh1ari0xMVEzZsw4b5+ioiIVFRW5Hufl5Uk6d/Msp9PpkTprSln9th9HbcO8+C7mxjcxLzWjsLBQu3btcqtP/ukifbRtr+rW36zwOo4q9+vQoYNCQ0PdLdGjqvr75fEwc/ToUUVGRpZri4yMVF5enk6fPq06depU6LNw4ULNmzevQvv69et97oWurvT0dG+XgEowL76LufFNzItn7d27V3fccUe1+j7g5vYPP/yw2rZtW62xPKWwsLBK23k8zFTHrFmzlJKS4nqcl5enmJgYJSQkKCIiwouV/XROp1Pp6ekaNmxYrbk1e23AvPgu5sY3MS81o7CwUP3793erz+4jJ/XH13fowV93Uvum9arczxfPzJRdmbkYj4eZqKgoZWdnl2vLzs5WREREpWdlJMnhcMjhqHhqLCgoqNb8p6lNx1KbMC++i7nxTcyLZ9WrV0+9e/d2q0/w/uNybDqrLj16qkfLyzxUWc2o6u+Wx79nJj4+Xhs2bCjXlp6ervj4eE8PDQAALgFuh5n8/Hxt3bpVW7dulXTuo9dbt27VgQMHJJ27RJSUlOTa/re//a2++uor3XXXXcrKytKTTz6pl19+WTNnzvx5jgAAAFzS3A4zn376qa644gpdccUVkqSUlBRdccUVmjNnjiTpyJEjrmAjSa1bt9aaNWuUnp6u7t276+GHH9bf//53PpYNAAB+Fm6vmRk0aJCMMed9vrJv9x00aJD+97//uTsUAADARXFvJgAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGqEGQAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqgd4uAACAS8W+3AIVFBV7dIy9OQWun4GBnnubD3MEqnWjMI/t3x2EGQAAasC+3AINfiijxsa749VtHh/j3TsH+USgIcwAAFADys7ILB7TQ+2ahHtunNNF+m/GJl0zKF5hdRweGWPPsXzNWLnV42eZqoowAwBADWrXJFxdmtXz2P6dTqeONpZ6tmygoKAgj43jS1gADAAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFitWmFmyZIlatWqlUJCQtSnTx99/PHHF9x+8eLF6tChg+rUqaOYmBjNnDlTZ86cqVbBAAAAP+R2mFm5cqVSUlKUmpqqLVu2qHv37kpMTNSxY8cq3f6FF17QPffco9TUVO3cuVP/+Mc/tHLlSt17770/uXgAAAC3w8wjjzyiKVOmaNKkSerUqZOWLl2q0NBQLVu2rNLtP/roI/Xr109jx45Vq1atlJCQoJtvvvmiZ3MAAACqItCdjc+ePavMzEzNmjXL1ebv76+hQ4dq06ZNlfb55S9/qRUrVujjjz9W79699dVXX+nNN9/U+PHjzztOUVGRioqKXI/z8vIkSU6nU06n052SfU5Z/bYfR23DvPgu5sY3MS/uKy4udv305OtWE3NT08dyMW6FmdzcXJWUlCgyMrJce2RkpLKysirtM3bsWOXm5qp///4yxqi4uFi//e1vL3iZaeHChZo3b16F9vXr1ys0NNSdkn1Wenq6t0tAJZgX38Xc+CbmpeoO5ktSoDZu3Kj94Z4fz5NzU1PHUlhYWKXt3Aoz1ZGRkaH77rtPTz75pPr06aM9e/Zo+vTpWrBggWbPnl1pn1mzZiklJcX1OC8vTzExMUpISFBERISnS/Yop9Op9PR0DRs2TEFBQd4uB99jXnwXc+ObmBf3fXE4Tw9t26z+/furc7Tn3stqYm5q6ljKrsxcjFthplGjRgoICFB2dna59uzsbEVFRVXaZ/bs2Ro/frxuvfVWSVLXrl1VUFCg2267TX/605/k719x2Y7D4ZDD4ajQHhQUVGv+09SmY6lNmBffxdz4Jual6gIDA10/a+I18+Tc1NSxVHXfbi0ADg4OVlxcnDZs2OBqKy0t1YYNGxQfH19pn8LCwgqBJSAgQJJkjHFneAAAgArcvsyUkpKiCRMmqFevXurdu7cWL16sgoICTZo0SZKUlJSkZs2aaeHChZKkkSNH6pFHHtEVV1zhusw0e/ZsjRw50hVqAAAAqsvtMDNmzBjl5ORozpw5Onr0qHr06KG1a9e6FgUfOHCg3JmYP//5z/Lz89Of//xnHTp0SI0bN9bIkSP117/+9ec7CgAAcMmq1gLg5ORkJScnV/pcRkZG+QECA5WamqrU1NTqDAUAAHBB3JsJAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGqEGQAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGqEGQAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAViPMAAAAqwV6uwBfkvnNfh0+lV3l7YuKzujQwQNujVFaWqJdWbv0lU7I3z/Arb7NYlrI4Qip8vbRdSMV17ylW2MAADzHLzBP+/J2yT8k3GNjFBcX63DxYe38dqcCAz3zNr8vL19+gXke2Xd1EGa+ty+3QDe/9KgcjTd4frAo6cMT/3G/37fubV6UM0TrJv5VrRuFuT8WAOBnF1T//+nej++rkbGeXPukR/cfVH+IpBEeHaOqCDPfKygqlvNEH82Iv04xDUOr1OennJnpENvBo2dmDn5bqAe/PKKComK3xgAAeI7zRB89fPVYtW3i2TMzH278UP369/PYmZm9x/I17V97PbLv6iDM/IApjtCVra5Ql2b1qt6ph3tjOJ1Ovak3NWLECAUFBbnX2Q3bD53UA8UFHts/AMB9pjhCrSM6qNNlbrzPuMnpdGpf4D51bNjRY+8zpWdOyhTneGTf1cECYAAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsVq0ws2TJErVq1UohISHq06ePPv744wtuf+LECU2dOlVNmzaVw+FQ+/bt9eabb1arYAAAgB9y+67ZK1euVEpKipYuXao+ffpo8eLFSkxM1K5du9SkSZMK2589e1bDhg1TkyZN9Oqrr6pZs2bav3+/6tev/3PUDwAALnFuh5lHHnlEU6ZM0aRJkyRJS5cu1Zo1a7Rs2TLdc889FbZftmyZvv32W3300UeuW5G3atXqp1UNAADwPbfCzNmzZ5WZmalZs2a52vz9/TV06FBt2rSp0j6rV69WfHy8pk6dqlWrVqlx48YaO3as7r77bgUEBFTap6ioSEVFRa7HeXl5kiSn0ymn0+lOyVVWXFzs+umpMSS59u3JMaSaO57aoqbmBe5jbnwT8+K+2vQ+U9PHcjFuhZnc3FyVlJQoMjKyXHtkZKSysrIq7fPVV1/pnXfe0bhx4/Tmm29qz549+v3vfy+n06nU1NRK+yxcuFDz5s2r0L5+/XqFhoa6U3KVHcyXpEBt3LhR+8M9MkQ56enpHt1/TR9PbeHpeUH1MTe+iXmputr0PlNTx1JYWFil7dy+zOSu0tJSNWnSRM8884wCAgIUFxenQ4cO6cEHHzxvmJk1a5ZSUlJcj/Py8hQTE6OEhARFRER4pM4vDufpoW2b1b9/f3WO9swY0rmUmZ6ermHDhrkuu3lCTR1PbVFT8wL3MTe+iXlxX216n6mpYym7MnMxboWZRo0aKSAgQNnZ2eXas7OzFRUVVWmfpk2bKigoqNwlpY4dO+ro0aM6e/asgoODK/RxOBxyOBwV2oOCgjw2MYGBga6fNfEf05PHItX88dQWnp4XVB9z45uYl6qrTe8zNXUsVd23Wx/NDg4OVlxcnDZs2OBqKy0t1YYNGxQfH19pn379+mnPnj0qLS11te3evVtNmzatNMgAAAC4w+3vmUlJSdGzzz6r5cuXa+fOnfrd736ngoIC16ebkpKSyi0Q/t3vfqdvv/1W06dP1+7du7VmzRrdd999mjp16s93FAAA4JLl9pqZMWPGKCcnR3PmzNHRo0fVo0cPrV271rUo+MCBA/L3/7+MFBMTo3Xr1mnmzJnq1q2bmjVrpunTp+vuu+/++Y4CAABcsqq1ADg5OVnJycmVPpeRkVGhLT4+Xps3b67OUAAAABfEvZkAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGqEGQAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGqEGQAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYL9HYBAABcCk47SyRJ2w+d9Og4BaeL9GmOFLX/O4XVcXhkjD3H8j2y3+oizAAAUAP2fh8A7vn3thoYLVDP7/nE46OEOXwjRvhGFQAA1HIJnaMkSW2bhKtOUIDHxtl15KTueHWbHr6+qzo0reexccIcgWrdKMxj+3cHYQYAgBrQMCxYN/Vu4fFxiouLJUltG4epSzPPhRlfwgJgAABgtWqFmSVLlqhVq1YKCQlRnz599PHHH1ep30svvSQ/Pz9dd9111RkWAACgArfDzMqVK5WSkqLU1FRt2bJF3bt3V2Jioo4dO3bBfl9//bXuvPNODRgwoNrFAgAA/JjbYeaRRx7RlClTNGnSJHXq1ElLly5VaGioli1bdt4+JSUlGjdunObNm6c2bdr8pIIBAAB+yK0FwGfPnlVmZqZmzZrlavP399fQoUO1adOm8/abP3++mjRposmTJ+uDDz646DhFRUUqKipyPc7Ly5MkOZ1OOZ1Od0qusrIFU8XFxR4bQ5Jr354cQ6q546ktampe4D7mxjcxL76rNv39r2r9boWZ3NxclZSUKDIyslx7ZGSksrKyKu2zceNG/eMf/9DWrVurPM7ChQs1b968Cu3r169XaGioOyVX2cF8SQrUxo0btT/cI0OUk56e7tH91/Tx1BaenhdUH3Pjm5gX31P293/z5s06tN3b1fw0hYWFVdrOox/NPnXqlMaPH69nn31WjRo1qnK/WbNmKSUlxfU4Ly9PMTExSkhIUEREhCdK1ReH8/TQts3q37+/Okd7ZgzpXMpMT0/XsGHDFBQU5LFxaup4aouamhe4j7nxTcyL7/rswLfStk/Vt29fdW/R0Nvl/CRlV2Yuxq0w06hRIwUEBCg7O7tce3Z2tqKioipsv3fvXn399dcaOXKkq620tPTcwIGB2rVrl9q2bVuhn8PhkMNR8SuYg4KCPPafJjAw0PWzJv5jevJYpJo/ntrC0/OC6mNufBPz4ntq09//qtbv1gLg4OBgxcXFacOGDa620tJSbdiwQfHx8RW2j42N1bZt27R161bXv1GjRmnw4MHaunWrYmJi3BkeAACgArcvM6WkpGjChAnq1auXevfurcWLF6ugoECTJk2SJCUlJalZs2ZauHChQkJC1KVLl3L969evL0kV2gHUfoWFheddX3c++aeL9NG2vWrQ6FOFu3nTvNjYWI+tswPgO9wOM2PGjFFOTo7mzJmjo0ePqkePHlq7dq1rUfCBAwfk788XCwOoKCsrS3FxcdXq+0A1+mRmZqpnz57VGg+APaq1ADg5OVnJycmVPpeRkXHBvmlpadUZEkAtEBsbq8zMTLf67DpyQimvbNMjN3RVh6b13R4PQO3HjSYB1JjQ0FC3z5T47z8uxwen1bFLd/VoeZmHKgNgM64HAQAAqxFmAACA1QgzAADAaqyZ+d5pZ4kkafuhkx4dp+B0kT7NkaL2f6cwNz9m6o49x/I9tm8AAHwJYeZ7e79/87/n39tqYLRAPb/nkxoYRwpzMMUAgNqNd7rvJXQ+dzuGtk3CVScowGPj7DpyUne8uk0PX99VHZrW89g40rkg07pRmEfHAADA2wgz32sYFqyberfw+Dhlt2Zv2zhMXZp5NswAAHApYAEwAACwGmEGAABYjTADAACsxpoZANW2L7dABUXFHh1jb06B62dgoOf+ZLFgHrAXYQZAtezLLdDghzJqbLw7XvX81ya8e+cgAg1gIcIMgGopOyOzeEwPtWsS7rlxThfpvxmbdM2geI990eSeY/masXKrx88yAfAMwgyAn6Rdk3CPfs2A0+nU0cZSz5YNFBQU5LFxANiLBcAAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGqEGQAAYDVuZwCg2vwC87Qvb5f8Qzx3b6bi4mIdLj6snd/u9Nhds/fl5csvMM8j+wbgeYQZANUWVP//6d6P76uRsZ5c+6RH9x9Uf4ikER4dA4BnEGYAVJvzRB89fPVYtfXgXbOLi4v14cYP1a9/P4+dmdl7LF/T/rXXI/sG4HmEGQDVZooj1Dqigzpd5tm7Zu8L3KeODTt67K7ZpWdOyhTneGTfADyPBcAAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGp8AzCAajntLJEkbT900qPjFJwu0qc5UtT+7xRWx+GRMfYcy/fIfgHUDMIMgGrZ+30AuOff22pgtEA9v+cTj48S5uBPImAj/ucCqJaEzlGSpLZNwlUnKMBj4+w6clJ3vLpND1/fVR2aeu4eUGGOQLVuFOax/QPwHMIMgGppGBasm3q38Pg4xcXFkqS2jcPUpZnnwgwAe7EAGAAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsxu0MANSYwsJCZWVludVn15ETKjq6Rzu311Hp8fpu9Y2NjVVoaKhbfQDYhzADoMZkZWUpLi6uWn3HLne/T2Zmpnr27Fmt8QDYgzADoMbExsYqMzPTrT75p4u05t1NunpwvMLrONweD0DtR5gBUGNCQ0PdPlPidDr1Xe4xxffupaCgIA9VBsBmLAAGAABWI8wAAACrEWYAAIDVCDMAAMBq1QozS5YsUatWrRQSEqI+ffro448/Pu+2zz77rAYMGKAGDRqoQYMGGjp06AW3BwAAcIfbYWblypVKSUlRamqqtmzZou7duysxMVHHjh2rdPuMjAzdfPPNevfdd7Vp0ybFxMQoISFBhw4d+snFAwAAuB1mHnnkEU2ZMkWTJk1Sp06dtHTpUoWGhmrZsmWVbv+vf/1Lv//979WjRw/Fxsbq73//u0pLS7Vhw4afXDwAAIBb3zNz9uxZZWZmatasWa42f39/DR06VJs2barSPgoLC+V0OtWwYcPzblNUVKSioiLX47y8PEnnvm/C6XS6U7LPKS4udv20/Vhqk7K5YE58D3Pjm5gX31Wb3meqWr9bYSY3N1clJSWKjIws1x4ZGVnl+63cfffdio6O1tChQ8+7zcKFCzVv3rwK7evXr7f+PisH8yUpUJs3b9ah7d6uBj+Wnp7u7RJwHsyNb2JefE9tep8pLCys0nY1+g3AixYt0ksvvaSMjAyFhIScd7tZs2YpJSXF9TgvL8+11iYiIqImSvWYzw58K237VH379lX3Fuc/O4Wa5XQ6lZ6ermHDhvEtsz6GufFNzIvvqk3vM2VXZi7GrTDTqFEjBQQEKDs7u1x7dna2oqKiLtj3oYce0qJFi/T222+rW7duF9zW4XDI4ah4D5agoCDr/9MEBga6ftp+LLVRbfgdq62YG9/EvPie2vQ+U9X63VoAHBwcrLi4uHKLd8sW88bHx5+33wMPPKAFCxZo7dq16tWrlztDAgAAXJDbl5lSUlI0YcIE9erVS71799bixYtVUFCgSZMmSZKSkpLUrFkzLVy4UJJ0//33a86cOXrhhRfUqlUrHT16VJIUHh6u8PDwn/FQAADApcjtMDNmzBjl5ORozpw5Onr0qHr06KG1a9e6FgUfOHBA/v7/d8Lnqaee0tmzZ3X99deX209qaqrmzp3706oHAACXvGotAE5OTlZycnKlz2VkZJR7/PXXX1dnCAAAgCrh3kwAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNVq9K7ZAACg6goLC5WVleVWn11HTqjo6B7t3F5HpcfrV7lfbGysQkND3azQNxBmAADwUVlZWYqLi6tW37HL3ds+MzNTPXv2rNZY3kaYAQDAR8XGxiozM9OtPvmni7Tm3U26enC8wus43BrLVoQZAAB8VGhoqNtnS5xOp77LPab43r0UFBTkocp8CwuAAQCA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGqEGQAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAVgv0dgE2KywsVFZWllt9dh05oaKje7Rzex2VHq/vVt/Y2FiFhoa61QcAgNqOMPMTZGVlKS4urlp9xy53v09mZqZ69uxZrfEAAKitCDM/QWxsrDIzM93qk3+6SGve3aSrB8crvI7D7fEAAEB5hJmfIDQ01O0zJU6nU9/lHlN8714KCgryUGUAAFw6WAAMAACsRpgBAABWI8wAAACrEWYAAIDVCDMAAMBqhBkAAGA1wgwAALAaYQYAAFiNMAMAAKxGmAEAAFYjzAAAAKsRZgAAgNUIMwAAwGpW3DXbGCNJysvL83IlP53T6VRhYaHy8vK4a7YPYV58F3Pjm5gX31Wb5qbsfb8sB5yPFWHm1KlTkqSYmBgvVwIAAGraqVOnVK9evfM+72cuFnd8QGlpqQ4fPqy6devKz8/P2+X8JHl5eYqJidHBgwcVERHh7XLwPebFdzE3vol58V21aW6MMTp16pSio6Pl73/+lTFWnJnx9/dX8+bNvV3GzyoiIsL6X7LaiHnxXcyNb2JefFdtmZsLnZEpwwJgAABgNcIMAACwGmGmhjkcDqWmpsrhcHi7FPwA8+K7mBvfxLz4rktxbqxYAAwAAHA+nJkBAABWI8wAAACrEWYAAIDVCDMAAMBqhJka8v7772vkyJGKjo6Wn5+f3njjDW+XBEkLFy7UL37xC9WtW1dNmjTRddddp127dnm7rEveU089pW7durm+9Cs+Pl5vvfWWt8vCjyxatEh+fn6aMWOGt0u55M2dO1d+fn7l/sXGxnq7rBpDmKkhBQUF6t69u5YsWeLtUvAD7733nqZOnarNmzcrPT1dTqdTCQkJKigo8HZpl7TmzZtr0aJFyszM1Keffqpf/epXuvbaa/XFF194uzR875NPPtHTTz+tbt26ebsUfK9z5846cuSI69/GjRu9XVKNseJ2BrXB8OHDNXz4cG+XgR9Zu3ZtucdpaWlq0qSJMjMzdeWVV3qpKowcObLc47/+9a966qmntHnzZnXu3NlLVaFMfn6+xo0bp2effVZ/+ctfvF0OvhcYGKioqChvl+EVnJkBfuDkyZOSpIYNG3q5EpQpKSnRSy+9pIKCAsXHx3u7HEiaOnWqrr76ag0dOtTbpeAHvvzyS0VHR6tNmzYaN26cDhw44O2SagxnZoDvlZaWasaMGerXr5+6dOni7XIuedu2bVN8fLzOnDmj8PBwvf766+rUqZO3y7rkvfTSS9qyZYs++eQTb5eCH+jTp4/S0tLUoUMHHTlyRPPmzdOAAQO0fft21a1b19vleRxhBvje1KlTtX379kvqOrMv69Chg7Zu3aqTJ0/q1Vdf1YQJE/Tee+8RaLzo4MGDmj59utLT0xUSEuLtcvADP1zG0K1bN/Xp00ctW7bUyy+/rMmTJ3uxsppBmAEkJScn67///a/ef/99NW/e3NvlQFJwcLDatWsnSYqLi9Mnn3yixx57TE8//bSXK7t0ZWZm6tixY+rZs6erraSkRO+//76eeOIJFRUVKSAgwIsVokz9+vXVvn177dmzx9ul1AjCDC5pxhj94Q9/0Ouvv66MjAy1bt3a2yXhPEpLS1VUVOTtMi5pQ4YM0bZt28q1TZo0SbGxsbr77rsJMj4kPz9fe/fu1fjx471dSo0gzNSQ/Pz8cgl537592rp1qxo2bKgWLVp4sbJL29SpU/XCCy9o1apVqlu3ro4ePSpJqlevnurUqePl6i5ds2bN0vDhw9WiRQudOnVKL7zwgjIyMrRu3Tpvl3ZJq1u3boX1ZGFhYbrssstYZ+Zld955p0aOHKmWLVvq8OHDSk1NVUBAgG6++WZvl1YjCDM15NNPP9XgwYNdj1NSUiRJEyZMUFpampeqwlNPPSVJGjRoULn25557ThMnTqz5giBJOnbsmJKSknTkyBHVq1dP3bp107p16zRs2DBvlwb4pG+++UY333yzjh8/rsaNG6t///7avHmzGjdu7O3SaoSfMcZ4uwgAAIDq4ntmAACA1QgzAADAaoQZAABgNcIMAACwGmEGAABYjTADAACsRpgBAABWI8wAAACrEWYAAIDVCDMAPCYnJ0e/+93v1KJFCzkcDkVFRSkxMVEffviht0sDUItwbyYAHjN69GidPXtWy5cvV5s2bZSdna0NGzbo+PHjHhnv7NmzCg4O9si+AfguzswA8IgTJ07ogw8+0P3336/BgwerZcuW6t27t2bNmqVRo0a5trn99tsVGRmpkJAQdenSRf/9739d+3jttdfUuXNnORwOtWrVSg8//HC5MVq1aqUFCxYoKSlJERERuu222yRJGzdu1IABA1SnTh3FxMRo2rRpKigoqLmDB1CjCDMAPCI8PFzh4eF64403VFRUVOH50tJSDR8+XB9++KFWrFihHTt2aNGiRQoICJAkZWZm6sYbb9RNN92kbdu2ae7cuZo9e3aFu8w/9NBD6t69u/73v/9p9uzZ2rt3r6666iqNHj1an3/+uVauXKmNGzcqOTm5Jg4bgBdw12wAHvPaa69pypQpOn36tHr27KmBAwfqpptuUrdu3bR+/XoNHz5cO3fuVPv27Sv0HTdunHJycrR+/XpX21133aU1a9boiy++kHTuzMwVV1yh119/3bXNrbfeqoCAAD399NOuto0bN2rgwIEqKChQSEiIB48YgDdwZgaAx4wePVqHDx/W6tWrddVVVykjI0M9e/ZUWlqatm7dqubNm1caZCRp586d6tevX7m2fv366csvv1RJSYmrrVevXuW2+eyzz5SWluY6MxQeHq7ExESVlpZq3759P/9BAvA6FgAD8KiQkBANGzZMw4YN0+zZs3XrrbcqNTVVd95558+y/7CwsHKP8/Pzdfvtt2vatGkVtm3RosXPMiYA30KYAVCjOnXqpDfeeEPdunXTN998o927d1d6dqZjx44VPsL94Ycfqn379q51NZXp2bOnduzYoXbt2v3stQPwTVxmAuARx48f169+9SutWLFCn3/+ufbt26dXXnlFDzzwgK699loNHDhQV155pUaPHq309HTt27dPb731ltauXStJuuOOO7RhwwYtWLBAu3fv1vLly/XEE09c9IzO3XffrY8++kjJycnaunWrvvzyS61atYoFwEAtxpkZAB4RHh6uPn366NFHH9XevXvldDoVExOjKVOm6N5775V0boHwnXfeqZtvvlkFBQVq166dFi1aJOncGZaXX35Zc+bM0YIFC9S0aVPNnz9fEydOvOC43bp103vvvac//elPGjBggIwxatu2rcaMGePpQwbgJXyaCQAAWI3LTAAAwGqEGQAAYDXCDAAAsBphBgAAWI0wAwAArEaYAQAAViPMAAAAqxFmAACA1QgzAADAaoQZAABgNcIMAACw2v8HL4V81+qInmgAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import statsmodels.api as sm\n",
    "\n",
    "\n",
    "correlation = X_test[['percentile_cosine_similarity', 'Score']].corr().values[0,1]\n",
    "print('Correlation between user & vector similarity percentile metric and review number of stars (score): %.2f%%' % (100*correlation))\n",
    "\n",
    "# boxplot of cosine similarity for each score\n",
    "X_test.boxplot(column='percentile_cosine_similarity', by='Score')\n",
    "plt.title('')\n",
    "plt.show()\n",
    "plt.close()\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe a weak trend, showing that the higher the similarity score between the user and the product embedding, the higher the review score. Therefore, the user and product embeddings can weakly predict the review score - even before the user receives the product!\n",
    "\n",
    "Because this signal works in a different way than the more commonly used collaborative filtering, it can act as an additional feature to slightly improve the performance on existing problems."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "be4b5d5b73a21c599de40d6deb1129796d12dc1cc33a738f7bac13269cfcafe8"
  },
  "kernelspec": {
   "display_name": "Python 3.7.3 64-bit ('base': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}