You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
225 lines
56 KiB
Plaintext
225 lines
56 KiB
Plaintext
2 years ago
|
{
|
||
|
|
"cells": [
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"## Zero-shot classification with embeddings\n",
|
||
|
|
"\n",
|
||
|
|
"In this notebook we will classify the sentiment of reviews using embeddings and zero labeled data! The dataset is created in the [Obtain_dataset Notebook](Obtain_dataset.ipynb).\n",
|
||
|
|
"\n",
|
||
|
|
"We'll define positive sentiment to be 4 and 5-star reviews, and negative sentiment to be 1 and 2-star reviews. 3-star reviews are considered neutral and we won't use them for this example.\n",
|
||
|
|
"\n",
|
||
|
|
"We will perform zero-shot classification by embedding descriptions of each class and then comparing new samples to those class embeddings."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 1,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [],
|
||
|
|
"source": [
|
||
|
|
"import pandas as pd\n",
|
||
|
|
"import numpy as np\n",
|
||
|
|
"\n",
|
||
|
|
"from sklearn.metrics import classification_report\n",
|
||
|
|
"\n",
|
||
|
|
"df = pd.read_csv('output/embedded_1k_reviews.csv')\n",
|
||
|
|
"df['babbage_similarity'] = df.babbage_similarity.apply(eval).apply(np.array)\n",
|
||
|
|
"df['babbage_search'] = df.babbage_search.apply(eval).apply(np.array)\n",
|
||
|
|
"\n",
|
||
|
|
"df= df[df.Score!=3]\n",
|
||
|
|
"df['sentiment'] = df.Score.replace({1:'negative', 2:'negative', 4:'positive', 5:'positive'})"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"### Zero-Shot Classification\n",
|
||
|
|
"To perform zero shot classification, we want to predict labels for our samples without any training. To do this, we can simply embed short descriptions of each label, such as positive and negative, and then compare the cosine distance between embeddings of samples and label descriptions. \n",
|
||
|
|
"\n",
|
||
|
|
"The highest similarity label to the sample input is the predicted label. We can also define a prediction score to be the difference between the cosine distance to the positive and to the negative label. This score can be used for plotting a precision-recall curve, which can be used to select a different tradeoff between precision and recall, by selecting a different threshold."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 16,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
" precision recall f1-score support\n",
|
||
|
|
"\n",
|
||
|
|
" negative 0.67 0.88 0.76 136\n",
|
||
|
|
" positive 0.98 0.93 0.95 789\n",
|
||
|
|
"\n",
|
||
|
|
" accuracy 0.92 925\n",
|
||
|
|
" macro avg 0.82 0.90 0.86 925\n",
|
||
|
|
"weighted avg 0.93 0.92 0.92 925\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAyIElEQVR4nO3dd5xV1b3//9ebJhZsgIoggqgxYAQVNaJGYlewm9iV2Fv0F6/eq9dIIolRY8kvxthyNWLDQoxdCSIoGk0oAoIoLaiMqAjSRNrM5/vH3oOH4czMBubMmfJ+Ph7zmLP7Z52B8zlrrb3XUkRgZmZWUZNiB2BmZnWTE4SZmeXlBGFmZnk5QZiZWV5OEGZmlpcThJmZ5eUEYTVKUj9JbxU7jpok6XRJ/8iw372Srq+NmGqDpJmSDklf/1rSo8WOyWqXE4QhaQNJD0j6WNIiSeMkHVnsuLJIP8S+lbRY0heSHpK0SU1eIyIei4jDMux3UUT8piavXU5SSPomLWeJpDskNS3EtczKOUEYQDPgU+BAYDPgl8BTkjoVM6i1cHREbALsAfQkiX81kprVelQ1r3tazgOBk4FzihxPjWogf6MGxQnCiIhvIuLXETEzIsoi4kXgP8CelR0jaTtJz0iaI2mupLsq2e+Pkj6VtFDSGEkH5GzbW9LodNsXku5I17eU9Gh63vmSRknaOkM5SoBXgF3T84SkSyVNBaam6/qmNaT5kv4pabfqypTbbKbEHyR9mcb9vqTy6z0k6bc55ztf0jRJ8yQ9L2nbnG0h6SJJU9NY/ixJ1ZUxLec04G2gR8751qVcXSS9nq77StJjkjbPEkNFko5Nr79Q0nRJR6TrVzVTpcurmqokdUrfh3MlfQK8LukVSZdVOPd4SSekr3eRNDR9Tz+S9NN1ideycYKwNaQfxjsDkyrZ3hR4EfgY6AS0B56o5HSjSD7ItgQeB56W1DLd9kfgjxGxKdAFeCpdfzZJTWY7oDVwEfBthri3A44C3stZfRywD9BV0u7Ag8CF6XnvA55Pm9iylukw4Eck789mwE+BuXliOQi4Kd3eLj1vxfP1BfYCdkv3O7y6Mqbn3gU4AJiWLq9ruZTGuC3wfZL3+9dZYqgQz97Aw8DVwOYk78/MtTjFgen1DwcGAafmnLsrsD3wkqSNgaEk/462Ak4B7k73sQJwgrDVSGoOPAYMjIgPK9ltb5IPlavT2sfSiMjbMR0Rj0bE3IhYGRG3AxsA30s3rwB2lNQmIhZHxLs561sDO0ZEaUSMiYiFVYT9rKT5wFvAG8DvcrbdFBHzIuJb4ALgvoj4V3regcAy4IdrUaYVQCtgF0ARMTkiZufZ73TgwYgYGxHLgGuBfSs0290cEfMj4hNgODk1gkqMlfQNMBkYAdydrl+nckXEtIgYGhHLImIOcAfJh/XaOjct69C0BlpSxb+dfH6dxvYt8Hegh6Tt022nA8+k72FfYGZE/DX99/Qe8DfgJ+sQs2XgBGGrSGoCPAIsBy7LWf+Kks7RxZJOJ/mm+XFErMxwzqskTZa0IP0Q3wxok24+l+Sb+IdpM1LfdP0jwBDgCUmfSfp9mrgqc1xEbB4R20fEJekHTblPc15vD/xX2gwzP41nO5IP0ExliojXgbuAPwNfSrpf0qZ5dt2W5Ft7+XGLSWoa7XP2+Tzn9RJgEwBJk3Le7wNy9tkj3edkklrRxutTLklbS3pCSaf3QuBRvvvbrI3tgOnrcFy5VX+jiFgEvERSO4CkNvFY+np7YJ8K5Twd2GY9rm1VcIIwIGlbBx4AtgZOjIgV5dsi4siI2CT9eYzkP3RHVdOpmH64/TdJ88kWEbE5sICkaYOImBoRp5I0F9wCDJa0cUSsiIgbIqIr0Ivkm+NZ61i03OGKPwVuTJNJ+c9GETEoa5nSuO+MiD2BriQJ7uo8u31G8oEGQNo80hooyXD+bjnv98gK2yIingLeAfqvZ7l+R/L+/CBt5juD9G+zlj4laSLM5xtgo5zlfB/mFYeUHgScKmlfoCVJ7ar8Om9UKOcmEXHxOsRsGThBWLl7SNqBj67wDTyffwOzgZslbaykU3m/PPu1AlYCc4BmkvoDq75tSzpDUtuIKAPmp6vLJP1Y0g/S9vOFJM06ZetTuNRfgIsk7aPExpL6SGqVtUyS9kqPb07y4be0ktgGAT+T1EPSBiQfxv+KiJk1UA6Am4HzJW2zHuVqBSwGFkhqT/5El8UDJGU9WFITSe3TfhKAccApkppL6gmclOF8L5Mk1wHAk+m/D0j6UnaWdGZ6vubp3+P76xi3VcMJwkjbey8kaQP/vEJz0hoiohQ4GtgR+ASYRdLsUdEQ4FVgCklzy1JWb/I5ApgkaTFJh/UpaXLaBhhMkhwmk/QrPLKexSQiRgPnkzQRfU3SydtvLcu0KckH8tdpmeYCt+a51mvA9SRt5LNJvmGfUnG/9SjL+8CbJH0L61quG0iarRaQNOs8s46x/Bv4GfCH9Fxv8F3t6XqSsn+dXu/xDOdblsZySO7+afPTYSTv42ckTXS3kPRrWQHIEwaZmVk+rkGYmVleThBmZpaXE4SZmeXlBGFmZnk1mMGx2rRpE506dSp2GGZm9cqYMWO+ioi2+bY1mATRqVMnRo8eXewwzMzqFUkfV7bNTUxmZpaXE4SZmeXlBGFmZnk5QZiZWV5OEGZmllfBEoSkB5VMyzixku2SdKeSKRknSNojZ9vZSqZinCrp7ELFaGZmlStkDeIhktE6K3MksFP6cwHJcNNI2hL4FcmEKHsDv5K0RQHjNDOzPAr2HEREvFlhesWKjgUejmQ42XclbS6pHdAbGBoR8wAkDSVJNIMKEeeS5Su5d8T6TIZlZnVVyxZN6derExu1aDCPfNWqYr5r7Vl9boBZ6brK1q9B0gUktQ86duy4TkF8u7yUPw2ftk7HmlndVT6TwS7btOKgXbYubjD1VL1OqxFxP3A/QM+ePddpYovWm2zAf27qU6NxmVnxTSxZQN8/vUVpTcxF2EgVM0GUkEx2Xq5Duq6EpJkpd/2IWovKzBqE8hrEKxNns2xlKaVlQVkEpWWwT+ct2W7Ljao+gRU1QTwPXCbpCZIO6QURMVvSEOB3OR3ThwHXFitIM6ufgiRDPDO2hGfGlqy27Yhu23DvmXsWI6x6pWAJQtIgkppAG0mzSO5Mag4QEfeSTEx+FMn8uUtI5rQlIuZJ+g0wKj3VgPIOazOzrHbrsDl/v6QXLZs3pWkT0USiaRNx4SOjWe52p0wKeRfTqdVsD+DSSrY9CDxYiLjMrPHYveOad8jPX7KC6XPm8NoHXzD3m2W0bN6UFaXBitIyVpaW8dmCpWzasvmq5RVlwafzlrDlxi1W229FWbBR86bccGy3BnuXVMMslZlZJb5ctAyA8x7ONj1Ai6ZNkGDZyjLabLIBLZqKZk2bsHxlGZ8vXMope3dkz+0b5qNaThBm1qiMuKo3H32xiHabtaRZkyY0aQIbNW9G82aiedMmNG/ShBbNmtC8adIkJSnved6YMoezH/x3LUdfu5wgzKxR6dRmYzq12bjYYdQLHqzPzMzycoIwM7O8nCDMzCwvJwgzM8vLCcLMzPJygjAzs7ycIMzMLC8nCDMzyyvTg3LpyKrbAt8CMyPCI12ZmTVwlSYISZuRDKZ3KtACmAO0BLaW9C5wd0QMr5Uozcys1lVVgxgMPAwcEBHzczdI2hM4U9IOEfFAAeMzM7MiqTRBRMShVWwbA4wpSERmZlYnVNtJrcQZkvqnyx0l7V340MzMrJiydFLfDZQBBwEDgEXA34C9ChiXmVmdtmJlcq/OdX9/n45bbkTTJmLSZwtZ8O0KIoLlpWXccEw3Tt6rY5EjXXdZEsQ+EbGHpPcAIuJrSS0KHJeZWZ22siyZ8/rDzxfx4eeLaNG0Cd/bphVbbNScTTdszjvT5zL1i8VFjnL9ZEkQKyQ1hWQGcEltSWoUZmaN1hG7bsP7vz6MDZo1pUWzNVvru/V/tQhR1awsD8rdCfwd2ErSjcBbwO8KGpWZWT3QqmXzvMmhoai2BhERj0kaAxwMCDguIiYXPDIzMyuqahOEpDuBJyL
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 432x288 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {
|
||
|
|
"needs_background": "light"
|
||
|
|
},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"from openai.embeddings_utils import cosine_similarity, get_embedding\n",
|
||
|
|
"from sklearn.metrics import PrecisionRecallDisplay\n",
|
||
|
|
"\n",
|
||
|
|
"def evaluate_emeddings_approach(\n",
|
||
|
|
" labels = ['negative', 'positive'], \n",
|
||
|
|
" engine = 'text-similarity-babbage-001',\n",
|
||
|
|
"):\n",
|
||
|
|
" label_embeddings = [get_embedding(label, engine=engine) for label in labels]\n",
|
||
|
|
"\n",
|
||
|
|
" def label_score(review_embedding, label_embeddings):\n",
|
||
|
|
" return cosine_similarity(review_embedding, label_embeddings[1]) - cosine_similarity(review_embedding, label_embeddings[0])\n",
|
||
|
|
"\n",
|
||
|
|
" engine_col_name = engine.replace('-','_').replace('_query','')\n",
|
||
|
|
" probas = df[engine_col_name].apply(lambda x: label_score(x, label_embeddings))\n",
|
||
|
|
" preds = probas.apply(lambda x: 'positive' if x>0 else 'negative')\n",
|
||
|
|
"\n",
|
||
|
|
" report = classification_report(df.sentiment, preds)\n",
|
||
|
|
" print(report)\n",
|
||
|
|
"\n",
|
||
|
|
" display = PrecisionRecallDisplay.from_predictions(df.sentiment, probas, pos_label='positive')\n",
|
||
|
|
" _ = display.ax_.set_title(\"2-class Precision-Recall curve\")\n",
|
||
|
|
"\n",
|
||
|
|
"evaluate_emeddings_approach(labels=['negative', 'positive'], engine='text-similarity-babbage-001')"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"We can see that this classifier already performs extremely well. We used similarity embeddings, and the simplest possible label name. Let's try to improve on this by using more descriptive label names, and search embeddings."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 17,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
" precision recall f1-score support\n",
|
||
|
|
"\n",
|
||
|
|
" negative 0.65 0.93 0.76 136\n",
|
||
|
|
" positive 0.99 0.91 0.95 789\n",
|
||
|
|
"\n",
|
||
|
|
" accuracy 0.92 925\n",
|
||
|
|
" macro avg 0.82 0.92 0.86 925\n",
|
||
|
|
"weighted avg 0.94 0.92 0.92 925\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAwoUlEQVR4nO3deZwU1bn/8c+XAQRRwCAigiyiRkEFFY2iROJucNeoGBc04pKo+cWruXqNRsmiRk3uNSYuuXpFY3Ahxl0JKihuUUBAcAMMIoiKsokswszz+6NqxmbsmSmY6emZ4ft+veY1Vaeqq57TA/10nVN1jiICMzOzypoVOwAzM2uYnCDMzCwvJwgzM8vLCcLMzPJygjAzs7ycIMzMLC8nCKtTkoZKerHYcdQlST+U9M8M+90q6Yr6iKk+SJot6cB0+SpJfy12TFa/nCAMSRtJukPSB5K+kDRZ0mHFjiuL9ENshaRlkj6RdJekTeryHBFxb0QcnGG/cyPiV3V57nKSQtKXaT3nSfq9pJJCnMusnBOEATQHPgT2A9oBvwAekNSjmEGtgyMiYhNgN6A/SfxrkdS83qOqe33Teu4HnAicWeR46lQT+Rs1KU4QRkR8GRFXRcTsiCiLiMeBfwO7V/UaSVtLekjSAkmfS7q5iv3+R9KHkpZKmihpYM62PSVNSLd9Iun3aXkrSX9Nj7tY0uuSOmWoxzzgKWCn9Dgh6SeSZgAz0rLD0yukxZJelrRLTXXKbTZT4g+SPk3jflNS+fnukvTrnOMNkzRT0kJJj0raKmdbSDpX0ow0lj9JUk11TOs5E3gJ6JdzvPWpVy9Jz6Vln0m6V1L7LDFUJumo9PxLJc2SdGhaXtFMla5XNFVJ6pG+Dz+SNAd4TtJTks6vdOwpko5Nl3eQNCZ9T9+VdML6xGvZOEHYN6QfxtsD06vYXgI8DnwA9AC6APdVcbjXST7IvgX8DXhQUqt02/8A/xMRbYFewANp+ekkVzJbAx2Ac4EVGeLeGvg+8EZO8dHAd4DeknYF7gTOSY97G/Bo2sSWtU4HA98leX/aAScAn+eJZX/gmnR75/S4lY93OLAHsEu63yE11TE99g7AQGBmur6+9VIa41bAjiTv91VZYqgUz57A3cAlQHuS92f2Ohxiv/T8hwAjgSE5x+4NdAeekNQGGEPy72gL4CTgz+k+VgBOELYWSS2Ae4EREfFOFbvtSfKhckl69bEyIvJ2TEfEXyPi84hYExE3AhsB3043rwa2lbR5RCyLiFdzyjsA20ZEaURMjIil1YT9sKTFwIvA88Bvc7ZdExELI2IFcDZwW0T8Kz3uCGAVsNc61Gk1sCmwA6CIeDsi5ufZ74fAnRExKSJWAZcBe1dqtrs2IhZHxBxgLDlXBFWYJOlL4G1gHPDntHy96hURMyNiTESsiogFwO9JPqzX1Y/Suo5Jr0DnVfNvJ5+r0thWAP8A+knqnm77IfBQ+h4eDsyOiP9L/z29Afwd+MF6xGwZOEFYBUnNgHuAr4Dzc8qfUtI5ukzSD0m+aX4QEWsyHPNiSW9LWpJ+iLcDNk83/4jkm/g7aTPS4Wn5PcBo4D5JH0n6XZq4qnJ0RLSPiO4R8eP0g6bchznL3YH/SJthFqfxbE3yAZqpThHxHHAz8CfgU0m3S2qbZ9etSL61l79uGcmVRpecfT7OWV4ObAIgaXrO+z0wZ5/d0n1OJLkqalObeknqJOk+JZ3eS4G/8vXfZl1sDcxaj9eVq/gbRcQXwBMkVweQXE3cmy53B75TqZ4/BLasxbmtGk4QBiRt68AdQCfguIhYXb4tIg6LiE3Sn3tJ/kN3Uw2diumH289Jmk82i4j2wBKSpg0iYkZEDCFpLrgOGCWpTUSsjoirI6I3MIDkm+Np61m13OGKPwR+kyaT8p+NI2Jk1jqlcd8UEbsDvUkS3CV5dvuI5AMNgLR5pAMwL8Px++S83+MrbYuIeAB4BbiylvX6Lcn7s3PazHcK6d9mHX1I0kSYz5fAxjnr+T7MKw8pPRIYImlvoBXJ1VX5eZ6vVM9NIuK89YjZMnCCsHK3kLQDH1HpG3g+rwHzgWsltVHSqbxPnv02BdYAC4Dmkq4EKr5tSzpFUseIKAMWp8Vlkr4naee0/XwpSbNOWW0ql/oLcK6k7yjRRtJgSZtmrZOkPdLXtyD58FtZRWwjgTMk9ZO0EcmH8b8iYnYd1APgWmCYpC1rUa9NgWXAEkldyJ/osriDpK4HSGomqUvaTwIwGThJUgtJ/YHjMxzvSZLkOhy4P/33AUlfyvaSTk2P1yL9e+y4nnFbDZwgjLS99xySNvCPKzUnfUNElAJHANsCc4C5JM0elY0GngbeI2luWcnaTT6HAtMlLSPpsD4pTU5bAqNIksPbJP0K99SymkTEBGAYSRPRIpJO3qHrWKe2JB/Ii9I6fQ5cn+dczwBXkLSRzyf5hn1S5f1qUZc3gRdI+hbWt15XkzRbLSFp1nloPWN5DTgD+EN6rOf5+urpCpK6L0rP97cMx1uVxnJg7v5p89PBJO/jRyRNdNeR9GtZAcgTBpmZWT6+gjAzs7ycIMzMLC8nCDMzy8sJwszM8moyg2Ntvvnm0aNHj2KHYWbWqEycOPGziOiYb1uTSRA9evRgwoQJxQ7DzKxRkfRBVdvcxGRmZnk5QZiZWV5OEGZmlpcThJmZ5eUEYWZmeRUsQUi6U8m0jNOq2C5JNymZknGqpN1ytp2uZCrGGZJOL1SMZmZWtUJeQdxFMlpnVQ4Dtkt/ziYZbhpJ3wJ+STIhyp7ALyVtVsA4zcwsj4I9BxERL1SaXrGyo4C7IxlO9lVJ7SV1BgYBYyJiIYCkMSSJZmQh4lz+1RpuHVebybDMrKnYeKPmDB3Qg1YtSoodSoNQzAflurD23ABz07Kqyr9B0tkkVx9069ZtvYJY8VUpfxw7c71ea2ZNR/nMB327tmfvXh2KG0wD0aifpI6I24HbAfr3779eE1t02GQj/n3N4DqNy8wan9f+vZATbnuFMs+RU6GYCWIeyWTn5bqmZfNImplyy8fVW1RmtkEqTwyPT53PZ8tWURZBWRmURrBqdSnLVpXSZqMSSsuCsoCysqA0IllPy0ojaNOyhDP26UnL5o3/JtFiJohHgfMl3UfSIb0kIuZLGg38Nqdj+mDgsmIFaWYbhtWlydTXI1+bw8jX5tTqWP17fIvduzf+e2sKliAkjSS5Ethc0lySO5NaAETErSQTk3+fZP7c5SRz2hIRCyX9Cng9PdTw8g5rM7NCGbhdRx75yT60bllCM4mSZqJEQoKSZsnvFiXNKJFo1kw0S8tz931p1mecesdrTaaZqpB3MQ2pYXsAP6li253AnYWIy8ysKn23bl+r1wvVTSANRONvJDMzs4JwgjAzs7ycIMzMLC8nCDMzy8sJwszM8nKCMDOzvJwgzMwsLycIMzPLywnCzMzycoIwM7O8nCDMzCyvTGMxpSOrbgWsAGZHRFlBozIzs6KrMkFIakcymN4QoCWwAGgFdJL0KvDniBhbL1GamVm9q+4KYhRwNzAwIhbnbpC0O3CqpG0i4o4CxmdmZkVSZYKIiIOq2TYRmFiQiMzMGqnPv1wFwA9ufYWn/99AdtiybZEjqp0aO6mVOEXSlel6N0l7Fj40M7PGZZeu7SuW35izuGhx1JUsdzH9GdibpC8C4AvgTwWLyMyskeq5eRteveyAYodRZ7LcxfSdiNhN0hsAEbFIUssCx2VmZkWW5QpitaQSIAAkdQR8m6uZWROXJUHcBPwD2ELSb4AXgd8WNCozMyu6GpuYIuJeSROBAwABR0fE2wWPzMzMiqrGBCHpJuC+iHDHtJnZBiRLJ/VE4BeSvk3S1HRfREwobFhmZo3bG3MW0aKkGV+tKaOkGRy7W1dalDSu4e+yNDGNAEZI+hZwHHCdpG4RsV3BozMza2TKIgB4YMJcHpgwt6K8V8dN6N/jW8UKa71kGqwvtS2wA9AdcB+EmVkeW7VvzdH9tqL3Vm3ZdotN+HzZV1wyaiqrS6PYoa2zLH0QvwOOAWYB9wO/qjw2k5mZfe2/T9q1YvmVWZ8XMZLayXIFMQv
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 432x288 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {
|
||
|
|
"needs_background": "light"
|
||
|
|
},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"evaluate_emeddings_approach(labels=['An Amazon review with a negative sentiment.', 'An Amazon review with a positive sentiment.'], engine='text-similarity-babbage-001')"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"Using the search embeddings and descriptive names leads to an additional improvement in performance."
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "code",
|
||
|
|
"execution_count": 11,
|
||
|
|
"metadata": {},
|
||
|
|
"outputs": [
|
||
|
|
{
|
||
|
|
"name": "stdout",
|
||
|
|
"output_type": "stream",
|
||
|
|
"text": [
|
||
|
|
" precision recall f1-score support\n",
|
||
|
|
"\n",
|
||
|
|
" negative 0.77 0.79 0.78 136\n",
|
||
|
|
" positive 0.96 0.96 0.96 789\n",
|
||
|
|
"\n",
|
||
|
|
" accuracy 0.94 925\n",
|
||
|
|
" macro avg 0.87 0.88 0.87 925\n",
|
||
|
|
"weighted avg 0.94 0.94 0.94 925\n",
|
||
|
|
"\n"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"data": {
|
||
|
|
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAsFUlEQVR4nO3deZhU1bnv8e+PQVEBNUwiqCBqDCgQRaIYI3FWcDbOGmdxvvGYHL25TmiixiEnMSZqjkQ0hmiIUVQUUUAxiVFQQHAARFQGFWVSmeG9f+zdbdFUd22gq6uH3+d5+mFPtfe7qpt6a62191qKCMzMzCpqVOoAzMysdnKCMDOzvJwgzMwsLycIMzPLywnCzMzycoIwM7O8nCCsWkk6S9IrpY6jOkk6TdLzGY67V9K1NRFTTZA0U9JB6fINkv5c6pisZjlBGJI2lfSApA8lfSlpgqTDSx1XFumH2FJJX0n6VNKDkppX5zUi4pGIOCTDcQMi4qbqvHYZSSHp67ScsyXdJalxMa5lVsYJwgCaAB8D+wNbAv8PeExSp1IGtR6OjIjmwB5AL5L41yKpSY1HVf16pOXcHzgJOKfE8VSrevI7qlecIIyI+DoiboiImRGxJiKeBj4A9qzsNZK2k/S4pHmSvpD0u0qO+42kjyUtljRe0n45+3pLGpfu+1TSXen2ZpL+nJ53oaTXJbXLUI7ZwLPAbul5QtIlkqYB09Jt/dMa0kJJ/5LUvVCZcpvNlPi1pM/SuN+SVHa9ByXdnHO+8yVNlzRf0jBJ2+bsC0kDJE1LY7lHkgqVMS3ndOCfQM+c821IubpIGpVu+1zSI5K2yhJDRZKOTq+/WNL7kg5Lt5c3U6Xr5U1Vkjql78O5kj4CRkl6VtKlFc49UdJx6fKukkam7+l7kk7ckHgtGycIW0f6YbwLMKWS/Y2Bp4EPgU5AB+CvlZzudZIPsm8BfwH+JqlZuu83wG8ioiXQBXgs3f5jkprMdkArYACwNEPc2wFHAG/mbD4G+B7QVdJ3gUHAhel57wOGpU1sWct0CPADkvdnS+BE4Is8sRwA3JLub5+et+L5+gN7Ad3T4w4tVMb03LsC+wHT0/UNLZfSGLcFvkPyft+QJYYK8fQGHgJ+CmxF8v7MXI9T7J9e/1BgCHBKzrm7AjsAz0jaAhhJ8nfUFjgZ+H16jBWBE4StRVJT4BFgcES8W8lhvUk+VH6a1j6WRUTejumI+HNEfBERqyLiTmBT4Nvp7pXATpJaR8RXEfFqzvZWwE4RsToixkfE4irCfkLSQuAV4CXglzn7bomI+RGxFLgAuC8i/pOedzCwHNh7Pcq0EmgB7AooIt6JiLl5jjsNGBQRb0TEcuAaYJ8KzXa3RsTCiPgIGE1OjaASb0j6GngHGAP8Pt2+QeWKiOkRMTIilkfEPOAukg/r9XVuWtaRaQ10dhV/O/nckMa2FPgH0FPSDum+04DH0/ewPzAzIv6U/j29Cfwd+NEGxGwZOEFYOUmNgIeBFcClOdufVdI5+pWk00i+aX4YEasynPMqSe9IWpR+iG8JtE53n0vyTfzdtBmpf7r9YWAE8FdJcyT9Kk1clTkmIraKiB0i4uL0g6bMxznLOwD/lTbDLEzj2Y7kAzRTmSJiFPA74B7gM0n3S2qZ59BtSb61l73uK5KaRoecYz7JWV4CNAeQNCXn/d4v55g90mNOIqkVbbEx5ZLUTtJflXR6Lwb+zDe/m/WxHfD+BryuTPnvKCK+BJ4hqR1AUpt4JF3eAfhehXKeBmyzEde2KjhBGJC0rQMPAO2A4yNiZdm+iDg8IpqnP4+Q/IfeXgU6FdMPt5+RNJ9sHRFbAYtImjaIiGkRcQpJc8FtwFBJW0TEyoi4MSK6An1IvjmeuYFFyx2u+GPgF2kyKfvZPCKGZC1TGvdvI2JPoCtJgvtpnsPmkHygAZA2j7QCZmc4f7ec93tshX0REY8B/wau28hy/ZLk/dk9beY7nfR3s54+JmkizOdrYPOc9Xwf5hWHlB4CnCJpH6AZSe2q7DovVShn84i4aANitgycIKzMH0jagY+s8A08n9eAucCtkrZQ0qm8b57jWgCrgHlAE0nXAeXftiWdLqlNRKwBFqab10j6oaTd0/bzxSTNOms2pnCpPwIDJH1PiS0k9ZPUImuZJO2Vvr4pyYffskpiGwKcLamnpE1JPoz/ExEzq6EcALcC50vaZiPK1QL4ClgkqQP5E10WD5CU9UBJjSR1SPtJACYAJ0tqKqkXcEKG8w0nSa4DgUfTvw9I+lJ2kXRGer6m6e/jOxsYtxXgBGGk7b0XkrSBf1KhOWkdEbEaOBLYCfgImEXS7FHRCOA5YCpJc8sy1m7yOQyYIukrkg7rk9PktA0wlCQ5vEPSr/DwRhaTiBgHnE/SRLSApJP3rPUsU0uSD+QFaZm+AG7Pc60XgGtJ2sjnknzDPrnicRtRlreAl0n6Fja0XDeSNFstImnWeXwDY3kNOBv4dXqul/im9nQtSdkXpNf7S4bzLU9jOSj3+LT56RCS93EOSRPdbST9WlYE8oRBZmaWj2sQZmaWlxOEmZnl5QRhZmZ5OUGYmVle9WZwrNatW0enTp1KHYaZWZ0yfvz4zyOiTb599SZBdOrUiXHjxpU6DDOzOkXSh5XtcxOTmZnl5QRhZmZ5OUGYmVleThBmZpaXE4SZmeVVtAQhaZCSaRknV7Jfkn6rZErGSZL2yNn3YyVTMU6T9ONixWhmZpUrZg3iQZLROitzOLBz+nMByXDTSPoWcD3JhCi9geslbV3EOM3MLI+iPQcRES9XmF6xoqOBhyIZTvZVSVtJag/0BUZGxHwASSNJEs2QYsS5ZMUq7h2zMZNhmZll9+1tWtKve/tSh5FJKR+U68DacwPMSrdVtn0dki4gqX2w/fbbb1AQS1es5u7R0zfotWZm6yMCWjRr4gRREyLifuB+gF69em3QxBatmm/KB7f0q9a4zMzyufnptxny2kelDiOzUt7FNJtksvMyHdNtlW03M6vTPvj8a75esZpfDn+Hu0ZOZemK1aUOqUqlTBDDgDPTu5n2BhZFxFySaSoPkbR12jl9SLrNzKxOW5PO4Pngv2by2xen8eZHC0ocUdWK1sQkaQhJh3NrSbNI7kxqChAR95JMTH4Eyfy5S0jmtCUi5ku6CXg9PdXAsg5rM7O67E9n9wbg9Znz+dG9/2ZNLZ/xuZh3MZ1SYH8Al1SybxAwqBhxmZmVWlnT0pWPTeB7O7aiSSPx34ftyjZbNitxZGvzk9RmZjVs262SRPDZl8t548MF/OPN2fzngy9KHNW66vRdTGZmddFObVsw89bk7skZ877igDtfKnFE+bkGYWZmeTlBmJlZXk4QZmaWlxOEmVkJLV2Z3NF096jaN+SPE4SZWQl9u10LAFatXlPiSNblu5jMzEqoSeNG7NVpa5o2rn3f12tfRGZmVis4QZiZWV5OEGZmlpcThJmZ5eUEYWZmeTlBmJlZXk4QZmaWlxOEmZnl5QRhZlZi733yJf96/wv+Of3zUoeyFicIM7MSW7xsFQA/GzqpxJGszQnCzKzEpt58OP27t2fVmto1HpMThJlZiW3SpBFvz13Mp4uXs2jJylKHU84JwsysFli+Mqk9vPnxghJH8g0nCDOzWuDuU79b6hDW4QRhZmZ5OUGYmVleThBmZpaXE4SZmeXlBGFmZnk5QZiZWV5NshwkqS2wL7AtsBSYDIyLiNr12J+ZmVWbKhOEpB8CVwPfAt4EPgOaAccAXSQNBe6MiMVFjtPMzGpYoRrEEcD5EfFRxR2SmgD9gYOBvxchNjMzK6EqE0RE/LSKfauAJ6o7IDMzqx02uJNa0tnVGYiZmdUuG3MX043VFoWZmdU6hTqpK5u9QkC76g/HzMxqi0Kd1O2AQ4GK488K+Fehk0s6DPgN0Bj434i4tcL+HYBBQBtgPnB6RMxK9/0K6EdSyxkJXBERUeiaZmZWPQo1MT0NNI+IDyv8zATGVPVCSY2Be4DDga7AKZK6VjjsDuChiOgODARuSV/bh+S5i+7AbsBewP7rUzAzM9s4VSaIiDg3Il6pZN+pBc7dG5g
|
||
|
|
"text/plain": [
|
||
|
|
"<Figure size 432x288 with 1 Axes>"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
"metadata": {
|
||
|
|
"needs_background": "light"
|
||
|
|
},
|
||
|
|
"output_type": "display_data"
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"source": [
|
||
|
|
"evaluate_emeddings_approach(labels=['An Amazon review with a negative sentiment.', 'An Amazon review with a positive sentiment.'], engine='text-similarity-babbage-001')"
|
||
|
|
]
|
||
|
|
},
|
||
|
|
{
|
||
|
|
"cell_type": "markdown",
|
||
|
|
"metadata": {},
|
||
|
|
"source": [
|
||
|
|
"As shown above, zero-shot classification with embeddings can lead to great results, especially when the labels are more descriptive than just simple words."
|
||
|
|
]
|
||
|
|
}
|
||
|
|
],
|
||
|
|
"metadata": {
|
||
|
|
"interpreter": {
|
||
|
|
"hash": "be4b5d5b73a21c599de40d6deb1129796d12dc1cc33a738f7bac13269cfcafe8"
|
||
|
|
},
|
||
|
|
"kernelspec": {
|
||
|
|
"display_name": "Python 3.7.3 64-bit ('base': conda)",
|
||
|
|
"name": "python3"
|
||
|
|
},
|
||
|
|
"language_info": {
|
||
|
|
"codemirror_mode": {
|
||
|
|
"name": "ipython",
|
||
|
|
"version": 3
|
||
|
|
},
|
||
|
|
"file_extension": ".py",
|
||
|
|
"mimetype": "text/x-python",
|
||
|
|
"name": "python",
|
||
|
|
"nbconvert_exporter": "python",
|
||
|
|
"pygments_lexer": "ipython3",
|
||
|
|
"version": "3.7.3"
|
||
|
|
},
|
||
|
|
"orig_nbformat": 4
|
||
|
|
},
|
||
|
|
"nbformat": 4,
|
||
|
|
"nbformat_minor": 2
|
||
|
|
}
|