You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
915 lines
34 KiB
Plaintext
915 lines
34 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "cb1537e6",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Running Hybrid VSS Queries with Redis and OpenAI\n",
|
|
"\n",
|
|
"This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings and running hybrid queries that combine VSS and lexical search using Redis Query and Search capability. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The Redis Query and Search capability allows you to index and search for vectors in Redis. This notebook will show you how to use the Redis Query and Search to index and search for vectors created by using the OpenAI API and stored in Redis.\n",
|
|
"\n",
|
|
"Hybrid queries combine vector similarity with traditional Redis Query and Search filtering capabilities on GEO, NUMERIC, TAG or TEXT data simplifying application code. A common example of a hybrid query in an e-commerce use case is to find items visually similar to a given query image limited to items available in a GEO location and within a price range."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f1a618c5",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Prerequisites\n",
|
|
"\n",
|
|
"Before we start this project, we need to set up the following:\n",
|
|
"\n",
|
|
"* start a Redis database with RediSearch (redis-stack)\n",
|
|
"* install libraries\n",
|
|
" * [Redis-py](https://github.com/redis/redis-py)\n",
|
|
"* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n",
|
|
"\n",
|
|
"===========================================================\n",
|
|
"\n",
|
|
"### Start Redis\n",
|
|
"\n",
|
|
"To keep this example simple, we will use the Redis Stack docker container which we can start as follows\n",
|
|
"\n",
|
|
"```bash\n",
|
|
"$ docker-compose up -d\n",
|
|
"```\n",
|
|
"\n",
|
|
"This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n",
|
|
"\n",
|
|
"You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b9babafe",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Install Requirements\n",
|
|
"\n",
|
|
"Redis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "2b04113f",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Defaulting to user installation because normal site-packages is not writeable\n",
|
|
"Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)\n",
|
|
"Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)\n",
|
|
"Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)\n",
|
|
"Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)\n",
|
|
"Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)\n",
|
|
"Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n",
|
|
"Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n",
|
|
"Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23.4)\n",
|
|
"Requirement already satisfied: requests>=2.20 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (2.28.1)\n",
|
|
"Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)\n",
|
|
"Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)\n",
|
|
"Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n",
|
|
"Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)\n",
|
|
"Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)\n",
|
|
"Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)\n",
|
|
"Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)\n",
|
|
"Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)\n",
|
|
"Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)\n",
|
|
"Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)\n",
|
|
"Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)\n",
|
|
"Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"! pip install redis pandas openai\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "36fe86f4",
|
|
"metadata": {},
|
|
"source": [
|
|
"===========================================================\n",
|
|
"## Prepare your OpenAI API key\n",
|
|
"\n",
|
|
"The `OpenAI API key` is used for vectorization of query data.\n",
|
|
"\n",
|
|
"If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n",
|
|
"\n",
|
|
"Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "88be138c",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"OPENAI_API_KEY is ready\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Test that your OpenAI API key is correctly set as an environment variable\n",
|
|
"# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n",
|
|
"import os\n",
|
|
"import openai\n",
|
|
"\n",
|
|
"os.environ[\"OPENAI_API_KEY\"] = '<YOUR_OPENAI_API_KEY>'\n",
|
|
"\n",
|
|
"if os.getenv(\"OPENAI_API_KEY\") is not None:\n",
|
|
" openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n",
|
|
" print (\"OPENAI_API_KEY is ready\")\n",
|
|
"else:\n",
|
|
" print (\"OPENAI_API_KEY environment variable not found\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "97fefe4c",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Load data\n",
|
|
"\n",
|
|
"In this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "9fbebe0d",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"<class 'pandas.core.frame.DataFrame'>\n",
|
|
"Index: 1978 entries, 0 to 1998\n",
|
|
"Data columns (total 10 columns):\n",
|
|
" # Column Non-Null Count Dtype \n",
|
|
"--- ------ -------------- ----- \n",
|
|
" 0 id 1978 non-null int64 \n",
|
|
" 1 gender 1978 non-null object\n",
|
|
" 2 masterCategory 1978 non-null object\n",
|
|
" 3 subCategory 1978 non-null object\n",
|
|
" 4 articleType 1978 non-null object\n",
|
|
" 5 baseColour 1978 non-null object\n",
|
|
" 6 season 1978 non-null object\n",
|
|
" 7 year 1978 non-null int64 \n",
|
|
" 8 usage 1978 non-null object\n",
|
|
" 9 productDisplayName 1978 non-null object\n",
|
|
"dtypes: int64(2), object(8)\n",
|
|
"memory usage: 170.0+ KB\n"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/html": [
|
|
"<div>\n",
|
|
"<style scoped>\n",
|
|
" .dataframe tbody tr th:only-of-type {\n",
|
|
" vertical-align: middle;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe tbody tr th {\n",
|
|
" vertical-align: top;\n",
|
|
" }\n",
|
|
"\n",
|
|
" .dataframe thead th {\n",
|
|
" text-align: right;\n",
|
|
" }\n",
|
|
"</style>\n",
|
|
"<table border=\"1\" class=\"dataframe\">\n",
|
|
" <thead>\n",
|
|
" <tr style=\"text-align: right;\">\n",
|
|
" <th></th>\n",
|
|
" <th>id</th>\n",
|
|
" <th>gender</th>\n",
|
|
" <th>masterCategory</th>\n",
|
|
" <th>subCategory</th>\n",
|
|
" <th>articleType</th>\n",
|
|
" <th>baseColour</th>\n",
|
|
" <th>season</th>\n",
|
|
" <th>year</th>\n",
|
|
" <th>usage</th>\n",
|
|
" <th>productDisplayName</th>\n",
|
|
" </tr>\n",
|
|
" </thead>\n",
|
|
" <tbody>\n",
|
|
" <tr>\n",
|
|
" <th>0</th>\n",
|
|
" <td>15970</td>\n",
|
|
" <td>Men</td>\n",
|
|
" <td>Apparel</td>\n",
|
|
" <td>Topwear</td>\n",
|
|
" <td>Shirts</td>\n",
|
|
" <td>Navy Blue</td>\n",
|
|
" <td>Fall</td>\n",
|
|
" <td>2011</td>\n",
|
|
" <td>Casual</td>\n",
|
|
" <td>Turtle Check Men Navy Blue Shirt</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>1</th>\n",
|
|
" <td>39386</td>\n",
|
|
" <td>Men</td>\n",
|
|
" <td>Apparel</td>\n",
|
|
" <td>Bottomwear</td>\n",
|
|
" <td>Jeans</td>\n",
|
|
" <td>Blue</td>\n",
|
|
" <td>Summer</td>\n",
|
|
" <td>2012</td>\n",
|
|
" <td>Casual</td>\n",
|
|
" <td>Peter England Men Party Blue Jeans</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>2</th>\n",
|
|
" <td>59263</td>\n",
|
|
" <td>Women</td>\n",
|
|
" <td>Accessories</td>\n",
|
|
" <td>Watches</td>\n",
|
|
" <td>Watches</td>\n",
|
|
" <td>Silver</td>\n",
|
|
" <td>Winter</td>\n",
|
|
" <td>2016</td>\n",
|
|
" <td>Casual</td>\n",
|
|
" <td>Titan Women Silver Watch</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>3</th>\n",
|
|
" <td>21379</td>\n",
|
|
" <td>Men</td>\n",
|
|
" <td>Apparel</td>\n",
|
|
" <td>Bottomwear</td>\n",
|
|
" <td>Track Pants</td>\n",
|
|
" <td>Black</td>\n",
|
|
" <td>Fall</td>\n",
|
|
" <td>2011</td>\n",
|
|
" <td>Casual</td>\n",
|
|
" <td>Manchester United Men Solid Black Track Pants</td>\n",
|
|
" </tr>\n",
|
|
" <tr>\n",
|
|
" <th>4</th>\n",
|
|
" <td>53759</td>\n",
|
|
" <td>Men</td>\n",
|
|
" <td>Apparel</td>\n",
|
|
" <td>Topwear</td>\n",
|
|
" <td>Tshirts</td>\n",
|
|
" <td>Grey</td>\n",
|
|
" <td>Summer</td>\n",
|
|
" <td>2012</td>\n",
|
|
" <td>Casual</td>\n",
|
|
" <td>Puma Men Grey T-shirt</td>\n",
|
|
" </tr>\n",
|
|
" </tbody>\n",
|
|
"</table>\n",
|
|
"</div>"
|
|
],
|
|
"text/plain": [
|
|
" id gender masterCategory subCategory articleType baseColour season \n",
|
|
"0 15970 Men Apparel Topwear Shirts Navy Blue Fall \\\n",
|
|
"1 39386 Men Apparel Bottomwear Jeans Blue Summer \n",
|
|
"2 59263 Women Accessories Watches Watches Silver Winter \n",
|
|
"3 21379 Men Apparel Bottomwear Track Pants Black Fall \n",
|
|
"4 53759 Men Apparel Topwear Tshirts Grey Summer \n",
|
|
"\n",
|
|
" year usage productDisplayName \n",
|
|
"0 2011 Casual Turtle Check Men Navy Blue Shirt \n",
|
|
"1 2012 Casual Peter England Men Party Blue Jeans \n",
|
|
"2 2016 Casual Titan Women Silver Watch \n",
|
|
"3 2011 Casual Manchester United Men Solid Black Track Pants \n",
|
|
"4 2012 Casual Puma Men Grey T-shirt "
|
|
]
|
|
},
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import pandas as pd\n",
|
|
"import numpy as np\n",
|
|
"from typing import List\n",
|
|
"\n",
|
|
"from utils.embeddings_utils import (\n",
|
|
" get_embeddings,\n",
|
|
" distances_from_embeddings,\n",
|
|
" tsne_components_from_embeddings,\n",
|
|
" chart_from_components,\n",
|
|
" indices_of_nearest_neighbors_from_distances,\n",
|
|
")\n",
|
|
"\n",
|
|
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
|
|
"\n",
|
|
"# load in data and clean data types and drop null rows\n",
|
|
"df = pd.read_csv(\"../../data/styles_2k.csv\", on_bad_lines='skip')\n",
|
|
"df.dropna(inplace=True)\n",
|
|
"df[\"year\"] = df[\"year\"].astype(int)\n",
|
|
"df.info()\n",
|
|
"\n",
|
|
"# print dataframe\n",
|
|
"n_examples = 5\n",
|
|
"df.head(n_examples)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "3ce1ec50",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"<class 'pandas.core.frame.DataFrame'>\n",
|
|
"Index: 1978 entries, 0 to 1998\n",
|
|
"Data columns (total 11 columns):\n",
|
|
" # Column Non-Null Count Dtype \n",
|
|
"--- ------ -------------- ----- \n",
|
|
" 0 product_id 1978 non-null int64 \n",
|
|
" 1 gender 1978 non-null object\n",
|
|
" 2 masterCategory 1978 non-null object\n",
|
|
" 3 subCategory 1978 non-null object\n",
|
|
" 4 articleType 1978 non-null object\n",
|
|
" 5 baseColour 1978 non-null object\n",
|
|
" 6 season 1978 non-null object\n",
|
|
" 7 year 1978 non-null int64 \n",
|
|
" 8 usage 1978 non-null object\n",
|
|
" 9 productDisplayName 1978 non-null object\n",
|
|
" 10 product_text 1978 non-null object\n",
|
|
"dtypes: int64(2), object(9)\n",
|
|
"memory usage: 185.4+ KB\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"df[\"product_text\"] = df.apply(lambda row: f\"name {row['productDisplayName']} category {row['masterCategory']} subcategory {row['subCategory']} color {row['baseColour']} gender {row['gender']}\".lower(), axis=1)\n",
|
|
"df.rename({\"id\":\"product_id\"}, inplace=True, axis=1)\n",
|
|
"\n",
|
|
"df.info()\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "13859ab5",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'"
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"# check out one of the texts we will use to create semantic embeddings\n",
|
|
"df[\"product_text\"][0]\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "91df4d5b",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Connect to Redis\n",
|
|
"\n",
|
|
"Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n",
|
|
"\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "cc662c1b",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"True"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"import redis\n",
|
|
"from redis.commands.search.indexDefinition import (\n",
|
|
" IndexDefinition,\n",
|
|
" IndexType\n",
|
|
")\n",
|
|
"from redis.commands.search.query import Query\n",
|
|
"from redis.commands.search.field import (\n",
|
|
" TagField,\n",
|
|
" NumericField,\n",
|
|
" TextField,\n",
|
|
" VectorField\n",
|
|
")\n",
|
|
"\n",
|
|
"REDIS_HOST = \"localhost\"\n",
|
|
"REDIS_PORT = 6379\n",
|
|
"REDIS_PASSWORD = \"\" # default for passwordless Redis\n",
|
|
"\n",
|
|
"# Connect to Redis\n",
|
|
"redis_client = redis.Redis(\n",
|
|
" host=REDIS_HOST,\n",
|
|
" port=REDIS_PORT,\n",
|
|
" password=REDIS_PASSWORD\n",
|
|
")\n",
|
|
"redis_client.ping()\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "7d3dac3c",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Creating a Search Index in Redis\n",
|
|
"\n",
|
|
"The below cells will show how to specify and create a search index in Redis. We will:\n",
|
|
"\n",
|
|
"1. Set some constants for defining our index like the distance metric and the index name\n",
|
|
"2. Define the index schema with RediSearch fields\n",
|
|
"3. Create the index"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 7,
|
|
"id": "f894b911",
|
|
"metadata": {
|
|
"scrolled": true
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Constants\n",
|
|
"INDEX_NAME = \"product_embeddings\" # name of the search index\n",
|
|
"PREFIX = \"doc\" # prefix for the document keys\n",
|
|
"DISTANCE_METRIC = \"L2\" # distance metric for the vectors (ex. COSINE, IP, L2)\n",
|
|
"NUMBER_OF_VECTORS = len(df)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 8,
|
|
"id": "15db8380",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Define RediSearch fields for each of the columns in the dataset\n",
|
|
"name = TextField(name=\"productDisplayName\")\n",
|
|
"category = TagField(name=\"masterCategory\")\n",
|
|
"articleType = TagField(name=\"articleType\")\n",
|
|
"gender = TagField(name=\"gender\")\n",
|
|
"season = TagField(name=\"season\")\n",
|
|
"year = NumericField(name=\"year\")\n",
|
|
"text_embedding = VectorField(\"product_vector\",\n",
|
|
" \"FLAT\", {\n",
|
|
" \"TYPE\": \"FLOAT32\",\n",
|
|
" \"DIM\": 1536,\n",
|
|
" \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
|
|
" \"INITIAL_CAP\": NUMBER_OF_VECTORS,\n",
|
|
" }\n",
|
|
")\n",
|
|
"fields = [name, category, articleType, gender, season, year, text_embedding]\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 9,
|
|
"id": "3658693c",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Check if index exists\n",
|
|
"try:\n",
|
|
" redis_client.ft(INDEX_NAME).info()\n",
|
|
" print(\"Index already exists\")\n",
|
|
"except:\n",
|
|
" # Create RediSearch Index\n",
|
|
" redis_client.ft(INDEX_NAME).create_index(\n",
|
|
" fields = fields,\n",
|
|
" definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n",
|
|
")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "775c15b4",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Generate OpenAI Embeddings and Load Documents into the Index\n",
|
|
"\n",
|
|
"Now that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 10,
|
|
"id": "852cff45",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# Use OpenAI get_embeddings batch requests to speed up embedding creation\n",
|
|
"def embeddings_batch_request(documents: pd.DataFrame):\n",
|
|
" records = documents.to_dict(\"records\")\n",
|
|
" print(\"Records to process: \", len(records))\n",
|
|
" product_vectors = []\n",
|
|
" docs = []\n",
|
|
" batchsize = 1000\n",
|
|
"\n",
|
|
" for idx,doc in enumerate(records,start=1):\n",
|
|
" # create byte vectors\n",
|
|
" docs.append(doc[\"product_text\"])\n",
|
|
" if idx % batchsize == 0:\n",
|
|
" product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n",
|
|
" docs.clear()\n",
|
|
" print(\"Vectors processed \", len(product_vectors), end='\\r')\n",
|
|
" product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n",
|
|
" print(\"Vectors processed \", len(product_vectors), end='\\r')\n",
|
|
" return product_vectors\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 11,
|
|
"id": "0d791186",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n",
|
|
" product_vectors = embeddings_batch_request(documents)\n",
|
|
" records = documents.to_dict(\"records\")\n",
|
|
" batchsize = 500\n",
|
|
"\n",
|
|
" # Use Redis pipelines to batch calls and save on round trip network communication\n",
|
|
" pipe = client.pipeline()\n",
|
|
" for idx,doc in enumerate(records,start=1):\n",
|
|
" key = f\"{prefix}:{str(doc['product_id'])}\"\n",
|
|
"\n",
|
|
" # create byte vectors\n",
|
|
" text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()\n",
|
|
"\n",
|
|
" # replace list of floats with byte vectors\n",
|
|
" doc[\"product_vector\"] = text_embedding\n",
|
|
"\n",
|
|
" pipe.hset(key, mapping = doc)\n",
|
|
" if idx % batchsize == 0:\n",
|
|
" pipe.execute()\n",
|
|
" pipe.execute()\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 12,
|
|
"id": "5bfaeafa",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Records to process: 1978\n",
|
|
"Loaded 1978 documents in Redis search index with name: product_embeddings\n",
|
|
"CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms\n",
|
|
"Wall time: 3.34 s\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"%%time\n",
|
|
"index_documents(redis_client, PREFIX, df)\n",
|
|
"print(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "46050ca9",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Simple Vector Search Queries with OpenAI Query Embeddings\n",
|
|
"\n",
|
|
"Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 13,
|
|
"id": "b044aa93",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"def search_redis(\n",
|
|
" redis_client: redis.Redis,\n",
|
|
" user_query: str,\n",
|
|
" index_name: str = \"product_embeddings\",\n",
|
|
" vector_field: str = \"product_vector\",\n",
|
|
" return_fields: list = [\"productDisplayName\", \"masterCategory\", \"gender\", \"season\", \"year\", \"vector_score\"],\n",
|
|
" hybrid_fields = \"*\",\n",
|
|
" k: int = 20,\n",
|
|
" print_results: bool = True,\n",
|
|
") -> List[dict]:\n",
|
|
"\n",
|
|
" # Use OpenAI to create embedding vector from user query\n",
|
|
" embedded_query = openai.Embedding.create(input=user_query,\n",
|
|
" model=\"text-embedding-3-small\",\n",
|
|
" )[\"data\"][0]['embedding']\n",
|
|
"\n",
|
|
" # Prepare the Query\n",
|
|
" base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n",
|
|
" query = (\n",
|
|
" Query(base_query)\n",
|
|
" .return_fields(*return_fields)\n",
|
|
" .sort_by(\"vector_score\")\n",
|
|
" .paging(0, k)\n",
|
|
" .dialect(2)\n",
|
|
" )\n",
|
|
" params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n",
|
|
"\n",
|
|
" # perform vector search\n",
|
|
" results = redis_client.ft(index_name).search(query, params_dict)\n",
|
|
" if print_results:\n",
|
|
" for i, product in enumerate(results.docs):\n",
|
|
" score = 1 - float(product.vector_score)\n",
|
|
" print(f\"{i}. {product.productDisplayName} (Score: {round(score ,3) })\")\n",
|
|
" return results.docs\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 14,
|
|
"id": "7e2025f6",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0. John Players Men Blue Jeans (Score: 0.791)\n",
|
|
"1. Lee Men Tino Blue Jeans (Score: 0.775)\n",
|
|
"2. Peter England Men Party Blue Jeans (Score: 0.763)\n",
|
|
"3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n",
|
|
"4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n",
|
|
"5. French Connection Men Blue Jeans (Score: 0.74)\n",
|
|
"6. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
|
|
"7. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
|
|
"8. Do U Speak Green Men Blue Shorts (Score: 0.736)\n",
|
|
"9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# Execute a simple vector search in Redis\n",
|
|
"results = search_redis(redis_client, 'man blue jeans', k=10)\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "2007be48",
|
|
"metadata": {},
|
|
"source": [
|
|
"## Hybrid Queries with Redis\n",
|
|
"\n",
|
|
"The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 15,
|
|
"id": "2c81fbb7",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0. John Players Men Blue Jeans (Score: 0.791)\n",
|
|
"1. Lee Men Tino Blue Jeans (Score: 0.775)\n",
|
|
"2. Peter England Men Party Blue Jeans (Score: 0.763)\n",
|
|
"3. French Connection Men Blue Jeans (Score: 0.74)\n",
|
|
"4. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
|
|
"5. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
|
|
"6. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n",
|
|
"7. Denizen Women Blue Jeans (Score: 0.725)\n",
|
|
"8. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n",
|
|
"9. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# improve search quality by adding hybrid query for \"man blue jeans\" in the product vector combined with a phrase search for \"blue jeans\"\n",
|
|
"results = search_redis(redis_client,\n",
|
|
" \"man blue jeans\",\n",
|
|
" vector_field=\"product_vector\",\n",
|
|
" k=10,\n",
|
|
" hybrid_fields='@productDisplayName:\"blue jeans\"'\n",
|
|
" )\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 16,
|
|
"id": "8a56633b",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)\n",
|
|
"1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)\n",
|
|
"2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n",
|
|
"3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n",
|
|
"4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)\n",
|
|
"5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)\n",
|
|
"6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)\n",
|
|
"7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# hybrid query for shirt in the product vector and only include results with the phrase \"slim fit\" in the title\n",
|
|
"results = search_redis(redis_client,\n",
|
|
" \"shirt\",\n",
|
|
" vector_field=\"product_vector\",\n",
|
|
" k=10,\n",
|
|
" hybrid_fields='@productDisplayName:\"slim fit\"'\n",
|
|
" )\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 17,
|
|
"id": "6c25ee8d",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0. Titan Women Gold Watch (Score: 0.544)\n",
|
|
"1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)\n",
|
|
"2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)\n",
|
|
"3. Titan Men Black Watch (Score: 0.543)\n",
|
|
"4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)\n",
|
|
"5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)\n",
|
|
"6. Titan Women Silver Watch (Score: 0.542)\n",
|
|
"7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)\n",
|
|
"8. Titan Raga Women Gold Watch (Score: 0.539)\n",
|
|
"9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# hybrid query for watch in the product vector and only include results with the tag \"Accessories\" in the masterCategory field\n",
|
|
"results = search_redis(redis_client,\n",
|
|
" \"watch\",\n",
|
|
" vector_field=\"product_vector\",\n",
|
|
" k=10,\n",
|
|
" hybrid_fields='@masterCategory:{Accessories}'\n",
|
|
" )\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 18,
|
|
"id": "2c0d11d8",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0. Enroute Teens Orange Sandals (Score: 0.701)\n",
|
|
"1. Fila Men Camper Brown Sandals (Score: 0.692)\n",
|
|
"2. Clarks Men Black Leather Closed Sandals (Score: 0.691)\n",
|
|
"3. Coolers Men Black Sandals (Score: 0.69)\n",
|
|
"4. Coolers Men Black Sandals (Score: 0.69)\n",
|
|
"5. Enroute Teens Brown Sandals (Score: 0.69)\n",
|
|
"6. Crocs Dora Boots Pink Sandals (Score: 0.69)\n",
|
|
"7. Enroute Men Leather Black Sandals (Score: 0.685)\n",
|
|
"8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)\n",
|
|
"9. Coolers Men Black Sports Sandals (Score: 0.684)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range\n",
|
|
"results = search_redis(redis_client,\n",
|
|
" \"sandals\",\n",
|
|
" vector_field=\"product_vector\",\n",
|
|
" k=10,\n",
|
|
" hybrid_fields='@year:[2011 2012]'\n",
|
|
" )\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 19,
|
|
"id": "7caad384",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0. ADIDAS Men Navy Blue Benton Sandals (Score: 0.691)\n",
|
|
"1. Enroute Teens Brown Sandals (Score: 0.681)\n",
|
|
"2. ADIDAS Women's Adi Groove Blue Flip Flop (Score: 0.672)\n",
|
|
"3. Enroute Women Turquoise Blue Flats (Score: 0.671)\n",
|
|
"4. Red Tape Men Black Sandals (Score: 0.67)\n",
|
|
"5. Enroute Teens Orange Sandals (Score: 0.661)\n",
|
|
"6. Vans Men Blue Era Scilla Plaid Shoes (Score: 0.658)\n",
|
|
"7. FILA Men Aruba Navy Blue Sandal (Score: 0.657)\n",
|
|
"8. Quiksilver Men Blue Flip Flops (Score: 0.656)\n",
|
|
"9. Reebok Men Navy Twist Sandals (Score: 0.656)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range from the summer season\n",
|
|
"results = search_redis(redis_client,\n",
|
|
" \"blue sandals\",\n",
|
|
" vector_field=\"product_vector\",\n",
|
|
" k=10,\n",
|
|
" hybrid_fields='(@year:[2011 2012] @season:{Summer})'\n",
|
|
" )\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 20,
|
|
"id": "f1232d3c",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"0. Wrangler Men Leather Brown Belt (Score: 0.67)\n",
|
|
"1. Wrangler Women Black Belt (Score: 0.639)\n",
|
|
"2. Wrangler Men Green Striped Shirt (Score: 0.575)\n",
|
|
"3. Wrangler Men Purple Striped Shirt (Score: 0.549)\n",
|
|
"4. Wrangler Men Griffith White Shirt (Score: 0.543)\n",
|
|
"5. Wrangler Women Stella Green Shirt (Score: 0.542)\n"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"# hybrid query for a brown belt filtering results by a year (NUMERIC) with a specific article types (TAG) and with a brand name (TEXT)\n",
|
|
"results = search_redis(redis_client,\n",
|
|
" \"brown belt\",\n",
|
|
" vector_field=\"product_vector\",\n",
|
|
" k=10,\n",
|
|
" hybrid_fields='(@year:[2012 2012] @articleType:{Shirts | Belts} @productDisplayName:\"Wrangler\")'\n",
|
|
" )\n"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.6"
|
|
},
|
|
"vscode": {
|
|
"interpreter": {
|
|
"hash": "9b1e6e9c2967143209c2f955cb869d1d3234f92dc4787f49f155f3abbdfb1316"
|
|
}
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|