"# Running Hybrid VSS Queries with Redis and OpenAI\n",
"# Running Hybrid VSS Queries with Redis and OpenAI\n",
"\n",
"\n",
"This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings and running hybrid queries that combine VSS and lexical search using Redis Search and Query capability. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The RediSearch module allows you to index and search for vectors in Redis. This notebook will show you how to use the RediSearch module to index and search for vectors created by using the OpenAI API and stored in Redis.\n",
"This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings and running hybrid queries that combine VSS and lexical search using Redis Query and Search capability. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The Redis Query and Search capability allows you to index and search for vectors in Redis. This notebook will show you how to use the Redis Query and Search to index and search for vectors created by using the OpenAI API and stored in Redis.\n",
"\n",
"\n",
"### What is Redis?\n",
"Hybrid queries combine vector similarity with traditional Redis Query and Search filtering capabilities on GEO, NUMERIC, TAG or TEXT data simplifying application code. A common example of a hybrid query in an e-commerce use case if to find items visually similar to a given query image limited to items available in a GEO location and within a price range."
"\n",
"Most developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.\n",
"\n",
"In addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new data types and commands. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).\n",
"\n",
"### What is RediSearch?\n",
"\n",
"RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [README](./README.md) or the [RediSearch documentation](https://redis.io/docs/stack/search/).\n",
"\n",
"### Deployment options\n",
"\n",
"There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.\n",
"\n",
"For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.\n",
"\n",
"Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.\n",
"# For using OpenAI to generate query embedding\n",
"# Execute a simple vector search in Redis\n",
"results = search_redis(redis_client, 'man blue jeans', k=10)"
"results = search_redis(redis_client, 'man blue jeans', k=10)"
]
]
},
},
{
"cell_type": "markdown",
"id": "2007be48",
"metadata": {},
"source": [
"## Hybrid Queries with Redis\n",
"\n",
"The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search."
]
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": 27,
"execution_count": 14,
"id": "93c4a696",
"id": "0c4f4d0f",
"metadata": {},
"metadata": {},
"outputs": [
"outputs": [
{
{
"name": "stdout",
"name": "stdout",
"output_type": "stream",
"output_type": "stream",
"text": [
"text": [
"0. John Players Men Blue Jeans (Score: 0.739)\n",
"0. John Players Men Blue Jeans (Score: 0.791)\n",
"1. Lee Men Tino Blue Jeans (Score: 0.72)\n",
"1. Lee Men Tino Blue Jeans (Score: 0.775)\n",
"2. Peter England Men Party Blue Jeans (Score: 0.718)\n",
"2. Peter England Men Party Blue Jeans (Score: 0.763)\n",
"3. Denizen Women Blue Jeans (Score: 0.715)\n",
"3. French Connection Men Blue Jeans (Score: 0.74)\n",
"4. Jealous 21 Women Washed Blue Jeans (Score: 0.708)\n",
"4. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
"5. Jealous 21 Women Washed Blue Jeans (Score: 0.708)\n",
"5. Locomotive Men Washed Blue Jeans (Score: 0.739)\n",
"6. Levis Kids Blue Solid Jean (Score: 0.706)\n",
"6. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n",
"7. French Connection Men Blue Jeans (Score: 0.705)\n",
"7. Denizen Women Blue Jeans (Score: 0.725)\n",
"8. Lee Men Blue Chicago Fit Jeans (Score: 0.705)\n",
"8. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n",
"9. Lee Men Blue Chicago Fit Jeans (Score: 0.705)\n"
"9. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n"
"The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search."
]
]
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": 28,
"execution_count": 15,
"id": "8a56633b",
"id": "8a56633b",
"metadata": {},
"metadata": {},
"outputs": [
"outputs": [
@ -735,7 +714,7 @@
"3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n",
"3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n",
"4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)\n",
"4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)\n",
"5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)\n",
"5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)\n",
"6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.555)\n",
"6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.559)\n",
"7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.553)\n"
"7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.553)\n"
]
]
}
}
@ -752,7 +731,7 @@
},
},
{
{
"cell_type": "code",
"cell_type": "code",
"execution_count": 29,
"execution_count": 16,
"id": "6c25ee8d",
"id": "6c25ee8d",
"metadata": {},
"metadata": {},
"outputs": [
"outputs": [
@ -765,11 +744,11 @@
"2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)\n",
"2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)\n",
"3. Titan Men Black Watch (Score: 0.543)\n",
"3. Titan Men Black Watch (Score: 0.543)\n",
"4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)\n",
"4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)\n",
"5. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.542)\n",
"5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)\n",
"6. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)\n",
"6. Titan Women Silver Watch (Score: 0.542)\n",
"7. Titan Women Silver Watch (Score: 0.542)\n",
"7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)\n",
"8. Titan Raga Women Gold Watch (Score: 0.539)\n",
"8. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)\n",
"9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)\n"