openai-cookbook/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "cb1537e6",
   "metadata": {},
   "source": [
    "# Using Redis for Embeddings Search\n",
    "\n",
    "This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n",
    "\n",
    "### What is a Vector Database\n",
    "\n",
    "A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n",
    "\n",
    "### Why use a Vector Database\n",
    "\n",
    "Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n",
    "\n",
    "\n",
    "### Demo Flow\n",
    "The demo flow is:\n",
    "- **Setup**: Import packages and set any required variables\n",
    "- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
    "- **Redis**\n",
    "    - *Setup*: Set up the Redis-Py client. For more details go [here](https://github.com/redis/redis-py)\n",
    "    - *Index Data*: Create the search index for vector search and hybrid search (vector + full-text search) on all available fields.\n",
    "    - *Search Data*: Run a few example queries with various goals in mind.\n",
    "\n",
    "Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2b59250",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Import the required libraries and set the embedding model that we'd like to use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8d8810f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We'll need to install the Redis client\n",
    "!pip install redis\n",
    "\n",
    "#Install wget to pull zip file\n",
    "!pip install wget"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "5be94df6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "\n",
    "from typing import List, Iterator\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import wget\n",
    "from ast import literal_eval\n",
    "\n",
    "# Redis client library for Python\n",
    "import redis\n",
    "\n",
    "# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\n",
    "EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
    "\n",
    "# Ignore unclosed SSL socket warnings - optional in case you get these errors\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\n",
    "warnings.filterwarnings(\"ignore\", category=DeprecationWarning) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5d9d2e1",
   "metadata": {},
   "source": [
    "## Load data\n",
    "\n",
    "In this section we'll load embedded data that we've prepared previous to this session."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dff8b55",
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n",
    "\n",
    "# The file is ~700 MB so this will take some time\n",
    "wget.download(embeddings_url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21097972",
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n",
    "    zip_ref.extractall(\"../data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "70bbd8ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "1721e45d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>url</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>title_vector</th>\n",
       "      <th>content_vector</th>\n",
       "      <th>vector_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/April</td>\n",
       "      <td>April</td>\n",
       "      <td>April is the fourth month of the year in the J...</td>\n",
       "      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n",
       "      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/August</td>\n",
       "      <td>August</td>\n",
       "      <td>August (Aug.) is the eighth month of the year ...</td>\n",
       "      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n",
       "      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Art</td>\n",
       "      <td>Art</td>\n",
       "      <td>Art is a creative activity that expresses imag...</td>\n",
       "      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n",
       "      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/A</td>\n",
       "      <td>A</td>\n",
       "      <td>A or a is the first letter of the English alph...</td>\n",
       "      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n",
       "      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>9</td>\n",
       "      <td>https://simple.wikipedia.org/wiki/Air</td>\n",
       "      <td>Air</td>\n",
       "      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n",
       "      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n",
       "      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                       url   title  \\\n",
       "0   1   https://simple.wikipedia.org/wiki/April   April   \n",
       "1   2  https://simple.wikipedia.org/wiki/August  August   \n",
       "2   6     https://simple.wikipedia.org/wiki/Art     Art   \n",
       "3   8       https://simple.wikipedia.org/wiki/A       A   \n",
       "4   9     https://simple.wikipedia.org/wiki/Air     Air   \n",
       "\n",
       "                                                text  \\\n",
       "0  April is the fourth month of the year in the J...   \n",
       "1  August (Aug.) is the eighth month of the year ...   \n",
       "2  Art is a creative activity that expresses imag...   \n",
       "3  A or a is the first letter of the English alph...   \n",
       "4  Air refers to the Earth's atmosphere. Air is a...   \n",
       "\n",
       "                                        title_vector  \\\n",
       "0  [0.001009464613161981, -0.020700545981526375, ...   \n",
       "1  [0.0009286514250561595, 0.000820168002974242, ...   \n",
       "2  [0.003393713850528002, 0.0061537534929811954, ...   \n",
       "3  [0.0153952119871974, -0.013759135268628597, 0....   \n",
       "4  [0.02224554680287838, -0.02044147066771984, -0...   \n",
       "\n",
       "                                      content_vector  vector_id  \n",
       "0  [-0.011253940872848034, -0.013491976074874401,...          0  \n",
       "1  [0.0003609954728744924, 0.007262262050062418, ...          1  \n",
       "2  [-0.004959689453244209, 0.015772193670272827, ...          2  \n",
       "3  [0.024894846603274345, -0.022186409682035446, ...          3  \n",
       "4  [0.021524671465158463, 0.018522677943110466, -...          4  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "article_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "960b82af",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read vectors from strings back into a list\n",
    "article_df['title_vector'] = article_df.title_vector.apply(literal_eval)\n",
    "article_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n",
    "\n",
    "# Set vector_id to be a string\n",
    "article_df['vector_id'] = article_df['vector_id'].apply(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a334ab8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 25000 entries, 0 to 24999\n",
      "Data columns (total 7 columns):\n",
      " #   Column          Non-Null Count  Dtype \n",
      "---  ------          --------------  ----- \n",
      " 0   id              25000 non-null  int64 \n",
      " 1   url             25000 non-null  object\n",
      " 2   title           25000 non-null  object\n",
      " 3   text            25000 non-null  object\n",
      " 4   title_vector    25000 non-null  object\n",
      " 5   content_vector  25000 non-null  object\n",
      " 6   vector_id       25000 non-null  object\n",
      "dtypes: int64(1), object(6)\n",
      "memory usage: 1.3+ MB\n"
     ]
    }
   ],
   "source": [
    "article_df.info(show_counts=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43bffd04",
   "metadata": {},
   "source": [
    "# Redis\n",
    "\n",
    "The next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.\n",
    "\n",
    "Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n",
    "\n",
    "| Project | Language | License | Author | Stars |\n",
    "|----------|---------|--------|---------|-------|\n",
    "| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n",
    "| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n",
    "| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n",
    "| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n",
    "| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |\n",
    "| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [![redisearch-api-rs-stars]][redisearch-api-rs-url] |\n",
    "\n",
    "[redis-url]: https://redis.com\n",
    "\n",
    "[redis-py-url]: https://github.com/redis/redis-py\n",
    "[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
    "[redis-py-package]: https://pypi.python.org/pypi/redis\n",
    "\n",
    "[jedis-url]: https://github.com/redis/jedis\n",
    "[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
    "[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis\n",
    "\n",
    "[nredisstack-url]: https://github.com/redis/nredisstack\n",
    "[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
    "[nredisstack-package]: https://www.nuget.org/packages/nredisstack/\n",
    "\n",
    "[node-redis-url]: https://github.com/redis/node-redis\n",
    "[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
    "[node-redis-package]: https://www.npmjs.com/package/redis\n",
    "\n",
    "[redis-om-python-url]: https://github.com/redis/redis-om-python\n",
    "[redis-om-python-author]: https://redis.com\n",
    "[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
    "\n",
    "[redisearch-go-url]: https://github.com/RediSearch/redisearch-go\n",
    "[redisearch-go-author]: https://redis.com\n",
    "[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
    "\n",
    "[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs\n",
    "[redisearch-api-rs-author]: https://redis.com\n",
    "[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000\n",
    "\n",
    "\n",
    "In the below cells, we will walk you through using Redis as a vector database. Since many of you are likely already used to the Redis API, this should be familiar to most."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "698e24f6",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "There are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment. For other deployment options, see the [redis directory](./redis) in this repo.\n",
    "\n",
    "For this tutorial, we will use Redis Stack on Docker.\n",
    "\n",
    "Start a version of Redis with RediSearch (Redis Stack) by running the following docker command\n",
    "\n",
    "```bash\n",
    "$ cd redis\n",
    "$ docker compose up -d\n",
    "```\n",
    "This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n",
    "\n",
    "You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d2ce669a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import redis\n",
    "from redis.commands.search.indexDefinition import (\n",
    "    IndexDefinition,\n",
    "    IndexType\n",
    ")\n",
    "from redis.commands.search.query import Query\n",
    "from redis.commands.search.field import (\n",
    "    TextField,\n",
    "    VectorField\n",
    ")\n",
    "\n",
    "REDIS_HOST =  \"localhost\"\n",
    "REDIS_PORT = 6379\n",
    "REDIS_PASSWORD = \"\" # default for passwordless Redis\n",
    "\n",
    "# Connect to Redis\n",
    "redis_client = redis.Redis(\n",
    "    host=REDIS_HOST,\n",
    "    port=REDIS_PORT,\n",
    "    password=REDIS_PASSWORD\n",
    ")\n",
    "redis_client.ping()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f6f0af9",
   "metadata": {},
   "source": [
    "## Creating a Search Index\n",
    "\n",
    "The below cells will show how to specify and create a search index in Redis. We will\n",
    "\n",
    "1. Set some constants for defining our index like the distance metric and the index name\n",
    "2. Define the index schema with RediSearch fields\n",
    "3. Create the index\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a7c64cb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "VECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors\n",
    "VECTOR_NUMBER = len(article_df)                 # initial number of vectors\n",
    "INDEX_NAME = \"embeddings-index\"                 # name of the search index\n",
    "PREFIX = \"doc\"                                  # prefix for the document keys\n",
    "DISTANCE_METRIC = \"COSINE\"                      # distance metric for the vectors (ex. COSINE, IP, L2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d95fcd06",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define RediSearch fields for each of the columns in the dataset\n",
    "title = TextField(name=\"title\")\n",
    "url = TextField(name=\"url\")\n",
    "text = TextField(name=\"text\")\n",
    "title_embedding = VectorField(\"title_vector\",\n",
    "    \"FLAT\", {\n",
    "        \"TYPE\": \"FLOAT32\",\n",
    "        \"DIM\": VECTOR_DIM,\n",
    "        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
    "        \"INITIAL_CAP\": VECTOR_NUMBER,\n",
    "    }\n",
    ")\n",
    "text_embedding = VectorField(\"content_vector\",\n",
    "    \"FLAT\", {\n",
    "        \"TYPE\": \"FLOAT32\",\n",
    "        \"DIM\": VECTOR_DIM,\n",
    "        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n",
    "        \"INITIAL_CAP\": VECTOR_NUMBER,\n",
    "    }\n",
    ")\n",
    "fields = [title, url, text, title_embedding, text_embedding]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "7418480d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check if index exists\n",
    "try:\n",
    "    redis_client.ft(INDEX_NAME).info()\n",
    "    print(\"Index already exists\")\n",
    "except:\n",
    "    # Create RediSearch Index\n",
    "    redis_client.ft(INDEX_NAME).create_index(\n",
    "        fields = fields,\n",
    "        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3563eec",
   "metadata": {},
   "source": [
    "## Load Documents into the Index\n",
    "\n",
    "Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "e98d63ad",
   "metadata": {},
   "outputs": [],
   "source": [
    "def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n",
    "    records = documents.to_dict(\"records\")\n",
    "    for doc in records:\n",
    "        key = f\"{prefix}:{str(doc['id'])}\"\n",
    "\n",
    "        # create byte vectors for title and content\n",
    "        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n",
    "        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n",
    "\n",
    "        # replace list of floats with byte vectors\n",
    "        doc[\"title_vector\"] = title_embedding\n",
    "        doc[\"content_vector\"] = content_embedding\n",
    "\n",
    "        client.hset(key, mapping = doc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "098d3c5a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 25000 documents in Redis search index with name: embeddings-index\n"
     ]
    }
   ],
   "source": [
    "index_documents(redis_client, PREFIX, article_df)\n",
    "print(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f646bff4",
   "metadata": {},
   "source": [
    "## Running Search Queries\n",
    "\n",
    "Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.\n",
    "\n",
    "1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.\n",
    "2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "508d1f89",
   "metadata": {},
   "outputs": [],
   "source": [
    "def search_redis(\n",
    "    redis_client: redis.Redis,\n",
    "    user_query: str,\n",
    "    index_name: str = \"embeddings-index\",\n",
    "    vector_field: str = \"title_vector\",\n",
    "    return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n",
    "    hybrid_fields = \"*\",\n",
    "    k: int = 20,\n",
    ") -> List[dict]:\n",
    "\n",
    "    # Creates embedding vector from user query\n",
    "    embedded_query = openai.Embedding.create(input=user_query,\n",
    "                                            model=EMBEDDING_MODEL,\n",
    "                                            )[\"data\"][0]['embedding']\n",
    "\n",
    "    # Prepare the Query\n",
    "    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n",
    "    query = (\n",
    "        Query(base_query)\n",
    "         .return_fields(*return_fields)\n",
    "         .sort_by(\"vector_score\")\n",
    "         .paging(0, k)\n",
    "         .dialect(2)\n",
    "    )\n",
    "    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n",
    "\n",
    "    # perform vector search\n",
    "    results = redis_client.ft(index_name).search(query, params_dict)\n",
    "    for i, article in enumerate(results.docs):\n",
    "        score = 1 - float(article.vector_score)\n",
    "        print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n",
    "    return results.docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "1f0eef07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0. Museum of Modern Art (Score: 0.875)\n",
      "1. Western Europe (Score: 0.867)\n",
      "2. Renaissance art (Score: 0.864)\n",
      "3. Pop art (Score: 0.86)\n",
      "4. Northern Europe (Score: 0.855)\n",
      "5. Hellenistic art (Score: 0.853)\n",
      "6. Modernist literature (Score: 0.847)\n",
      "7. Art film (Score: 0.843)\n",
      "8. Central Europe (Score: 0.843)\n",
      "9. European (Score: 0.841)\n"
     ]
    }
   ],
   "source": [
    "# For using OpenAI to generate query embedding\n",
    "openai.api_key = os.getenv(\"OPENAI_API_KEY\", \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\n",
    "results = search_redis(redis_client, 'modern art in Europe', k=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "7b805a81",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0. Battle of Bannockburn (Score: 0.869)\n",
      "1. Wars of Scottish Independence (Score: 0.861)\n",
      "2. 1651 (Score: 0.853)\n",
      "3. First War of Scottish Independence (Score: 0.85)\n",
      "4. Robert I of Scotland (Score: 0.846)\n",
      "5. 841 (Score: 0.844)\n",
      "6. 1716 (Score: 0.844)\n",
      "7. 1314 (Score: 0.837)\n",
      "8. 1263 (Score: 0.836)\n",
      "9. William Wallace (Score: 0.835)\n"
     ]
    }
   ],
   "source": [
    "results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ed0b34e",
   "metadata": {},
   "source": [
    "## Hybrid Queries with Redis\n",
    "\n",
    "The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "c94d5cce",
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_hybrid_field(field_name: str, value: str) -> str:\n",
    "    return f'@{field_name}:\"{value}\"'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "bfcd31c2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0. First War of Scottish Independence (Score: 0.892)\n",
      "1. Wars of Scottish Independence (Score: 0.889)\n",
      "2. Second War of Scottish Independence (Score: 0.879)\n",
      "3. List of Scottish monarchs (Score: 0.873)\n",
      "4. Scottish Borders (Score: 0.863)\n"
     ]
    }
   ],
   "source": [
    "# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\n",
    "results = search_redis(redis_client,\n",
    "                       \"Famous battles in Scottish history\",\n",
    "                       vector_field=\"title_vector\",\n",
    "                       k=5,\n",
    "                       hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n",
    "                       )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "28ab1e30",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0. Art (Score: 1.0)\n",
      "1. Paint (Score: 0.896)\n",
      "2. Renaissance art (Score: 0.88)\n",
      "3. Painting (Score: 0.874)\n",
      "4. Renaissance (Score: 0.846)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\n",
    "results = search_redis(redis_client,\n",
    "                       \"Art\",\n",
    "                       vector_field=\"title_vector\",\n",
    "                       k=5,\n",
    "                       hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n",
    "                       )\n",
    "\n",
    "# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\n",
    "mention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\n",
    "mention"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f94b5be2",
   "metadata": {},
   "source": [
    "For more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0119d87a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "vector_db_split",
   "language": "python",
   "name": "vector_db_split"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  },
  "vscode": {
   "interpreter": {
    "hash": "fd16a328ca3d68029457069b79cb0b38eb39a0f5ccc4fe4473d3047707df8207"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}