2023-03-15 01:06:03 +00:00
{
"cells": [
{
"cell_type": "markdown",
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-03-15 01:06:03 +00:00
"source": [
"# Redis\n",
"\n",
2023-04-29 02:26:50 +00:00
">[Redis (Remote Dictionary Server)](https://en.wikipedia.org/wiki/Redis) is an in-memory data structure store, used as a distributed, in-memory key– value database, cache and message broker, with optional durability.\n",
"\n",
2023-07-17 14:18:51 +00:00
"This notebook shows how to use functionality related to the [Redis vector database](https://redis.com/solutions/use-cases/vector-database/).\n",
"\n",
"As database either Redis standalone server or Redis Sentinel HA setups are supported for connections with the \"redis_url\"\n",
"parameter. More information about the different formats of the redis connection url can be found in the LangChain\n",
2023-07-17 14:53:11 +00:00
"[Redis Readme](/docs/modules/data_connection/vectorstores/integrations/redis) file"
2023-03-27 02:49:46 +00:00
]
2023-03-15 01:06:03 +00:00
},
2023-05-18 22:35:47 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installing"
]
},
2023-03-15 01:06:03 +00:00
{
"cell_type": "code",
2023-04-29 02:26:50 +00:00
"execution_count": null,
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"!pip install redis"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We want to use `OpenAIEmbeddings` so we have to get the OpenAI API Key."
]
},
{
"cell_type": "code",
"execution_count": null,
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-03-15 01:06:03 +00:00
"outputs": [],
2023-04-29 02:26:50 +00:00
"source": [
"import os\n",
"import getpass\n",
"\n",
2023-06-16 18:52:56 +00:00
"os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")"
2023-04-29 02:26:50 +00:00
]
},
2023-05-18 22:35:47 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Example"
]
},
2023-04-29 02:26:50 +00:00
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"tags": []
},
"outputs": [],
2023-03-15 01:06:03 +00:00
"source": [
2023-04-07 06:21:22 +00:00
"from langchain.embeddings import OpenAIEmbeddings\n",
2023-03-15 01:06:03 +00:00
"from langchain.text_splitter import CharacterTextSplitter\n",
"from langchain.vectorstores.redis import Redis"
2023-03-27 02:49:46 +00:00
]
2023-03-15 01:06:03 +00:00
},
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-03-15 01:06:03 +00:00
"outputs": [],
"source": [
"from langchain.document_loaders import TextLoader\n",
2023-04-07 06:21:22 +00:00
"\n",
2023-06-16 18:52:56 +00:00
"loader = TextLoader(\"../../../state_of_the_union.txt\")\n",
2023-03-15 01:06:03 +00:00
"documents = loader.load()\n",
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
"docs = text_splitter.split_documents(documents)\n",
"\n",
"embeddings = OpenAIEmbeddings()"
2023-03-27 02:49:46 +00:00
]
2023-03-15 01:06:03 +00:00
},
2023-06-22 20:26:47 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you're not interested in the keys of your entries you can also create your redis instance from the documents."
]
},
2023-03-15 01:06:03 +00:00
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-03-15 01:06:03 +00:00
"outputs": [],
"source": [
2023-06-16 18:52:56 +00:00
"rds = Redis.from_documents(\n",
" docs, embeddings, redis_url=\"redis://localhost:6379\", index_name=\"link\"\n",
")"
2023-03-27 02:49:46 +00:00
]
2023-03-15 01:06:03 +00:00
},
2023-06-22 20:26:47 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you're interested in the keys of your entries you have to split your docs in texts and metadatas"
]
},
2023-03-15 01:06:03 +00:00
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-06-22 20:26:47 +00:00
"outputs": [],
"source": [
"texts = [d.page_content for d in docs]\n",
"metadatas = [d.metadata for d in docs]\n",
"\n",
Fix `make docs_build` and related scripts (#7276)
**Description: a description of the change**
Fixed `make docs_build` and related scripts which caused errors. There
are several changes.
First, I made the build of the documentation and the API Reference into
two separate commands. This is because it takes less time to build. The
commands for documents are `make docs_build`, `make docs_clean`, and
`make docs_linkcheck`. The commands for API Reference are `make
api_docs_build`, `api_docs_clean`, and `api_docs_linkcheck`.
It looked like `docs/.local_build.sh` could be used to build the
documentation, so I used that. Since `.local_build.sh` was also building
API Rerefence internally, I removed that process. `.local_build.sh` also
added some Bash options to stop in error or so. Futher more added `cd
"${SCRIPT_DIR}"` at the beginning so that the script will work no matter
which directory it is executed in.
`docs/api_reference/api_reference.rst` is removed, because which is
generated by `docs/api_reference/create_api_rst.py`, and added it to
.gitignore.
Finally, the description of CONTRIBUTING.md was modified.
**Issue: the issue # it fixes (if applicable)**
https://github.com/hwchase17/langchain/issues/6413
**Dependencies: any dependencies required for this change**
`nbdoc` was missing in group docs so it was added. I installed it with
the `poetry add --group docs nbdoc` command. I am concerned if any
modifications are needed to poetry.lock. I would greatly appreciate it
if you could pay close attention to this file during the review.
**Tag maintainer**
- General / Misc / if you don't know who to tag: @baskaryan
If this PR needs any additional changes, I'll be happy to make them!
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-12 02:05:14 +00:00
"rds, keys = Redis.from_texts_return_keys(\n",
" texts, embeddings, redis_url=\"redis://localhost:6379\", index_name=\"link\"\n",
")"
2023-06-22 20:26:47 +00:00
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
2023-03-15 01:06:03 +00:00
"source": [
"rds.index_name"
2023-03-27 02:49:46 +00:00
]
2023-03-23 02:57:56 +00:00
},
2023-03-15 01:06:03 +00:00
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-06-22 20:26:47 +00:00
"outputs": [],
2023-03-15 01:06:03 +00:00
"source": [
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"results = rds.similarity_search(query)\n",
"print(results[0].page_content)"
2023-03-27 02:49:46 +00:00
]
2023-03-15 01:06:03 +00:00
},
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-06-22 20:26:47 +00:00
"outputs": [],
2023-03-15 01:06:03 +00:00
"source": [
"print(rds.add_texts([\"Ankush went to Princeton\"]))"
2023-03-27 02:49:46 +00:00
]
2023-03-15 01:06:03 +00:00
},
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-06-22 20:26:47 +00:00
"outputs": [],
2023-03-15 01:06:03 +00:00
"source": [
"query = \"Princeton\"\n",
"results = rds.similarity_search(query)\n",
"print(results[0].page_content)"
2023-03-27 02:49:46 +00:00
]
2023-03-15 01:06:03 +00:00
},
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-27 02:49:46 +00:00
"metadata": {},
2023-06-22 20:26:47 +00:00
"outputs": [],
2023-03-23 02:57:56 +00:00
"source": [
2023-04-07 06:21:22 +00:00
"# Load from existing index\n",
2023-06-16 18:52:56 +00:00
"rds = Redis.from_existing_index(\n",
" embeddings, redis_url=\"redis://localhost:6379\", index_name=\"link\"\n",
")\n",
2023-03-23 02:57:56 +00:00
"\n",
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"results = rds.similarity_search(query)\n",
"print(results[0].page_content)"
2023-03-27 02:49:46 +00:00
]
2023-03-28 02:51:23 +00:00
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2023-05-18 22:35:47 +00:00
"## Redis as Retriever\n",
2023-03-28 02:51:23 +00:00
"\n",
"Here we go over different options for using the vector store as a retriever.\n",
"\n",
"There are three different search methods we can use to do retrieval. By default, it will use semantic similarity."
]
},
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-28 02:51:23 +00:00
"metadata": {},
"outputs": [],
"source": [
"retriever = rds.as_retriever()"
]
},
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-28 02:51:23 +00:00
"metadata": {},
"outputs": [],
"source": [
"docs = retriever.get_relevant_documents(query)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also use similarity_limit as a search method. This is only return documents if they are similar enough"
]
},
{
"cell_type": "code",
2023-06-22 20:26:47 +00:00
"execution_count": null,
2023-03-28 02:51:23 +00:00
"metadata": {},
"outputs": [],
"source": [
"retriever = rds.as_retriever(search_type=\"similarity_limit\")"
]
},
{
"cell_type": "code",
2023-04-07 06:21:22 +00:00
"execution_count": null,
2023-03-28 02:51:23 +00:00
"metadata": {},
2023-04-07 06:21:22 +00:00
"outputs": [],
2023-03-28 02:51:23 +00:00
"source": [
"# Here we can see it doesn't return any results because there are no relevant documents\n",
"retriever.get_relevant_documents(\"where did ankush go to college?\")"
]
2023-06-22 20:26:47 +00:00
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Delete keys"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To delete your entries you have to address them by their keys."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"Redis.delete(keys, redis_url=\"redis://localhost:6379\")"
]
2023-07-17 14:18:51 +00:00
},
{
"cell_type": "markdown",
2023-07-17 14:53:11 +00:00
"metadata": {},
2023-07-17 14:18:51 +00:00
"source": [
"### Redis connection Url examples\n",
"\n",
"Valid Redis Url scheme are:\n",
"1. `redis://` - Connection to Redis standalone, unencrypted\n",
"2. `rediss://` - Connection to Redis standalone, with TLS encryption\n",
"3. `redis+sentinel://` - Connection to Redis server via Redis Sentinel, unencrypted\n",
"4. `rediss+sentinel://` - Connection to Redis server via Redis Sentinel, booth connections with TLS encryption\n",
"\n",
"More information about additional connection parameter can be found in the redis-py documentation at https://redis-py.readthedocs.io/en/stable/connections.html"
2023-07-17 14:53:11 +00:00
]
2023-07-17 14:18:51 +00:00
},
{
"cell_type": "code",
"execution_count": null,
2023-07-17 14:53:11 +00:00
"metadata": {},
2023-07-17 14:18:51 +00:00
"outputs": [],
"source": [
"# connection to redis standalone at localhost, db 0, no password\n",
2023-07-17 16:39:11 +00:00
"redis_url = \"redis://localhost:6379\"\n",
2023-07-17 14:18:51 +00:00
"# connection to host \"redis\" port 7379 with db 2 and password \"secret\" (old style authentication scheme without username / pre 6.x)\n",
2023-07-17 16:39:11 +00:00
"redis_url = \"redis://:secret@redis:7379/2\"\n",
2023-07-17 14:18:51 +00:00
"# connection to host redis on default port with user \"joe\", pass \"secret\" using redis version 6+ ACLs\n",
2023-07-17 16:39:11 +00:00
"redis_url = \"redis://joe:secret@redis/0\"\n",
2023-07-17 14:18:51 +00:00
"\n",
"# connection to sentinel at localhost with default group mymaster and db 0, no password\n",
2023-07-17 16:39:11 +00:00
"redis_url = \"redis+sentinel://localhost:26379\"\n",
2023-07-17 14:18:51 +00:00
"# connection to sentinel at host redis with default port 26379 and user \"joe\" with password \"secret\" with default group mymaster and db 0\n",
2023-07-17 16:39:11 +00:00
"redis_url = \"redis+sentinel://joe:secret@redis\"\n",
2023-07-17 14:18:51 +00:00
"# connection to sentinel, no auth with sentinel monitoring group \"zone-1\" and database 2\n",
2023-07-17 16:39:11 +00:00
"redis_url = \"redis+sentinel://redis:26379/zone-1/2\"\n",
2023-07-17 14:18:51 +00:00
"\n",
"# connection to redis standalone at localhost, db 0, no password but with TLS support\n",
2023-07-17 16:39:11 +00:00
"redis_url = \"rediss://localhost:6379\"\n",
2023-07-17 14:18:51 +00:00
"# connection to redis sentinel at localhost and default port, db 0, no password\n",
"# but with TLS support for booth Sentinel and Redis server\n",
2023-07-17 16:39:11 +00:00
"redis_url = \"rediss+sentinel://localhost\""
2023-07-17 14:53:11 +00:00
]
2023-03-15 01:06:03 +00:00
}
],
"metadata": {
"kernelspec": {
2023-03-27 02:49:46 +00:00
"display_name": "Python 3 (ipykernel)",
2023-03-15 01:06:03 +00:00
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
2023-03-27 02:49:46 +00:00
"version": 3
2023-03-15 01:06:03 +00:00
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
2023-03-27 02:49:46 +00:00
"pygments_lexer": "ipython3",
2023-07-17 14:53:11 +00:00
"version": "3.11.3"
2023-03-15 01:06:03 +00:00
}
},
"nbformat": 4,
2023-04-29 02:26:50 +00:00
"nbformat_minor": 4
2023-03-23 02:57:56 +00:00
}