"- **Setup**: Import packages and set any required variables\n",
"- **Load data**: Load a dataset and embed it using OpenAI embeddings\n",
"- **Pinecone**\n",
" - *Setup*: Here we setup the Python client for Pinecone. For more details go [here](https://docs.pinecone.io/docs/quickstart)\n",
" - *Setup*: Here we'll setup the Python client for Pinecone. For more details go [here](https://docs.pinecone.io/docs/quickstart)\n",
" - *Index Data*: We'll create an index with namespaces for __titles__ and __content__\n",
" - *Search Data*: We'll test out both namespaces with search queries to confirm it works\n",
"- **Weaviate**\n",
" - *Setup*: Here we setup the Python client for Weaviate. For more details go [here](https://weaviate.io/developers/weaviate/current/client-libraries/python.html)\n",
" - *Setup*: Here we'll setup the Python client for Weaviate. For more details go [here](https://weaviate.io/developers/weaviate/current/client-libraries/python.html)\n",
" - *Index Data*: We'll create an index with __title__ search vectors in it\n",
" - *Search Data*: We'll run a few searches to confirm it works\n",
"- **Qdrant**\n",
" - *Setup*: Here we setup the Python client for Qdrant. For more details go [here](https://github.com/qdrant/qdrant_client)\n",
" - *Setup*: Here we'll setup the Python client for Qdrant. For more details go [here](https://github.com/qdrant/qdrant_client)\n",
" - *Index Data*: We'll create a collection with vectors for __titles__ and __content__\n",
" - *Search Data*: We'll run a few searches to confirm it works\n",
"\n",
@ -55,7 +55,7 @@
"metadata": {},
"outputs": [],
"source": [
"# Here we install the clients for all vector databases\n",
"# We'll need to install the clients for all vector databases\n",
"!pip install pinecone-client\n",
"!pip install weaviate-client\n",
"!pip install qdrant-client"
@ -63,7 +63,7 @@
},
{
"cell_type": "code",
"execution_count": 1,
"execution_count": 3,
"id": "5be94df6",
"metadata": {},
"outputs": [],
@ -111,7 +111,7 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 4,
"id": "bd99e08e",
"metadata": {},
"outputs": [],
@ -172,32 +172,10 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": null,
"id": "0c1c73cb",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Found cached dataset wikipedia (/Users/colin.jarvis/.cache/huggingface/datasets/wikipedia/20220301.simple/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559)\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6aa3a11d70424916915334e267f4964b",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"outputs": [],
"source": [
"# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\n",
"First we need to create an index, which we'll call `wikipedia-articles`. Once we have an index, we can create multiple namespaces, which can make a single index searchable for various use cases. For more details, consult [Pinecone documentation](https://docs.pinecone.io/docs/namespaces#:~:text=Pinecone%20allows%20you%20to%20partition,different%20subsets%20of%20your%20index.).\n",
"First we will need to create an index, which we'll call `wikipedia-articles`. Once we have an index, we can create multiple namespaces, which can make a single index searchable for various use cases. For more details, consult [Pinecone documentation](https://docs.pinecone.io/docs/namespaces#:~:text=Pinecone%20allows%20you%20to%20partition,different%20subsets%20of%20your%20index.).\n",
"\n",
"If you want to batch insert to your index in parallel to increase insertion speed then there is a great guide in the Pinecone documentation on [batch inserts in parallel](https://docs.pinecone.io/docs/insert-data#sending-upserts-in-parallel)."
]
},
{
"cell_type": "code",
"execution_count": 19,
"execution_count": 108,
"id": "0a71c575",
"metadata": {},
"outputs": [],
@ -483,7 +461,7 @@
},
{
"cell_type": "code",
"execution_count": 20,
"execution_count": 124,
"id": "7ea9ad46",
"metadata": {},
"outputs": [
@ -493,7 +471,7 @@
"['wikipedia-articles']"
]
},
"execution_count": 20,
"execution_count": 124,
"metadata": {},
"output_type": "execute_result"
}
@ -516,7 +494,7 @@
},
{
"cell_type": "code",
"execution_count": 22,
"execution_count": 126,
"id": "5daeba00",
"metadata": {},
"outputs": [
@ -529,7 +507,7 @@
}
],
"source": [
"# Upsert content vectors in content namespace\n",
"# Upsert content vectors in content namespace - this can take a few minutes\n",
"print(\"Uploading vectors to content namespace..\")\n",
"Most similar results to modern art in Europe in \"title\" namespace:\n",
"\n",
"Museum of Modern Art (score = 0.875286043)\n",
"Western Europe (score = 0.867383599)\n",
"Renaissance art (score = 0.864250064)\n",
"Pop art (score = 0.860506058)\n",
"Northern Europe (score = 0.854678154)\n",
"\n",
"\n"
]
}
],
"outputs": [],
"source": [
"query_output = query_article('modern art in Europe','title')"
]
},
{
"cell_type": "code",
"execution_count": 30,
"execution_count": null,
"id": "3e7ac79b",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Most similar results to Famous battles in Scottish history in \"content\" namespace:\n",
"\n",
"Battle of Bannockburn (score = 0.869324744)\n",
"Wars of Scottish Independence (score = 0.861479)\n",
"1651 (score = 0.852555931)\n",
"First War of Scottish Independence (score = 0.84969604)\n",
"Robert I of Scotland (score = 0.846192539)\n",
"\n",
"\n"
]
}
],
"outputs": [],
"source": [
"content_query_output = query_article(\"Famous battles in Scottish history\",'content')"
]
@ -728,7 +672,7 @@
"source": [
"### Setup\n",
"\n",
"To get Weaviate running locally we used Docker and followed the instructions contained the Weaviate documentation here: https://weaviate.io/developers/weaviate/current/installation/docker-compose.html\n",
"To get Weaviate running locally we will use Docker and follow the instructions contained in the Weaviate documentation here: https://weaviate.io/developers/weaviate/current/installation/docker-compose.html\n",
"\n",
"For an example docker-compose.yaml file please refer to `./weaviate/docker-compose.yaml` in this repo\n",
"\n",
@ -737,7 +681,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 113,
"id": "b9ea472d",
"metadata": {},
"outputs": [],
@ -747,10 +691,21 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 114,
"id": "13be220d",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"{'classes': []}"
]
},
"execution_count": 114,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.schema.delete_all()\n",
"client.schema.get()"
@ -758,10 +713,21 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 115,
"id": "73d33184",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"text/plain": [
"True"
]
},
"execution_count": 115,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"client.is_ready()"
]
@ -782,7 +748,7 @@
},
{
"cell_type": "code",
"execution_count": 36,
"execution_count": 116,
"id": "e868d143",
"metadata": {},
"outputs": [
@ -824,7 +790,7 @@
" 'vectorizer': 'none'}]}"
]
},
"execution_count": 36,
"execution_count": 116,
"metadata": {},
"output_type": "execute_result"
}
@ -854,7 +820,7 @@
},
{
"cell_type": "code",
"execution_count": 39,
"execution_count": 117,
"id": "786d437f",
"metadata": {},
"outputs": [
@ -895,7 +861,7 @@
},
{
"cell_type": "code",
"execution_count": 47,
"execution_count": 118,
"id": "3658693c",
"metadata": {},
"outputs": [
@ -903,26 +869,28 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Cave Story\n",
"is a freeware video game released in 2004 for PC. It was thought of and created over five years by Daisuke Amaya, known by his pseudonym, or art name, Pixel. The game is an action-adventure game, and is similar to the Castlevania and Metroid games. It was first made in Japanese, and was translated to English by the fan translating group, Aeon Genesis.\n",
"Kim Jong-nam\n",
"Kim Jong-nam (May 10, 1971 - February 13, 2017) was the eldest son of Kim Jong-il, the former leader of North Korea.\n",
"\n",
"He tried to enter Japan using a fake passport in May 2001. This was to visit Disneyland. This caused his father to not approve of him. Kim Jong-nam's younger half-brother Kim Jong-un was made the heir in September 2010.\n",
"\n",
"In June 2010, Kim Jong-nam gave a brief interview to the Associated Press in Macau. He told the reporter that he had \"no plans\" to defect to Europe. The press had recently said this. Kim Jong-nam lived in an apartment on the southern tip of Macau's Coloane Island until 2007. An anonymous South Korean official reported in October 2010 that Jong-nam had not lived in Macau for \"months\", and now goes between China and \"another country.\"\n",
"\n",
"When his father died, Kim Jong-nam did not attend the funeral. This was to avoid rumours on the succession.\n",
"\n",
"He was assassinated in Malaysia on February 13, 2017, which is believed to be ordered by his half-brother Kim Jong-un.\n",
"\n",
"Personal life\n",
"The South Korean newspaper The Chosun Ilbo said that Kim Jong-nam has two wives, at least one mistress, and several children. His first wife Shin Jong-hui (born c. 1980) and their son Kum-sol (born c. 1996) live at a home called Dragon Villa on the northern outskirts of Beijing. His second wife Lee Hye-kyong (born c. 1970), their son Han-sol (born c. 1995) and their daughter Sol-hui (born c. 1998) live in an apartment building in Macau. Jong-nam's mistress, former Air Koryo flight attendant So Yong-la (born c. 1980), also lives in Macau. \n",
"\n",
"References \n",
"Jong-nam is often given attention by the media for his gambling and extravagant spending.\n",
"2. First War of Scottish Independence (Score: 0.946)\n",
"3. Battle of Bannockburn (Score: 0.946)\n",
"4. Wars of Scottish Independence (Score: 0.944)\n",
"5. Second War of Scottish Independence (Score: 0.94)\n",
"5. Second War of Scottish Independence (Score: 0.939)\n",
"6. List of Scottish monarchs (Score: 0.937)\n",
"7. Scottish Borders (Score: 0.932)\n",
"8. Braveheart (Score: 0.929)\n",
@ -1095,7 +1063,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 99,
"id": "76d697e9",
"metadata": {
"ExecuteTime": {
@ -1110,7 +1078,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 100,
"id": "1deeb539",
"metadata": {
"ExecuteTime": {
@ -1125,7 +1093,7 @@
"CollectionsResponse(collections=[])"
]
},
"execution_count": 8,
"execution_count": 100,
"metadata": {},
"output_type": "execute_result"
}
@ -1143,12 +1111,12 @@
"\n",
"Qdrant stores data in __collections__ where each object is described by at least one vector and may contain an additional metadata called __payload__. Our collection will be called **Articles** and each object will be described by both **title** and **content** vectors.\n",
"\n",
"We're going to be using an official [qdrant-client](https://github.com/qdrant/qdrant_client) package that has all the utility methods already built-in."
"We'll be using an official [qdrant-client](https://github.com/qdrant/qdrant_client) package that has all the utility methods already built-in."
"# Check the collection size to make sure all the points have been stored\n",
"qdrant.count(collection_name='Articles')"
@ -1263,12 +1209,12 @@
"source": [
"### Search Data\n",
"\n",
"Once the data is put into Qdrant we can start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search."
"Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search."
]
},
{
"cell_type": "code",
"execution_count": 49,
"execution_count": null,
"id": "f1bac4ef",
"metadata": {
"ExecuteTime": {
@ -1299,7 +1245,7 @@
},
{
"cell_type": "code",
"execution_count": 50,
"execution_count": null,
"id": "aa92f3d3",
"metadata": {
"ExecuteTime": {
@ -1307,34 +1253,7 @@
"start_time": "2023-01-18T09:50:35.711020Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0. Art (Score: 0.841)\n",
"1. Europe (Score: 0.839)\n",
"2. Italy (Score: 0.816)\n",
"3. Architecture (Score: 0.815)\n",
"4. Madrid (Score: 0.815)\n",
"5. France (Score: 0.812)\n",
"6. Belgium (Score: 0.808)\n",
"7. Austria (Score: 0.802)\n",
"8. London (Score: 0.799)\n",
"9. History (Score: 0.797)\n",
"10. Creativity (Score: 0.796)\n",
"11. Archaeology (Score: 0.795)\n",
"12. Cartography (Score: 0.794)\n",
"13. Denmark (Score: 0.793)\n",
"14. Finland (Score: 0.79)\n",
"15. English (Score: 0.789)\n",
"16. Catharism (Score: 0.788)\n",
"17. Dublin (Score: 0.787)\n",
"18. Ireland (Score: 0.787)\n",
"19. Japan (Score: 0.787)\n"
]
}
],
"outputs": [],
"source": [
"query_results = query_qdrant('modern art in Europe', 'Articles')\n",
"for i, article in enumerate(query_results):\n",
@ -1343,7 +1262,7 @@
},
{
"cell_type": "code",
"execution_count": 51,
"execution_count": null,
"id": "7ed116b8",
"metadata": {
"ExecuteTime": {
@ -1351,36 +1270,9 @@
"start_time": "2023-01-18T09:52:55.248029Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1. History (Score: 0.797)\n",
"2. Dublin (Score: 0.787)\n",
"3. Ireland (Score: 0.786)\n",
"4. History of Australia (Score: 0.782)\n",
"5. Historian (Score: 0.778)\n",
"6. Belgium (Score: 0.776)\n",
"7. Black pudding (Score: 0.773)\n",
"8. London (Score: 0.769)\n",
"9. History of Spain (Score: 0.768)\n",
"10. Cartography (Score: 0.763)\n",
"11. March (Score: 0.762)\n",
"12. France (Score: 0.761)\n",
"13. Bubonic plague (Score: 0.76)\n",
"14. Great Lakes (Score: 0.759)\n",
"15. Inch (Score: 0.758)\n",
"16. Dissolution of the monasteries (Score: 0.758)\n",
"17. Austria (Score: 0.757)\n",
"18. English (Score: 0.757)\n",
"19. British English (Score: 0.757)\n",
"20. Armenia (Score: 0.756)\n"
]
}
],
"outputs": [],
"source": [
"# This time we're going to query using content vector\n",
"# This time we'll query using content vector\n",
"query_results = query_qdrant('Famous battles in Scottish history', 'Articles', 'content')\n",
colin-openai
2 years ago