"Found cached dataset wikipedia (/Users/colin.jarvis/.cache/huggingface/datasets/wikipedia/20220301.simple/2.0.0/aa542ed919df55cc5d3347f42dd4521d05ca68751f50dbc32bae2a7f1e167559)\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "edbff2615b964463be20d0a2ac33e4ab",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/1 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"outputs": [],
"source": [
"# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\n",
" \"\"\" Makes chunks out of an input DataFrame. \"\"\"\n",
" splits = self.splits_num(df.shape[0])\n",
" if splits <= 1:\n",
" yield df\n",
" else:\n",
" for chunk in np.array_split(df, splits):\n",
" yield chunk\n",
" \n",
"\n",
" # Determines how many chunks DataFrame contains\n",
" def splits_num(self, elements: int) -> int:\n",
" \"\"\" Determines how many chunks DataFrame contians. \"\"\"\n",
" return round(elements / self.batch_size)\n",
" \n",
" __call__ = to_batches\n",
@ -435,112 +429,40 @@
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": 99,
"id": "7ea9ad46",
"metadata": {},
"outputs": [],
"source": [
"# Pick a name for the new index\n",
"index_name = 'wikipedia-articles'"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "3ff8eca1",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/colin.jarvis/Documents/dev/vectordb_cookbook/vectordb/lib/python3.10/site-packages/pinecone/core/client/rest.py:45: DeprecationWarning: HTTPResponse.getheader() is deprecated and will be removed in urllib3 v2.1.0. Instead use HTTResponse.headers.get(name, default).\n",
"Most similar results querying modern art in Europe in \"title\" namespace:\n",
"\n",
"Result 1 with a score of 0.891034067 is Early modern Europe\n",
"Result 2 with a score of 0.87504226 is Museum of Modern Art\n",
"Result 3 with a score of 0.867497 is Western Europe\n",
"Result 4 with a score of 0.864146471 is Renaissance art\n",
"Result 5 with a score of 0.860363305 is Pop art\n",
"Result 1 with a score of 0.890994787 is Early modern Europe\n",
"Result 2 with a score of 0.875286043 is Museum of Modern Art\n",
"Result 3 with a score of 0.867404044 is Western Europe\n",
"Result 4 with a score of 0.864250064 is Renaissance art\n",
"Result 5 with a score of 0.860506058 is Pop art\n",
"\n",
"\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/colin.jarvis/Documents/dev/vectordb_cookbook/vectordb/lib/python3.10/site-packages/pinecone/core/client/rest.py:45: DeprecationWarning: HTTPResponse.getheader() is deprecated and will be removed in urllib3 v2.1.0. Instead use HTTResponse.headers.get(name, default).\n",
"print(\"Uploading vectors to article schema..\")\n",
"uuids = []\n",
@ -1013,18 +857,18 @@
},
{
"cell_type": "code",
"execution_count": 48,
"execution_count": 112,
"id": "3658693c",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'content': 'Sociedade Esportiva Palmeiras, usually called Palmeiras, is a Brazilian football team. They are from São Paulo, Brazil. The team was founded by an Italian-speaking community on August 26, 1914, as Palestra Itália. They changed to the name used now on September 14, 1942.\\n\\nThey play in green shirts, white shorts and green socks and are one of the most popular and traditional Brazilian clubs.\\n\\nPalmeiras plays at the Palestra Itália stadium, which has seats for 32,000. But in the past, local derbies against São Paulo or Corinthians were usually played in Morumbi stadium. However, the Arena Palestra Itália is under construction with capacity for 45,000 people, expected to be finalized in 2013.\\n\\nName \\n 1914–1942 S.S. Palestra Italia\\n 1942–present S.E. Palmeiras\\n\\nMain titles \\n Copa Rio: 1951\\n Libertadores Cup: 1999 and 2020\\n Copa Mercosul: 1998\\n Campeonato Brasileiro: 1960, 1967, 1967, 1969, 1972, 1973, 1993, 1994, 2016 and 2018 – greatest champion\\n Copa do Brasil: 1998, 2012, 2015 and 2020/21\\n Copa dos Campeões: 2000\\n Campeão do Século\\n Torneio Rio-SP: 1933, 1951, 1965, 1993 and 2000\\n Campeonato Paulista: 1920, 1926 (unbeaten), 1927, 1932 (unbeaten), 1933, 1934, 1936, 1940, 1942, 1944, 1947,1950, 1959 (super champions), 1963, 1966, 1972 (unbeaten), 1974, 1976, 1993, 1994, 1996, 2008 and 2020.\\n Campeonato Paulista Extra: 1926 (unbeaten) and 1938\\n\\nRelated pages\\n List of Brazilian football teams\\n\\nOther websites \\n Palmeiras official site \\n\\nFootball clubs in São Paulo (state)\\n1914 establishments in Brazil',\n",
" 'title': 'Sociedade Esportiva Palmeiras'}"
"{'content': 'Eddie Cantor (January 31, 1892 - October 10, 1964) was an American comedian, singer, actor, songwriter. Familiar to Broadway, radio and early television audiences, this \"Apostle of Pep\" was regarded almost as a family member by millions because his top-rated radio shows revealed intimate stories and amusing anecdotes about his wife Ida and five daughters. His eye-rolling song-and-dance routines eventually led to his nickname, Banjo Eyes, and in 1933, the artist Frederick J. Garner caricatured Cantor with large round and white eyes resembling the drum-like pot of a banjo. Cantor\\'s eyes became his trademark, often exaggerated in illustrations, and leading to his appearance on Broadway in the musical Banjo Eyes (1941). He was the original singer of 1929 hit song \"Makin\\' Whoopie\".\\n\\nReferences\\n\\nPresidents of the Screen Actors Guild\\nAmerican stage actors\\nComedians from New York City\\nAmerican Jews\\nActors from New York City\\nSingers from New York City\\nAmerican television actors\\nAmerican radio actors\\n1892 births\\n1964 deaths',\n",
" 'title': 'Eddie Cantor'}"
]
},
"execution_count": 48,
"execution_count": 112,
"metadata": {},
"output_type": "execute_result"
}
@ -1038,21 +882,21 @@
"id": "46050ca9",
"metadata": {},
"source": [
"### Search Data"
"### Search Data\n",
"\n",
"As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors"
]
},
{
"cell_type": "code",
"execution_count": 41,
"execution_count": 113,
"id": "5acd5437",
"metadata": {},
"outputs": [],
"source": [
"def query_weaviate(query, schema, top_k=20):\n",
" '''Queries an article using its title in the specified\n",
" namespace and prints results.'''\n",
"\n",
" # Create vector embeddings based on the title column\n",
" # Creates embedding vector from user query\n",
" embedded_query = openai.Embedding.create(\n",
" input=query,\n",
" model=MODEL,\n",
@ -1060,19 +904,18 @@
" \n",
" near_vector = {\"vector\": embedded_query}\n",
"\n",
" # Query namespace passed as parameter using title vector\n",
" # Queries input schema with vectorised user query\n",
"Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through the cookbook examples here:\n",
"\n",
"TODO: Make other cool things to link to"
"Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo"
Colin Jarvis
1 year ago