Archives/openai-cookbook
2
0
Fork 0
mirror of https://github.com/openai/openai-cookbook synced 2024-11-11 13:11:02 +00:00
Code Issues Packages Projects Releases Wiki Activity
2399022424
openai-cookbook/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb
Logan Kilpatrick f1e13cfcc7
Misc updates (#1022)
2024-01-25 11:59:05 -06:00

893 lines
133 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{
"cells": [
{
"cell_type": "markdown",
"id": "f96b815e",
"metadata": {},
"source": [
"# Retrieval augmented generation using Elasticsearch and OpenAI"
]
},
{
"cell_type": "markdown",
"id": "e0f537af",
"metadata": {},
"source": [
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb)\n"
]
},
{
"cell_type": "markdown",
"id": "349e0e74",
"metadata": {},
"source": [
"This notebook demonstrates how to: \n",
"- Index the OpenAI Wikipedia vector dataset into Elasticsearch \n",
"- Embed a question with the OpenAI [`embeddings`](https://platform.openai.com/docs/api-reference/embeddings) endpoint\n",
"- Perform semantic search on the Elasticsearch index using the encoded question\n",
"- Send the top search results to the OpenAI [Chat Completions](https://platform.openai.com/docs/guides/gpt/chat-completions-api) API endpoint for retrieval augmented generation (RAG)\n",
"\n",
"ℹ️ If you've already worked through our semantic search notebook, you can skip ahead to the final step!"
]
},
{
"cell_type": "markdown",
"id": "aa9576ca",
"metadata": {},
"source": [
"## Install packages and import modules "
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "8c304b93",
"metadata": {},
"outputs": [],
"source": [
"# install packages\n",
"\n",
"!python3 -m pip install -qU openai pandas wget elasticsearch\n",
"\n",
"# import modules\n",
"\n",
"from getpass import getpass\n",
"from elasticsearch import Elasticsearch, helpers\n",
"import wget\n",
"import zipfile\n",
"import pandas as pd\n",
"import json\n",
"import openai"
]
},
{
"cell_type": "markdown",
"id": "de32a789",
"metadata": {},
"source": [
"## Connect to Elasticsearch\n",
"\n",
"ℹ️ We're using an Elastic Cloud deployment of Elasticsearch for this notebook.\n",
"If you don't already have an Elastic deployment, you can sign up for a free [Elastic Cloud trial](https://cloud.elastic.co/registration?utm_source=github&utm_content=openai-cookbook).\n",
"\n",
"To connect to Elasticsearch, you need to create a client instance with the Cloud ID and password for your deployment.\n",
"\n",
"Find the Cloud ID for your deployment by going to https://cloud.elastic.co/deployments and selecting your deployment."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "3a57b6a8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'name': 'instance-0000000001', 'cluster_name': '29ef9817e13142f5ba0ea7b29c2a86e2', 'cluster_uuid': 'absjWgQvRw63IlwWKisN8w', 'version': {'number': '8.9.1', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'a813d015ef1826148d9d389bd1c0d781c6e349f0', 'build_date': '2023-08-10T05:02:32.517455352Z', 'build_snapshot': False, 'lucene_version': '9.7.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}\n"
]
}
],
"source": [
"CLOUD_ID = getpass(\"Elastic deployment Cloud ID\")\n",
"CLOUD_PASSWORD = getpass(\"Elastic deployment Password\")\n",
"client = Elasticsearch(\n",
" cloud_id = CLOUD_ID,\n",
" basic_auth=(\"elastic\", CLOUD_PASSWORD) # Alternatively use `api_key` instead of `basic_auth`\n",
")\n",
"\n",
"# Test connection to Elasticsearch\n",
"print(client.info())"
]
},
{
"cell_type": "markdown",
"id": "80b55952",
"metadata": {},
"source": [
"## Download the dataset \n",
"\n",
"In this step we download the OpenAI Wikipedia embeddings dataset, and extract the zip file."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "c584f15c",
"metadata": {},
"outputs": [],
"source": [
"embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n",
"wget.download(embeddings_url)\n",
"\n",
"with zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\n",
"\"r\") as zip_ref:\n",
" zip_ref.extractall(\"data\")"
]
},
{
"cell_type": "markdown",
"id": "9654ac08",
"metadata": {},
"source": [
"## Read CSV file into a Pandas DataFrame.\n",
"\n",
"Next we use the Pandas library to read the unzipped CSV file into a DataFrame. This step makes it easier to index the data into Elasticsearch in bulk."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "76347d10",
"metadata": {},
"outputs": [],
"source": [
"\n",
"wikipedia_dataframe = pd.read_csv(\"data/vector_database_wikipedia_articles_embedded.csv\")"
]
},
{
"cell_type": "markdown",
"id": "6af9f5ad",
"metadata": {},
"source": [
"## Create index with mapping\n",
"\n",
"Now we need to create an Elasticsearch index with the necessary mappings. This will enable us to index the data into Elasticsearch.\n",
"\n",
"We use the `dense_vector` field type for the `title_vector` and `content_vector` fields. This is a special field type that allows us to store dense vectors in Elasticsearch.\n",
"\n",
"Later, we'll need to target the `dense_vector` field for kNN search.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "681989b3",
"metadata": {},
"outputs": [],
"source": [
"index_mapping= {\n",
" \"properties\": {\n",
" \"title_vector\": {\n",
" \"type\": \"dense_vector\",\n",
" \"dims\": 1536,\n",
" \"index\": \"true\",\n",
" \"similarity\": \"cosine\"\n",
" },\n",
" \"content_vector\": {\n",
" \"type\": \"dense_vector\",\n",
" \"dims\": 1536,\n",
" \"index\": \"true\",\n",
" \"similarity\": \"cosine\"\n",
" },\n",
" \"text\": {\"type\": \"text\"},\n",
" \"title\": {\"type\": \"text\"},\n",
" \"url\": { \"type\": \"keyword\"},\n",
" \"vector_id\": {\"type\": \"long\"}\n",
" \n",
" }\n",
"}\n",
"\n",
"client.indices.create(index=\"wikipedia_vector_index\", mappings=index_mapping)"
]
},
{
"cell_type": "markdown",
"id": "c2fb582e",
"metadata": {},
"source": [
"## Index data into Elasticsearch \n",
"\n",
"The following function generates the required bulk actions that can be passed to Elasticsearch's Bulk API, so we can index multiple documents efficiently in a single request.\n",
"\n",
"For each row in the DataFrame, the function yields a dictionary representing a single document to be indexed. "
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "efee9b97",
"metadata": {},
"outputs": [],
"source": [
"def dataframe_to_bulk_actions(df):\n",
" for index, row in df.iterrows():\n",
" yield {\n",
" \"_index\": 'wikipedia_vector_index',\n",
" \"_id\": row['id'],\n",
" \"_source\": {\n",
" 'url' : row[\"url\"],\n",
" 'title' : row[\"title\"],\n",
" 'text' : row[\"text\"],\n",
" 'title_vector' : json.loads(row[\"title_vector\"]),\n",
" 'content_vector' : json.loads(row[\"content_vector\"]),\n",
" 'vector_id' : row[\"vector_id\"]\n",
" }\n",
" }"
]
},
{
"cell_type": "markdown",
"id": "b8164b38",
"metadata": {},
"source": [
"As the dataframe is large, we will index data in batches of `100`. We index the data into Elasticsearch using the Python client's [helpers](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/client-helpers.html#bulk-helpers) for the bulk API."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "aacb5e9c",
"metadata": {},
"outputs": [],
"source": [
"start = 0\n",
"end = len(wikipedia_dataframe)\n",
"batch_size = 100\n",
"for batch_start in range(start, end, batch_size):\n",
" batch_end = min(batch_start + batch_size, end)\n",
" batch_dataframe = wikipedia_dataframe.iloc[batch_start:batch_end]\n",
" actions = dataframe_to_bulk_actions(batch_dataframe)\n",
" helpers.bulk(client, actions)"
]
},
{
"cell_type": "markdown",
"id": "091ffc51",
"metadata": {},
"source": [
"Let's test the index with a simple match query."
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "2ccc8955",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'took': 10, 'timed_out': False, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 4, 'relation': 'eq'}, 'max_score': 14.917897, 'hits': [{'_index': 'wikipedia_vector_index', '_id': '34227', '_score': 14.917897, '_source': {'url': 'https://simple.wikipedia.org/wiki/Hummingbird', 'title': 'Hummingbird', 'text': \"Hummingbirds are small birds of the family Trochilidae.\\n\\nThey are among the smallest of birds: most species measure 7.5–13\\xa0cm (3–5\\xa0in). The smallest living bird species is the 2–5\\xa0cm Bee Hummingbird. They can hover in mid-air by rapidly flapping their wings 12–80 times per second (depending on the species). They are also the only group of birds able to fly backwards. Their rapid wing beats do actually hum. They can fly at speeds over 15\\xa0m/s (54\\xa0km/h, 34\\xa0mi/h).\\n\\nEating habits and pollination \\nHummingbirds help flowers to pollinate, though most insects are best known for doing so. The hummingbird enjoys nectar, like the butterfly and other flower-loving insects, such as bees.\\n\\nHummingbirds do not have a good sense of smell; instead, they are attracted to color, especially the color red. Unlike the butterfly, the hummingbird hovers over the flower as it drinks nectar from it, like a moth. When it does so, it flaps its wings very quickly to stay in one place, which makes it look like a blur and also beats so fast it makes a humming sound. A hummingbird sometimes puts its whole head into the flower to drink the nectar properly. When it takes its head back out, its head is covered with yellow pollen, so that when it moves to another flower, it can pollinate. Or sometimes it may pollinate with its beak.\\n\\nLike bees, hummingbirds can assess the amount of sugar in the nectar they eat. They reject flowers whose nectar has less than 10% sugar. Nectar is a poor source of nutrients, so hummingbirds meet their needs for protein, amino acids, vitamins, minerals, etc. by preying on insects and spiders.\\n\\nFeeding apparatus \\nMost hummingbirds have bills that are long and straight or nearly so, but in some species the bill shape is adapted for specialized feeding. Thornbills have short, sharp bills adapted for feeding from flowers with short corollas and piercing the bases of longer ones. The Sicklebills' extremely decurved bills are adapted to extracting nectar from the curved corollas of flowers in the family Gesneriaceae. The bill of the Fiery-tailed Awlbill has an upturned tip, as in the Avocets. The male Tooth-billed Hummingbird has barracuda-like spikes at the tip of its long, straight bill.\\n\\nThe two halves of a hummingbird's bill have a pronounced overlap, with the lower half (mandible) fitting tightly inside the upper half (maxilla). When hummingbirds feed on nectar, the bill is usually only opened slightly, allowing the tongue to dart out into the nectar.\\n\\nLike the similar nectar-feeding sunbirds and unlike other birds, hummingbirds drink by using grooved or trough-like tongues which they can stick out a long way.\\nHummingbirds do not spend all day flying, as the energy cost would be prohibitive; the majority of their activity consists simply of sitting or perching. Hummingbirds feed in many small meals, consuming many small invertebrates and up to twelve times their own body weight in nectar each day. They spend an average of 10–15% of their time feeding and 75–80% sitting and digesting.\\n\\nCo-evolution with flowers\\n\\nSince hummingbirds are specialized nectar-eaters, they are tied to the bird-flowers they feed upon. Some species, especially those with unusual bill shapes such as the Sword-billed Hummingbird and the sicklebills, are co-evolved with a small number of flower species.\\n\\nMany plants pollinated by hummingbirds produce flowers in shades of red, orange, and bright pink, though the birds will take nectar from flowers of many colors. Hummingbirds can see wavelengths into the near-ultraviolet. However, their flowers do not reflect these wavelengths as many insect-pollinated flowers do. The narrow color spectrum may make hummingbird-pollinated flowers inconspicuous to insects, thereby reducing nectar robbing by insects. Hummingbird-pollinated flowers also produce relatively weak nectar (averaging 25% sugars w/w) containing high concentrations of sucrose, whereas insect-pollinated flowers typically produce more concentrated nectars dominated by fructose and glucose.\\n\\nTaxonomy \\nHummingbirds have traditionally been a part of the bird order Apodiformes. This order includes the hummingbirds, the swifts and the tree swifts. The Sibley-Ahlquist taxonomy of birds, based on DNA studies done in the 1970s and 1980s, changed the classification of hummingbirds. Instead of being in the same order as the swifts, the hummingbirds were made an order, the Trochiliformes. Their previous order, Apodiformes was changed to the superorder Apodimorphae. This superorder contains the three families of birds which were in it when it was an order.\\n\\nReferences\", 'vector_id': 10024}}, {'_index': 'wikipedia_vector_index', '_id': '84773', '_score': 10.951234, '_source': {'url': 'https://simple.wikipedia.org/wiki/Inagua', 'title': 'Inagua', 'text': \"Inagua is the southernmost district of the Bahamas. It is the islands of Great Inagua and Little Inagua.\\n\\nGreat Inagua is the third largest island in the Bahamas at 596 square miles (1544\\xa0km²) and lies about 55 miles (90\\xa0km) from the eastern tip of Cuba. The island is about 55 × 19 miles (90 × 30\\xa0km) in extent, the highest point being 108\\xa0ft (33 m) on East Hill. It encloses several lakes, most notably the 12-mile long Lake Windsor (also called Lake Rosa) which occupies nearly ¼ of the interior. The population of Great Inagua is 969 (2000 census).\\n\\nThe island's capital and only harbour is Matthew Town.\\n\\nThere is a large bird sanctuary in the centre of the island. There are more than 80,000 West Indian Flamingoes and many other exotic birds such as the native Bahama Parrot, the Bahama woodstar hummingbird, Bahama pintails, Brown pelicans, Tri-colored herons, Snowy egrets, Reddish egrets, Stripe-headed tanangers, Cormorants, Roseate spoonbills, American kestrels, and Burrowing owls.\\n\\nDistricts of the Bahamas\\nIslands of the Bahamas\\n1999 establishments in the Bahamas\", 'vector_id': 22383}}, {'_index': 'wikipedia_vector_index', '_id': '3707', '_score': 1.1967773, '_source': {'url': 'https://simple.wikipedia.org/wiki/Bird', 'title': 'Bird', 'text': 'Birds (Aves) are a group of animals with backbones which evolved from dinosaurs. Technically speaking, they are dinosaurs. \\n\\nBirds are endothermic. The heat loss from their bodies is slowed down by their feathers. \\nModern birds are toothless: they have beaked jaws. They lay hard-shelled eggs. They have a high metabolic rate, a four-chambered heart and a strong yet lightweight skeleton.\\n\\nBirds live all over the world. They range in size from the 5 cm (2 in) bee hummingbird to the 2.70 m (9 ft) ostrich. They are the tetrapods with the most living species: about ten thousand. More than half of these are passerines, sometimes known as perching birds.\\n\\nBirds are the closest living relatives of the Crocodilia. This is because they are the two main survivors of a once huge group called the Archosaurs. \\n\\nModern birds are not descended from Archaeopteryx. According to DNA evidence, modern birds (Neornithes) evolved in the long Upper Cretaceous period. More recent estimates showed that modern birds originated early in the Upper Cretaceous.\\n\\nPrimitive bird-like dinosaurs are in the broader group Avialae. They have been found back to the mid-Jurassic period, around 170 million years ago. Many of these early \"stem-birds\", such as Anchiornis, were not yet capable of fully powered flight. Many had primitive characteristics like teeth in their jaws and long bony tails.p274\\n\\nThe Cretaceous–Palaeogene extinction event 66 million years ago killed off all the non-avian dinosaur lines. Birds, especially those in the southern continents, survived this event and then migrated to other parts of the world. Diversification occurred around the Cretaceous–Palaeogene extinction event. \\n\\nBirds have wings which are more or less developed depending on the species. The only known groups without wings are the extinct moa and elephant birds. Wings, which evolved from forelimbs, gave birds the ability to fly. Later, many groups evolved with reduced wings, such as ratites, penguins and many island species of birds. The digestive and respiratory systems of birds are also adapted for flight. Some bird species in aquatic environments, particularly seabirds and some waterbirds, have evolved as good swimmers.\\n\\nIn general, birds are effective, and inherit their behaviour almost entirely. The key elements of their life are inherited. It was a great discovery that birds never learn to fly. \\nSo it is quite wrong to say, when a chick waves its wings in the nest \"It\\'s learning to fly\". What the chick is doing is exercising its muscles. They develop the ability to fly automatically (assuming they are species that do fly). And if they are species which migrate, that behaviour is also inherited. Many species migrate over great distances each year. Other main features of their life may be inherited, though they can and do learn. Birds have good memories which they use, for example, when they search for food.\\n\\nSeveral bird species make and use tools. Some social species pass on some knowledge across generations, a form of culture. Birds are social. They communicate with visual signals, calls and bird songs. Most of their social behaviours are inherited, such as cooperative breeding and hunting, flocking and mobbing of predators.\\n\\nMost bird species are socially monogamous, usually for one breeding season at a time, sometimes for years, but rarely for life. Other species are polygynous (one male with many females) or, rarely, polyandrous (one female with many males). Birds produce offspring by laying eggs which are fertilised by sexual reproduction. They are often laid in a nest and incubated by the parents. Most birds have an extended period of parental care after hatching. Some birds, such as hens, lay eggs even when not fertilised, though unfertilised eggs do not produce offspring.\\n\\nMany species of birds are eaten by humans. Domesticated and undomesticated birds are sources of eggs, meat, and feathers. In English, domesticated birds are often called poultry, undomesticated birds are called game. Songbirds, parrots and other species are popular as pets. Guano, which is bird manure, is harvested for use as a fertiliser. Birds figure throughout human culture. About 120–130 species have become extinct due to human activity since the 17th century and hundreds more before then. Human activity threatens about 1,200 bird species with extinction, though efforts are underway to protect them. Recreational bird-watching is an important part of the ecotourism industry.\\n\\nBird colours \\n\\nBirds come in a huge range of colours. These colours can be useful to a bird in two ways. Camouflage colours help to hide the bird, and bright colours identify the bird to others of the same species. Often the male is brightly coloured while the female is camouflaged. The logic is as follows: the female carries the \"precious package\" of developing eggs. The male has to defend a territory, and the function of his colour and song is to let others know that \"this place is occupied\".\\n\\nBird camouflage \\n\\nMany birds are brown, green or grey. These colours make a bird harder to be seen: they camouflage the bird. Brown is the most common colour. Brown birds include: sparrows, emus, thrushes, larks, eagles and falcons and the female birds of many species such as: wrens, ducks, blackbirds and peafowls. When a brown bird is in long grass or among tree trunks or rocks, it is camouflaged. Birds that live in long grass often have brown feathers streaked with black which looks like shadows. A bittern is almost invisible in long reeds because its camouflage is helped by its posture (beak and head pointed upwards). Other birds, including starlings and mynas, are quite dark in colour, but are flecked with little spots that look like raindrops on leaves. Bird may also camouflage their nests.\\n\\nMany birds from hot countries are green or have some green feathers, particularly parrots. Birds that live in green trees often have green backs, even if they have bright-coloured breasts. From the back, the birds are camouflaged. This is very useful when sitting on a nest. The bird\\'s bright-coloured breast is hidden. Budgerigars are bred in different colours such as blue, white and mauve, but in the wild, they are nearly all green and yellow. Even though they fly very well, they normally spend a lot of time on the ground, eating grass seeds. Their yellow and black striped back helps to hide them in the shadows made by long dry grass, while their green breasts are a similar colour to the leaves of gum trees.\\n\\nGrey birds include most pigeons and doves, cranes, storks and herons. Grey birds are often rock-living birds like pigeons or birds that sit on dead tree trunks looking like a broken branch. Water birds like herons often have a pale grey colour which makes it harder for a fish to notice that the bird is standing, looking down for something to catch. Water birds, no matter what colour they are on top, are often white underneath, so that when a fish looks up, the bird looks like part of the sky.\\n\\nBlack birds include crows, ravens and male blackbirds. Some birds that are dark colours spend quite a lot of time on the ground, hopping around in the shadows under bushes. Among these birds are the male blackbird and the satin bowerbird which is not black but very dark blue. Crows and ravens often perch high on bare trees in the winter, where their black shape against the sky looks like the dark bare branches.\\n\\nNoticeable colours \\n\\nMany birds are not camouflaged, but stand out with vivid colours. They are usually male birds whose females are dull and camouflaged. The function of the colours is two-fold. First, the colours help them get mates, and second, the colours identify them to other males of the same species. Many birds are territorial, especially in the nesting season. They give out territory sounds and are easily seen. This lets other males know they will defend their territory. It sends out a \"look elsewhere\" signal to their competitors.\\n\\nSome birds are famous for their colour and are named for it, such as the bluebird, the azure kingfisher, the golden pheasant, the scarlet macaw, the violet wren and the robin.\\n\\nMany other birds are very brightly coloured, in countless combinations. Some of the most colourful birds are quite common, like pheasants, peacocks, domestic fowl and parrots. Colourful small birds include blue tits, the goldfinches, hummingbirds, fairy wrens and bee eaters (which are also called rainbow birds). Some birds, like those of the bird of paradise in Papua New Guinea have such beautiful feathers that they have been hunted for them.\\n\\nThe peafowl is the best example of a display of colour to attract a mate. Also the male domestic fowl and junglefowl have long shiny feathers above his tail and also long neck feathers that may be a different colour to his wings and body. There are only a very few types of birds (like the eclectus parrot) where the female is more colourful than the male.\\n\\n\\'\\'Pied birds\\'\\' are black and white. Black and white birds include magpies, pied geese, pelicans and Australian magpies (which are not really magpies at all). Pied birds often have brightly coloured beaks and legs of yellow or red. The silver pheasant, with its long white tail striped with fine bars of black, has a brightly coloured face.\\n\\nFlight \\nMost birds can fly, and if they do, then the ability is inherited, not learnt. They fly by pushing through the air with their wings. The curved surfaces of the wings cause air currents (wind) which lift the bird. Flapping keeps the air current moving to create lift and also moves the bird forward.\\n\\nSome birds can glide on air currents without flapping. Many birds use this method when they are about to land. Some birds can also hover in the air. This method is used by birds of prey such as falcons that are looking for something to eat. Seagulls are also good at hovering, particularly if there is a strong breeze. The most expert hovering birds are tiny hummingbirds which can beat their wings both backwards and forwards and can stay quite still in the air while they dip their long beaks into flowers to feed on the sweet nectar.\\n\\nTypes of flight \\nDifferent types of birds have different needs. Their wings have evolved to suit their lifestyle. Large birds of prey, such as eagles, spend a lot of time soaring on the wind. They have wings that are large and broad. The main flight feathers are long and wide. They help the eagle to stay on rising air currents without using much energy, while the eagle looks at the ground below, to find the next meal. When the eagle sees some small creature move, it can close its wings and fall from the sky like a missile, opening its great wings again to slow down as it comes to land. The world\\'s largest eagle, the Philippine eagle has a wingspan of about 2 m (6.7\\xa0ft) wide.\\n\\nBirds that live in grassland areas or open forests and feed on fruit, insects and reptiles often spend a lot of time flying short journeys looking for food and water. They have wings that are shaped in a similar way to eagles, but rounder and not as good for soaring. These include many Australian birds like cockatoos.\\n\\nBirds such as geese that migrate from one country to another fly very long distances. Their wings are big and strong, because the birds are large. They stock up on food for the long flight. Migrating water birds usually form family groups of 1230 birds. They fly very high, making use of long streams of air that blow from north to south in different seasons. They are well organised, often flying in a V pattern. The geese at the back do not have to flap so hard; they are pulled on by the wind of the ones at the front. Every so often, they change the leader so that the front bird, who does most work and sets the pace, can have a rest. Geese and swans are the highest-flying birds, reaching 8,000 metres or more when on migration. Geese often honk loudly while they are flying. It is thought that they do this to support the leader and help the young ones.\\n\\nBirds that fly very quickly, such as swifts and swallows, have long narrow pointed wings. These birds need great speed because they eat insects, catching most of them while they are flying. These birds also migrate. They often collect in huge flocks of thousands of birds that move together like a whirling cloud.\\n\\nBirds that live in bushes and branches have triangular wings that help the bird change direction. Many forest birds are expert at getting up speed by flapping and then gliding steadily among the trees, tilting to avoid things as they go. Members of the kingfisher family are expert at this type of flying.\\n\\nBirds such as owls that hunt at night have wings with soft rounded feathers so that they do not flap loudly. Birds that are awake at night are called nocturnal birds. Birds that are awake during the day are diurnal.\\n\\nWandering albatross might spend several years without coming to land. They can sleep while gliding. Arctic terns nest every one to three years.\\n\\nFlocks \\nFlocks of birds can be very highly organised in a way that takes care of all the flock members. Studies of small flocking birds like tree sparrows show that they clearly communicate with each other, as sometimes thousands of birds may fly in close formation and spiral patterns without colliding (or flying into each other).\\n\\nTwo common behaviours in flocking birds are guarding and reconnaissance. When a flock of birds is feeding it is common for one bird to perch on a high place to keep guard over the flock. In the same way, when a flock is asleep, often, one bird will remain awake. It is also common for large flocks to send one or two birds ahead of them when they are flying to a new area. The look-out birds can spy the lie of the land to find food, water and good places to perch. Mixed feeding flocks occur, and can help to spot predators.\\n\\nFlightless birds \\nSome birds do not fly. Flightlessness in birds has evolved many times.\\nThese include running birds like ostriches and emus and ocean-living birds, the large penguin family. Birds on islands have usually lost the power of flight. This is to their advantage because birds with the power of flight can be blown off their island during a storm. The same ability which got them to the island may later take them away in a storm.\\n\\nOstriches and emus do not need to fly because although they feed and nest on the ground, their great size and their speed is their protection. Some other ground-feeding birds have not been so lucky. Some birds such as the dodo and the kiwi were ground-feeding birds that lived in safety on islands where there was nothing dangerous to eat them. They lost the power of flight. Kiwis are endangered because European settlement to New Zealand brought animals like cats, dogs and rats which kill kiwis and eat their eggs. However, kiwis and also the rare New Zealand ground parrot have survived. In the case of dodos, they were fat and disgusting in taste. All the same, they were killed and eaten by sailors until there was none left. Other flightless birds which have disappeared are the great auk and the moa.\\n\\nPenguins are a very successful group of birds. They are a clade. They spend half their time on land. Their wings are adapted to life in the sea and have become flippers which let them in swim fast. They catch fish at sea, where they are in danger from seals.\\n\\nDigestion \\nModern birds do not have teeth, and many swallow their prey whole. Nevertheless, they must break up food before it is digested. First of all, along their throat (oesophagus) they have a crop. This stores food items before digestion. That way a bird can eat several items, and then fly off to a quiet spot to digest them. \\n\\nTheir stomach comes next, with two very different parts. One part is like a straight hollow rod (the proventriculus) which secretes mild hydrochloric acid and an enzyme to break down protein. The other part of the stomach is the gizzard. This is muscular, and grinds up the contents. In herbivorous birds the gizzard contains some gastroliths (small stones or pieces of grit). Bones of fish will mostly be dissolved by the stomach acid. The partly digested and ground-up food now goes to the intestine, where digestion is completed, and most contents are absorbed. Anything indigestible, for example remains of feathers, is regurgitated via the mouth, not the cloaca.\\n\\nThe system is effective, and carnivorous birds can swallow quite large prey. A blue heron can swallow a fish as large as a carp successfully. Raptors eat by holding the prey down with a foot, and tearing it apart with their beak.\\n\\nReproduction\\n\\nMating \\nAlthough birds are warm-blooded creatures like mammals, they do not give birth to live young. They lay eggs as reptiles do, but the shell of a bird\\'s egg is hard. The baby bird grows inside the egg, and after a few weeks hatches (breaks out of the egg).\\n\\nBirds in cold climates usually have a breeding season once a year in the spring. Migratory birds can have two springs and two mating seasons in a year. \\n\\nNinety-five per cent of bird species are socially monogamous. These birds pair for at least the length of the breeding season. In some cases this arrangement lasts until the death of one of the pair. Monogamy clearly helps if females need males\\' help to raise a brood successfully. It has other practical advantages: the nest is never left without defence. Birds are generally small, and they have many potential enemies.\\n\\nSome birds mate for life, like married couples. These birds include pigeons, geese, and cranes. Other birds look for new partners each year. For birds that choose new mates, part of the breeding season is display. The male bird will do all sorts of things to attract females. These include singing, dancing, showing off the feathers and building a beautiful nest. Some male birds have splendid feathers for attracting females. The most famous is the peacock who can spread the feathers above his tail into a huge fan. \\n\\nOther mating systems do occur in some species. Polygyny, polyandry, polygamy, polygynandry, and promiscuity do happen. Polygamous breeding systems arise when females are able to raise broods without the help of males. Some species may use more than one system depending on the circumstances.\\n\\nNesting \\nOnce the birds have found partners, they find a suitable place to lay eggs. The idea of what is a suitable place differs between species, but most build bird nests. The bird is driven by a hormone (estradiol E2) to prepare a place for the eggs to hatch. Birds\\' nests may be up a tree, in a cliff or on the ground according to species. When filled with eggs they are almost always guarded by one of the pair. In fact it is virtually impossible for the eggs to survive if one of the parents dies.\\n\\nRobins will make a beautiful little round nest of woven grass and carefully line it with feathers, bits of fluff and other soft things. Swallows like to nest near other swallows. They make nests from little blobs of clay, often on a beam near the roof of a building where it is well sheltered. Many birds like a hollow tree to nest in. Eagle\\'s nests are often just piles of dead wood on the top of the tallest tree or mountain. Scrub turkeys scratch together a huge pile of leaves that may be 10 metres across. Guillemots lay their eggs on rock shelves with no nest at all. Their eggs are shaped so that they roll around in circles and do not fall off cliffs. A cuckoo does not make its own nest. It lays its egg in the nest of another bird and leaves it for them to care for. The cuckoo eggs are camouflaged to look like the host\\'s eggs.\\n\\nWhen the nest has been prepared, the birds mate so that the eggs are fertilised and the chicks will start growing. Unlike mammals, birds (and reptiles) only have one opening as the exit hole for body fluids, and for reproduction. The opening is called the cloaca. A female bird, called a hen, has two ovaries, of which the left one usually produces eggs.\\n\\nMost male birds have no sex organs that can be seen. But inside the male are two testes which produce sperm which is stored in the cloaca. Birds mate by rubbing their cloacas together, although with some birds, particularly large water birds, the male has a sort of a penis inside the cloaca.\\n\\nHatching \\nOnce the hen has mated, she produces fertile eggs which have chicks growing inside them. She lays the eggs in the nest. There might be just one egg or a number of them, called a clutch. Emus might lay as many as fifteen huge dark green eggs in a clutch. After the eggs are laid, they are incubated, or kept warm so the chicks form inside. Most birds stay together for the whole nesting season, and one advantage is that the work is shared. Many birds take turns sitting on the eggs, so that each adult can feed.\\n\\nThis is not always the case. With emus, the male does all the sitting and all the baby-minding. With emperor penguins it is also the male that cares for the egg. There is only one egg, which he keeps on his feet and under his feathers, standing in a big group of males without feeding until the chick is hatched. While the eggs are hatching, the females are at sea, feeding, so that they can care for the chicks when they return.\\n\\nSome birds put the eggs inside or on top of the mound of leaves and twigs. The mound acts like a compost heap. The decomposition of the rotting leaves causes the temperature to rise. This is heat released by the chemical action of bacterial and fungal respiration. It is the same reaction as that which keeps mammals and birds at a high temperature. The parents leave the mound. When the chicks hatch, they are able to feed themselves.\\n\\nMany small birds take 2–4 weeks to hatch eggs. Albatrosses take 80 days. During this time the female loses a lot of her body weight.\\n\\nThe quickest hatching time is for the cuckoo. Some types of cuckoos take only 10 days. This means that when they hatch in the nest of their \\'\\'foster parents\\'\\', the eggs that the parents have laid are not yet ready. Newborn cuckoos are naked, blind and ugly, but they are strong. They get under any eggs that are in the nest and throw them out before they hatch. That means that the cuckoo has the whole care of both parents. Baby cuckoos grow fast and often get bigger than the parents who feed them.\\n\\nWhen baby birds hatch, in most types of birds, they are fed by both parents, and sometimes by older aunties as well. Their mouths are open all the time and are often very brightly coloured, which acts as a releaser\\'\\', a trigger which stimulates the parent to feed them. For birds that eat grain and fruit, the parents eat and partly digest the food for the babies. It is then vomited carefully into the baby\\'s mouth.\\n \\n\\n Families \\nMany birds, particularly those that mate for life, are very sociable and keep together in a family group which might be anything from 4 or 6 adult birds and their young to a very large flock.\\n\\nAs chicks grow they change the fluffy down that covers them as babies for real feathers. At this stage they are called fledglings. Other family members may help care for fledgling chicks, feeding them and protecting them from attack while parents are feeding. When the fledglings have their new feathers, they come out of the nest to learn to fly. In some types of birds, like pigeons, the parents watch over this and as the young ones get stronger, will give them flying lessons, teaching them how to glide, how to fly in spirals and how to land like an expert.\\n\\n Communication \\nMost birds are social animals, at least part of the time. They communicate to each other using sounds and displays.\\n\\nAlmost all birds make sounds to communicate. The types of noises that vary greatly. Some birds can sing, and they are called songbirds or passerines. Examples are robins, larks, canaries, thrushes, nightingales. Corvids are passerines, but they do not sing. Birds that are not songbirds include: pigeons, seagulls, eagles, owls and ducks. Parrots are not songbirds, even though they can be taught to sing human songs.\\n\\n Songbirds \\nAll birds make noises (\\'\\'bird vocalisation\\'\\'), but not all sing. Songbirds are passerines, many of which have beautiful melodic songs. Songs have different functions. Danger cries are different from territorial songs and mating calls are a third type. Fledgling may also have different calls from adults. Recognition calls for partners are quite common.\\n\\nAs to where the song comes from, there are three kinds of species:\\nThose where the song is mainly inherited, and the bird always sings the same song in the same situations. The capacity is inherited, and only details are learnt from its neighbours.\\nThose where the song is partly inherited, but the bird tunes it in by copying others. In this case the slight differences between the calls of different birds may be used by partners for identification.\\nThose where the song is entirely learnt, and the bird often copies sounds from its environment. Only the capacity to sing is inherited.\\n\\nMost singing birds that are kept as pets, like canaries, have several tunes and some variations.\\n\\nThe same species of bird will sing different songs in different regions. A good example of this is the currawong. This is an Australia bird which is like a black and white crow. In the autumn, families get together in large flocks and do a lot of singing. Currawongs from some areas sing much more complex songs than others. Generally, currawongs from the Blue Mountains are the finest singers. The song of the currawong can be sung as a solo, but is often performed as a choir. One bird will take the lead and sing \"Warble-warble-warble-warble!\" All the other birds will join in and sing \"Wooooooo!\". When all the birds know the song, the choir will sing the \"Warble\" part and the soloist will sing the \"Woo!\". The song changes from year to year and from place to place.\\n\\n Lorenz\\'s studies \\nThe Austrian naturalist Konrad Lorenz studied the way in which birds communicate, or talk to each other. He found that each type of bird had a number of sounds which they made automatically, when ever they felt a certain way. Every sound had an action that went with it. So, if the bird was frightened, it acted frightened and made a frightened sound. This told the other birds around it that something frightening was happening.\\n\\nIf a flock of birds were flying over a field, they would be calling \"Fly! Fly!\" But a hungry bird, seeing something good to eat down below might start calling \"Food! Food!\" If other birds were also hungry, they would make the same call until more birds were calling \"Food! Food!\" than \"Fly! Fly!\". At this point, the mind of the flock would be changed. Some of the birds would start to yell \"Fly downwards! Fly downwards!\" as they sank from the sky, until the whole flock was all noisily calling the same thing.\\n\\nThese communication sounds are often short hard sounds like: chirps, squeaks, squawks and twitters. Sometimes the calls are longer and more musical. They include the \"Rookety-coo\" sound of a pigeon and the \"Cockadoodledoo!\" of a rooster. The bird cannot change these sounds. They always make them in the same way. The bird is locked into making each sound every time a particular idea comes into its head. The connection between how they feel and how they call is innate: they are born with it. Some calls in some species are learnt. Then, it is the tendency to learn which is inherited.\\n\\n The Jackdaw of Altenberg \\nKonrad Lorenz noticed that when birds sing, they often use a lot of their regular calls as part of the song. Lorenz had a flock of jackdaws which were scattered during World War II. One day, an old bird returned. For many months she sat on the chimney singing her song, but in the song she kept making the call which Lorenz knew meant \"Come home! Come home!\" One day, to the great surprise of Lorenz, a male bird flew from a passing flock and joined her on the chimney. Lorenz was sure that it was her long-lost \"husband\" who had found his way home at last.\\n\\n Evolution and taxonomy \\n\\nPalaeontologists have found some exceptional places (lagerstätten) where fossils of early birds are found. The preservation is so good that on the best examples impressions of their feathers can be seen, and sometimes even the remains of meals they have eaten. From these remains we know that birds evolved from small carnivorous dinosaurs (theropods) in the Jurassic period. They radiated into a huge variety in the Lower Cretaceous. At the same time, their direct competitors, the pterosaurs, dwindled in numbers and variety, and became extinct at the end of the Mesozoic.\\n\\nBirds are classified by taxonomists as \\'Aves\\' (Avialae). Birds are the only living descendants of dinosaurs (strictly speaking, they are dinosaurs). Birds and Crocodilia are the only living members of the once-dominant Archosaur reptiles.\\n\\n Definition \\nThe class Aves is was defined (1990) as all the descendants of the most recent common ancestor of modern birds and Archaeopteryx lithographica. But Archaeopteryx is almost certainly not the ancestor of modern birds. The transition to flight happened a number of times. The researchers offered four definitions. Birds can be: \\nAll archosaurs closer to birds than crocodiles (Avemetatarsalia).\\nAdvanced archosaurs with feathers (Avofilopluma).\\nThose feathered dinosaurs that fly (or Avialae)\\nAves can mean the last common ancestor of all living birds and all of its descendants (a \"crown group\", in this sense synonymous with Neornithes).\\n\\n The first bird-like creatures Archaeopteryx, from the Upper Jurassic some 150–145 million years ago (mya), was for a long time the earliest known bird which could fly. It is famous, because it was one of the first important fossils found after Charles Darwin published his ideas about evolution in the 19th century. By modern standards, Archaeopteryx could not fly very well. Other early fossil birds are, for example, Confuciusornis, Anchiornis huxlei and other Paraves.\\n\\nMany fossils of early birds and small dinosaurs have been discovered in the Liaoning Province of Northeast China. These include Anchiornis huxlei, from about 160 mya. The fossils show that most small theropod dinosaurs had feathers. These deposits have preserved them so well that the impressions of their feathers can be clearly seen. This leads us to think that feathers evolved first as heat insulation and only later for flight. The origin of birds lies in these small feathered dinosaurs.\\n\\nPalaeontologists now agree that birds are included in Maniraptora group of dinosaurs. This explains why we say that birds are living dinosaurs.\\n\\n Evolution of modern birds \\nA leading authority says \"Most living birds have fossil representatives in the Cenozoic\"... \"Key problems remain in understanding bird phylogeny... we seem to understand as little about the relationships among living birds as among Cretaceous birds\".\\n\\n Origin of birds\\n Paraves\\n\\nBirds and people \\n\\nSome birds are eaten as food. Most usually it is the chicken and its eggs, but people often also eat geese, pheasants, turkeys and ducks. Other birds are sometimes eaten are: emus, ostriches, pigeons, grouse, quails, doves, woodcocks and even songbirds. Some species have died out because they have been hunted for food, for example the dodo and the passenger pigeon.\\n\\nMany species have learned how to get food from people. The number of birds of these species has grown because of it. Seagulls and crows find food from garbage dumps. The feral pigeon (Columba livia), sparrows (Passer domesticus and starlings (Sturnus vulgaris) live in large numbers in towns and cities all over the world.\\n\\nSometimes people also use working birds. For example, homing pigeons carry messages. Nowadays people sometimes race them for sport. People also use falcons for hunting, and cormorants for fishing. In the past, people in mines often used a canary to see if there were bad gas methane in the air.\\n\\nPeople often have colorful birds such as parrots and mynahs as pets. These intelligent birds are popular because they can copy human talking. Because of this, some people trap birds and take them to other countries to sell. This is not usually allowed these days. Most pet birds are specially bred and are sold in pet shops.\\n\\nPeople can catch some bird diseases, for example: psittacosis, salmonellosis, campylobacteriosis, Newcastle\\'s disease, mycobacteriosis, influenza, giardiasis and cryptosporiadiosis. In 2005, there was an epidemic of bird influenza spreading through some parts of the world, often called avian flu.\\n\\nSome people have birdboxes in their gardens to give birds a place to nest and bird tables where birds can get food and water in very cold or very dry weather. This lets people see some small birds close up which are normally hidden away in bushes and trees.\\n\\nBird orders \\nThe following is a listing of all bird orders:\\n Infraclass Palaeognathae\\n Superorder Struthionimorphae\\n Struthioniformes\\n Superorder Notopalaeognathae\\n Rheiformes\\n Tinamiformes\\n Casuariiformes\\n Apterygiformes\\n Infraclass Neognathae\\n Superorder Galloanserae\\n Galliformes\\n Anseriformes\\n Superorder Neoaves\\n Phoenicopteriformes\\n Podicipediformes\\n Columbiformes\\n Mesitornithiformes\\n Pteroclidiformes\\n Apodiformes\\n Caprimulgiformes\\n Cuculiformes\\n Otidiformes\\n Musophagiformes\\n Opisthocomiformes\\n Gruiformes\\n Charadriiformes\\n Gaviiformes\\n Procellariiformes\\n Sphenisciformes\\n Ciconiiformes\\n Suliformes\\n Pelecaniformes\\n Eurypygiformes\\n Phaethontiformes\\n Cathartiformes\\n Accipitriformes\\n Strigiformes\\n Coliiformes\\n Leptosomiformes\\n Trogoniformes\\n Bucerotiformes\\n Coraciiformes\\n Piciformes\\n Cariamiformes\\n Falconiformes\\n Psittaciformes\\n Passeriformes\\n\\nBird population decreasing\\nA report produced by BirdLife International every five years measures the population of birds worldwide. One in every eight types of birds is now \"in decline\".\\n\\nReferences\\n\\nOther websites \\n\\n Avibase - The World Bird Database \\n Bird Hybrids Database - Search by bird name, use Sibley classification\\n International Ornithological Committee \\n\\nBasic English 850 words', 'vector_id': 898}}, {'_index': 'wikipedia_vector_index', '_id': '42874', '_score': 0.89821434, '_source': {'url': 'https://simple.wikipedia.org/wiki/History%20of%20the%20world', 'title': 'History of the world', 'text': 'The history of the world (also called human history) is the study of what the entire human race did in the past. It includes the time from prehistory to the present day. It is different from natural history.\\n\\nDevelopment of the human species \\n\\nModern human beings are called Homo sapiens (\\'wise man\\'). They have existed for about 250,000 years. Biologists believe that Homo sapiens evolved in Africa.\\n\\nHomo sapiens, lived at the same time as other species of human. These included Homo erectus (\\'standing man\\') and Homo neanderthalensis (\\'man from Neanderthal\\'). The theory of human evolution says that modern humans, Neanderthals, and Homo erectus slowly developed from other earlier species of human-like creatures.\\n\\nHomo neanderthalensis are the first humans scientists discovered which were not Homo sapiens. Homo neanderthalensis are usually called Neanderthal Man. They were discovered when the cranium of a skull was found in the Neanderthal Valley in 1856. It was different from a modern human skull so scientists believed it was from a new species. Entire Neanderthal skeletons have been found in other places since then. When ancient stone tools are found, their style often shows whether they were made by Homo sapiens or Neanderthals (see Palaeolithic). Neanderthals existed before modern humans. They knew how to use tools and fire.\\n\\nScientists believe that Homo sapiens spread from Africa to all other parts of the world, replacing Homo neanderthalensis in Europe and Homo erectus in Asia. By the end of the Stone Age, it is believed that Homo sapiens were the only type of humans left.\\n\\nInfluence of climate \\n\\nClimate is the normal weather in a place. It changes from one part of the world to another. Some areas are hot all year, and some are cold all year. Some areas are dry all year, and others are wet all year. Most areas have climates that are warmer in the summer and cooler in the winter. Most parts of the world get rain at some times of the year and do not get rain at other times of the year. Some parts of the world have oceanic climates and others have alpine climates.\\n\\nClimate affects what food people eat. This is because climate affects what foods can grow. If one food is easier to grow, people usually eat that food more often than other foods. Foods that people eat more of than other foods are called staple foods. Staple foods are usually grains or vegetables because they are easy to grow. Wheat, maize, millet, rice, oats, rye, potatoes, yams, breadfruit and beans are examples of different staple foods from around the world.\\n\\nClimate can affect the way people live in many other ways. It affects the types of animals that can live in any area, which affect the types of meats that are available to eat.\\nClimate also affects the buildings that people make, the clothes that they wear and the way that they travel.\\n\\nClimate change \\n\\nThe climate on earth has not stayed the same through human history. There are long periods of time when it is generally warmer, and there are long periods of time when it is generally colder. When it is generally colder, there is more ice on the poles of the planet. A cold period is called an ice age. There have been many ice ages in the history of the earth. Two have affected humans.\\n\\nFrom 70,000 to around 10,000 years ago there was a big ice age which affected humans and the way that they lived. Between 1600\\xa0AD and 1900\\xa0AD there was a period called the Little Ice Age when the climate was a little bit colder than usual.\\n\\nPrehistory \\n\\nThe word \"Prehistory\" means \"before history\". It is used for the long period of time before humans began to write about their lives. This time is divided into two main ages: the Paleolithic Age (or Early Stone Age) and the Neolithic Age (or late Stone Age). The two ages did not start and end at the same time everywhere. A place moved from one age to another depending on when people changed their technology.\\n\\nThe end of prehistory varies from one place to another. It depends on the date when that place began to use writing. In Egypt the first written documents date from around 3200\\xa0BC. In Australia the first written records date from 1788 and in New Guinea from about 1900.\\n\\nPaleolithic Era \\n\\nThe Paleolithic Era is by far the longest age of humanity\\'s time, about 99% of human history. The Paleolithic Age started about 2.6 million years ago and ended around 10,000\\xa0BC. The age began when hominids (early humans) started to use stones as tools for bashing, cutting and scraping. The age ended when humans began to plant crops and have other types of agriculture. In some areas, such as Western Europe, the way that people lived was affected by the Ice age. In these places, people moved towards agriculture quicker than in warmer places where there was always lots of food to gather. Their culture is sometimes called the Mesolithic Era (Middle Stone Age).\\n\\nDuring the Paleolithic Era humans grouped together in small bands. They lived by gathering plants and hunting wild animals. This way of living is called a \"hunter-gatherer society\". People hunted small burrowing animals like rabbits, as well as birds and herds of animals like deer and cattle. They also gathered plants to eat, including grains. Grain often grows on grasslands where herds of grass-eating animals are found. People also gathered root vegetables, green vegetables, beans, fruit, seeds, berries, nuts, eggs, insects and small reptiles.\\n\\nMany Paleolithic bands were nomadic. They moved from place to place as the weather changed. They followed herds of animals that they hunted from their winter feeding places to their summer feeding places. If there was a drought,flood, or some other disaster, the herds and the people might haved moved a long distance, looking for food. During the \"Ice Age\" a lot of the water on Earth turned to ice. This made sea much lower than it is now. People were able to walk through Beringia from Siberia to Alaska. Bands of Homo sapiens ( another word for people) travelled to that area from Asia. At that time there were rich grasslands with many large animals that are now extinct. It is believed that many groups of people travelled there over a long time and later spread to other parts of America, as the weather changed.\\n\\nPaleolithic people used stone tools. Sometimes a stone tool was just a rock. It might have been useful for smashing a shell or an animal\\'s skull, or for grinding grain on another rock. Other tools were made by breaking rocks to make a sharp edge. The next development in stone tool making was to chip all the edges of a rock so that it made a pointed shape, useful for a spearhead, or arrow tip. Some stone tools are carefully \"flaked\" at the edges to make them sharp, and symmetrically shaped. Paleolithic people also used tools of wood and bone. They probably also used leather and vegetable fibers but these have not lasted from that time. Paleolithic people also knew how to make fire which they used for warmth and cooking.\\n\\nThe Neolithic\\n\\nSettling down \\n\\nIn the Paleolithic Era there were many different human species. According to current research, only the modern human reached the Neolithic Era.\\n\\nThe Neolithic era was marked by changes in society. During the Neolithic era, people started to settle down. They developed agriculture and domesticated animals, both of which took a very long time. Because of these two things, people did not have to migrate as much any more. Villages could grow to much larger sizes than before. Over time, villages fought and spread their control over larger areas and some became civilisations. During this time, humankind also developed further intellectually, militarily and spiritually.\\n\\nWhen humans started to grow crops and domesticate certain animals such as dogs, goats, sheep, and cattle; their societies changed. Because people now grew crops and raised livestock, they started to stay in the same place and build permanent settlements. In most places, this happened between 10,000 and 12,000 years ago. Their diet also changed. People ate more cereals and vegetables. They started to keep extra foods and seeds for later. In some years there were surpluses (extras) that could be traded for other goods.\\n\\nThese changes happened independently in many parts of the world. They did not happen in the same order though. For example, the earliest farming societies in the Near East did not use pottery. No one is sure if Britain had agriculture, or if permanent villages existed there at all. Early Japanese societies used pottery before developing agriculture.\\n\\nVere Gordon Childe gave the name Neolithic Revolution to this process in the 1920s. He thought that it was as important as the Industrial Revolution (which happened in the 18th and 19th century).\\n\\nAncient history – the early civilizations \\n\\nAncient history was the time from the development of writing to the fall of the Roman Empire. The fall of the Roman Empire caused chaos in Europe, leading to the Middle Ages (also called the Dark Ages or the Age of Faith).\\n\\nThe first civilizations were built along major river systems. These civilizations are called river valley civilizations. River valley civilizations were the most powerful civilizations in this time period because water was needed to have an agricultural society.\\n\\nThese civilizations were similar in that:\\n They developed along river systems\\n They had polytheistic religions\\n They used writing systems\\n\\nMiddle East and North Africa\\n\\nSumer \\n\\nSumer was the world\\'s first known ancient civilization. The Sumerians took over the fertile crescent region of Mesopotamia around 3300 BCE. They grew crops on the Tigris and Euphrates rivers. By 3000 BCE, many cities had been built in parts of Sumerian Mesopotamia. They formed independently and each had their own government. They were called city-states and often fought with each other.\\n\\nA surplus in food led to a Division of labour. This means that some people were able to stop growing crops and do other jobs, since enough crops were already grown. This brought a split in society. Today, such a split is called social pyramid. In a social pyramid, people are grouped into social classes based on their wealth and power. In Sumer, the king, priests, and government officials were at the top of the social pyramid. Below them were the artisans, merchants, farmers, and fishers. At the bottom of the pyramid were slaves. Slaves were often prisoners of war, criminals, or people working to pay off debt.\\n\\nThe Sumerians created the world\\'s first system of writing; it was called cuneiform. The oldest versions of one of the world\\'s first literary works, the Epic of Gilgamesh, go back to this time. In Sumer, only the sons of the rich and powerful learned how to read and write. They went to a school called edubba. Only the boys who went to edubba could become scribes.\\n\\nThe Sumerians also invented sun-dried bricks, the wheel, the ox plow, and were skilled at making pottery. They are also thought to have invented the sailboat.\\n\\nAfter the Sumerians, the civilizations of Babylonia and then Assyria rose to power in Mesopotamia.\\n\\nBabylonia had a king named Hammurabi. He is famous for the Codex Hammurabi.\\n\\nJust to the east was the long-lasting civilization of Elam.\\n\\nAncient Egypt \\n\\nAncient Egypt grew along the Nile river. It was created around 3500\\xa0BC. It was most powerful in the second millennium BC. When it was its biggest, it went all the way from the Nile delta to a mountain called Jebel Barkal in Sudan. It probably ended at about 30\\xa0BC when the country was invaded by the Roman Empire.\\n\\nThe society of ancient Egypt depended on a balance of natural and human resources, especially the irrigation of the Nile Valley so that Egyptians could grow crops.\\n\\nThere was a great difference between classes in this society. Most of the people were farmers but they did not own the agricultural products they produced. These were property of the state, temple, or noble family that owned the land. There was slavery, but it is not clear how it was practiced.\\nThe Religion of Ancient Egypt encouraged people to respect their rulers and their past.\\n\\nThe Egyptians are known for writing in hieroglyphs, building the famous pyramids, and building other sorts of tombs and big temples and for their military.\\n\\nThe religion of Judaism formed about 1500 BCE around the Egyptian and Babylonian civilizations.\\n\\nMid and Eastern Asia\\n\\nAncient China \\n\\nChina began as city-states in the Yellow River valley. The Shang Dynasty (商朝) was the first dynasty of Ancient China.Turtle shells with writing on them have been carbon dated to about 1500\\xa0BC.\\n\\nThe Zhou Dynasty came after the Shang Dynasty. Kong Fuzi and Laozi lived at the end of the Zhou Dynasty. They were the greatest Chinese philosophers. They founded new philosophies, or ways of thinking. Confucius founded Confucianism and Laozi founded Daoism.\\n\\nAfter the Zhou Dynasty came the Warring States Period.\\n\\nThe Qin (秦) dynasty came after the Warring States Period. The Qin emperor Qin Shi Huang created the first centralized state in China in 221\\xa0BC. It was based on his based on his political philosophy of legalism. He made everyone write the same way. He fought against Confucianism. He also started building what would later become the Great Wall.\\n\\nIn 202\\xa0BC the Han Dynasty took over. It was about as strong as the Roman Empire. Towards the end of the Han Dynasty, Buddhism became influential in China.\\n\\nAncient India/Pakistan \\n\\nThe Indus Valley Civilization lasted from about 2600\\xa0BC to 1900\\xa0BC. It was the first urban civilization on the subcontinent. It was centered on the Indus River and its tributaries. The civilization is famous for its brick cities that had road-side drainage systems and multi-storied houses.\\n\\nThe Maurya dynasty started in 321 BCE. This was the first time most of the Indian subcontinent was united under a single government. Ashoka the Great was a famous Mauryan emperor. When he started ruling, he sought to expand his empire, but then followed a policy of ahimsa (non-violence) after converting to Buddhism. He wrote about this in the Edicts of Ashoka. The Edicts of Ashoka are the oldest historical documents from India that still exist. While Ashoka ruled, Buddhist ideals spread across all of East Asia and South-East Asia.\\n\\nThe Gupta dynasty ruled from around 320 to 550\\xa0AD. The Gupta Empire included only Central India, and the area east of current day Bangladesh. This empire never included present-day Pakistan to the west. Gupta society was ordered in accordance with Hindu beliefs. Historians place the Gupta dynasty alongside with the Han Dynasty, Tang Dynasty and Roman Empire as a model of a classical civilization.\\n\\nThe Americas\\n\\nAncient Maya \\n\\nThe Maya civilization is a civilization that started in Central America. They lived mostly on the Yucatán Peninsula in what is now known as Mexico, but also Honduras, Belize and Guatemala. They were the only known civilization of pre-Columbian America to have a fully developed written language. They also made great achievements in art and architecture and had a very advanced system of mathematics and astronomy.\\n\\nThe area where the Maya civilization developed was inhabited from around the 10th millennium BC. The first Maya settlements were built there in about 1800\\xa0BC, in the Soconusco region. This is in the modern-day state of Chiapas in Mexico, on the Pacific Ocean. Today, this is called the Early Preclassic period. At the time, humans began to settle down permanently. They started to grow livestock. Pottery and small clay figures were made. They constructed simple burial mounds. Later they developed these mounds into step pyramids. There were other civilizations around, especially in the north, such as the Olmec, the Mixe-Zoque, and Zapotec civilizations. These people mostly lived in the area of the modern-day state Oaxaca. The exact borders of the Maya empire in the north are unclear. There were probably areas where Maya culture overlapped with other cultures. Many of the earliest significant inscriptions and buildings appeared in this overlapping zone. These cultures and the Maya probably influenced one another.\\n\\nAustralia \\nThere has been a long history of contact between Papuan peoples of the Papua New Guinea and the Aboriginal people. Aboriginal people seem to have lived a long time in the same environment as the now extinct Australian megafauna. Stories about that are told in the oral culture of many Aboriginal groups.\\n\\nAncient Europe\\n\\nHallstatt culture \\n\\nThe Hallstatt era is named after the city Hallstatt in Austria, where the first artifacts were found. It lasted from about 1200\\xa0BC to about 275\\xa0BC. There were different periods, which today are mainly told apart by the kinds of brooches used at the time. These brooches changed rather rapidly, and can therefore give us good guesses at to what time they came from. Hallstatt culture sites have been found in the east of France, in Switzerland, in the south of Germany, in Austria, in Slovenia and Croatia, northwestern Hungary, southwestern Slovakia and southern Moravia. The culture can be divided into an eastern and a western one quite easily; the dividing line runs through the Czech Republic, and Austria, between longitudes 14 and 15 degrees east.\\n\\nIn this time, the social structure developed into a hierarchy. This can be documented by various things that were added to graves. In the Bronze Age, people used to live in big settlements. As iron became available, trade routes changed. A new richer class evolved. Unlike before, these richer class people liked to live in big houses in the countryside, as a demonstration of their wealth. Funerals also changed, from cremation burials, to burials with stone coffins. The new upper class used their wealth for import goods, mostly from the Mediterranean.\\n\\nLa Tène culture \\n\\nThe La Tène culture is a culture that lasted from about 500\\xa0BC to about 100\\xa0AD. It is named after the city of La Tène (today, Marin-Epagnier, next to Neuchâtel). It was influenced a lot by the Roman and Greek cultures. There are two sources for this:\\n Objects found there\\n Romans and Greeks came in contact with the culture. They called them Celts, usually. They wrote about them. The most important work about them was written by Julius Caesar. It is called On the Gallic War (De bello gallico).\\n\\nThe Celts basically lived in clans. Each clan was headed by a leader, which came from the Druids or the Bards. Women were much better off than with the Romans, they were almost equal to men. There was polygamy and polyandry (A man could have several women, a woman could have several men).\\n\\nIllyria \\n\\nIllyria is the part of west-south Balkan Peninsula populated by Illyrians whose descendants are Albanians.\\nIllyrians lived in tribunes such as Epirus, Dardania, Taulantia etc.\\nThey had their own language, the Illyrian language that was different from the Greek language and Latin.\\nAt the year 1000\\xa0BC the population of Illyria is estimated to be around 500,000.\\n\\nAncient Greece \\n\\nWhat is known today as Ancient Greece is a very important period in history. Most people agree that it came after the Minoan and Mycenaean civilizations. It ended when the Romans invaded Greece, in 146\\xa0BC. Greek culture had a very powerful influence on later civilizations, especially the Romans. The Greeks developed what is now called a city-state, or a polis. There were many polises. Some of the more important ones were Athens, Sparta, Corinth and Thebes. The word politics comes from there. It literally means: things that are about the polis. Greek cities did not have much contact with each other, because of the mountains and many islands Greece is made up of. When a city no longer had enough food to care for all its citizens, some people were sent out to set up a new city. This was called a colony. Each city was independent, and ruled by someone within that city. Colonies also looked to the city where they originally came from for guidance.\\n\\nWhen Greece went to war (for example against the Persian Empire), there was an alliance of such city states, against the Persians. There were also many wars between different city states.\\n\\nThere were many artists and philosophers who lived in that period. Most of them are still important for philosophy today. A well-known artist was Homer. He wrote epics about the war against the Trojans, and the early history of Greece. Other well-known artists were Aristophanes and Sappho. Well-known philosophers include Socrates, Plato, and Aristotle. A well known mathematician of the time was Euclid. Statesmen of the time were Pericles and Alexander the Great.\\n\\nAncient Rome \\n\\nAncient Rome was a civilization that started in modern-day Italy, in the 8th Century before Christ. The civilization lasted for 12 centuries. It ended, when Mehmed II conquered Constantinople, on May 29, 1453. According to legend, the Roman civilization was founded by Romulus and Remus, in the year 753\\xa0BC. The Roman Empire developed in wars against Carthage and the Seleucid Empire. Julius Caesar conquered Gaul, modern France, and Augustus ended the Roman republic by becoming emperor. At its biggest extent, the empire covered all of the Mediterranean. Rome became so big, because it led war against other nations and then assimilated their culture.\\n\\nSplit of the Empire into East and West \\nIn 293, Diocletian organized a separate administration of the western and the eastern part of the empire. The capital of the western part was Rome, the capital of the eastern part was Constantinople. Constantine I was the first to stop discrimination against Christians (313). Christianity became state religion under the reign of Theodosius I.\\n\\nThe western part of the empire had many problems with barbarians. In the 5th century, the Huns migrated westwards. This meant that the Visigoths moved into the empire, to seek protection. Rome was sacked by barbarians multiple times. On September 4, 476, the Germanic chief Odoacer forced the last Roman emperor in the west, Romulus Augustus, to quit. After about 1200 years, the rule of Rome in the West came to an end.\\n\\nThe eastern part had similar problems. Justinian I managed to conquer parts of North Africa and Italy. Shortly after he died, all that was left were parts of Southern Italy, and Sicily. In the east, the empire was threatened by the Sassanid Empire.\\n\\nNew departures and continuity \\nAfter the fall of Western Rome, the Germanic tribes that took over tried to learn from Roman civilization, but much was forgotten and up to the Renaissance not many achievements happened in Europe. But with the rise of Islam, many changes happened during the Islamic Golden Age. The Greek and Roman traditions were kept and further development took place. The Chinese civilization had a Golden Age during the Tang period, when their capital was the biggest in the world. During the Renaissance, Europe developed and made great advancements in many areas as well.\\n\\nAsia\\n\\nMiddle East – Islamic rise, Byzantine decline \\n\\nIn Arabia, Muhammad founded Islam in 632. His followers rapidly conquered territories in Syria and Egypt. They soon were a danger to the Byzantine Empire. In the 8th and 9th centuries, the Byzantine Empire stopped Islamic expansion and reconquered some lost territories. In 1000 A.D. the eastern Empire was at its height: Basileios II reconquered Bulgaria and Armenia. Culture and trade flourished. In 1071 the Battle of Manzikert led the empire into a dramatic decline. For the Byzantine Empire this meant centuries of civil wars and Turkic invasions. The Muslim caliphate had an Golden Age under the Abbasids.\\n\\nTheir power forced Emperor Alexius I Comnenus of the Byzantine Empire to send a call for help to the West in 1095. The West sent the Crusades. These eventually led to the Sack of Constantinople in the Fourth Crusade in 1204. Because of this, what was left of the Empire broke into successor states. The winner of these disputes was that of Nicaea. After Constantinople was again conquered by imperial forces, the empire was little more than a Greek state on the Aegean coast. The Eastern Empire came to an end when Mehmed II conquered Constantinople on May 29, 1453. The Ottoman Empire took its place and from 1400 to 1600 was the most powerful empire in the Middle East and ruled at the southern and eastern coast of the Mediterranean Sea.\\n\\nChina \\nThe Tang Dynasty (618–907), with its capital at Chang\\'an (today Xi\\'an), was the biggest city in the world at the time and is considered by historians as a high point in Chinese civilization as well as a golden age of cosmopolitan culture. The Ming Dynasty ruled from 1368 to 1644. The Ming built a vast army and navy.\\n\\nIndia \\nFrom around the 6th–7th century. In South India, Chola kings ruled Tamil Nadu, and Chera kings ruled Kerala. They had trading relationships with the Roman Empire to the west and Southeast Asia to the east. In north India, Rajputs ruled in many kingdoms.\\n\\nIn 1336, two brothers named Harihara I and Bukka founded the Vijayanagara Empire in an area which is now in the Karnataka state of India. The most famous king of this empire was Krishnadevaraya. In 1565, rulers of this empire were defeated in a battle. But the empire continued for about the next one hundred years.\\nNorthern India was ruled by Islamic sultans.\\n\\nJapan \\nThe Heian Period in Japan is famous for its art, poetry and literature. The writing system, Kana, was developed. It was followed by the feudal period (1185–1853) during which samurai and daimyos were the leading figures and the shogun the real monarch whereas the tennō had only a role as religious head. Between the years 1272 and 1281 the Mongols tried to invade but were driven out by the Japanese.\\nIn 1542, a Portuguese ship reached Japan. Japanese learned about guns and firearms from them.\\n\\nMongols \\nGenghis Khan in 1209 brought together the Mongol tribes and founded the Mongol Empire, one of the largest land empires in history. Later Kublai Khan would go on to expand the empire and found the Mongol-ruled Yuan Dynasty of China. The empire later broke into several empires, all of which were later destroyed.\\n\\nEuropean Middle Ages \\n\\nThe Middle Ages was the time from the fall of the Roman empire until the middle of the 15th century. From 500 to about 800 there was some decline compared with the Roman civilization. European villages were often destroyed and looted by barbarians such as the Vikings. During the High Middle Ages magnificent castles and large churches called cathedrals were built and important works of literature were written. In the later Middle Ages, there was a plague called the Black Death. The Black Death killed one-third to one-half of Europe\\'s population.\\n\\nA system called feudalism was a very important part of the Middle Ages. In this system, the king was at the top of the social pyramid. The king gave land to the lord in exchange for loyalty. The lords were the next in the pyramid. The lords gave land (called a fief) to knights in exchange for loyalty and protection. The knights came next in the pyramid. Peasants were not part of the feudal system because they did not give or receive land. They worked on a lord\\'s manor in exchange for protection.\\n\\nThe Crusades were also fought during the Middle Ages. There is a theory that says the Crusades helped end the Middle Ages along with the Black Death, increased trade and better farming technology.\\n\\nRenaissance \\n\\nThe Renaissance started in Italy. Renaissance is a French word meaning \"rebirth\". The Renaissance meant that people learned from the ancient Greek and Roman or \"classical\" cultures that had been forgotten for some time. Artists learned from classical paintings and sculptures. So they reinvented perspective and the art of free standing realistic sculptures that had been characteristic in Greek and Roman art. Some famous Renaissance artists are Leonardo da Vinci, Michelangelo, and Raphael. The Gutenberg printing press, invented by Johannes Gutenberg, was also developed during this time.\\n\\nThe Renaissance was also a time of great achievements in science (Galileo Galilei, Francis Bacon), philosophy (Thomas More) and literature (Dante Alighieri, William Shakespeare).\\n\\nAmerica\\n\\nMaya civilization (classical period) \\n\\nWhat is known as the classical period lasted from about 250 to about 900. During this time, many monuments were constructed. There are also many big inscriptions from then. In this period, the Maya moved to building large cities. This is known as urbanism. Many important intellectual and artistic developments happened in an area that is known as the southern lowlands.\\n\\nLike the Ancient Greek, the Maya civilization was made of many independent city-states. Agriculture was important around these city states like Tikal and Copán.\\nThe most important monuments are the pyramids they built in their religious centers and the palaces of their rulers. The palace at Cancuén is the largest in the Maya area. There are no pyramids in the area of the palace. Other important things the archaeologists found include the carved stone slabs usually called stelae (the Maya called them tetun, or \"tree-stones\"). These slabs show rulers along with hieroglyphic texts describing their genealogy, military victories, and other accomplishments. In North America, they made Mississipian culture with the largest land field from around 800 CE to 1600.\\n\\nTrade with other civilizations \\nThe Maya also had trade routes that ran over long distances. They traded with many of the other Mesoamerican cultures, such as Teotihuacan, the Zapotec, and other groups in central and gulf-coast Mexico. They also traded with non-Mesoamerican groups, that were farther away. Archaeologists have found gold from Panama in the Sacred Cenote of Chichen Itza.\\n\\nImportant trade goods were cacao, salt, sea shells, jade and obsidian.\\n\\nSudden collapse \\nIn the 8th and 9th century, the cities in the southern lowlands had problems, and declined. At the same time, the Maya stopped making big monuments and inscriptions. Shortly afterwards, these cities were abandoned. Currently, archaeologists are not sure why this happened. There are different theories. Either ecological factors played a role in this, or the cause of this abandonment was not related to the environment.\\n\\nPost-classical period and decline \\n\\nIn the north, development went on, form the 10th to about the 16th century. The influences from the outside left more traces in the Maya culture at that time. Some of the important sites in this era were Chichen Itza, Uxmal, and Coba. At some point, the ruling dynasties of Chichen and Uxmal declined. Afterwards, Mayapan ruled all of Yucatán until a revolt in 1450. The area then degenerated into competing city-states until the Yucatán was conquered by the Spanish.\\n\\nBy 1250, there developed other city-states. The Itza maintained their capital at Tayasal. It ruled over an area extending across the Peten Lakes region, including the community of Ekckixil on Lake Quexil. Postclassic Maya states also survived in the southern highlands. One of the Maya kingdoms in this area is responsible for the best-known Maya work of historiography and mythology, the Popol Vuh.\\n\\nThe Spanish started to conquer Maya lands. This took them much longer than with the Inca or Aztecs, because there was no capital city. This meant that when they had conquered one city, this had little influence on the whole empire. The last Maya states were finally subdued in 1697.\\n\\nThe Maya people did not disappear though. There are still about 6 million of them. Some are well-integrated, others continue speak one of the Maya languages and uphold their cultural heritage.\\n\\nThe Aztecs \\n\\nThe Aztecs built an empire in Central America, mainly in Mexico. The empire lasted from the 14th to the 16th century. They spoke the Nahuatl language. Their capital was Tenochtitlan. It was built on islands in a lake. Tenochtitlan was one of the greatest cities of the world in that time.\\n\\nThe Aztecs believed in polytheism. Quetzalcoatl (feathered snake), Huitzilopochtli (hummingbird of the south) and Tezcatlipoca (smoking mirror) were the most important Gods. Sometimes the Aztecs killed humans to please their gods. Between 1519 and 1521 the Spanish leader Hernán Cortés defeated the Aztecs and took their empire. Some Aztecs did not want to fight against the soldiers of Cortés, because they thought they were Gods.\\n\\nToday many Mexicans have Aztec and other Native American forefathers. People still use Aztec symbols in Mexico. On the Mexican flag there is a picture of an eagle on a cactus with a snake in its mouth. This was an Aztec symbol. Also the name Mexico is an Aztec word.\\n\\nThe Aztecs ate a lot of plants and vegetables that could be grown easily in the Mexico area. The main food that they ate was corn, which they called maize. Another food that they ate was squash.\\n\\nAztecs also had a lot of harsh punishments for certain crimes. For the following crimes the punishment was death: adultery, wearing cotton clothes (cotton clothes were only for the nobles), cutting down a living tree, moving a field boundary making your land bigger, making someone else\\'s smaller, major theft and treason.\\n\\nThe Incas \\n\\nThe Incas were a civilized empire in western South America. The Incas are called a \"pre-Columbian\" empire. This means that their country was here before Christopher Columbus. They ruled parts of South America around what is now Peru for a little over 100 years, until the Spanish invasion in the 16th century.\\n\\nThe Incan empire or , meaning four regions in Quechua, only lasted for about 100 years as the arrival of the Spaniards in 1532 conquered them. Their main language was Quechua, but as the Incas were basically made up of many different groups there were probably many other different languages.\\n\\nTheir capital was in the city of Cusco, or Qosqo, in what is now southern Peru.\\n\\nManco Capac founded the first Inca state around 1200. It covered the area around Cusco. In the 1400s, Pachacuti began to absorb other people in the Andes. The expansion of the Inca Empire had started. The Inca Empire would become the biggest empire in the Americas before Columbus.\\n\\nIn 1532, the civil war ended. The brothers Huascar and Atahualpa, fought for who would succeed their father. During this time, the Spanish conquerors took possession of the Inca territory. They were led by Francisco Pizarro. In the following years the conquistadors managed to extend their power over the whole Andean region. They suppressed successive Inca rebellions until the establishment of the Viceroyalty of Perú in 1542 and the fall of the resistance of the last Incas of Vilcabamba in 1572. The Inca civilization ends at that time, but many cultural traditions remain in some ethnic groups as Quechuas and Aymara people.\\n\\nAfrica \\n\\nAncient Egypt and Carthage are well known civilizations of ancient Africa. But because there are not many written sources in large parts of Sub-Saharan Africa, the history of Africa is not easy to write about. But with new techniques such as the recording of oral history, historical linguistics and archeology knowledge has improved, not only for the empires and kingdoms of Ethiopia, Ghana, Mali, Nubia, Kush and Kerma.\\n\\nGlobalization\\n\\nFrom colonialization to imperialism\\n\\nThe rise of Europe\\n\\nColonization \\n\\nColonization happened after Christopher Columbus came to the Americas. European countries such as England, France, and Spain built colonies in the Americas. These settlers fought the Native Americans to take over their land. The colonisation of the Americas was the beginning of modern times.\\n\\nAn important part about contact with the Americas was the Columbian Exchange The Columbian Exchange brought new foods, ideas, and diseases to the Old World and New World, changing the way people lived. Historians believe that almost everyone as far as Asia was affected in some way by the Columbian Exchange.\\n\\nReformation and Counter-Reformation \\nProtestant Reformation started with Martin Luther and the posting of the 95 theses on the door of the castle church in Wittenberg, Germany. At first he protested against corruption such as simony or the sale of indulgences. But then it became clear that he had different ideas about the church doctrine. He thought that Christians should only read the Bible to find out what God wants from them. That meant that they did not need priests (see: Five solas). The three most important traditions that came directly from the Protestant Reformation were the Lutheran, Reformed (Calvinist, Presbyterian, etc.), and Anglican traditions.\\n\\nThe Counter-Reformation, or Catholic Reformation, was the Catholic Church fighting the Protestant Reformation. New religious orders, such as the Jesuits were founded and missionaries sent around the world. Decisions were taken at the Council of Trent (1545–1563).\\n\\nIndustrial revolution \\nThe Industrial Revolution started in Great Britain. It brought many advances in the way goods were produced. These advances allowed people to produce much more than they needed for living. The early British Empire split as its colonies in America revolted to establish a representative government.\\n\\nFrom nationalism to imperialism \\nThe French Revolution lead to massive political change in continental Europe, as people following the ideas of Enlightenment asked for human rights with the slogan liberté, egalité, fraternité (liberty, equality, fraternity). That led to the Declaration of the Rights of Man and of the Citizen, but also to terror and the execution of King Louis XVI. The French leader, Napoleon Bonaparte, conquered and changed Europe through war up to 1815. As more and more small property holders were granted the vote, in France and the UK, socialist and trade union activity developed and revolution gripped Europe in 1848. The last vestiges of serfdom were abolished in Austria-Hungary in 1848. Russian serfdom was abolished in 1861. The Balkan nations began to regain their independence from the Ottoman Empire. After the Franco-Prussian War, Italy and Germany became unified in 1870 and 1871. Conflict spread across the globe, in a chase for empires. The search for a \"place in the sun\" ended with the outbreak of World War I. In the desperation of war, the Russian Revolution promised the people \"peace, bread and land\". The defeat of Germany came at the price of economic destruction, which was written down in the Treaty of Versailles.\\n\\nAsia\\n\\nChina – continuity \\nFrom 1644 to 1912 the Qing or Manchu Dynasty ruled China. The dynasty was founded by the Manchu clan in northeast China (Manchuria). It expanded into China proper and its surrounding territories, establishing the Empire of the Great Qing.\\nIts military power weakened during the 1800s, and faced with international pressure, massive rebellions and defeats in wars, the Qing Dynasty declined after the mid-19th century. It was overthrown in 1912.\\n\\nJapan \\nDuring the Edo period, Japan had many small rulers. There were about 200 of them, called the daimyo. Out of them, the Tokugawa clan was most powerful. They ruled from a place called Edo. This place was around the present day’s Tokyo. For fifteen generations they were the most powerful clan in Japan.\\n\\nBeginning from the early 17th century, the rulers (known as shogunate) started a policy of seclusion (stopping some people coming in), known as sakoku in Japanese language. They suspected that traders, merchants and missionaries wanted to bring Japan under the control of European powers. Except the Dutch and the Chinese, all foreigners, traders and merchants from other countries, missionaries were no longer allowed into Japan.\\n\\nStill even during the period of seclusion, Japanese continued to gain information and knowledge about other parts of the world.\\nThis policy of seclusion lasted for about 200 years. It ended 1868 with Meiji Restoration, when the emperor took over again and started a lot of reforms.\\n\\nIndia – Mughal Empire \\n\\nThe Mughal Empire existed from 1526 to 1857. When it was biggest it ruled most of the Indian subcontinent, then known as Hindustan, and parts of what is now Afghanistan. It was founded by Babur in 1526 and ruled until 1530. Its most important ruler was Akbar (1556–1605). After the death of Aurangjeb (1658–1707), the Mughal Empire became weak. It continued until 1857. By that time, India came under the British Raj.\\n\\nAmerica \\n\\nSettlement by the Spanish started the European colonization of the Americas, it meant genocide of the native Indians. The Spanish gained control of most of the Caribbean and conquered the Aztecs. So they founded the Spanish Empire in the New World.\\n\\nThe first successful English settlements were in North America at Jamestown (Virginia), 1607 (along with its satellite, Bermuda in 1609) and Plymouth (Massachusetts), 1620. The first French settlements were Port Royal (1604) and Quebec City (1608). The Fur Trade soon became the primary business on the continent and as a result transformed the Native Americans lifestyle. Plantation slavery of the West Indies lead to the beginning of the Atlantic slave trade.\\n\\nRivalry between the European powers created a series of wars on the North American landmass. The American Revolution led to the creation of the United States of America. Spain\\'s hold on its colonies weakened till it had to give them independence.\\n\\nThe United States expanded quickly to the west. At the same time, British built more in Canada.\\n\\nAfrica \\nDuring the 15th century the Portuguese began exploring Africa. At the Guinea coast they built their first fort in 1482. They started slave trade after the first European contact with America in 1492 to supply settlers from there with workers. Soon English, Spanish, Dutch, French and Danish merchants also built forts. But their influence on the inland was minor (except from decimation of population by slave trade) till during the 19th century larger colonies were founded.\\n\\nTwentieth Century onward \\n\\nThe 20th century was a very important time in history. New technology and different ideas led to many worldwide changes in the time of just 100 years.\\n\\nWorld Wars\\n\\nThe First World War \\n\\nWorld War I was a war fought from 1914 to 1918. During the time of the war, it was called \"The Great War\", or \"The War to End All Wars\". Chemical poisons, tanks, aeroplanes, and bombs were used for the first time.\\n\\nThere were four main causes of the war:\\n Imperialism\\n Nationalism\\n Alliances\\n Militarism\\n\\nThese were causes that made it likely that a war would start in Europe. The \"spark\" that started the war was the assassination of the heir to the throne in Austria-Hungary: Archduke Franz Ferdinand by a group of young Serbians. Austria-Hungary declared war on Serbia and each country\\'s allies then joined the war. This created a bigger conflict which turned into World War I.\\n\\nEurope divided into two groups of allies: the Central Powers and the Allied Powers (the \"Allies\"). The Central Powers were made up of Germany, Austria-Hungary, the Ottoman Empire and Bulgaria. The Allies were made up of Britain, France, Russia, Italy and the United States.\\n\\nWorld War I was fought on two fronts; the Eastern Front and the Western Front. Trench warfare was commonly used on the Eastern Front.\\n\\nBecause of a British blockade, Germany began using U-boats, or submarines, to sink British ships. After the sinking of two ships with Americans on board, and the public release of the Zimmermann Note, The U.S. declared war on Germany, joining the Allies.\\n\\nOn November 11, 1918, Germany signed the armistice, meaning \"the laying down of arms\", to end the war. After the war ended, the Treaty of Versailles was written and Germany was made to sign it. They had to pay $33 million in reparations (payment for damage). The influenza pandemic of 1918 spread around the world, killing millions.\\n\\nAfter the First War \\nAfter the war the German Empire, the Russian Empire, the Ottoman Empire and Austrian Empire ended and France and Britain got weaker.\\nThe 1920s and 1930s had military-related fascist dictators take control of Italy, Germany, Japan and Spain. They were helped by the Great Depression starting in 1929. When Hitler in 1933 had gained power in Germany he prepared World War II.\\n\\nThe Second World War \\n\\nOf all the wars ever fought, World War II involved the most countries and killed the most people. More than 60 million people died, making it the worst disaster of all time. It lasted six years in Europe, from 1939 to 1945.\\nIt was fought between the Axis Powers (Germany, Italy and Japan) and the Allied Powers. At first the Axis Powers were successful, but that ended in Europe with the Battle of Stalingrad in 1943 and the invasion in Normandy in 1944. But Hitler was able to pursue his plan to annihilate Jews nearly all over Europe. Today, this plan is called the Holocaust.\\nIn the Pacific it ended with the battles of Midway and Guadalcanal. Germany surrendered on May 8. The Soviet invasion of Japan led Japan to surrender on August 15, 1945.\\n\\nAfter World War II \\nAfter World War II the United Nations was founded in the hope that it could solve arguments among nations and keep wars from happening. Communism spread to Central and Eastern Europe, Yugoslavia, Bulgaria, Romania, Albania, North Vietnam and North Korea. In 1949, China became communist. During the 1950s and 1960s, many third world countries became communist.\\n\\nThis led to the Cold War, a forty-year argument between the United States, the Soviet Union, and their allies (mainly countries that were members of NATO or the Warsaw Pact). Each country wanted to promote their type of government. The Soviet Union wanted to spread communism, and the United States wanted to spread democracy. People across the world feared a nuclear war because of the tension.\\n\\nCommunism became less popular when it became clear that it could not promote economic growth as well as Western states and that it was not suited for a reform that allowed freedom of speech for everybody. Therefore, the Soviet Union forced Hungary to give up its reform in 1956, it favored the building of the Berlin Wall in 1961 and it stopped reform in Czechoslovakia in 1968. When in 1988/89 Gorbachev made clear that he would not force the countries of the East block to stick to Communism the Berlin Wall was torn down in 1989 and the Soviet Union collapsed (1991). Then the United States was the only superpower left.\\n\\nAfter Mao Zedong\\'s death China\\'s communist party proved that economic reform was possible without political freedom and paved the way for enormous economic growth.\\n\\nAs the 20th century ended, the European Union began to rise and included former satellite states and even parts of the Soviet Union. States in Asia, Africa and South America tried to copy the European Union.\\n\\nThe twentieth century was a time of great progress in terms of technology. People began to live longer because of better medicine and medical technology. New communications and transportation technologies connected the world. But these advances also helped cause problems with the environment.\\n\\nThe last half of the century had smaller wars. Improved information technology and globalization increased trade and cultural exchange. Space exploration expanded through the solar system. The structure of DNA was discovered.\\n\\nThe same period also raised questions about the end of human history because of global dangers: nuclear weapons, greenhouse effect and other problems in the environment.\\n\\n21st century \\n\\nAs the 20th century ended, globalization has continued. During this period, communications with mobile phones and the Internet have expanded, which has led to fundamental social changes in corporation, political, and individuals\\' personal lives. Due to the population of growth and industrialization, worldwide resource competition is becoming increasingly highly, especially in India, China and Brazil. The increasing demand on the environmental degradation and global warming.\\n\\nA new Great Recession affected the world in the late 2000s and the early 2010s, and the COVID-19 pandemic spread in 2020, causing further economic and political disruption. Some scientists referred to this as a \"Planetary Phase of Civilization\".\\n\\nRelated pages \\n History of Africa\\n History of America\\n History of Asia\\n History of Australia\\n History of Europe\\n History of the Earth\\n\\nReferences\\n\\nFurther reading \\n \\nEnglish translation by Paul G. Bahn from the French edition La Grotte Chauvet\\n \\n \\nTranslation of La Grotte Chauvet, l\\'art des origins, Éditions du Seuil, 2001\\n\\nOther websites \\n Universal Concise History of the World, 1832 Full text, free to read, American book on the history of the world with the intriguing perspective of 1832 America.\\n WWW-VL: World History at European University Institute\\n Five Epochs of Civilization A scheme of organization which divides world history into five epochs marked by changes in communication technology\\n World history -Citizendium\\n\\n+\\nFormer good articles', 'vector_id': 11672}}]}}\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
"/var/folders/vz/v2f6_x6s0kg51j2vbm5rlhww0000gn/T/ipykernel_27978/2105931364.py:1: DeprecationWarning: The 'body' parameter is deprecated and will be removed in a future version. Instead use individual parameters.\n",
" print(client.search(index=\"wikipedia_vector_index\", body={\n"
]
}
],
"source": [
"print(client.search(index=\"wikipedia_vector_index\", body={\n",
" \"_source\": {\n",
" \"excludes\": [\"title_vector\", \"content_vector\"]\n",
" },\n",
" \"query\": {\n",
" \"match\": {\n",
" \"text\": {\n",
" \"query\": \"Hummingbird\"\n",
" }\n",
" }\n",
" }\n",
"}))"
]
},
{
"cell_type": "markdown",
"id": "992b6804",
"metadata": {},
"source": [
"## Encode a question with OpenAI embedding model\n",
"\n",
"To perform kNN search, we need to encode queries with the same embedding model used to encode the documents at index time.\n",
"In this example, we need to use the `text-embedding-3-small` model.\n",
"\n",
"You'll need your OpenAI [API key](https://platform.openai.com/account/api-keys) to generate the embeddings."
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "57385c69",
"metadata": {},
"outputs": [],
"source": [
"# Get OpenAI API key\n",
"OPENAI_API_KEY = getpass(\"Enter OpenAI API key\")\n",
"\n",
"# Set API key\n",
"openai.api_key = OPENAI_API_KEY\n",
"\n",
"# Define model\n",
"EMBEDDING_MODEL = \"text-embedding-3-small\"\n",
"\n",
"# Define question\n",
"question = 'Is the Atlantic the biggest ocean in the world?'\n",
"\n",
"# Create embedding\n",
"question_embedding = openai.Embedding.create(input=question, model=EMBEDDING_MODEL)\n"
]
},
{
"cell_type": "markdown",
"id": "c7e6bf5d",
"metadata": {},
"source": [
"## Run semantic search queries\n",
"\n",
"Now we're ready to run queries against our Elasticsearch index using our encoded question. We'll be doing a k-nearest neighbors search, using the Elasticsearch [kNN query](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html) option.\n",
"\n",
"First, we define a small function to pretty print the results."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "b1291434",
"metadata": {},
"outputs": [],
"source": [
"# Function to pretty print Elasticsearch results\n",
"\n",
"def pretty_response(response):\n",
" for hit in response['hits']['hits']:\n",
" id = hit['_id']\n",
" score = hit['_score']\n",
" title = hit['_source']['title']\n",
" text = hit['_source']['text']\n",
" pretty_output = (f\"\\nID: {id}\\nTitle: {title}\\nSummary: {text}\\nScore: {score}\")\n",
" print(pretty_output)"
]
},
{
"cell_type": "markdown",
"id": "ed8a497f",
"metadata": {},
"source": [
"Now let's run our `kNN` query."
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "fc834fdd",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"ID: 1936\n",
"Title: Atlantic Ocean\n",
"Summary: The Atlantic Ocean is the world's second largest ocean. It covers a total area of about . It covers about 20 percent of the Earth's surface. It is named after the god Atlas from Greek mythology.\n",
"\n",
"Geologic history \n",
"The Atlantic formed when the Americas moved west from Eurasia and Africa. This began sometime in the Cretaceous period, roughly 135 million years ago. It was part of the break-up of the supercontinent Pangaea.\n",
"\n",
"The east coast of South America is shaped somewhat like the west coast of Africa, and this gave a clue that continents moved over long periods of time (continental drift). The Atlantic Ocean is still growing now, because of sea-floor spreading from the mid-Atlantic Ridge, while the Pacific Ocean is said to be shrinking because the sea floor is folding under itself or subducting into the mantle.\n",
"\n",
"Geography\n",
"The Atlantic Ocean is bounded on the west by North and South America. It connects to the Arctic Ocean through the Denmark Strait, Greenland Sea, Norwegian Sea and Barents Sea. It connects with the Mediterranean Sea through the Strait of Gibraltar.\n",
"\n",
"In the southeast, the Atlantic merges into the Indian Ocean. The 20° East meridian defines its border.\n",
"\n",
"In the southwest, the Drake Passage connects it to the Pacific Ocean. The Panama Canal links the Atlantic and Pacific.\n",
"\n",
"The Atlantic Ocean is second in size to the Pacific. It occupies an area of about . The volume of the Atlantic, along with its adjacent seas (the seas next to it), is 354,700,000 cubic kilometres.\n",
"\n",
"The average depth of the Atlantic, along with its adjacent seas, is . The greatest depth is Milwaukee Deep near Puerto Rico, where the Ocean is deep.\n",
"\n",
"Gulf Stream \n",
"The Atlantic Ocean has important ocean currents. One of these, called the Gulf Stream, flows across the North Atlantic. Water gets heated by the sun in the Caribbean Sea and then moves northwest toward the North Pole. This makes France, the British Isles, Iceland, and Norway in Europe much warmer in winter than Newfoundland and Nova Scotia in Canada. Without the Gulf Stream, the climates of northeast Canada and northwest Europe might be the same, because these places are about the same distance from the North Pole.\n",
"\n",
"There are currents in the South Atlantic too, but the shape of this sea means that it has less effect on South Africa.\n",
"\n",
"Geology \n",
"The main feature of the Atlantic Ocean's seabed is a large underwater mountain chain called the Mid-Atlantic Ridge. It runs from north to south under the Ocean. This is at the boundary of four tectonic plates: Eurasian, North American, South American and African. The ridge extends from Iceland in the north to about 58° south.\n",
"\n",
"The salinity of the surface waters of the open ocean ranges from 3337 parts per thousand and varies with latitude and season.\n",
"\n",
"References\n",
"\n",
"Other websites \n",
"LA Times special Altered Oceans \n",
"Oceanography Image of the Day, from the Woods Hole Oceanographic Institution\n",
"National Oceanic and Atmospheric Administration \n",
"NOAA In-situ Ocean Data Viewer Plot and download ocean observations\n",
"\n",
"www.cartage.org.lb \n",
"www.mnsu.edu\n",
"Score: 0.93641126\n",
"\n",
"ID: 1975\n",
"Title: Pacific Ocean\n",
"Summary: The Pacific Ocean is the body of water between Asia and Australia in the west, the Americas in the east, the Southern Ocean to the south, and the Arctic Ocean to the north. It is the largest named ocean and it covers one-third of the surface of the entire world. It joins the Atlantic Ocean at a line drawn south from Cape Horn, Chile/Argentina to Antarctica, and joins the Indian Ocean at a line drawn south from Tasmania, Australia to Antarctica.\n",
"\n",
"As the Atlantic slowly gets wider, the Pacific is slowly shrinking. It does this by folding the sea floor in towards the centre of the Earth - this is called subduction. This bumping and grinding is hard so there are many earthquakes and volcanoes when the pressure builds up and is quickly released as large explosions of hot rocks and dust. When an earthquake happens under the sea, the quick jerk causes a tsunami. This is why tsunamis are more common around the edge of the Pacific than anywhere else. Many of the Earth's volcanoes are either islands in the Pacific, or are on continents within a few hundred kilometers of the ocean's edge. Plate tectonics are another reason which makes Pacific Ocean smaller.\n",
"\n",
"Other websites \n",
"\n",
" EPIC Pacific Ocean Data Collection Viewable on-line collection of observational data\n",
" NOAA In-situ Ocean Data Viewer plot and download ocean observations\n",
" NOAA PMEL Argo profiling floats Realtime Pacific Ocean data\n",
" NOAA TAO El Niño data Realtime Pacific Ocean El Niño buoy data\n",
" NOAA Ocean Surface Current Analyses – Realtime (OSCAR) Near-realtime Pacific Ocean Surface Currents derived from satellite altimeter and scatterometer data\n",
"Score: 0.9177895\n",
"\n",
"ID: 11124\n",
"Title: List of seas\n",
"Summary: The sea is the interconnected system of all the Earth's oceanic waters, including the Atlantic, Pacific, Indian, Southern and Arctic Oceans. However, the word \"sea\" can also be used for many specific, much smaller bodies of seawater, such as the North Sea or the Red Sea.There are 78 seas in the world\n",
"\n",
"List of seas, by ocean\n",
"\n",
"Pacific Ocean \n",
" Bering Sea\n",
" Gulf of Alaska\n",
" Seck Sea (Gulf of California)\n",
" Sea of Okhotsk\n",
" Sea of Japan\n",
" Seto Inland Sea\n",
" East China Sea\n",
" South China Sea\n",
" Beibu Gulf\n",
" Sulu Sea\n",
" Celebes Sea\n",
" Bohol Sea (Mindanao Sea)\n",
" Philippine Sea\n",
" Flores Sea\n",
" Banda Sea\n",
" Arafura Sea\n",
" Tasman Sea\n",
" Yellow Sea\n",
" Bohai Sea\n",
" Coral Sea\n",
" Gulf of Carpentaria\n",
"\n",
"Atlantic Ocean \n",
" Hudson Bay\n",
" James Bay\n",
" Baffin Bay init fam\n",
" Gulf of St. Lawrence\n",
" Gulf of Guinea\n",
" Caribbean Sea\n",
" Gulf of Mexico\n",
" Sargasso Sea\n",
" North Sea\n",
" Baltic Sea\n",
" Gulf of Bothnia\n",
" Irish Sea\n",
" Celtic Sea\n",
" English Channel\n",
" Mediterranean Sea\n",
" Adriatic Sea\n",
" Aegean Sea\n",
" Black Sea\n",
" Sea of Azov\n",
" Ionian Sea\n",
" Ligurian Sea\n",
" Mirtoon Sea\n",
" Tyrrhenian Sea\n",
" Gulf of Sidra\n",
" Sea of Marmara\n",
" Sea of Crete\n",
"\n",
"Indian Ocean \n",
" Red Sea\n",
" Gulf of Aden\n",
" Persian Gulf\n",
" Gulf of Oman\n",
" Arabian Sea\n",
" Bay of Bengal\n",
" Gulf of Thailand\n",
" Java Sea\n",
" Timor Sea\n",
" Gulf of Kutch\n",
" Gulf of Khambhat\n",
"\n",
"Arctic Ocean \n",
" Barents Sea\n",
" Kara Sea\n",
" Beaufort Sea\n",
" Amundsen Gulf\n",
" Greenland Sea\n",
" Chukchi Sea\n",
" Laptev Sea\n",
" East Siberian Sea\n",
"\n",
"Southern Ocean \n",
" Amundsen Sea\n",
" Weddell Sea\n",
" Ross Sea\n",
" Great Australian Bight\n",
" Gulf St. Vincent\n",
" Spencer Gulf\n",
"\n",
"Seas which have land around them (these are landlocked) \n",
" Aral Sea\n",
" Caspian Sea\n",
" Dead Sea\n",
" Sea of Galilee (we call this a sea, but it is really a small freshwater lake)\n",
" Salton Sea\n",
"\n",
"Seas which are not on Earth \n",
"Lunar maria are very big areas on the Moon. In the past, people thought they were water and called them \"seas\". \n",
"\n",
"Scientists think that there is liquid water under the ground on some moons, for example Europa.\n",
"\n",
"Scientists also think that there are liquid hydrocarbons on Titan. \n",
"\n",
"Basic English 850 words\n",
"\n",
"Geography-related lists\n",
"Score: 0.9160284\n",
"\n",
"ID: 2033\n",
"Title: Southern Ocean\n",
"Summary: The Southern Ocean is the ocean around Antarctica. It means the waters of the Atlantic, Pacific, and Indian Oceans around the continent of Antarctica. Since the 1770s geographers have discussed its limits. Nowadays, sixty degrees south latitude is often accepted. Some people call this ocean the Antarctic Ocean.\n",
"\n",
"The total area is 20,327,000 km², and the coastline length is 17,968 km.\n",
"\n",
"Other websites \n",
"\n",
" Oceanography Image of the Day, from the Woods Hole Oceanographic Institution\n",
" The CIA World Factbook's entry on the Southern Ocean\n",
" The Fifth Ocean from Geography.About.com\n",
" NOAA In-situ Ocean Data Viewer Plot and download ocean observations\n",
" NOAA FAQ about the number of oceans \n",
"\n",
" \n",
"Geography of Antarctica\n",
"Score: 0.9083342\n",
"\n",
"ID: 1978\n",
"Title: Indian Ocean\n",
"Summary: The Indian Ocean is the ocean surrounded by Asia to the north, Australia and the Pacific Ocean to the east, the Southern Ocean to the south, and Africa and the Atlantic Ocean to the west. It is named for the river Indus and Ancient India on its north shore. The Bay of Bengal, the Arabian Sea, the Persian Gulf and the Red Sea are all parts of this ocean.\n",
"\n",
"The deepest point in the Indian Ocean is in the Java Trench near the Sunda Islands in the east, 7500 m (25,344 feet) deep. The average depth is 3,890 m (12,762 ft). The Indian Ocean is the third largest ocean, 28,350,000 square miles in size. The majority is in the southern hemisphere.\n",
"\n",
"Other websites \n",
"\n",
" Maps of the indian Ocean\n",
" Océan Indien in easy French\n",
" NOAA In-situ Ocean Data Viewer Plot and download ocean observations\n",
" The Indian Ocean in World History: Educational Website Interactive resource from the Sultan Qaboos Cultural Center\n",
" The Regional Tuna Tagging Project-Indian Ocean with details of the importance of Tuna in the Indian Ocean.. \n",
" Detailed maps of the Indian Ocean\n",
" The Indian Ocean Trade: A Classroom Simulation \n",
" CIA - The World Factbook, Oceans: Indian Ocean\n",
"Score: 0.90738976\n",
"\n",
"ID: 1980\n",
"Title: Arctic Ocean\n",
"Summary: The Arctic Ocean is the ocean around the North Pole. The most northern parts of Eurasia and North America are around the Arctic Ocean. Thick pack ice and snow cover almost all of this ocean in winter, and most of it in summer. An icebreaker or a nuclear-powered submarine can use the Northwest Passage through the Arctic Ocean to go between the Pacific and Atlantic oceans.\n",
"\n",
"The ocean's area is about 14.056 million km2, which is the smallest of the world's 5 oceans, and it has of coastline. The central surface covered by ice about thick. The biology there is quite special. Endangered species there include walruses, whales and polar bears. Year by year the Arctic Ocean is becoming less icy, as a result of global warming.\n",
"\n",
"The average depth of the Arctic Ocean is . The deepest point is in the Eurasian Basin, at .\n",
"\n",
"Geography \n",
"The Arctic Ocean covers an area of about 14,056,000 km2. The coastline is 45,390 km (28,200 mi) long It is surrounded by Eurasia, North America, Greenland, and by several islands.\n",
"\n",
"It is generally taken to include Baffin Bay, Barents Sea, Beaufort Sea, Chukchi Sea, East Siberian Sea, Greenland Sea, Hudson Bay, Hudson Strait, Kara Sea, Laptev Sea, White Sea and other bodies of water. It is connected to the Pacific Ocean by the Bering Strait and to the Atlantic Ocean through the Greenland Sea and Labrador Sea.\n",
"\n",
"Countries bordering the Arctic Ocean are: Russia, Norway, Iceland, Greenland, Canada and the United States.\n",
"\n",
"Climate \n",
"The Arctic Ocean is in a polar climate. Winters are characterized by the polar night, cold and stable weather conditions, and clear skies.\n",
"\n",
"The temperature of the surface of the Arctic Ocean is fairly constant, near the freezing point of seawater. Arctic Ocean consists of saltwater but its salinity is less than other oceans. The temperature must reach −1.8 °C (28.8 °F) before freezing occurs.\n",
"\n",
"Ice covers most of the Arctic Ocean. It covers almost the whole ocean in late winter and the majority of the ocean in late summer. Much of the Arctic ice pack is covered in snow for about 10 months of the year. The maximum snow cover is in March or April — about 20 to 50 cm (7.9 to 19.7 in).\n",
"\n",
"The climate of the Arctic region has varied significantly in the past. As recently as 55 million years ago, during the eocene epoch, the region reached an average annual temperature of 10–20 °C (50–68 °F). The surface waters of the Arctic Ocean warmed enough to support tropical lifeforms.\n",
"\n",
"Animal and plant life \n",
"Endangered marine species in the Arctic Ocean include walruses and whales. The area has a fragile ecosystem. The Arctic Ocean has relatively little plant life except for phytoplankton. Phytoplankton are a crucial part of the ocean. They feed on nutrients from rivers and the currents of the Atlantic and Pacific oceans.\n",
"\n",
"References\n",
"\n",
"Other websites \n",
"\n",
" The Hidden Ocean Arctic 2005 Daily logs, photos and video from exploration mission.\n",
" Oceanography Image of the Day, from the Woods Hole Oceanographic Institution\n",
" Arctic Council\n",
" The Northern Forum\n",
" Arctic Environmental Atlas Interactive map\n",
" NOAA Arctic Theme Page\n",
" \n",
" Daily Arctic Ocean Rawinsonde Data from Soviet Drifting Ice Stations (1954–1990) at NSIDC\n",
" Arctic time series: The Unaami Data collection \n",
" NOAA North Pole Web Cam Images from Web Cams deployed in spring on an ice floe\n",
" NOAA Near-realtime North Pole Weather Data Data from instruments deployed on an ice floe\n",
" Search for Arctic Life Heats Up by Stephen Leahy\n",
" International Polar Foundation\n",
" National Snow and Ice Data Center – Daily report of Arctic ice cover based on satellite data\n",
" Marine Biodiversity Wiki \n",
"\n",
"Oceans\n",
"Arctic\n",
"Score: 0.9073119\n",
"\n",
"ID: 15220\n",
"Title: Caribbean Sea\n",
"Summary: The Caribbean Sea is a tropical sea in the center of the Caribbean area. The body of water is part of the Atlantic Ocean. The sea is southeast of the Gulf of Mexico. The Caribbean Sea has many islands, which are popular among North American tourists because of their tropical climate. The Caribbean Sea is famous around the world as a tourist destination.\n",
"\n",
"History \n",
"Christopher Columbus came across a group of islands in the Caribbean region. When he did so, he thought he had reached another part of the world. Because of this, he named the islands the ‘West Indies’. However, later it was realized that he found an entire region. It still had its natural resources. The name ‘Caribbean’ was later given to it by the Amerindian tribe, the Caribs. That is how it got its name: the Caribbean Sea.\n",
"\n",
"This entire region covers an area of 1,063,000 sq. miles. It covers from Mexico to the boundaries of South America.\n",
"\n",
"This sea is just as deep as it is wide. Its deepest point is believed to be even lower than 25,220 ft, 7,686 m. That makes this point one of the lowest points on the surface of the earth, and the Caribbean Sea one of the deepest seas in the world.\n",
"\n",
"Other websites \n",
"\n",
"Seas of the Atlantic Ocean\n",
"Score: 0.9067033\n",
"\n",
"ID: 21206\n",
"Title: Irish Sea\n",
"Summary: The Irish Sea (sometimes called the Manx Sea) is a body of water that separates Ireland and Great Britain. It is known to be one of the most polluted seas in the world including the North Sea and the Mediterranean Sea. The sea is important to regional trade, shipping and fishing. It is a source of power generation in the form of wind power and nuclear plants. Annual traffic between Great Britain and Ireland amounts to over 12 million passengers and of traded goods.\n",
"\n",
"Economics \n",
"It covers and at its deepest point is deep. In 2008, about of fish were caught. Shell fish made up three quarters of this amount.\n",
"\n",
"The Irish Sea has 17 active oil and gas drilling platforms. It is estimated there are about 1.6 billion barrels of oil in the Barryroe oil field alone.\n",
"\n",
"Sealife \n",
"At least thirty species of shark can be found in the Irish Sea at different times. These include the basking, thresher, blue, mako and porbeagle sharks. There are about 12 species of Dolphin, porpoise and whales in the Irish Sea. These include the common dolphin, bottlenose dolphin and the harbor porpoise.\n",
"\n",
"References \n",
"\n",
"Seas of the Atlantic Ocean\n",
"Ireland\n",
"Geography of the United Kingdom\n",
"Score: 0.90408546\n",
"\n",
"ID: 6308\n",
"Title: North Sea\n",
"Summary: The North Sea is a sea that is part of the Atlantic Ocean in northern Europe. The North Sea is between Norway and Denmark in the east, Scotland and England in the west, Germany, the Netherlands, Belgium and France in the south.\n",
"\n",
"Borders \n",
"The Skagerrak connects the North Sea to the Baltic Sea. In the south, the North Sea becomes the English Channel, a sea between England and France. This is called the Dover Straits and is very busy with ships.\n",
"\n",
"The border between the North Sea and the Skagerrak is at an imagined line between Lindesnes in Norway, and Hanstholm in Denmark. In the North, the North sea is open towards the Atlantic. The border between the two is an imagined line from Northern Scotland, to Shetland, and then to Ålesund in Norway. According to the Oslo-Paris Treaty of 1962 it is a bit more to the west and the north though. The treaty puts it at 5° East longitude, and 62° North latitude. That is at the parallel of the Geirangerfjord in Norway.\n",
"\n",
"Various statistical data \n",
"On average, the North Sea has a depth of only 94 meters. About 80 million people live near the North Sea, at most 150 km away from the coast. Together with the English Channel in the south, the southern North Sea is the busiest body of water in the world.\n",
"\n",
"Rivers that drain into it \n",
"Well-known rivers that drain into the North Sea include the Tay (at Dundee), the Forth (at Edinburgh), the Tyne (South Shields), the Wear (at Sunderland), the Tees (near Middlesbrough), the Elbe (at Cuxhaven), the Weser (at Bremerhaven), the Rhine and Meuse or Maas (at Rotterdam), the Scheldt (at Flushing or Vlissingen), the Thames, and the Humber (at Hull), and the river Nairn (at Nairn)\n",
"\n",
"The Kiel Canal, one of the world's busiest artificial waterways, connects the North Sea with the Baltic.\n",
"\n",
"Name \n",
"Its name comes from its relationship to the land of the Frisians (see Frisia). They live directly to the south of the North Sea, and to the west of the East Sea (Oostzee, the Baltic Sea), the former South Sea (Zuiderzee, today's IJsselmeer) and the today reclaimed Middle Sea (Middelzee). But the spread of the name could also be from the view of the cities of the Hanseatic League. Some of its main cities, like Lübeck, Bremen or Hamburg had the same view.\n",
"\n",
"In classical times this body of water was also called the Oceanum Germanicum or Mare Germanicum, meaning German Ocean or Sea. This name was commonly used in English and other languages along with the name North Sea, until the early eighteenth century. By the late nineteenth century, German Sea was a rare, scholarly usage even in Germany. In Danish the North Sea is also named Vesterhavet (besides Nordsøen), meaning Western Ocean because it is west of Denmark.\n",
"\n",
"Geographic divisions \n",
"\n",
"Most of the North sea is on the European Continental shelf. On average, the depth is about 93 to 94 meters only. In the south it is very shallow, only 25 to 35 meters. In the north in the bathyal zone north of Shetland, this depth increases to between 100 and 200 metres. In the south, the depth is at most 50 metres. An exception to this is the Norwegian Trench. It is deepest there, with a depth of 725 metres. The most shallow part of it is a sand bank called Dogger Bank. In the southern part, there are many sand banks.\n",
"\n",
"Looking at the satellite picture it is easy to see the geographic divisions of the North Sea:\n",
"a generally shallow southern North Sea\n",
"the central North Sea\n",
" the northern North Sea, with the Norwegian Trench, near the Skagerrak.\n",
"\n",
"The southern north sea is composed of the Southern Bight, before the coast of Belgium and the Netherlands and the German Bight before the coastline of Germany. The Dogger Bank is the limit between the southern and central parts. The Waddenzee runs all the way from Den Helder in the Netherlands to Esbjerg in Denmark.\n",
"\n",
"The Dogger Bank covers an area about half the size of the Netherlands. There, the North Sea has a depth of between 13 and 20 metres only. The area is very famous for fishing. With some storms there are even waves breaking there.\n",
"\n",
"The Norwegian Trench has an average depth of around 250 to 300 metres; at the entrance to the Skagerrak, the depth increases up to 725 meters. Along the trench is the Norwegian Current, which brings most of the waters of the North Sea into the Atlantic Ocean. Also, most of the waters of the Baltic Sea flow northwards here.\n",
"\n",
"About 200 km east of the Scottish city of Dundee there are more trenches, known collectively as the Devil's hole. Generally, the water is about 90 meters deep there. The trenches very often are only a few kilometers in length. In these trenches, the depth increases to up to 230 meters.\n",
"\n",
"In the Dover Strait the water is about 30 meters deep. At the end of the English Channel, this depth increases to about 100 meters.\n",
"\n",
"History \n",
"In the last ice age the North Sea was covered by large areas of ice called glaciers. About 20,000 years ago the ice melted and the North Sea was formed (made).\n",
"\n",
"North Sea oil \n",
"In the 1960s, geologists found large areas of oil and natural gas under the North Sea. Most of the oil fields are owned by the United Kingdom and Norway but some belong to Denmark, the Netherlands and Germany. Drilling began in the 1960s and led to a famous argument between England and Scotland about how the revenue (money) from the oil should be spent.\n",
"\n",
"Animal life \n",
"\n",
"People have been fishing in the North Sea for thousands of years. However, so many fish are now caught there that new ones may not be able to grow fast enough to keep the fishery going.\n",
"\n",
"Terns, Atlantic puffins, razorbills, kittiwakes and other seabirds live on the North Sea coast. Many coastal areas are protected nature reserves.\n",
"\n",
"Other websites\n",
"\n",
"Seas of the Atlantic Ocean\n",
"Bodies of water of Europe\n",
"Score: 0.9021919\n",
"\n",
"ID: 6278\n",
"Title: Atlantis\n",
"Summary: Atlantis is a name for a fictional large island or small continent that was (in the legend) in the Atlantic Ocean many years before it sank into the depth of the sea .\n",
"\n",
"The name Atlantis first appears in the writings of Herodotus - he describes the western ocean as \"Sea of Atlantis.\" Then, one generation later, Atlantis is described in detail in the stories Timaeus and Critias by the Greek philosopher Plato. He used this story to help explain his ideas about government and philosophy. Plato was the only ancient writer who wrote specific things about Atlantis.\n",
"\n",
"According to Plato, the Atlanteans lived 9000 years before his own time and were half human and half god. They created a very good human society. When they stopped being good people and did bad things, the gods sent earthquakes and fire to destroy Atlantis.\n",
"\n",
"Many scholars think Plato could have been thinking of a real place when he wrote about Atlantis. Many, many people have thought of many, many places where the real place that inspired Atlantis could have been. For example, there was a Minoan kingdom on the island of Santorini. The Minoan kingdom was very powerful thousands of years before Plato, and their society was damaged when a volcano erupted on their island. According to Plato, Atlantis was very large, as big as North Africa, so it should not have been hard to find.\n",
"\n",
"After the discovery of the Americas, some people in Europe thought they might be Atlantis. However, after Plato, the idea of Atlantis was mostly forgotten until 1882, when a writer named Ignatius Donnelly wrote a book saying that Atlantis was real and that the culture of Atlantis had started many other ancient cultures, such as the Egyptian and Mayan. Then other people became interested in Atlantis. \n",
"\n",
"Atlantis has appeared in many works of fiction. In Marvel Comics, Atlantis is at the bottom of the ocean and exists in modern times, with people who breathe water. Other works of fiction use Atlantis as background. For example, Robert E. Howard set his Conan the Barbarian stories in a fictional time called the Hyborian Age, which began with the destruction of Atlantis and ended when real written history started.\n",
"\n",
"References\n",
"\n",
"Greek mythology\n",
"Ancient history\n",
"Score: 0.9008117\n"
]
}
],
"source": [
"response = client.search(\n",
" index = \"wikipedia_vector_index\",\n",
" knn={\n",
" \"field\": \"content_vector\",\n",
" \"query_vector\": question_embedding[\"data\"][0][\"embedding\"],\n",
" \"k\": 10,\n",
" \"num_candidates\": 100\n",
" }\n",
")\n",
"pretty_response(response)\n",
"\n",
"top_hit_summary = response['hits']['hits'][0]['_source']['text'] # Store content of top hit for final step"
]
},
{
"cell_type": "markdown",
"id": "276c1147",
"metadata": {},
"source": [
"Success! We've used kNN to perform semantic search over our dataset and found the top results.\n",
"\n",
"Now we can use the Chat Completions API to work some generative AI magic using the top search result as additional context."
]
},
{
"cell_type": "markdown",
"id": "8abac103",
"metadata": {},
"source": [
"## Use Chat Completions API for retrieval augmented generation\n",
"\n",
"Now we can send the question and the text to OpenAI's chat completion API.\n",
"\n",
"Using a LLM model together with a retrieval model is known as retrieval augmented generation (RAG). We're using Elasticsearch to do what it does best, retrieve relevant documents. Then we use the LLM to do what it does best, tasks like generating summaries and answering questions, using the retrieved documents as context. \n",
"\n",
"The model will generate a response to the question, using the top kNN hit as context. Use the `messages` list to shape your prompt to the model. In this example, we're using the `gpt-3.5-turbo` model."
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "5cfb3153",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"------------------------------------------------------------\n",
"No, the Atlantic Ocean is not the biggest ocean in the world. It is the second largest ocean, covering about 20 percent of the Earth's surface. The Pacific Ocean is the largest ocean in the world.\n",
"------------------------------------------------------------\n"
]
}
],
"source": [
"summary = openai.ChatCompletion.create(\n",
" model=\"gpt-3.5-turbo\",\n",
" messages=[\n",
" {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n",
" {\"role\": \"user\", \"content\": \"Answer the following question:\" \n",
" + question \n",
" + \"by using the following text:\" \n",
" + top_hit_summary},\n",
" ]\n",
")\n",
"\n",
"choices = summary.choices\n",
"\n",
"for choice in choices:\n",
" print(\"------------------------------------------------------------\")\n",
" print(choice.message.content)\n",
" print(\"------------------------------------------------------------\")"
]
},
{
"cell_type": "markdown",
"id": "2a3f33fa",
"metadata": {},
"source": []
},
{
"cell_type": "markdown",
"id": "868ab150",
"metadata": {},
"source": [
"### Code explanation\n",
"\n",
"Here's what that code does:\n",
"\n",
"- Uses OpenAI's model to generate a response\n",
"- Sends a conversation containing a system message and a user message to the model\n",
"- The system message sets the assistant's role as \"helpful assistant\"\n",
"- The user message contains a question as specified in the original kNN query and some input text\n",
"- The response from the model is stored in the `summary.choices` variable"
]
},
{
"cell_type": "markdown",
"id": "aa0eec27",
"metadata": {},
"source": [
"## Next steps\n",
"\n",
"That was just one example of how to combine Elasticsearch with the power of OpenAI's models, to enable retrieval augmented generation. RAG allows you to avoid the costly and complex process of training or fine-tuning models, by leveraging out-of-the-box models, enhanced with additional context.\n",
"\n",
"Use this as a blueprint for your own experiments.\n",
"\n",
"To adapt the conversation for different use cases, customize the system message to define the assistant's behavior or persona. Adjust the user message to specify the task, such as summarization or question answering, along with the desired format of the response."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.11.3 64-bit",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.12"
},
"vscode": {
"interpreter": {
"hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
}
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Reference in New Issue View Git Blame Copy Permalink