{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "Note: To answer questions based on text documents, we recommend the procedure in Question Answering using Embeddings. Some of the code below may rely on deprecated API endpoints." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 1. Collect Wikipedia data about Olympic Games 2020\n", "\n", "The idea of this project is to create a question answering model, based on a few paragraphs of provided text. Base GPT-3 models do a good job at answering questions when the answer is contained within the paragraph, however if the answer isn't contained, the base models tend to try their best to answer anyway, often leading to confabulated answers. \n", "\n", "To create a model which answers questions only if there is sufficient context for doing so, we first create a dataset of questions and answers based on paragraphs of text. In order to train the model to answer only when the answer is present, we also add adversarial examples, where the question doesn't match the context. In those cases, we ask the model to output \"No sufficient context for answering the question\". \n", "\n", "We will perform this task in three notebooks:\n", "1. The first (this) notebook focuses on collecting recent data, which GPT-3 didn't see during it's pre-training. We picked the topic of Olympic Games 2020 (which actually took place in the summer of 2021), and downloaded 713 unique pages. We organized the dataset by individual sections, which will serve as context for asking and answering the questions.\n", "2. The [second notebook](olympics-2-create-qa.ipynb) will utilize Davinci-instruct to ask a few questions based on a Wikipedia section, as well as answer those questions, based on that section.\n", "3. The [third notebook](olympics-3-train-qa.ipynb) will utilize the dataset of context, question and answer pairs to additionally create adversarial questions and context pairs, where the question was not generated on that context. In those cases the model will be prompted to answer \"No sufficient context for answering the question\". We will also train a discriminator model, which predicts whether the question can be answered based on the context or not." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.1 Data extraction using the wikipedia API\n", "Extracting the data will take about half an hour, and processing will likely take about as much." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "909" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "import wikipedia\n", "\n", "\n", "def filter_olympic_2020_titles(titles):\n", " \"\"\"\n", " Get the titles which are related to Olympic games hosted in 2020, given a list of titles\n", " \"\"\"\n", " titles = [title for title in titles if '2020' in title and 'olympi' in title.lower()]\n", " \n", " return titles\n", "\n", "def get_wiki_page(title):\n", " \"\"\"\n", " Get the wikipedia page given a title\n", " \"\"\"\n", " try:\n", " return wikipedia.page(title)\n", " except wikipedia.exceptions.DisambiguationError as e:\n", " return wikipedia.page(e.options[0])\n", " except wikipedia.exceptions.PageError as e:\n", " return None\n", "\n", "def recursively_find_all_pages(titles, titles_so_far=set()):\n", " \"\"\"\n", " Recursively find all the pages that are linked to the Wikipedia titles in the list\n", " \"\"\"\n", " all_pages = []\n", " \n", " titles = list(set(titles) - titles_so_far)\n", " titles = filter_olympic_2020_titles(titles)\n", " titles_so_far.update(titles)\n", " for title in titles:\n", " page = get_wiki_page(title)\n", " if page is None:\n", " continue\n", " all_pages.append(page)\n", "\n", " new_pages = recursively_find_all_pages(page.links, titles_so_far)\n", " for pg in new_pages:\n", " if pg.title not in [p.title for p in all_pages]:\n", " all_pages.append(pg)\n", " titles_so_far.update(page.links)\n", " return all_pages\n", "\n", "\n", "pages = recursively_find_all_pages([\"2020 Summer Olympics\"])\n", "len(pages)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.2 Filtering the Wikipedia pages and splitting them into sections by headings\n", "We remove sections unlikely to contain textual information, and ensure that each section is not longer than the token limit" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "('Bermuda at the 2020 Summer Olympics',\n", " 'Equestrian',\n", " \"Bermuda entered one dressage rider into the Olympic competition by finishing in the top four, outside the group selection, of the individual FEI Olympic Rankings for Groups D and E (North, Central, and South America), marking the country's recurrence to the sport after an eight-year absence. The quota was later withdrawn, following an injury of Annabelle Collins' main horse Joyero and a failure to obtain minimum eligibility requirements (MER) aboard a new horse Chuppy Checker.\",\n", " 104)" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "\n", "import re\n", "from typing import Set\n", "from transformers import GPT2TokenizerFast\n", "\n", "import numpy as np\n", "from nltk.tokenize import sent_tokenize\n", "\n", "tokenizer = GPT2TokenizerFast.from_pretrained(\"gpt2\")\n", "\n", "def count_tokens(text: str) -> int:\n", " \"\"\"count the number of tokens in a string\"\"\"\n", " return len(tokenizer.encode(text))\n", "\n", "def reduce_long(\n", " long_text: str, long_text_tokens: bool = False, max_len: int = 590\n", ") -> str:\n", " \"\"\"\n", " Reduce a long text to a maximum of `max_len` tokens by potentially cutting at a sentence end\n", " \"\"\"\n", " if not long_text_tokens:\n", " long_text_tokens = count_tokens(long_text)\n", " if long_text_tokens > max_len:\n", " sentences = sent_tokenize(long_text.replace(\"\\n\", \" \"))\n", " ntokens = 0\n", " for i, sentence in enumerate(sentences):\n", " ntokens += 1 + count_tokens(sentence)\n", " if ntokens > max_len:\n", " return \". \".join(sentences[:i]) + \".\"\n", "\n", " return long_text\n", "\n", "discard_categories = ['See also', 'References', 'External links', 'Further reading', \"Footnotes\",\n", " \"Bibliography\", \"Sources\", \"Citations\", \"Literature\", \"Footnotes\", \"Notes and references\",\n", " \"Photo gallery\", \"Works cited\", \"Photos\", \"Gallery\", \"Notes\", \"References and sources\",\n", " \"References and notes\",]\n", "\n", "\n", "def extract_sections(\n", " wiki_text: str,\n", " title: str,\n", " max_len: int = 1500,\n", " discard_categories: Set[str] = discard_categories,\n", ") -> str:\n", " \"\"\"\n", " Extract the sections of a Wikipedia page, discarding the references and other low information sections\n", " \"\"\"\n", " if len(wiki_text) == 0:\n", " return []\n", "\n", " # find all headings and the coresponding contents\n", " headings = re.findall(\"==+ .* ==+\", wiki_text)\n", " for heading in headings:\n", " wiki_text = wiki_text.replace(heading, \"==+ !! ==+\")\n", " contents = wiki_text.split(\"==+ !! ==+\")\n", " contents = [c.strip() for c in contents]\n", " assert len(headings) == len(contents) - 1\n", "\n", " cont = contents.pop(0).strip()\n", " outputs = [(title, \"Summary\", cont, count_tokens(cont)+4)]\n", " \n", " # discard the discard categories, accounting for a tree structure\n", " max_level = 100\n", " keep_group_level = max_level\n", " remove_group_level = max_level\n", " nheadings, ncontents = [], []\n", " for heading, content in zip(headings, contents):\n", " plain_heading = \" \".join(heading.split(\" \")[1:-1])\n", " num_equals = len(heading.split(\" \")[0])\n", " if num_equals <= keep_group_level:\n", " keep_group_level = max_level\n", "\n", " if num_equals > remove_group_level:\n", " if (\n", " num_equals <= keep_group_level\n", " ):\n", " continue\n", " keep_group_level = max_level\n", " if plain_heading in discard_categories:\n", " remove_group_level = num_equals\n", " keep_group_level = max_level\n", " continue\n", " nheadings.append(heading.replace(\"=\", \"\").strip())\n", " ncontents.append(content)\n", " remove_group_level = max_level\n", "\n", " # count the tokens of each section\n", " ncontent_ntokens = [\n", " count_tokens(c)\n", " + 3\n", " + count_tokens(\" \".join(h.split(\" \")[1:-1]))\n", " - (1 if len(c) == 0 else 0)\n", " for h, c in zip(nheadings, ncontents)\n", " ]\n", "\n", " # Create a tuple of (title, section_name, content, number of tokens)\n", " outputs += [(title, h, c, t) if t 1024). Running this sequence through the model will result in indexing errors\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
titleheadingcontenttokens
02020 Summer OlympicsSummaryThe 2020 Summer Olympics (Japanese: 2020年夏季オリン...713
12020 Summer OlympicsHost city selectionThe International Olympic Committee (IOC) vote...126
22020 Summer OlympicsImpact of the COVID-19 pandemicIn January 2020, concerns were raised about th...369
32020 Summer OlympicsQualifying event cancellation and postponementConcerns about the pandemic began to affect qu...298
42020 Summer OlympicsEffect on doping testsMandatory doping tests were being severely res...163
\n", "
" ], "text/plain": [ " title heading \\\n", "0 2020 Summer Olympics Summary \n", "1 2020 Summer Olympics Host city selection \n", "2 2020 Summer Olympics Impact of the COVID-19 pandemic \n", "3 2020 Summer Olympics Qualifying event cancellation and postponement \n", "4 2020 Summer Olympics Effect on doping tests \n", "\n", " content tokens \n", "0 The 2020 Summer Olympics (Japanese: 2020年夏季オリン... 713 \n", "1 The International Olympic Committee (IOC) vote... 126 \n", "2 In January 2020, concerns were raised about th... 369 \n", "3 Concerns about the pandemic began to affect qu... 298 \n", "4 Mandatory doping tests were being severely res... 163 " ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "res = []\n", "for page in pages:\n", " res += extract_sections(page.content, page.title)\n", "df = pd.DataFrame(res, columns=[\"title\", \"heading\", \"content\", \"tokens\"])\n", "df = df[df.tokens>40]\n", "df = df.drop_duplicates(['title','heading'])\n", "df = df.reset_index().drop('index',axis=1) # reset index\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Save the section dataset\n", "We will save the section dataset, for the [next notebook](olympics-2-create-qa.ipynb)" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "df.to_csv('olympics-data/olympics_sections.csv', index=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## 1.3 (Optional) Exploring the data " ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "Concerns and controversies at the 2020 Summer Olympics 51\n", "United States at the 2020 Summer Olympics 46\n", "Great Britain at the 2020 Summer Olympics 42\n", "Canada at the 2020 Summer Olympics 39\n", "Olympic Games 39\n", "Name: title, dtype: int64" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.title.value_counts().head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There appear to be winter and summer Olympics 2020. We chose to leave a little ambiguity and noise in the dataset, even though we were interested in only Summer Olympics 2020." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "True 3567\n", "False 305\n", "Name: title, dtype: int64" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.title.str.contains('Summer').value_counts()" ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "False 3774\n", "True 98\n", "Name: title, dtype: int64" ] }, "execution_count": 7, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df.title.str.contains('Winter').value_counts()" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Z1A+gAAAACXBIWXMAAAsTAAALEwEAmpwYAAAr20lEQVR4nO3deZwcVbn/8c+XsCdI2MwNEAibelEuCBHxojgBZRfUHyhcwABRREFB8UrABRQRXABFEERZXQiIC7sIXCIqsgWBsEqAsIRNIIQEBAl5fn+c06Sm6emumUzNFJnv+/Xq11SdU8vTNdX1dNU5XaWIwMzMrJ3FBjsAMzOrPycLMzPryMnCzMw6crIwM7OOnCzMzKwjJwszM+vIyaJA0qmSvtZPy1pD0lxJw/L4FEmf7I9l5+VdLmlCfy2vF+v9lqSnJT0x0OtuiqNL0qODuP6PSHok/4/f2Q/LC0nr9kdsfVh3t3214nW99hlr9z+U9D5J91YUw1mSvlX1evqDpMMl/Wyw44AhlCwkzZD0L0lzJD0n6TpJ+0t6bRtExP4RcVTJZX2g3TQR8XBEjIiIV/sh9iMl/aJp+dtFxNkLu+xexrEGcAiwfkT8x0Cuu4a+DxyY/8d/b64czIN/by3MvirpXkkfL4xvnt97c9kcSYuX/YxFxJ8j4q29jae3Bmo9ZbRKnhHx7Yjoty+ZC2PIJIvsQxGxHLAmcCxwKHB6f69E0uL9vcyaWAN4JiKeGuxA+lMf/19rAnf2dyxvQNcCWxTGtwDuaVH2t4iYN5CBWT+LiCHxAmYAH2gq2xSYD7wjj58FfCsPrwxcAjwHPAv8mZRcf57n+RcwF/gyMBYIYCLwMOkD1ChbPC9vCnAMcCPwPHAhsGKu6wIebRUvsC3wb+CVvL7bCsv7ZB5eDPgq8BDwFHAOsHyua8QxIcf2NPCVNttp+Tz/P/PyvpqX/4H8nufnOM5qMW8X8Cjp7OMp4HFgn0L9azHn8b2BvxTGA/gscB8wBzgKWAe4Lm+z84Elm9Z1eH5PM4A9CstaivTt/2HgSeBUYJmmeQ8FngB+3uK9tNymeblzc6wvAPe3mPfaQv1c4OO5/FPAdNL+dBGwatN7XzcPvxd4BOjK4/sCdwOzgCuANZvm2z9vs+eAkwHlunWBPwGz8zY6r4f/eWMfKe6rRwF/zf+HPwIr9zDvXsC0wvhl+f/aXPbVFp+xLgr7PfB54C5g9RZ1M4DDcv0s4Exg6UL9jsCteRtcB/xXoe6dwC35vZwHTG4TwyTg/jztXcBH2nxWNgVuJu2bTwLHF+o2y3E8B9zW+F/muhVz/I/l9/J7YDjdP19zgVWBI4FfFObdifQl5bn8f/rPpm30JeD2/D8/r7GN6OF41qtjaJUH6Dq9aJEscvnDwGda7MjHkA4wS+TX+1jwIey2LBZ82M7J//RlaP0BnAm8I0/zm8ZO0LzDNq+jeYcpLK+RLPYlHYTWBkYAvyUfAAtx/DTHtSHwcnEna1ruOaREtlye9x/AxJ7ibJq3C5gHfDNvs+2BF4EVmmPO43vz+mRxIfAm4O05zqvz+1qe9OGd0LSu40kH8PeTDs5vzfUnkA7IK+b3cjFwTNO838nzLtPivfS4TQuxrttmW3SrB7YkHbA3zuv8EXBt8/SkLwePAJvm8p1zHP8JLE5KYNc1zXcJMJJ05vdPYNtcdy7wFVLiWxp4bw+xNvaR4r56P/CWvM9MAY7tYd41SQe4FfN6nsrzPFIomw1s0eIz1kXen4Cvkw7oq7Ta10ifhzuAMXm5fy0s5515ve8GhpG+GM3I23lJUsL/Ammf3IX0xaunZLEr6SC9GPBx0j41uof3/jdgrzw8AtgsD68GPEPa/xcDPpjHG+/tUtKBfIUc0/vbHAeOZMFx4i05ng/m+b5M2jeWLGyjG3P8K5K+YOzf6XhW9jXULkO18hhpwzZ7BRhN+hb3SqRrm9FhWUdGxAsR8a8e6n8eEXdExAvA14CP9VOj4h6kbzUPRMRc0jew3Zour3wjIv4VEbeRvuls2LyQHMtuwGERMSciZgDHkb49lvUK8M28zS4jfUPqzTXh70bE8xFxJ+ng8Mf8vmYDl5MODEVfi4iXI+JPpA/hxyQJ2A/4QkQ8GxFzgG/n99YwHzgiz9vq/1Vmm/bGHsAZEXFLRLycl/ceSWML0+wK/ATYLiJuzGX7k5Lc3ZEu43wb2EjSmoX5jo2I5yLiYeAaYKNc/grpYL5qRLwUEX/pRbxnRsQ/8rY5v7DMbiLiIdIXrveR9qn78jx/LZQtCdzQw3ok6Xhga2B8RPyzTUwnRcQjEfEscDSwey7fD/hJRNwQEa9Gast7mfTtfjPSwfEHeZ+8ALippxVExK8j4rGImB8R55HO2DbtYfJXgHUlrRwRcyPi+ly+J3BZRFyWl3Ml6Qxke0mjge1IB/FZOaY/tXnPRR8HLo2IKyPiFdKZ8zLAfxemOTHH/yzpC9JGhVh7ezzrxskifQt4tkX590hZ+4+SHpA0qcSyHulF/UOknXjlUlG2t2peXnHZiwOjCmXF3ksvkr4JNVs5x9S8rNV6Ecsz0f3adE/r6smTheF/tRgvLmtWTrwND5G2xSrAssDU3JnhOeAPubzhnxHxUps4ymzT3ui2vJyAnqH7tj0YOD8i7iiUrQn8sPA+ngXUNF9P/9sv52lvlHSnpH17EW+Z/aWh0W6xBenyBsBfCmU35gTZykjSwf6Y/IWgnebPz6p5eE3gkMY2yttpTK5fFZjZdGAs/l+7kfQJSbcWlvMOev6MTiR9279H0k2SdizEs2tTPO8lHazHAM9GxKwO77WV5n1oPmmblNkX+nI862ZIJwtJ7yJt6Nd948rfrA+JiLVJ1wm/KGmrRnUPi+yUqccUhtcgZfunSaeWyxbiGkb3A1un5T5G2kGLy55H9wNtGU+z4NtocVkze7mcnnR7n8DC9qhaQdLwwvgapG3xNCmxvD0iRubX8hFRPOAN1DZtubwc90p037a7Ah+WdFCh7BHg04X3MTIilomI6zqtMCKeiIhPRcSqwKeBH1fUQ6uRLN7HgmTx50LZtW3mnUVqbzhT0uYd1tP8+XksDz8CHN20jZaNiHNJ7War5bPN4ryvk8/WfgocCKwUESNJZ7dqNX1E3BcRuwNvJl3SvCD/Xx8hXUUoxjM8Io7NdStKGtlqkR3ef/M+JNI26fj57HA8K2VIJgtJb8rfAiaTrgdOazHNjpLWzf+Q2cCrpEsXkA4Ya/dh1XtKWl/SsqTr+hdE6q74D2BpSTtIWoJ0XXqpwnxPAmOL3XybnAt8QdJakkaQLlWcF73sfZJjOR84WtJy+cPzReAX7ecs7Vbgo5KWzQetif2wzG9IWlLS+0gHnV/nb1w/BU6Q9GYASatJ2qYXy13Ybdq8j5wL7CNpI0lL5eXdkC/1NTwGbAUcJOkzuexU4DBJb8/vY3lJu5YJQNKuklbPo7NIB6P5bWbpq2tJlwe3IF1+ApgGrAWMp32yICKmkC7T/VZST5d8AA6QtLqkFUltMefl8p8C+0t6t5Lh+bO0HKldYR7weUlLSPooPV9WGk7aRv8EkLQP6cyiJUl7Slol72/P5eL5pM/LhyRtI2mYpKVzt9jVI+Jx0uXUH0taIcfU6Dn2JLCSpOV7WOX5wA6StsrHiUNIl9s6fnHocDwrZagli4slzSFl96+QGkf36WHa9YCrSNfc/wb8OCKuyXXHAF/Np5hf6sX6f05q4HuC1OD4eYB8+v1Z4GekbwkvkHrrNPw6/31G0i0tlntGXva1wIPAS8DnehFX0efy+h8gnXH9Ki+/P5xA6tn1JHA28MuFXN4TpIPgY3lZ+0fEPbnuUNJp9/WSnif9L3vTdrKw2/RI4Oy8j3wsIq4itVP9hvRtdx26t6EA6TcPpIQxSdInI+J3pG+tk/P7uIN0zbuMdwE3SJpLauw/KCIe6MV7KCUi/kE6wD4REc/lsvmkxtY3UeJglq/r70v6jG7cw2S/IvXMeoDUAP+tPO/NpJ5mJ5H2h+mkzhNExL+Bj+bxZ0nX/X/bQwx3kdro/kbaRzdgQfJrZVvgzrx9fwjsFqld8BFSx4TDSdvlEeB/WXC83Yt0Bn8PqWH+4Lz+e0hfKh7I+82qhXUREfeS2kN+RDp7/hDp5wD/bhNjQ7vjWSmN3j1mZrUlaQapJ91Vgx3LUDXUzizMzKwPnCzMzKwjX4YyM7OOfGZhZmYdLZI3vFt55ZVj7NixLeteeOEFhg8f3rKubhxrNRxrNRxrNQYy1qlTpz4dEau0rIyFvOdSHV+bbLJJ9OSaa67psa5uHGs1HGs1HGs1BjJW4ObwvaHMzKyvnCzMzKwjJwszM+vIycLMzDpysjAzs46cLMzMrCMnCzMz68jJwszMOnKyMDOzjhbJ230srLGTLh2U9c44dodBWa+ZWSc+szAzs46cLMzMrCMnCzMz68jJwszMOnKyMDOzjpwszMysIycLMzPrqLJkIWlpSTdKuk3SnZK+kcvXknSDpOmSzpO0ZC5fKo9Pz/VjC8s6LJffK2mbqmI2M7PWqjyzeBnYMiI2BDYCtpW0GfAd4ISIWBeYBUzM008EZuXyE/J0SFof2A14O7At8GNJwyqM28zMmlSWLPIjXefm0SXyK4AtgQty+dnAh/PwznmcXL+VJOXyyRHxckQ8CEwHNq0qbjMze71K2ywkDZN0K/AUcCVwP/BcRMzLkzwKrJaHVwMeAcj1s4GViuUt5jEzswFQ6b2hIuJVYCNJI4HfAW+ral2S9gP2Axg1ahRTpkxpOd3cuXN7rGs4ZIN5beur0hxXmVjrwrFWw7FWw7H23oDcSDAinpN0DfAeYKSkxfPZw+rAzDzZTGAM8KikxYHlgWcK5Q3FeYrrOA04DWDcuHHR1dXVMpYpU6bQU13D3oN1I8E9urqNl4m1LhxrNRxrNRxr71XZG2qVfEaBpGWADwJ3A9cAu+TJJgAX5uGL8ji5/v8iInL5brm31FrAesCNVcVtZmavV+WZxWjg7NxzaTHg/Ii4RNJdwGRJ3wL+Dpyepz8d+Lmk6cCzpB5QRMSdks4H7gLmAQfky1tmZjZAKksWEXE78M4W5Q/QojdTRLwE7NrDso4Gju7vGM3MrBz/gtvMzDpysjAzs46cLMzMrCMnCzMz68jJwszMOnKyMDOzjpwszMysIycLMzPrqGOykLS5pOF5eE9Jx0tas/rQzMysLsqcWZwCvChpQ+AQ0m3Gz6k0KjMzq5UyyWJevqHfzsBJEXEysFy1YZmZWZ2UuTfUHEmHAXsCW0hajPTUOzMzGyLKnFl8nPQ87YkR8QTpeRLfqzQqMzOrlY5nFjlBHF8Yfxi3WZiZDSllekN9VNJ9kmZLel7SHEnPD0RwZmZWD2XaLL4LfCgi7q46GDMzq6cybRZPOlGYmQ1tZc4sbpZ0HvB7UkM3ABHx26qCMjOzeimTLN4EvAhsXSgLwMnCzGyIKNMbap+BCMTMzOqrTG+o1SX9TtJT+fUbSasPRHBmZlYPZRq4zwQuAlbNr4tzmZmZDRFlksUqEXFmRMzLr7OAVSqOy8zMaqRMsngm35p8WH7tCTxTdWBmZlYfZZLFvsDHgCeAx4FdgI6N3pLGSLpG0l2S7pR0UC4/UtJMSbfm1/aFeQ6TNF3SvZK2KZRvm8umS5rU2zdpZmYLp0xvqIeAnfqw7HnAIRFxi6TlgKmSrsx1J0TE94sTS1of2A14O6lt5CpJb8nVJwMfBB4FbpJ0UUTc1YeYzMysD3pMFpK+HBHflfQj0u8quomIz7dbcEQ8TjoTISLmSLobWK3NLDsDkyPiZeBBSdOBTXPd9Ih4IMc1OU/rZGFmNkCUnmvUokL6UERcLGlCq/qIOLv0SqSxwLXAO4AvAnsDzwM3k84+Zkk6Cbg+In6R5zkduDwvYtuI+GQu3wt4d0Qc2LSO/YD9AEaNGrXJ5MmTW8Yyd+5cRowY0TbeaTNnl31r/WqD1ZbvNl4m1rpwrNVwrNVwrK2NHz9+akSMa1XX45lFRFycB1+MiF8X6yTtWnblkkYAvwEOjojnJZ0CHEU6WzkKOI7ULrJQIuI04DSAcePGRVdXV8vppkyZQk91DXtPunRhw+mTGXt0dRsvE2tdONZqONZqONbeK9PAfVjJsteRtAQpUfyycS+piHgyIl6NiPnAT1lwqWkmMKYw++q5rKdyMzMbIO3aLLYDtgdWk3RioepNpMbrtiQJOB24OyKOL5SPzu0ZAB8B7sjDFwG/knQ8qYF7PeBGQMB6ktYiJYndgP8p9/bMzKw/tOsN9RipTWEnYGqhfA7whRLL3hzYC5gm6dZcdjiwu6SNSJehZgCfBoiIOyWdT2q4ngccEBGvAkg6ELgCGAacERF3lli/mZn1k3ZtFrcBt0n6HfBC4cA9DFiq04Ij4i+ks4Jml7WZ52jg6Bbll7Wbz8zMqlWmzeKPwDKF8WWAq6oJx8zM6qhMslg6IuY2RvLwstWFZGZmdVMmWbwgaePGiKRNgH9VF5KZmdVNmSflHQz8WtJjpDaI/wA+XmVQZmZWL2XuDXWTpLcBb81F90bEK9WGZWZmdVLmSXnLAocCB0XEHcBYSTtWHpmZmdVG2Sfl/Rt4Tx6fCXyrsojMzKx2yiSLdSLiu8ArABHxIq1/P2FmZouoMsni35KWId+mXNI6wMuVRmVmZrVSpjfUEcAfgDGSfkm6jcfeVQZlZmb1UqY31JWSbgE2I11+Oiginq48MjMzq40yvaE2B16KiEuBkcDhktasOjAzM6uPMm0WpwAvStqQ9JS7+4FzKo3KzMxqpUyymBfp2as7AydHxMnActWGZWZmdVKmgXuOpMOAPYEtJC0GLFFtWGZmVidlziw+TuoqOzEiniA91vR7lUZlZma1UqY31BPA8YXxh3GbhZnZkFLmzMLMzIY4JwszM+vIycLMzDrq2GYhaT3gGGB9YOlGeUSsXWFcZmZWI2VvUX4KMA8YT2rc/kWVQZmZWb2USRbLRMTVgCLioYg4Etih2rDMzKxOyiSLl/MP8e6TdKCkjwAjOs0kaYykayTdJelOSQfl8hUlXSnpvvx3hVwuSSdKmi7pdkkbF5Y1IU9/n6QJfXyvZmbWR2WSxUHAssDngU2AvYAyB+x5wCERsT7pjrUHSFofmARcHRHrAVfncYDtgPXyaz/SpS8krUi6Tfq7gU2BIxoJxszMBkaZH+XdlAfnAvuUXXBEPA48nofnSLobWI10j6muPNnZwBTSM753Bs7J96G6XtJISaPztFdGxLMAkq4EtgXOLRuLmZktnB6ThaQfRMTBki4mPyWvKCJ2KrsSSWOBdwI3AKNyIgF4AhiVh1cDHinM9mgu66nczMwGSLszi5/nv99fmBVIGgH8Bjg4Ip6XFjy+OyJC0usSUR/Xsx/p8hWjRo1iypQpLaebO3duj3UNh2wwrz9C6rXmuMrEWheOtRqOtRqOtfd6TBYRMTX//VNfFy5pCVKi+GVE/DYXPylpdEQ8ni8zPZXLZwJjCrOvnstmsuCyVaN8Sot4TwNOAxg3blx0dXU1TwKkA3JPdQ17T7q0bX1VZuzR1W28TKx14Vir4Vir4Vh7r8cGbknTcq+klq9OC1Y6hTgduDsiji9UXcSCBvIJwIWF8k/kXlGbAbPz5aorgK0lrZAbtrfOZWZmNkDaXYbaMf89IP9tXJbakxZtGC1sTuo5NU3SrbnscOBY4HxJE4GHgI/lusuA7YHpwIvkxvSIeFbSUUCjof2bjcZuMzMbGO0uQz0EIOmDEfHOQtWhkm5hQZfXnub/C6AeqrdqMX2wIDE1150BnNFufWZmVp0yv7OQpM0LI/9dcj4zM1tElHms6kTgDEnL5/HngH0ri8jMzGqnzI/ypgIbNpJFRMyuPCozM6uVjpeTJI2SdDowOSJmS1o/N06bmdkQUabt4SxSV9VV8/g/gIMrisfMzGqoTLJYOSLOB+YDRMQ84NVKozIzs1opkyxekLQS+bcVjR/MVRqVmZnVSpneUF8k/bp6HUl/BVYBdqk0KjMzq5UyvaFukfR+4K2kH9ndGxGvVB6ZmZnVRsdkIWlp4LPAe0mXov4s6dSIeKnq4MzMrB7KXIY6B5gD/CiP/w/pPlG7VhWUmZnVS5lk8Y78aNSGayTdVVVAZmZWP2V6Q92Se0ABIOndwM3VhWRmZnVT5sxiE+A6SQ/n8TWAeyVNI90s9r8qi87MzGqhTLLYtvIozMys1npMFpLeFBHPkxq3X8cPIDIzGzranVn8ivS0vKmkLrPFBxkFsHaFcZmZWY20e1LejvnvWs11+fnaZmY2RJS5Rfk3m8YXA35RWURmZlY7ZbrOjpF0GICkpYDfAfdVGpWZmdVKmWSxL7BBThgXA9dExJGVRmVmZrXSrjfUxoXRHwI/Af4KXCtp44i4pergzMysHtr1hjquaXwWsH4uD2DLqoIyM7N6adcbavxABmJmZvXVY5uFpD3z3y+2enVasKQzJD0l6Y5C2ZGSZkq6Nb+2L9QdJmm6pHslbVMo3zaXTZc0qe9v1czM+qrdZajh+e9yfVz2WcBJpFucF50QEd8vFkhaH9gNeDuwKnCVpLfk6pOBDwKPAjdJuigifNdbM7MB1C5ZPCnpzRHxjb4sOCKulTS25OQ7A5Mj4mXgQUnTgU1z3fSIeABA0uQ8rZOFmdkAUkS0rpAuAN4DvAhcR+oJdV1E3NFyhtbLGAtcEhHvyONHAnsDz5Nuc35IRMySdBJwfUT8Ik93OnB5Xsy2EfHJXL4X8O6IOLDFuvYD9gMYNWrUJpMnT24Z09y5cxkxYkTbuKfNnF32LfarDVZbvtt4mVjrwrFWw7FWw7G2Nn78+KkRMa5VXbsG7l0AJK1FShr/DXxa0hrATRGxfU/ztnEKcBSpN9VRpJ5V+/ZhOa3iPQ04DWDcuHHR1dXVcropU6bQU13D3pMu7Y+Qem3GHl3dxsvEWheOtRqOtRqOtfc63qI8Ih7Mv9xeJr+Wzn97LSKebAxL+ilwSR6dCYwpTLp6LqNNuZmZDZB2vaEOl3SxpOuBw4AlSQ3W/9XXbrWSRhdGPwI0LmldBOwmaal8JrMecCNwE7CepLUkLUlqBL+oL+s2M7O+a3dm8QngBdItPq4DboiI0hfzJZ0LdAErS3oUOALokrQR6TLUDODTABFxp6TzSQ3X84ADIuLVvJwDgSuAYcAZEXFnL96fmZn1g3ZtFm+TtCKpraILmCRpBHAbqaH7zHYLjojdWxSf3mb6o4GjW5RfBlzWbl1mZlattm0W+Wl4l0j6A+lZ3FuQzgb2BdomCzMzW3S0u5HgTqSzis1JP5a7k9R99hDSZSkzMxsi2p1Z7E1KDl8GpkbEvwckIjMzq512bRYfHchAzMysvso8/MjMzIY4JwszM+uo3Y/yrs5/vzNw4ZiZWR21a+AeLem/gZ3y3V5VrPRjVc3Mho52yeLrwNdI92M6vqnOj1U1MxtC2vWGugC4QNLXIuKoAYzJzMxqpsxdZ4/KP9DbIhdNiYhL2s1jZmaLlo69oSQdAxxEusnfXcBBkr5ddWBmZlYfHc8sgB2AjSJiPoCks4G/A4dXGZiZmdVH2d9ZjCwML9/TRGZmtmgqc2ZxDPB3SdeQus9uAUyqNCozM6uVMg3c50qaArwrFx0aEU9UGpWZmdVKmTMLIuJx/DhTM7Mhy/eGMjOzjpwszMyso7bJQtIwSfcMVDBmZlZPbZNFRLwK3CtpjQGKx8zMaqhMA/cKwJ2SbgReaBRGxE6VRWVmZrVSJll8rfIozMys1sr8zuJPktYE1ouIqyQtCwyrPjQzM6uLMjcS/BRwAfCTXLQa8PsS850h6SlJdxTKVpR0paT78t8VcrkknShpuqTbJW1cmGdCnv4+SRN6+f7MzKwflOk6ewCwOfA8QETcB7y5xHxnAds2lU0Cro6I9YCrWXDbkO2A9fJrP+AUSMkFOAJ4N7ApcEQjwZiZ2cApkyxejoh/N0YkLU56Ul5bEXEt8GxT8c7A2Xn4bODDhfJzIrkeGClpNLANcGVEPBsRs4AreX0CMjOziimi/XFf0neB54BPAJ8DPgvcFRFf6bhwaSxwSUS8I48/FxEj87CAWRExUtIlwLER8ZdcdzVwKNAFLB0R38rlXwP+FRHfb7Gu/UhnJYwaNWqTyZMnt4xp7ty5jBgxom3c02bO7vTWKrHBat1v6Fsm1rpwrNVwrNVwrK2NHz9+akSMa1VXpjfUJGAiMA34NHAZ8LOFDSoiQlLHM5ReLO804DSAcePGRVdXV8vppkyZQk91DXtPurS/wuqVGXt0dRsvE2tdONZqONZqONbeK9Mban5+4NENpMtP90an05GePSlpdEQ8ni8zPZXLZwJjCtOtnstmks4uiuVT+rhuMzProzK9oXYA7gdOBE4Cpkvaro/ruwho9GiaAFxYKP9E7hW1GTA73+n2CmBrSSvkhu2tc5mZmQ2gMpehjgPGR8R0AEnrAJcCl7ebSdK5pLOClSU9SurVdCxwvqSJwEPAx/LklwHbA9OBF4F9ACLiWUlHATfl6b4ZEc2N5mZmVrEyyWJOI1FkDwBzOs0UEbv3ULVVi2mD1EW31XLOAM4oEaeZmVWkx2Qh6aN58GZJlwHnk9osdmXBN30zMxsC2p1ZfKgw/CTw/jz8T2CZyiIyM7Pa6TFZRMQ+AxmImZnVV8c2C0lrkX6MN7Y4vW9RbmY2dJRp4P49cDpwMTC/0mjMzKyWyiSLlyLixMojMTOz2iqTLH4o6Qjgj8DLjcKIuKWyqMzMrFbKJIsNgL2ALVlwGSryuJmZDQFlksWuwNrF25SbmdnQUuZ5FncAIyuOw8zMaqzMmcVI4B5JN9G9zcJdZ83MhogyyeKIyqMwM7NaK/M8iz8NRCBmZlZfZX7BPYcFz9xeElgCeCEi3lRlYGZmVh9lziyWawzn52bvDGxWZVBmZlYvZXpDvSaS3wPbVBOOmZnVUZnLUB8tjC4GjANeqiwiMzOrnTK9oYrPtZgHzCBdijIzsyGiTJuFn2thZjbEtXus6tfbzBcRcVQF8ZiZWQ21O7N4oUXZcGAisBLgZGFmNkS0e6zqcY1hScsBBwH7AJOB43qaz8zMFj1t2ywkrQh8EdgDOBvYOCJmDURgZmZWHz3+zkLS94CbgDnABhFxZH8lCkkzJE2TdKukm3PZipKulHRf/rtCLpekEyVNl3S7pI37IwYzMyuv3Y/yDgFWBb4KPCbp+fyaI+n5flj3+IjYKCLG5fFJwNURsR5wdR4H2A5YL7/2A07ph3WbmVkvtGuz6NWvu/vBzkBXHj4bmAIcmsvPiYgArpc0UtLoiHh8gOMzMxuylI7BA7xS6UFgFukGhT+JiNMkPRcRI3O9gFkRMVLSJcCxEfGXXHc1cGhE3Ny0zP1IZx6MGjVqk8mTJ7dc99y5cxkxYkTb+KbNnL0wb6/PNlht+W7jZWKtC8daDcdaDcfa2vjx46cWrvZ0U+YX3FV4b0TMlPRm4EpJ9xQrIyIk9SqLRcRpwGkA48aNi66urpbTTZkyhZ7qGvaedGlvVt1vZuzR1W28TKx14Vir4Vir4Vh7b6AvNQEQETPz36eA3wGbAk9KGg2Q/z6VJ58JjCnMvnouMzOzATLgyULS8Py7DSQNB7YmPef7ImBCnmwCcGEevgj4RO4VtRkw2+0VZmYDazAuQ40CfpeaJVgc+FVE/CE/4/t8SROBh4CP5ekvA7YHpgMvkn4YaGZmA2jAk0VEPABs2KL8GWCrFuUBHDAAoQ26sU1tJYdsMG/A2k9mHLvDgKzHzN6YBqXNwszM3licLMzMrCMnCzMz68jJwszMOnKyMDOzjpwszMysIycLMzPryMnCzMw6crIwM7OOnCzMzKwjJwszM+vIycLMzDpysjAzs46cLMzMrCMnCzMz68jJwszMOnKyMDOzjgbjsapWQ81P6eutvj7Vz0/oM3tj8JmFmZl15GRhZmYdOVmYmVlHThZmZtaRk4WZmXXkZGFmZh29YbrOStoW+CEwDPhZRBw7yCFZP1jYLrt90dduvv3F3YXtjegNcWYhaRhwMrAdsD6wu6T1BzcqM7Oh441yZrEpMD0iHgCQNBnYGbhrUKMy64PenE0N9llQb7SL1WdTb3yKiMGOoSNJuwDbRsQn8/hewLsj4sDCNPsB++XRtwL39rC4lYGnKwy3PznWajjWajjWagxkrGtGxCqtKt4oZxYdRcRpwGmdppN0c0SMG4CQFppjrYZjrYZjrUZdYn1DtFkAM4ExhfHVc5mZmQ2AN0qyuAlYT9JakpYEdgMuGuSYzMyGjDfEZaiImCfpQOAKUtfZMyLizj4uruOlqhpxrNVwrNVwrNWoRaxviAZuMzMbXG+Uy1BmZjaInCzMzKyjIZMsJG0r6V5J0yVNqkE8YyRdI+kuSXdKOiiXryjpSkn35b8r5HJJOjHHf7ukjQch5mGS/i7pkjy+lqQbckzn5c4HSFoqj0/P9WMHOM6Rki6QdI+kuyW9p67bVdIX8v//DknnSlq6LttV0hmSnpJ0R6Gs19tR0oQ8/X2SJgxgrN/L+8Dtkn4naWSh7rAc672StimUV36caBVroe4QSSFp5Tw+qNu1m4hY5F+kRvH7gbWBJYHbgPUHOabRwMZ5eDngH6RbmXwXmJTLJwHfycPbA5cDAjYDbhiEmL8I/Aq4JI+fD+yWh08FPpOHPwucmod3A84b4DjPBj6Zh5cERtZxuwKrAQ8CyxS259512a7AFsDGwB2Fsl5tR2BF4IH8d4U8vMIAxbo1sHge/k4h1vXzMWApYK18bBg2UMeJVrHm8jGkTjwPASvXYbt2i6/qD0QdXsB7gCsK44cBhw12XE0xXgh8kPTL89G5bDRwbx7+CbB7YfrXphug+FYHrga2BC7JO+/ThQ/ja9s47/DvycOL5+k0QHEunw/Aaiqv3XYlJYtH8gd+8bxdt6nTdgXGNh2Ae7Udgd2BnxTKu01XZaxNdR8BfpmHu33+G9t1II8TrWIFLgA2BGawIFkM+nZtvIbKZajGh7Lh0VxWC/lywjuBG4BREfF4rnoCGJWHB/s9/AD4MjA/j68EPBcR81rE81qsuX52nn4grAX8EzgzXzL7maTh1HC7RsRM4PvAw8DjpO00lXpu14bebsfB3m8b9iV9Q4caxippZ2BmRNzWVFWbWIdKsqgtSSOA3wAHR8TzxbpIXxkGvW+zpB2BpyJi6mDHUsLipFP8UyLincALpMslr6nRdl2BdEPMtYBVgeHAtoMaVC/UZTt2IukrwDzgl4MdSyuSlgUOB74+2LG0M1SSRS1vFyJpCVKi+GVE/DYXPylpdK4fDTyVywfzPWwO7CRpBjCZdCnqh8BISY0fdhbjeS3WXL888MwAxfoo8GhE3JDHLyAljzpu1w8AD0bEPyPiFeC3pG1dx+3a0NvtOKifPUl7AzsCe+TkRpuYBivWdUhfGG7Ln7HVgVsk/UedYh0qyaJ2twuRJOB04O6IOL5QdRHQ6NkwgdSW0Sj/RO4dsRkwu3A5oFIRcVhErB4RY0nb7v8iYg/gGmCXHmJtvIdd8vQD8g00Ip4AHpH01ly0FelW9rXbrqTLT5tJWjbvD41Ya7ddC3q7Ha8Atpa0Qj6T2jqXVU7pgWlfBnaKiBeb3sNuuXfZWsB6wI0M0nEiIqZFxJsjYmz+jD1K6vzyBHXarlU2iNTpRepV8A9Sb4ev1CCe95JO4W8Hbs2v7UnXoK8G7gOuAlbM04v0AKj7gWnAuEGKu4sFvaHWJn3IpgO/BpbK5Uvn8em5fu0BjnEj4Oa8bX9P6i1Sy+0KfAO4B7gD+Dmph04ttitwLqkt5RXSAWxiX7Yjqb1gen7tM4CxTidd1298vk4tTP+VHOu9wHaF8sqPE61ibaqfwYIG7kHdrsWXb/dhZmYdDZXLUGZmthCcLMzMrCMnCzMz68jJwszMOnKyMDOzjpws7A0v36XzuML4lyQd2U/LPkvSLp2nXOj17Kp0h9xrmsrHSvqfEvPvLemk6iK0oc7JwhYFLwMfbdzWuS4Kv8IuYyLwqYgY31Q+FuiYLMyq5mRhi4J5pOcUf6G5ovnMQNLc/LdL0p8kXSjpAUnHStpD0o2Spklap7CYD0i6WdI/8n2yGs/2+J6km/JzBj5dWO6fJV1E+jV2czy75+XfIek7uezrpB9pni7pe02zHAu8T9KtSs++WFrSmXkZf5fUnFyQtIOkv0laWdLWefgWSb/O9yJD0gxJ38jl0yS9LZe/P6/r1rz85cr/G2xR5mRhi4qTgT0kLd+LeTYE9gf+E9gLeEtEbAr8DPhcYbqxwKbADsCpkpYmnQnMjoh3Ae8CPpVvHQHpXlQHRcRbiiuTtCrpuQpbkn5l/i5JH46Ib5J+cb5HRPxvU4yTgD9HxEYRcQJwAOkefhuQblN9do6nsY6P5Hm2z0VfBT4QERvndXyxsOync/kpwJdy2ZeAAyJiI+B9wL/ab0IbKpwsbJEQ6Y695wCf78VsN0XE4xHxMul2Cn/M5dNICaLh/IiYHxH3kR4y8zbSvXg+IelW0q3lVyLdYwjgxoh4sMX63gVMiXTjwMZdULfoRbyQzkB+ARAR95AelNNISlsChwI7RMQs0sNy1gf+muOcAKxZWFbj5pVTC+/3r8Dxkj4PjIwFt0q3Ic7JwhYlPyB94x9eKJtH3s8lLUZ6AlrDy4Xh+YXx+aRbnTc03xMnSPfs+Vz+xr9RRKwVEY1k88LCvImFcD/pqYuN5CHgykKM60fExML0jff7Kvn9RsSxwCeBZUhJ5m0DE7rVnZOFLTIi4lnSI0mLB8QZwCZ5eCdgiT4seldJi+V2jLVJN5+7AviM0m3mkfQWpYcstXMj8P7cljCMdBnpTx3mmUNKAA1/BvZorBNYI8cD6Szj/wHnSHo7cD2wuaR18/TD8zw9krROpLugfod0F1YnCwOcLGzRcxxQ7BX1U9IB+jbSYzP78q3/YdKB/nJg/4h4idSucRfpuQN3kB5r2bb3U6RbS08i3YL8NmBqRFzYbh7SnXNflXSbpC8APwYWkzQNOA/YO19Ga6zjHlIy+TXwJtIzvc+VdDvwNzof/A/Oje+3k+6KenmH6W2I8F1nzcysI59ZmJlZR04WZmbWkZOFmZl15GRhZmYdOVmYmVlHThZmZtaRk4WZmXX0/wFZfduL32Si2AAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import pandas as pd\n", "from matplotlib import pyplot as plt\n", "\n", "df = pd.read_csv('olympics-data/olympics_sections.csv')\n", "df[['tokens']].hist()\n", "# add axis descriptions and title\n", "plt.xlabel('Number of tokens')\n", "plt.ylabel('Number of Wikipedia sections')\n", "plt.title('Distribution of number of tokens in Wikipedia sections')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see that the majority of section are fairly short (less than 500 tokens)." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3.9.9 64-bit ('3.9.9')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.9" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "cb9817b186a29e4e9713184d901f26c1ee05ad25243d878baff7f31bb1fef480" } } }, "nbformat": 4, "nbformat_minor": 2 }