langchain/docs/modules/indexes/document_loaders/examples/wikipedia.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bda1f3f5",
   "metadata": {},
   "source": [
    "# Wikipedia\n",
    "\n",
    ">[Wikipedia](https://wikipedia.org/) is a multilingual free online encyclopedia written and maintained by a community of volunteers, known as Wikipedians, through open collaboration and using a wiki-based editing system called MediaWiki. `Wikipedia` is the largest and most-read reference work in history.\n",
    "\n",
    "This notebook shows how to load wiki pages from `wikipedia.org` into the Document format that we use downstream."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b7a1eef-7bf7-4e7d-8bfc-c4e27c9488cb",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2abd5578-aa3d-46b9-99af-8b262f0b3df8",
   "metadata": {},
   "source": [
    "First, you need to install `wikipedia` python package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b674aaea-ed3a-4541-8414-260a8f67f623",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#!pip install wikipedia"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95f05e1c-195e-4e2b-ae8e-8d6637f15be6",
   "metadata": {},
   "source": [
    "## Examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e29b954c-1407-4797-ae21-6ba8937156be",
   "metadata": {},
   "source": [
    "`WikipediaLoader` has these arguments:\n",
    "- `query`: free text which used to find documents in Wikipedia\n",
    "- optional `lang`: default=\"en\". Use it to search in a specific language part of Wikipedia\n",
    "- optional `load_max_docs`: default=100. Use it to limit number of downloaded documents. It takes time to download all 100 documents, so use a small number for experiments. There is a hard limit of 300 for now.\n",
    "- optional `load_all_available_meta`: default=False. By default only the most important fields downloaded: `Published` (date when document was published/last updated), `title`, `Summary`. If True, other fields also downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9bfd5e46",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders import WikipediaLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "700e4ef2",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = WikipediaLoader(query='HUNTER X HUNTER', load_max_docs=2).load()\n",
    "len(docs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8977bac0-0042-4f23-9754-247dbd32439b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs[0].metadata  # meta-information of the Document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46969806-45a9-4c4d-a61b-cfb9658fc9de",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "docs[0].page_content[:400]  # a content of the Document \n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}