langchain/docs/extras/modules/data_connection/document_loaders/integrations/grobid.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bdccb278",
   "metadata": {},
   "source": [
    "# Grobid\n",
    "\n",
    "GROBID is a machine learning library for extracting, parsing, and re-structuring raw documents.\n",
    "\n",
    "It is particularly good for sturctured PDFs, like academic papers.\n",
    "\n",
    "This loader uses GROBIB to parse PDFs into `Documents` that retain metadata associated with the section of text.\n",
    "\n",
    "---\n",
    "\n",
    "For users on `Mac` - \n",
    "\n",
    "(Note: additional instructions can be found [here](https://python.langchain.com/docs/ecosystem/integrations/grobid.mdx).)\n",
    "\n",
    "Install Java (Apple Silicon):\n",
    "```\n",
    "$ arch -arm64 brew install openjdk@11\n",
    "$ brew --prefix openjdk@11\n",
    "/opt/homebrew/opt/openjdk@ 11\n",
    "```\n",
    "\n",
    "In `~/.zshrc`:\n",
    "```\n",
    "export JAVA_HOME=/opt/homebrew/opt/openjdk@11\n",
    "export PATH=$JAVA_HOME/bin:$PATH\n",
    "```\n",
    "\n",
    "Then, in Terminal:\n",
    "```\n",
    "$ source ~/.zshrc\n",
    "```\n",
    "\n",
    "Confirm install:\n",
    "```\n",
    "$ which java\n",
    "/opt/homebrew/opt/openjdk@11/bin/java\n",
    "$ java -version \n",
    "openjdk version \"11.0.19\" 2023-04-18\n",
    "OpenJDK Runtime Environment Homebrew (build 11.0.19+0)\n",
    "OpenJDK 64-Bit Server VM Homebrew (build 11.0.19+0, mixed mode)\n",
    "```\n",
    "\n",
    "Then, get [Grobid](https://grobid.readthedocs.io/en/latest/Install-Grobid/#getting-grobid):\n",
    "```\n",
    "$ curl -LO https://github.com/kermitt2/grobid/archive/0.7.3.zip\n",
    "$ unzip 0.7.3.zip\n",
    "```\n",
    "                   \n",
    "Build\n",
    "```\n",
    "$ ./gradlew clean install\n",
    "```\n",
    "\n",
    "Then, run the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2d8992fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "! get_ipython().system_raw('nohup ./gradlew run > grobid.log 2>&1 &')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b41bfb1",
   "metadata": {},
   "source": [
    "Now, we can use the data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "640e9a4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.document_loaders.parsers import GrobidParser\n",
    "from langchain.document_loaders.generic import GenericLoader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ecdc1fb9",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = GenericLoader.from_filesystem(\n",
    "    \"../Papers/\",\n",
    "    glob=\"*\",\n",
    "    suffixes=[\".pdf\"],\n",
    "    parser= GrobidParser(segment_sentences=False)\n",
    ")\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "efe9e356",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g.\"Books -2TB\" or \"Social media conversations\").There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[3].page_content"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "5be03d17",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': 'Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g.\"Books -2TB\" or \"Social media conversations\").There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.',\n",
       " 'para': '2',\n",
       " 'bboxes': \"[[{'page': '1', 'x': '317.05', 'y': '509.17', 'h': '207.73', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '522.72', 'h': '220.08', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '536.27', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '549.82', 'h': '218.65', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '563.37', 'h': '136.98', 'w': '9.46'}], [{'page': '1', 'x': '446.49', 'y': '563.37', 'h': '78.11', 'w': '9.46'}, {'page': '1', 'x': '304.69', 'y': '576.92', 'h': '138.32', 'w': '9.46'}], [{'page': '1', 'x': '447.75', 'y': '576.92', 'h': '76.66', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '590.47', 'h': '219.63', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '604.02', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '617.56', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '631.11', 'h': '220.18', 'w': '9.46'}]]\",\n",
       " 'pages': \"('1', '1')\",\n",
       " 'section_title': 'Introduction',\n",
       " 'section_number': '1',\n",
       " 'paper_title': 'LLaMA: Open and Efficient Foundation Language Models',\n",
       " 'file_path': '/Users/31treehaus/Desktop/Papers/2302.13971.pdf'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "docs[3].metadata"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}