mirror of
https://github.com/hwchase17/langchain
synced 2024-10-29 17:07:25 +00:00
20c6ade2fc
### Scientific Article PDF Parsing via Grobid `Description:` This change adds the GrobidParser class, which uses the Grobid library to parse scientific articles into a universal XML format containing the article title, references, sections, section text etc. The GrobidParser uses a local Grobid server to return PDFs document as XML and parses the XML to optionally produce documents of individual sentences or of whole paragraphs. Metadata includes the text, paragraph number, pdf relative bboxes, pages (text may overlap over two pages), section title (Introduction, Methodology etc), section_number (i.e 1.1, 2.3), the title of the paper and finally the file path. Grobid parsing is useful beyond standard pdf parsing as it accurately outputs sections and paragraphs within them. This allows for post-fitering of results for specific sections i.e. limiting results to the methodology section or results. While sections are split via headings, ideally they could be classified specifically into introduction, methodology, results, discussion, conclusion. I'm currently experimenting with chatgpt-3.5 for this function, which could later be implemented as a textsplitter. `Dependencies:` For use, the grobid repo must be cloned and Java must be installed, for colab this is: ``` !apt-get install -y openjdk-11-jdk -q !update-alternatives --set java /usr/lib/jvm/java-11-openjdk-amd64/bin/java !git clone https://github.com/kermitt2/grobid.git os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-11-openjdk-amd64" os.chdir('grobid') !./gradlew clean install ``` Once installed the server is ran on localhost:8070 via ``` get_ipython().system_raw('nohup ./gradlew run > grobid.log 2>&1 &') ``` @rlancemartin, @eyurtsev Twitter Handle: @Corranmac Grobid Demo Notebook is [here](https://colab.research.google.com/drive/1X-St_mQRmmm8YWtct_tcJNtoktbdGBmd?usp=sharing). --------- Co-authored-by: rlm <pexpresss31@gmail.com>
181 lines
6.0 KiB
Plaintext
181 lines
6.0 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bdccb278",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Grobid\n",
|
|
"\n",
|
|
"GROBID is a machine learning library for extracting, parsing, and re-structuring raw documents.\n",
|
|
"\n",
|
|
"It is particularly good for sturctured PDFs, like academic papers.\n",
|
|
"\n",
|
|
"This loader uses GROBIB to parse PDFs into `Documents` that retain metadata associated with the section of text.\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"For users on `Mac` - \n",
|
|
"\n",
|
|
"(Note: additional instructions can be found [here](https://python.langchain.com/docs/ecosystem/integrations/grobid.mdx).)\n",
|
|
"\n",
|
|
"Install Java (Apple Silicon):\n",
|
|
"```\n",
|
|
"$ arch -arm64 brew install openjdk@11\n",
|
|
"$ brew --prefix openjdk@11\n",
|
|
"/opt/homebrew/opt/openjdk@ 11\n",
|
|
"```\n",
|
|
"\n",
|
|
"In `~/.zshrc`:\n",
|
|
"```\n",
|
|
"export JAVA_HOME=/opt/homebrew/opt/openjdk@11\n",
|
|
"export PATH=$JAVA_HOME/bin:$PATH\n",
|
|
"```\n",
|
|
"\n",
|
|
"Then, in Terminal:\n",
|
|
"```\n",
|
|
"$ source ~/.zshrc\n",
|
|
"```\n",
|
|
"\n",
|
|
"Confirm install:\n",
|
|
"```\n",
|
|
"$ which java\n",
|
|
"/opt/homebrew/opt/openjdk@11/bin/java\n",
|
|
"$ java -version \n",
|
|
"openjdk version \"11.0.19\" 2023-04-18\n",
|
|
"OpenJDK Runtime Environment Homebrew (build 11.0.19+0)\n",
|
|
"OpenJDK 64-Bit Server VM Homebrew (build 11.0.19+0, mixed mode)\n",
|
|
"```\n",
|
|
"\n",
|
|
"Then, get [Grobid](https://grobid.readthedocs.io/en/latest/Install-Grobid/#getting-grobid):\n",
|
|
"```\n",
|
|
"$ curl -LO https://github.com/kermitt2/grobid/archive/0.7.3.zip\n",
|
|
"$ unzip 0.7.3.zip\n",
|
|
"```\n",
|
|
" \n",
|
|
"Build\n",
|
|
"```\n",
|
|
"$ ./gradlew clean install\n",
|
|
"```\n",
|
|
"\n",
|
|
"Then, run the server:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "2d8992fc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"! get_ipython().system_raw('nohup ./gradlew run > grobid.log 2>&1 &')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4b41bfb1",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now, we can use the data loader."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "640e9a4b",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.document_loaders.parsers import GrobidParser\n",
|
|
"from langchain.document_loaders.generic import GenericLoader"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "ecdc1fb9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"loader = GenericLoader.from_filesystem(\n",
|
|
" \"../Papers/\",\n",
|
|
" glob=\"*\",\n",
|
|
" suffixes=[\".pdf\"],\n",
|
|
" parser= GrobidParser(segment_sentences=False)\n",
|
|
")\n",
|
|
"docs = loader.load()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "efe9e356",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g.\"Books -2TB\" or \"Social media conversations\").There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.'"
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"docs[3].page_content"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "5be03d17",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'text': 'Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g.\"Books -2TB\" or \"Social media conversations\").There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.',\n",
|
|
" 'para': '2',\n",
|
|
" 'bboxes': \"[[{'page': '1', 'x': '317.05', 'y': '509.17', 'h': '207.73', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '522.72', 'h': '220.08', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '536.27', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '549.82', 'h': '218.65', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '563.37', 'h': '136.98', 'w': '9.46'}], [{'page': '1', 'x': '446.49', 'y': '563.37', 'h': '78.11', 'w': '9.46'}, {'page': '1', 'x': '304.69', 'y': '576.92', 'h': '138.32', 'w': '9.46'}], [{'page': '1', 'x': '447.75', 'y': '576.92', 'h': '76.66', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '590.47', 'h': '219.63', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '604.02', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '617.56', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '631.11', 'h': '220.18', 'w': '9.46'}]]\",\n",
|
|
" 'pages': \"('1', '1')\",\n",
|
|
" 'section_title': 'Introduction',\n",
|
|
" 'section_number': '1',\n",
|
|
" 'paper_title': 'LLaMA: Open and Efficient Foundation Language Models',\n",
|
|
" 'file_path': '/Users/31treehaus/Desktop/Papers/2302.13971.pdf'}"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"docs[3].metadata"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.16"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|