mirror of
https://github.com/hwchase17/langchain
synced 2024-10-31 15:20:26 +00:00
181 lines
6.0 KiB
Plaintext
181 lines
6.0 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bdccb278",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Grobid\n",
|
|
"\n",
|
|
"GROBID is a machine learning library for extracting, parsing, and re-structuring raw documents.\n",
|
|
"\n",
|
|
"It is particularly good for sturctured PDFs, like academic papers.\n",
|
|
"\n",
|
|
"This loader uses GROBIB to parse PDFs into `Documents` that retain metadata associated with the section of text.\n",
|
|
"\n",
|
|
"---\n",
|
|
"\n",
|
|
"For users on `Mac` - \n",
|
|
"\n",
|
|
"(Note: additional instructions can be found [here](https://python.langchain.com/docs/ecosystem/integrations/grobid.mdx).)\n",
|
|
"\n",
|
|
"Install Java (Apple Silicon):\n",
|
|
"```\n",
|
|
"$ arch -arm64 brew install openjdk@11\n",
|
|
"$ brew --prefix openjdk@11\n",
|
|
"/opt/homebrew/opt/openjdk@ 11\n",
|
|
"```\n",
|
|
"\n",
|
|
"In `~/.zshrc`:\n",
|
|
"```\n",
|
|
"export JAVA_HOME=/opt/homebrew/opt/openjdk@11\n",
|
|
"export PATH=$JAVA_HOME/bin:$PATH\n",
|
|
"```\n",
|
|
"\n",
|
|
"Then, in Terminal:\n",
|
|
"```\n",
|
|
"$ source ~/.zshrc\n",
|
|
"```\n",
|
|
"\n",
|
|
"Confirm install:\n",
|
|
"```\n",
|
|
"$ which java\n",
|
|
"/opt/homebrew/opt/openjdk@11/bin/java\n",
|
|
"$ java -version \n",
|
|
"openjdk version \"11.0.19\" 2023-04-18\n",
|
|
"OpenJDK Runtime Environment Homebrew (build 11.0.19+0)\n",
|
|
"OpenJDK 64-Bit Server VM Homebrew (build 11.0.19+0, mixed mode)\n",
|
|
"```\n",
|
|
"\n",
|
|
"Then, get [Grobid](https://grobid.readthedocs.io/en/latest/Install-Grobid/#getting-grobid):\n",
|
|
"```\n",
|
|
"$ curl -LO https://github.com/kermitt2/grobid/archive/0.7.3.zip\n",
|
|
"$ unzip 0.7.3.zip\n",
|
|
"```\n",
|
|
" \n",
|
|
"Build\n",
|
|
"```\n",
|
|
"$ ./gradlew clean install\n",
|
|
"```\n",
|
|
"\n",
|
|
"Then, run the server:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "2d8992fc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"! get_ipython().system_raw('nohup ./gradlew run > grobid.log 2>&1 &')"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4b41bfb1",
|
|
"metadata": {},
|
|
"source": [
|
|
"Now, we can use the data loader."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "640e9a4b",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.document_loaders.parsers import GrobidParser\n",
|
|
"from langchain.document_loaders.generic import GenericLoader"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "ecdc1fb9",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"loader = GenericLoader.from_filesystem(\n",
|
|
" \"../Papers/\",\n",
|
|
" glob=\"*\",\n",
|
|
" suffixes=[\".pdf\"],\n",
|
|
" parser=GrobidParser(segment_sentences=False),\n",
|
|
")\n",
|
|
"docs = loader.load()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 5,
|
|
"id": "efe9e356",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g.\"Books -2TB\" or \"Social media conversations\").There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.'"
|
|
]
|
|
},
|
|
"execution_count": 5,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"docs[3].page_content"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "5be03d17",
|
|
"metadata": {},
|
|
"outputs": [
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"{'text': 'Unlike Chinchilla, PaLM, or GPT-3, we only use publicly available data, making our work compatible with open-sourcing, while most existing models rely on data which is either not publicly available or undocumented (e.g.\"Books -2TB\" or \"Social media conversations\").There exist some exceptions, notably OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) and GLM (Zeng et al., 2022), but none that are competitive with PaLM-62B or Chinchilla.',\n",
|
|
" 'para': '2',\n",
|
|
" 'bboxes': \"[[{'page': '1', 'x': '317.05', 'y': '509.17', 'h': '207.73', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '522.72', 'h': '220.08', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '536.27', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '549.82', 'h': '218.65', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '563.37', 'h': '136.98', 'w': '9.46'}], [{'page': '1', 'x': '446.49', 'y': '563.37', 'h': '78.11', 'w': '9.46'}, {'page': '1', 'x': '304.69', 'y': '576.92', 'h': '138.32', 'w': '9.46'}], [{'page': '1', 'x': '447.75', 'y': '576.92', 'h': '76.66', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '590.47', 'h': '219.63', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '604.02', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '617.56', 'h': '218.27', 'w': '9.46'}, {'page': '1', 'x': '306.14', 'y': '631.11', 'h': '220.18', 'w': '9.46'}]]\",\n",
|
|
" 'pages': \"('1', '1')\",\n",
|
|
" 'section_title': 'Introduction',\n",
|
|
" 'section_number': '1',\n",
|
|
" 'paper_title': 'LLaMA: Open and Efficient Foundation Language Models',\n",
|
|
" 'file_path': '/Users/31treehaus/Desktop/Papers/2302.13971.pdf'}"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"docs[3].metadata"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.16"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|