blob42
/
langchain
forked from Archives/langchain
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat: Add support for the Solidity language (#6054)

Browse Source 
<!--
Thank you for contributing to LangChain! Your PR will appear in our
release under the title you set. Please make sure it highlights your
valuable contribution.

Replace this with a description of the change, the issue it fixes (if
applicable), and relevant context. List any dependencies required for
this change.

After you're done, someone will review your PR. They may suggest
improvements. If no one reviews your PR within a few days, feel free to
@-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd
like us to shout you out on Twitter, please also include your handle!
-->

## Add Solidity programming language support for code splitter.

Twitter: @0xjord4n_

<!-- If you're adding a new integration, please include:

1. a test for the integration - favor unit tests that does not rely on
network access.
2. an example notebook showing its use


See contribution guidelines for more information on how to write tests,
lint
etc:


https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->
#### Who can review?

Tag maintainers/contributors who might be interested:
@hwchase17

<!-- For a quicker response, figure out the right person to tag with @

  @hwchase17 - project lead

  Tracing / Callbacks
  - @agola11

  Async
  - @agola11

  DataLoaders
  - @eyurtsev

  Models
  - @hwchase17
  - @agola11

  Agents / Tools / Toolkits
  - @hwchase17

  VectorStores / Retrievers / Memory
  - @dev2049

 -->
searx_updates
0xJordan 11 months ago committed by GitHub
parent 17c4ec4812
commit c5a46e7435
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
3 changed files with 502 additions and 399 deletions
Show Stats Download Patch File Download Diff File

840
docs/modules/indexes/text_splitters/examples/code_splitter.ipynb
Unescape Escape View File

@ -1,413 +1,457 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# CodeTextSplitter\n",
"\n",
"CodeTextSplitter allows you to split your code with multiple language support. Import enum `Language` and specify the language. "
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from langchain.text_splitter import (\n",
" RecursiveCharacterTextSplitter,\n",
" Language,\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['cpp',\n",
" 'go',\n",
" 'java',\n",
" 'js',\n",
" 'php',\n",
" 'proto',\n",
" 'python',\n",
" 'rst',\n",
" 'ruby',\n",
" 'rust',\n",
" 'scala',\n",
" 'swift',\n",
" 'markdown',\n",
" 'latex',\n",
" 'html']"
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# CodeTextSplitter\n",
"\n",
"CodeTextSplitter allows you to split your code with multiple language support. Import enum `Language` and specify the language. "
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Full list of support languages\n",
"[e.value for e in Language]"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['\\nclass ', '\\ndef ', '\\n\\tdef ', '\\n\\n', '\\n', ' ', '']"
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from langchain.text_splitter import (\n",
" RecursiveCharacterTextSplitter,\n",
" Language,\n",
")"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# You can also see the separators used for a given language\n",
"RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python\n",
"\n",
"Here's an example using the PythonTextSplitter"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='def hello_world():\\n print(\"Hello, World!\")', metadata={}),\n",
" Document(page_content='# Call the function\\nhello_world()', metadata={})]"
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['cpp',\n",
" 'go',\n",
" 'java',\n",
" 'js',\n",
" 'php',\n",
" 'proto',\n",
" 'python',\n",
" 'rst',\n",
" 'ruby',\n",
" 'rust',\n",
" 'scala',\n",
" 'swift',\n",
" 'markdown',\n",
" 'latex',\n",
" 'html',\n",
" 'sol']"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Full list of support languages\n",
"[e.value for e in Language]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PYTHON_CODE = \"\"\"\n",
"def hello_world():\n",
" print(\"Hello, World!\")\n",
"\n",
"# Call the function\n",
"hello_world()\n",
"\"\"\"\n",
"python_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.PYTHON, chunk_size=50, chunk_overlap=0\n",
")\n",
"python_docs = python_splitter.create_documents([PYTHON_CODE])\n",
"python_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## JS\n",
"Here's an example using the JS text splitter"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='function helloWorld() {\\n console.log(\"Hello, World!\");\\n}', metadata={}),\n",
" Document(page_content='// Call the function\\nhelloWorld();', metadata={})]"
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"['\\nclass ', '\\ndef ', '\\n\\tdef ', '\\n\\n', '\\n', ' ', '']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# You can also see the separators used for a given language\n",
"RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON)"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"JS_CODE = \"\"\"\n",
"function helloWorld() {\n",
" console.log(\"Hello, World!\");\n",
"}\n",
"\n",
"// Call the function\n",
"helloWorld();\n",
"\"\"\"\n",
"\n",
"js_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.JS, chunk_size=60, chunk_overlap=0\n",
")\n",
"js_docs = js_splitter.create_documents([JS_CODE])\n",
"js_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Markdown\n",
"\n",
"Here's an example using the Markdown text splitter."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"markdown_text = \"\"\"\n",
"# 🦜️🔗 LangChain\n",
"\n",
"⚡ Building applications with LLMs through composability ⚡\n",
"\n",
"## Quick Install\n",
"\n",
"```bash\n",
"# Hopefully this code block isn't split\n",
"pip install langchain\n",
"```\n",
"\n",
"As an open source project in a rapidly developing field, we are extremely open to contributions.\n",
"\"\"\"\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='# 🦜️🔗 LangChain', metadata={}),\n",
" Document(page_content='⚡ Building applications with LLMs through composability ⚡', metadata={}),\n",
" Document(page_content='## Quick Install', metadata={}),\n",
" Document(page_content=\"```bash\\n# Hopefully this code block isn't split\", metadata={}),\n",
" Document(page_content='pip install langchain', metadata={}),\n",
" Document(page_content='```', metadata={}),\n",
" Document(page_content='As an open source project in a rapidly developing field, we', metadata={}),\n",
" Document(page_content='are extremely open to contributions.', metadata={})]"
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Python\n",
"\n",
"Here's an example using the PythonTextSplitter"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"md_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"md_docs = md_splitter.create_documents([markdown_text])\n",
"md_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Latex\n",
"\n",
"Here's an example on Latex text"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"latex_text = \"\"\"\n",
"\\documentclass{article}\n",
"\n",
"\\begin{document}\n",
"\n",
"\\maketitle\n",
"\n",
"\\section{Introduction}\n",
"Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.\n",
"\n",
"\\subsection{History of LLMs}\n",
"The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.\n",
"\n",
"\\subsection{Applications of LLMs}\n",
"LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.\n",
"\n",
"\\end{document}\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='\\\\documentclass{article}\\n\\n\\x08egin{document}\\n\\n\\\\maketitle', metadata={}),\n",
" Document(page_content='\\\\section{Introduction}', metadata={}),\n",
" Document(page_content='Large language models (LLMs) are a type of machine learning', metadata={}),\n",
" Document(page_content='model that can be trained on vast amounts of text data to', metadata={}),\n",
" Document(page_content='generate human-like language. In recent years, LLMs have', metadata={}),\n",
" Document(page_content='made significant advances in a variety of natural language', metadata={}),\n",
" Document(page_content='processing tasks, including language translation, text', metadata={}),\n",
" Document(page_content='generation, and sentiment analysis.', metadata={}),\n",
" Document(page_content='\\\\subsection{History of LLMs}', metadata={}),\n",
" Document(page_content='The earliest LLMs were developed in the 1980s and 1990s,', metadata={}),\n",
" Document(page_content='but they were limited by the amount of data that could be', metadata={}),\n",
" Document(page_content='processed and the computational power available at the', metadata={}),\n",
" Document(page_content='time. In the past decade, however, advances in hardware and', metadata={}),\n",
" Document(page_content='software have made it possible to train LLMs on massive', metadata={}),\n",
" Document(page_content='datasets, leading to significant improvements in', metadata={}),\n",
" Document(page_content='performance.', metadata={}),\n",
" Document(page_content='\\\\subsection{Applications of LLMs}', metadata={}),\n",
" Document(page_content='LLMs have many applications in industry, including', metadata={}),\n",
" Document(page_content='chatbots, content creation, and virtual assistants. They', metadata={}),\n",
" Document(page_content='can also be used in academia for research in linguistics,', metadata={}),\n",
" Document(page_content='psychology, and computational linguistics.', metadata={}),\n",
" Document(page_content='\\\\end{document}', metadata={})]"
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='def hello_world():\\n print(\"Hello, World!\")', metadata={}),\n",
" Document(page_content='# Call the function\\nhello_world()', metadata={})]"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PYTHON_CODE = \"\"\"\n",
"def hello_world():\n",
" print(\"Hello, World!\")\n",
"\n",
"# Call the function\n",
"hello_world()\n",
"\"\"\"\n",
"python_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.PYTHON, chunk_size=50, chunk_overlap=0\n",
")\n",
"python_docs = python_splitter.create_documents([PYTHON_CODE])\n",
"python_docs"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"latex_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"latex_docs = latex_splitter.create_documents([latex_text])\n",
"latex_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## HTML\n",
"\n",
"Here's an example using an HTML text splitter"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"html_text = \"\"\"\n",
"<!DOCTYPE html>\n",
"<html>\n",
" <head>\n",
" <title>🦜️🔗 LangChain</title>\n",
" <style>\n",
" body {\n",
" font-family: Arial, sans-serif;\n",
" }\n",
" h1 {\n",
" color: darkblue;\n",
" }\n",
" </style>\n",
" </head>\n",
" <body>\n",
" <div>\n",
" <h1>🦜️🔗 LangChain</h1>\n",
" <p>⚡ Building applications with LLMs through composability ⚡</p>\n",
" </div>\n",
" <div>\n",
" As an open source project in a rapidly developing field, we are extremely open to contributions.\n",
" </div>\n",
" </body>\n",
"</html>\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='<!DOCTYPE html>\\n<html>\\n <head>', metadata={}),\n",
" Document(page_content='<title>🦜️🔗 LangChain</title>\\n <style>', metadata={}),\n",
" Document(page_content='body {', metadata={}),\n",
" Document(page_content='font-family: Arial, sans-serif;', metadata={}),\n",
" Document(page_content='}\\n h1 {', metadata={}),\n",
" Document(page_content='color: darkblue;\\n }', metadata={}),\n",
" Document(page_content='</style>\\n </head>\\n <body>\\n <div>', metadata={}),\n",
" Document(page_content='<h1>🦜️🔗 LangChain</h1>', metadata={}),\n",
" Document(page_content='<p>⚡ Building applications with LLMs through', metadata={}),\n",
" Document(page_content='composability ⚡</p>', metadata={}),\n",
" Document(page_content='</div>\\n <div>', metadata={}),\n",
" Document(page_content='As an open source project in a rapidly', metadata={}),\n",
" Document(page_content='developing field, we are extremely open to contributions.', metadata={}),\n",
" Document(page_content='</div>\\n </body>\\n</html>', metadata={})]"
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## JS\n",
"Here's an example using the JS text splitter"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"html_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"html_docs = html_splitter.create_documents([html_text])\n",
"html_docs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='function helloWorld() {\\n console.log(\"Hello, World!\");\\n}', metadata={}),\n",
" Document(page_content='// Call the function\\nhelloWorld();', metadata={})]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"JS_CODE = \"\"\"\n",
"function helloWorld() {\n",
" console.log(\"Hello, World!\");\n",
"}\n",
"\n",
"// Call the function\n",
"helloWorld();\n",
"\"\"\"\n",
"\n",
"js_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.JS, chunk_size=60, chunk_overlap=0\n",
")\n",
"js_docs = js_splitter.create_documents([JS_CODE])\n",
"js_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Solidity\n",
"Here's an example using the Solidity text splitter"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='pragma solidity ^0.8.20;', metadata={}),\n",
" Document(page_content='contract HelloWorld {\\n function add(uint a, uint b) pure public returns(uint) {\\n return a + b;\\n }\\n}', metadata={})]"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"SOL_CODE = \"\"\"\n",
"pragma solidity ^0.8.20;",
"\n",
"contract HelloWorld {\n",
" function add(uint a, uint b) pure public returns(uint) {\n",
" return a + b;\n",
" }\n",
"}\n",
"\"\"\"\n",
"\n",
"sol_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.SOL, chunk_size=128, chunk_overlap=0\n",
")\n",
"sol_docs = sol_splitter.create_documents([SOL_CODE])\n",
"sol_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Markdown\n",
"\n",
"Here's an example using the Markdown text splitter."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"markdown_text = \"\"\"\n",
"# 🦜️🔗 LangChain\n",
"\n",
"⚡ Building applications with LLMs through composability ⚡\n",
"\n",
"## Quick Install\n",
"\n",
"```bash\n",
"# Hopefully this code block isn't split\n",
"pip install langchain\n",
"```\n",
"\n",
"As an open source project in a rapidly developing field, we are extremely open to contributions.\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='# 🦜️🔗 LangChain', metadata={}),\n",
" Document(page_content='⚡ Building applications with LLMs through composability ⚡', metadata={}),\n",
" Document(page_content='## Quick Install', metadata={}),\n",
" Document(page_content=\"```bash\\n# Hopefully this code block isn't split\", metadata={}),\n",
" Document(page_content='pip install langchain', metadata={}),\n",
" Document(page_content='```', metadata={}),\n",
" Document(page_content='As an open source project in a rapidly developing field, we', metadata={}),\n",
" Document(page_content='are extremely open to contributions.', metadata={})]"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"md_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"md_docs = md_splitter.create_documents([markdown_text])\n",
"md_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Latex\n",
"\n",
"Here's an example on Latex text"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"latex_text = \"\"\"\n",
"\\documentclass{article}\n",
"\n",
"\\begin{document}\n",
"\n",
"\\maketitle\n",
"\n",
"\\section{Introduction}\n",
"Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.\n",
"\n",
"\\subsection{History of LLMs}\n",
"The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.\n",
"\n",
"\\subsection{Applications of LLMs}\n",
"LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.\n",
"\n",
"\\end{document}\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='\\\\documentclass{article}\\n\\n\\x08egin{document}\\n\\n\\\\maketitle', metadata={}),\n",
" Document(page_content='\\\\section{Introduction}', metadata={}),\n",
" Document(page_content='Large language models (LLMs) are a type of machine learning', metadata={}),\n",
" Document(page_content='model that can be trained on vast amounts of text data to', metadata={}),\n",
" Document(page_content='generate human-like language. In recent years, LLMs have', metadata={}),\n",
" Document(page_content='made significant advances in a variety of natural language', metadata={}),\n",
" Document(page_content='processing tasks, including language translation, text', metadata={}),\n",
" Document(page_content='generation, and sentiment analysis.', metadata={}),\n",
" Document(page_content='\\\\subsection{History of LLMs}', metadata={}),\n",
" Document(page_content='The earliest LLMs were developed in the 1980s and 1990s,', metadata={}),\n",
" Document(page_content='but they were limited by the amount of data that could be', metadata={}),\n",
" Document(page_content='processed and the computational power available at the', metadata={}),\n",
" Document(page_content='time. In the past decade, however, advances in hardware and', metadata={}),\n",
" Document(page_content='software have made it possible to train LLMs on massive', metadata={}),\n",
" Document(page_content='datasets, leading to significant improvements in', metadata={}),\n",
" Document(page_content='performance.', metadata={}),\n",
" Document(page_content='\\\\subsection{Applications of LLMs}', metadata={}),\n",
" Document(page_content='LLMs have many applications in industry, including', metadata={}),\n",
" Document(page_content='chatbots, content creation, and virtual assistants. They', metadata={}),\n",
" Document(page_content='can also be used in academia for research in linguistics,', metadata={}),\n",
" Document(page_content='psychology, and computational linguistics.', metadata={}),\n",
" Document(page_content='\\\\end{document}', metadata={})]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"latex_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"latex_docs = latex_splitter.create_documents([latex_text])\n",
"latex_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## HTML\n",
"\n",
"Here's an example using an HTML text splitter"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"html_text = \"\"\"\n",
"<!DOCTYPE html>\n",
"<html>\n",
" <head>\n",
" <title>🦜️🔗 LangChain</title>\n",
" <style>\n",
" body {\n",
" font-family: Arial, sans-serif;\n",
" }\n",
" h1 {\n",
" color: darkblue;\n",
" }\n",
" </style>\n",
" </head>\n",
" <body>\n",
" <div>\n",
" <h1>🦜️🔗 LangChain</h1>\n",
" <p>⚡ Building applications with LLMs through composability ⚡</p>\n",
" </div>\n",
" <div>\n",
" As an open source project in a rapidly developing field, we are extremely open to contributions.\n",
" </div>\n",
" </body>\n",
"</html>\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='<!DOCTYPE html>\\n<html>\\n <head>', metadata={}),\n",
" Document(page_content='<title>🦜️🔗 LangChain</title>\\n <style>', metadata={}),\n",
" Document(page_content='body {', metadata={}),\n",
" Document(page_content='font-family: Arial, sans-serif;', metadata={}),\n",
" Document(page_content='}\\n h1 {', metadata={}),\n",
" Document(page_content='color: darkblue;\\n }', metadata={}),\n",
" Document(page_content='</style>\\n </head>\\n <body>\\n <div>', metadata={}),\n",
" Document(page_content='<h1>🦜️🔗 LangChain</h1>', metadata={}),\n",
" Document(page_content='<p>⚡ Building applications with LLMs through', metadata={}),\n",
" Document(page_content='composability ⚡</p>', metadata={}),\n",
" Document(page_content='</div>\\n <div>', metadata={}),\n",
" Document(page_content='As an open source project in a rapidly', metadata={}),\n",
" Document(page_content='developing field, we are extremely open to contributions.', metadata={}),\n",
" Document(page_content='</div>\\n </body>\\n</html>', metadata={})]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"html_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"html_docs = html_splitter.create_documents([html_text])\n",
"html_docs"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.1"
}
},
"nbformat": 4,
"nbformat_minor": 2
}

33
langchain/text_splitter.py
Unescape Escape View File

@ -565,6 +565,7 @@ class Language(str, Enum):
MARKDOWN = "markdown"
LATEX = "latex"
HTML = "html"
SOL = "sol"
class RecursiveCharacterTextSplitter(TextSplitter):
@ -895,7 +896,7 @@ class RecursiveCharacterTextSplitter(TextSplitter):
"\n\\\begin{quotation}",
"\n\\\begin{verse}",
"\n\\\begin{verbatim}",
## Now split by math environments
# Now split by math environments
"\n\\\begin{align}",
"$$",
"$",
@ -935,6 +936,36 @@ class RecursiveCharacterTextSplitter(TextSplitter):
"<title",
"",
]
elif language == Language.SOL:
return [
# Split along compiler informations definitions
"\npragma ",
"\nusing ",
# Split along contract definitions
"\ncontract ",
"\ninterface ",
"\nlibrary ",
# Split along method definitions
"\nconstructor ",
"\ntype ",
"\nfunction ",
"\nevent ",
"\nmodifier ",
"\nerror ",
"\nstruct ",
"\nenum ",
# Split along control flow statements
"\nif ",
"\nfor ",
"\nwhile ",
"\ndo while ",
"\nassembly ",
# Split by the normal type of lines
"\n\n",
"\n",
" ",
"",
]
else:
raise ValueError(
f"Language {language} is not supported! "

28
tests/unit_tests/test_text_splitter.py
Unescape Escape View File

@ -798,3 +798,31 @@ def test_md_header_text_splitter_3() -> None:
]
assert output == expected_output
def test_solidity_code_splitter() -> None:
splitter = RecursiveCharacterTextSplitter.from_language(
Language.SOL, chunk_size=CHUNK_SIZE, chunk_overlap=0
)
code = """pragma solidity ^0.8.20;
contract HelloWorld {
function add(uint a, uint b) pure public returns(uint) {
return a + b;
}
}
"""
chunks = splitter.split_text(code)
assert chunks == [
"pragma solidity",
"^0.8.20;",
"contract",
"HelloWorld {",
"function",
"add(uint a,",
"uint b) pure",
"public",
"returns(uint) {",
"return a",
"+ b;",
"}\n }",
]

Write Preview
Loading…
Cancel
Save