Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# CodeTextSplitter\n",
"\n",
"CodeTextSplitter allows you to split your code with multiple language support. Import enum `Language` and specify the language. "
]
},
{
"cell_type": "code",
2023-05-31 14:11:53 +00:00
"execution_count": 1,
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"metadata": {},
"outputs": [],
"source": [
"from langchain.text_splitter import (\n",
2023-05-31 14:11:53 +00:00
" RecursiveCharacterTextSplitter,\n",
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
" Language,\n",
")"
]
},
{
2023-05-31 14:11:53 +00:00
"cell_type": "code",
"execution_count": 2,
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"metadata": {},
2023-05-31 14:11:53 +00:00
"outputs": [
{
"data": {
"text/plain": [
"['cpp',\n",
" 'go',\n",
" 'java',\n",
" 'js',\n",
" 'php',\n",
" 'proto',\n",
" 'python',\n",
" 'rst',\n",
" 'ruby',\n",
" 'rust',\n",
" 'scala',\n",
" 'swift',\n",
" 'markdown',\n",
" 'latex',\n",
" 'html']"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"source": [
2023-05-31 14:11:53 +00:00
"# Full list of support languages\n",
"[e.value for e in Language]"
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
]
},
{
"cell_type": "code",
2023-05-31 14:11:53 +00:00
"execution_count": 3,
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"metadata": {},
2023-05-31 14:11:53 +00:00
"outputs": [
{
"data": {
"text/plain": [
"['\\nclass ', '\\ndef ', '\\n\\tdef ', '\\n\\n', '\\n', ' ', '']"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"source": [
2023-05-31 14:11:53 +00:00
"# You can also see the separators used for a given language\n",
"RecursiveCharacterTextSplitter.get_separators_for_language(Language.PYTHON)"
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
2023-05-31 14:11:53 +00:00
"## Python\n",
"\n",
"Here's an example using the PythonTextSplitter"
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
]
},
{
"cell_type": "code",
2023-05-31 14:11:53 +00:00
"execution_count": 4,
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
2023-05-31 14:11:53 +00:00
"[Document(page_content='def hello_world():\\n print(\"Hello, World!\")', metadata={}),\n",
" Document(page_content='# Call the function\\nhello_world()', metadata={})]"
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
]
},
2023-05-31 14:11:53 +00:00
"execution_count": 4,
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"PYTHON_CODE = \"\"\"\n",
"def hello_world():\n",
" print(\"Hello, World!\")\n",
"\n",
"# Call the function\n",
"hello_world()\n",
"\"\"\"\n",
2023-05-31 14:11:53 +00:00
"python_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.PYTHON, chunk_size=50, chunk_overlap=0\n",
")\n",
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"python_docs = python_splitter.create_documents([PYTHON_CODE])\n",
"python_docs"
]
},
2023-05-31 14:11:53 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## JS\n",
"Here's an example using the JS text splitter"
]
},
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
{
"cell_type": "code",
2023-05-31 14:11:53 +00:00
"execution_count": 5,
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
2023-05-31 14:11:53 +00:00
"[Document(page_content='function helloWorld() {\\n console.log(\"Hello, World!\");\\n}', metadata={}),\n",
" Document(page_content='// Call the function\\nhelloWorld();', metadata={})]"
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
]
},
2023-05-31 14:11:53 +00:00
"execution_count": 5,
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"JS_CODE = \"\"\"\n",
"function helloWorld() {\n",
" console.log(\"Hello, World!\");\n",
"}\n",
"\n",
"// Call the function\n",
"helloWorld();\n",
"\"\"\"\n",
"\n",
2023-05-31 14:11:53 +00:00
"js_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.JS, chunk_size=60, chunk_overlap=0\n",
")\n",
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"js_docs = js_splitter.create_documents([JS_CODE])\n",
"js_docs"
]
},
2023-05-31 14:11:53 +00:00
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Markdown\n",
"\n",
"Here's an example using the Markdown text splitter."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [],
"source": [
"markdown_text = \"\"\"\n",
"# 🦜️🔗 LangChain\n",
"\n",
"⚡ Building applications with LLMs through composability ⚡\n",
"\n",
"## Quick Install\n",
"\n",
"```bash\n",
"# Hopefully this code block isn't split\n",
"pip install langchain\n",
"```\n",
"\n",
"As an open source project in a rapidly developing field, we are extremely open to contributions.\n",
"\"\"\"\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='# 🦜️🔗 LangChain', metadata={}),\n",
" Document(page_content='⚡ Building applications with LLMs through composability ⚡', metadata={}),\n",
" Document(page_content='## Quick Install', metadata={}),\n",
" Document(page_content=\"```bash\\n# Hopefully this code block isn't split\", metadata={}),\n",
" Document(page_content='pip install langchain', metadata={}),\n",
" Document(page_content='```', metadata={}),\n",
" Document(page_content='As an open source project in a rapidly developing field, we', metadata={}),\n",
" Document(page_content='are extremely open to contributions.', metadata={})]"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"md_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"md_docs = md_splitter.create_documents([markdown_text])\n",
"md_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Latex\n",
"\n",
"Here's an example on Latex text"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"latex_text = \"\"\"\n",
"\\documentclass{article}\n",
"\n",
"\\begin{document}\n",
"\n",
"\\maketitle\n",
"\n",
"\\section{Introduction}\n",
"Large language models (LLMs) are a type of machine learning model that can be trained on vast amounts of text data to generate human-like language. In recent years, LLMs have made significant advances in a variety of natural language processing tasks, including language translation, text generation, and sentiment analysis.\n",
"\n",
"\\subsection{History of LLMs}\n",
"The earliest LLMs were developed in the 1980s and 1990s, but they were limited by the amount of data that could be processed and the computational power available at the time. In the past decade, however, advances in hardware and software have made it possible to train LLMs on massive datasets, leading to significant improvements in performance.\n",
"\n",
"\\subsection{Applications of LLMs}\n",
"LLMs have many applications in industry, including chatbots, content creation, and virtual assistants. They can also be used in academia for research in linguistics, psychology, and computational linguistics.\n",
"\n",
"\\end{document}\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='\\\\documentclass{article}\\n\\n\\x08egin{document}\\n\\n\\\\maketitle', metadata={}),\n",
" Document(page_content='\\\\section{Introduction}', metadata={}),\n",
" Document(page_content='Large language models (LLMs) are a type of machine learning', metadata={}),\n",
" Document(page_content='model that can be trained on vast amounts of text data to', metadata={}),\n",
" Document(page_content='generate human-like language. In recent years, LLMs have', metadata={}),\n",
" Document(page_content='made significant advances in a variety of natural language', metadata={}),\n",
" Document(page_content='processing tasks, including language translation, text', metadata={}),\n",
" Document(page_content='generation, and sentiment analysis.', metadata={}),\n",
" Document(page_content='\\\\subsection{History of LLMs}', metadata={}),\n",
" Document(page_content='The earliest LLMs were developed in the 1980s and 1990s,', metadata={}),\n",
" Document(page_content='but they were limited by the amount of data that could be', metadata={}),\n",
" Document(page_content='processed and the computational power available at the', metadata={}),\n",
" Document(page_content='time. In the past decade, however, advances in hardware and', metadata={}),\n",
" Document(page_content='software have made it possible to train LLMs on massive', metadata={}),\n",
" Document(page_content='datasets, leading to significant improvements in', metadata={}),\n",
" Document(page_content='performance.', metadata={}),\n",
" Document(page_content='\\\\subsection{Applications of LLMs}', metadata={}),\n",
" Document(page_content='LLMs have many applications in industry, including', metadata={}),\n",
" Document(page_content='chatbots, content creation, and virtual assistants. They', metadata={}),\n",
" Document(page_content='can also be used in academia for research in linguistics,', metadata={}),\n",
" Document(page_content='psychology, and computational linguistics.', metadata={}),\n",
" Document(page_content='\\\\end{document}', metadata={})]"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"latex_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"latex_docs = latex_splitter.create_documents([latex_text])\n",
"latex_docs"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## HTML\n",
"\n",
"Here's an example using an HTML text splitter"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"html_text = \"\"\"\n",
"<!DOCTYPE html>\n",
"<html>\n",
" <head>\n",
" <title>🦜️🔗 LangChain</title>\n",
" <style>\n",
" body {\n",
" font-family: Arial, sans-serif;\n",
" }\n",
" h1 {\n",
" color: darkblue;\n",
" }\n",
" </style>\n",
" </head>\n",
" <body>\n",
" <div>\n",
" <h1>🦜️🔗 LangChain</h1>\n",
" <p>⚡ Building applications with LLMs through composability ⚡</p>\n",
" </div>\n",
" <div>\n",
" As an open source project in a rapidly developing field, we are extremely open to contributions.\n",
" </div>\n",
" </body>\n",
"</html>\n",
"\"\"\""
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[Document(page_content='<!DOCTYPE html>\\n<html>\\n <head>', metadata={}),\n",
" Document(page_content='<title>🦜️🔗 LangChain</title>\\n <style>', metadata={}),\n",
" Document(page_content='body {', metadata={}),\n",
" Document(page_content='font-family: Arial, sans-serif;', metadata={}),\n",
" Document(page_content='}\\n h1 {', metadata={}),\n",
" Document(page_content='color: darkblue;\\n }', metadata={}),\n",
" Document(page_content='</style>\\n </head>\\n <body>\\n <div>', metadata={}),\n",
" Document(page_content='<h1>🦜️🔗 LangChain</h1>', metadata={}),\n",
" Document(page_content='<p>⚡ Building applications with LLMs through', metadata={}),\n",
" Document(page_content='composability ⚡</p>', metadata={}),\n",
" Document(page_content='</div>\\n <div>', metadata={}),\n",
" Document(page_content='As an open source project in a rapidly', metadata={}),\n",
" Document(page_content='developing field, we are extremely open to contributions.', metadata={}),\n",
" Document(page_content='</div>\\n </body>\\n</html>', metadata={})]"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"html_splitter = RecursiveCharacterTextSplitter.from_language(\n",
" language=Language.MARKDOWN, chunk_size=60, chunk_overlap=0\n",
")\n",
"html_docs = html_splitter.create_documents([html_text])\n",
"html_docs"
]
},
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
2023-05-31 14:11:53 +00:00
"display_name": "Python 3 (ipykernel)",
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
2023-05-31 14:11:53 +00:00
"version": "3.9.1"
}
Add more code splitters (go, rst, js, java, cpp, scala, ruby, php, swift, rust) (#5171)
As the title says, I added more code splitters.
The implementation is trivial, so i don't add separate tests for each
splitter.
Let me know if any concerns.
Fixes # (issue)
https://github.com/hwchase17/langchain/issues/5170
## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@eyurtsev @hwchase17
---------
Signed-off-by: byhsu <byhsu@linkedin.com>
Co-authored-by: byhsu <byhsu@linkedin.com>
2023-05-30 15:04:05 +00:00
},
"nbformat": 4,
"nbformat_minor": 2
}