mirror of
https://github.com/hwchase17/langchain
synced 2024-11-06 03:20:49 +00:00
e5472b5eb8
## Description I am submitting this for a school project as part of a team of 5. Other team members are @LeilaChr, @maazh10, @Megabear137, @jelalalamy. This PR also has contributions from community members @Harrolee and @Mario928. Initial context is in the issue we opened (#11229). This pull request adds: - Generic framework for expanding the languages that `LanguageParser` can handle, using the [tree-sitter](https://github.com/tree-sitter/py-tree-sitter#py-tree-sitter) parsing library and existing language-specific parsers written for it - Support for the following additional languages in `LanguageParser`: - C - C++ - C# - Go - Java (contributed by @Mario928 https://github.com/ThatsJustCheesy/langchain/pull/2) - Kotlin - Lua - Perl - Ruby - Rust - Scala - TypeScript (contributed by @Harrolee https://github.com/ThatsJustCheesy/langchain/pull/1) Here is the [design document](https://docs.google.com/document/d/17dB14cKCWAaiTeSeBtxHpoVPGKrsPye8W0o_WClz2kk) if curious, but no need to read it. ## Issues - Closes #11229 - Closes #10996 - Closes #8405 ## Dependencies `tree_sitter` and `tree_sitter_languages` on PyPI. We have tried to add these as optional dependencies. ## Documentation We have updated the list of supported languages, and also added a section to `source_code.ipynb` detailing how to add support for additional languages using our framework. ## Maintainer - @hwchase17 (previously reviewed https://github.com/langchain-ai/langchain/pull/6486) Thanks!! ## Git commits We will gladly squash any/all of our commits (esp merge commits) if necessary. Let us know if this is desirable, or if you will be squash-merging anyway. <!-- Thank you for contributing to LangChain! Replace this entire comment with: - **Description:** a description of the change, - **Issue:** the issue # it fixes (if applicable), - **Dependencies:** any dependencies required for this change, - **Tag maintainer:** for a quicker response, tag the relevant maintainer (see below), - **Twitter handle:** we announce bigger features on Twitter. If your PR gets announced, and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/extras` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --> --------- Co-authored-by: Maaz Hashmi <mhashmi373@gmail.com> Co-authored-by: LeilaChr <87657694+LeilaChr@users.noreply.github.com> Co-authored-by: Jeremy La <jeremylai511@gmail.com> Co-authored-by: Megabear137 <zubair.alnoor27@gmail.com> Co-authored-by: Lee Harrold <lhharrold@sep.com> Co-authored-by: Mario928 <88029051+Mario928@users.noreply.github.com> Co-authored-by: Bagatur <baskaryan@gmail.com> Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> |
||
|---|---|---|
| .. | ||
| blob_loaders | ||
| loaders | ||
| parsers | ||
| sample_documents | ||
| test_docs | ||
| __init__.py | ||
| test_airbyte.py | ||
| test_arcgis_loader.py | ||
| test_assemblyai.py | ||
| test_base.py | ||
| test_bibtex.py | ||
| test_bshtml.py | ||
| test_confluence.py | ||
| test_couchbase.py | ||
| test_csv_loader.py | ||
| test_cube_semantic.py | ||
| test_detect_encoding.py | ||
| test_directory.py | ||
| test_evernote_loader.py | ||
| test_generic_loader.py | ||
| test_git.py | ||
| test_github.py | ||
| test_hugging_face.py | ||
| test_imports.py | ||
| test_json_loader.py | ||
| test_lakefs.py | ||
| test_mediawikidump.py | ||
| test_mhtml.py | ||
| test_mongodb.py | ||
| test_obsidian.py | ||
| test_onenote.py | ||
| test_pebblo.py | ||
| test_psychic.py | ||
| test_readthedoc.py | ||
| test_rspace_loader.py | ||
| test_rss.py | ||
| test_trello.py | ||
| test_web_base.py | ||
| test_youtube.py | ||