You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
langchain/libs/experimental
Raviraj 858ce264ef
SemanticChunker : Feature Addition ("Semantic Splitting with gradient") (#22895)
```SemanticChunker``` currently provide three methods to split the texts semantically:
- percentile
- standard_deviation
- interquartile

I propose new method ```gradient```. In this method, the gradient of distance is used to split chunks along with the percentile method (technically) . This method is useful when chunks are highly correlated with each other or specific to a domain e.g. legal or medical. The idea is to apply anomaly detection on gradient array so that the distribution become wider and easy to identify boundaries in highly semantic data.
I have tested this merge on a set of 10 domain specific documents (mostly legal).

Details : 
    - **Issue:** Improvement
    - **Dependencies:** NA
    - **Twitter handle:** [x.com/prajapat_ravi](https://x.com/prajapat_ravi)


@hwchase17

---------

Co-authored-by: Raviraj Prajapat <raviraj.prajapat@sirionlabs.com>
Co-authored-by: isaac hershenson <ihershenson@hmc.edu>
3 months ago
..
langchain_experimental SemanticChunker : Feature Addition ("Semantic Splitting with gradient") (#22895) 3 months ago
scripts infra: add print rule to ruff (#16221) 7 months ago
tests Fix: lint errors and update Field alias in models.py and AutoSelectionScorer initialization (#22846) 3 months ago
LICENSE Library Licenses (#13300) 10 months ago
Makefile create mypy cache dir if it doesn't exist (#14579) 9 months ago
README.md CONTRIBUTING.md Quick Start: focus on langchain core; clarify docs and experimental are separate (#10906) 12 months ago
extended_testing_deps.txt ci: add testing with Python 3.12 (#22813) 3 months ago
poetry.lock experimental: release 0.0.61 (#22924) 3 months ago
poetry.toml Harrison/move experimental (#8084) 1 year ago
pyproject.toml experimental: release 0.0.61 (#22924) 3 months ago

README.md

🦜🧪 LangChain Experimental

This package holds experimental LangChain code, intended for research and experimental uses.

[!WARNING] Portions of the code in this package may be dangerous if not properly deployed in a sandboxed environment. Please be wary of deploying experimental code to production unless you've taken appropriate precautions and have already discussed it with your security team.

Some of the code here may be marked with security notices. However, given the exploratory and experimental nature of the code in this package, the lack of a security notice on a piece of code does not mean that the code in question does not require additional security considerations in order to be safe to use.