mirror of
https://github.com/hwchase17/langchain
synced 2024-10-29 17:07:25 +00:00
8d9e9e013c
# Token text splitter for sentence transformers The current TokenTextSplitter only works with OpenAi models via the `tiktoken` package. This is not clear from the name `TokenTextSplitter`. In this (first PR) a token based text splitter for sentence transformer models is added. In the future I think we should work towards injecting a tokenizer into the TokenTextSplitter to make ti more flexible. Could perhaps be reviewed by @dev2049 --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> |
||
---|---|---|
.. | ||
_static | ||
additional_resources | ||
ecosystem | ||
getting_started | ||
integrations | ||
modules | ||
reference | ||
templates | ||
tracing | ||
use_cases | ||
conf.py | ||
dependents.md | ||
index.rst | ||
integrations.rst | ||
make.bat | ||
Makefile | ||
reference.rst | ||
requirements.txt |