mirror of
https://github.com/hwchase17/langchain
synced 2024-10-29 17:07:25 +00:00
8d9e9e013c
# Token text splitter for sentence transformers The current TokenTextSplitter only works with OpenAi models via the `tiktoken` package. This is not clear from the name `TokenTextSplitter`. In this (first PR) a token based text splitter for sentence transformer models is added. In the future I think we should work towards injecting a tokenizer into the TokenTextSplitter to make ti more flexible. Could perhaps be reviewed by @dev2049 --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> |
||
---|---|---|
.. | ||
agent | ||
cache | ||
callbacks | ||
chains | ||
chat_models | ||
client | ||
document_loaders | ||
embeddings | ||
examples | ||
llms | ||
memory | ||
prompts | ||
retrievers | ||
utilities | ||
vectorstores | ||
__init__.py | ||
.env.example | ||
conftest.py | ||
test_document_transformers.py | ||
test_nlp_text_splitters.py | ||
test_pdf_pagesplitter.py | ||
test_schema.py | ||
test_text_splitter.py |