forked from Archives/langchain
8d9e9e013c
# Token text splitter for sentence transformers The current TokenTextSplitter only works with OpenAi models via the `tiktoken` package. This is not clear from the name `TokenTextSplitter`. In this (first PR) a token based text splitter for sentence transformer models is added. In the future I think we should work towards injecting a tokenizer into the TokenTextSplitter to make ti more flexible. Could perhaps be reviewed by @dev2049 --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> |
||
---|---|---|
.. | ||
agents | ||
callbacks | ||
chains | ||
indexes | ||
memory | ||
models | ||
prompts | ||
utils/examples | ||
agents.rst | ||
chains.rst | ||
indexes.rst | ||
memory.rst | ||
models.rst | ||
paul_graham_essay.txt | ||
prompts.rst | ||
state_of_the_union.txt |