Jens Madsen 8d9e9e013c refactor: extract token text splitter function (#5179) ... # Token text splitter for sentence transformers The current TokenTextSplitter only works with OpenAi models via the `tiktoken` package. This is not clear from the name `TokenTextSplitter`. In this (first PR) a token based text splitter for sentence transformer models is added. In the future I think we should work towards injecting a tokenizer into the TokenTextSplitter to make ti more flexible. Could perhaps be reviewed by @dev2049 --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>		2023-06-04 14:41:44 -07:00
..
examples	refactor: extract token text splitter function (#5179)	2023-06-04 14:41:44 -07:00
getting_started.ipynb	big docs refactor (#1978)	2023-03-26 19:49:46 -07:00