3 Commits (21a353e9c25017a8a66661edb2ed98e9b152572b)

Author	SHA1	Message	Date
Jona Sassenhagen	7ffc431b3a	Add spacy sentencizer (#7442) ... `SpacyTextSplitter` currently uses spacy's statistics-based `en_core_web_sm` model for sentence splitting. This is a good splitter, but it's also pretty slow, and in this case it's doing a lot of work that's not needed given that the spacy parse is then just thrown away. However, there is also a simple rules-based spacy sentencizer. Using this is at least an order of magnitude faster than using `en_core_web_sm` according to my local tests. Also, spacy sentence tokenization based on `en_core_web_sm` can be sped up in this case by not doing the NER stage. This shaves some cycles too, both when loading the model and when parsing the text. Consequently, this PR adds the option to use the basic spacy sentencizer, and it disables the NER stage for the current approach, *which is kept as the default*. Lastly, when extracting the tokenized sentences, the `text` attribute is called directly instead of doing the string conversion, which is IMO a bit more idiomatic.	1 year ago
Samantha Whitmore	a0780cc930	OptimizedPrompt -- k-shot example choice backed by semantic search (#91)	2 years ago
Delip Rao	3ee6e332dd	Implements NLTK and Spacy-based TextSplitters (#103) ... This PR is for Issue #88 - [x] `make format` - [x] `make lint` - [x] `make tests`	2 years ago