[Minor] Remove tagger from spacy sentencizer (#7534)

@svlandeg gave me a tip for how to improve a bit on
https://github.com/hwchase17/langchain/pull/7442 for some extra speed
and memory gains. The tagger isn't needed for sentencization, so can be
disabled too.
pull/7564/head
Jona Sassenhagen 1 year ago committed by GitHub
parent 1f83b5f47e
commit 0ea7224535
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

@ -47,7 +47,7 @@ def _make_spacy_pipeline_for_splitting(pipeline: str) -> Any: # avoid importing
sentencizer = English()
sentencizer.add_pipe("sentencizer")
else:
sentencizer = spacy.load(pipeline, disable=["ner"])
sentencizer = spacy.load(pipeline, exclude=["ner", "tagger"])
return sentencizer

Loading…
Cancel
Save