You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
langchain/libs/text-splitters/langchain_text_splitters
Matthew DeGenaro 66828f4ecc
text-splitters[patch]: Modified SpacyTextSplitter to fully keep whitespace when strip_whitespace is false (#23272)
Previously, regardless of whether or not strip_whitespace was set to
true or false, the strip text method in the SpacyTextSplitter class used
`sent.text` to get the sentence. I modified this to include a ternary
such that if strip_whitespace is false, it uses `sent.text_with_ws`
I also modified the project.toml to include the spacy pipeline package
and to lock the numpy version, as higher versions break spacy.

- **Issue:** N/a
- **Dependencies:** None
3 weeks ago
..
xsl text-splitters[minor]: Adding a new section aware splitter to langchain (#16526) 6 months ago
__init__.py text-splitters[minor]: Adding a new section aware splitter to langchain (#16526) 6 months ago
base.py langchain : text_splitters Added PowerShell (#24582) 2 months ago
character.py text-splitters[patch]: fix typing for `keep_separator` (#25706) 1 month ago
html.py text_splitters: add request parameters for function HTMLHeaderTextSplitter.split_text… (#24178) 2 months ago
json.py text-splitters: Fix/recursive json splitter data persistence issue (#21529) 3 months ago
konlpy.py text-splitters[minor], langchain[minor], community[patch], templates, docs: langchain-text-splitters 0.0.1 (#18346) 7 months ago
latex.py text-splitters[minor], langchain[minor], community[patch], templates, docs: langchain-text-splitters 0.0.1 (#18346) 7 months ago
markdown.py text-splitters: Introduce Experimental Markdown Syntax Splitter (#22257) 3 months ago
nltk.py text-splitters[minor], langchain[minor], community[patch], templates, docs: langchain-text-splitters 0.0.1 (#18346) 7 months ago
py.typed text-splitters[minor], langchain[minor], community[patch], templates, docs: langchain-text-splitters 0.0.1 (#18346) 7 months ago
python.py text-splitters[minor], langchain[minor], community[patch], templates, docs: langchain-text-splitters 0.0.1 (#18346) 7 months ago
sentence_transformers.py text-splitters[minor], langchain[minor], community[patch], templates, docs: langchain-text-splitters 0.0.1 (#18346) 7 months ago
spacy.py text-splitters[patch]: Modified SpacyTextSplitter to fully keep whitespace when strip_whitespace is false (#23272) 3 weeks ago