forked from Archives/langchain
4a8f5cdf4b
This does not involve a separator, and will naively chunk input text at the appropriate boundaries in token space. This is helpful if we have strict token length limits that we need to strictly follow the specified chunk size, and we can't use aggressive separators like spaces to guarantee the absence of long strings. CharacterTextSplitter will let these strings through without splitting them, which could cause overflow errors downstream. Splitting at arbitrary token boundaries is not ideal but is hopefully mitigated by having a decent overlap quantity. Also this results in chunks which has exact number of tokens desired, instead of sometimes overcounting if we concatenate shorter strings. Potentially also helps with #528. |
||
|---|---|---|
| .. | ||
| chains | ||
| embeddings | ||
| llms | ||
| vectorstores | ||
| __init__.py | ||
| test_googlesearch_api.py | ||
| test_ngram_overlap_example_selector.py | ||
| test_nlp_text_splitters.py | ||
| test_serpapi.py | ||
| test_text_splitter.py | ||
| test_wolfram_alpha_api.py | ||