You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

felpigeon 2791a753bf Add start index to metadata in TextSplitter (#5912) ... <!-- Thank you for contributing to LangChain! Your PR will appear in our release under the title you set. Please make sure it highlights your valuable contribution. Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change. After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost. Finally, we'd love to show appreciation for your contribution - if you'd like us to shout you out on Twitter, please also include your handle! --> #### Add start index to metadata in TextSplitter - Modified method `create_documents` to track start position of each chunk - The `start_index` is included in the metadata if the `add_start_index` parameter in the class constructor is set to `True` This enables referencing back to the original document, particularly useful when a specific chunk is retrieved. <!-- If you're adding a new integration, please include: 1. a test for the integration - favor unit tests that does not rely on network access. 2. an example notebook showing its use See contribution guidelines for more information on how to write tests, lint etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md --> #### Who can review? Tag maintainers/contributors who might be interested: @eyurtsev @agola11 <!-- For a quicker response, figure out the right person to tag with @ @hwchase17 - project lead Tracing / Callbacks - @agola11 Async - @agola11 DataLoaders - @eyurtsev Models - @hwchase17 - @agola11 Agents / Tools / Toolkits - @vowelparrot VectorStores / Retrievers / Memory - @dev2049 -->		1 year ago
..
agents	Raise an exception in MKRL and Chat Output Parsers if parsing text which contains both an action and a final answer (#5609)	1 year ago
callbacks	py tracer fixes (#5377)	1 year ago
chains	support returning run info for llms, chat models and chains (#5666)	1 year ago
chat_models	Add ChatModel, LLM, and Embeddings for Google's PaLM APIs (#3575)	1 year ago
client	Use client from LCP-SDK (#5695)	1 year ago
data	Prompt from file proof of concept using plain text (#127)	2 years ago
docstore	Add `DocstoreFn` - lookup doc via arbitrary function (#3760)	1 year ago
document_loaders	YoutubeAudioLoader and updates to OpenAIWhisperParser (#5772)	1 year ago
evaluation	Adding an in-context QA evaluation chain + chain of thought reasoning chain for improved accuracy (#2444)	2 years ago
examples	feat #4479: TextLoader auto detect encoding and improved exceptions (#4927)	1 year ago
llms	Add Invocation Params (#4509)	1 year ago
memory	Implemented appending arbitrary messages (#5293)	1 year ago
output_parsers	convert the parameter 'text' to uppercase in the function 'parse' of the class BooleanOutputParser (#5397)	1 year ago
prompts	Harrison/pipeline prompt (#5540)	1 year ago
retrievers	Zep Hybrid Search (#5742)	1 year ago
tools	Harrison/pubmed integration (#5664)	1 year ago
utilities	Fix graphql tool (#4984)	1 year ago
vectorstores	Add maximal relevance search to SKLearnVectorStore (#5430)	1 year ago
__init__.py	initial commit	2 years ago
conftest.py	Add pytest --only-extended and --only-core options (#4494)	1 year ago
test_bash.py	Add Mastodon toots loader (#5036)	1 year ago
test_dependencies.py	Use client from LCP-SDK (#5695)	1 year ago
test_document_transformers.py	Contextual compression retriever (#2915)	1 year ago
test_formatting.py	initial commit	2 years ago
test_math_utils.py	add get_top_k_cosine_similarity method to get max top k score and index (#5059)	1 year ago
test_pytest_config.py	Block sockets for unit-tests (#4803)	1 year ago
test_python.py	option for csv agent to not include df in prompt (#4610)	1 year ago
test_schema.py	[simple][test] Added test case for schema.py (#3692)	1 year ago
test_sql_database.py	Fix SQLAlchemy truncating text when it is too big (#5206)	1 year ago
test_sql_database_schema.py	Suppress duckdb warning in unit tests explicitly (#3653)	1 year ago
test_text_splitter.py	Add start index to metadata in TextSplitter (#5912)	1 year ago