langchain

Commit Graph

Author	SHA1	Message	Date
Christos Boulmpasakos	c3bcfad66d	text-splitters[patch]: Extend TextSplitter:keep_separator functionality (#21130 ) Description: Added extra functionality to `CharacterTextSplitter`, `TextSplitter` classes. The user can select whether to append the separator to the previous chunk with `keep_separator='end' ` or else prepend to the next chunk. Previous functionality prepended by default to next chunk. Issue: Fixes #20908 --------- Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	6 days ago
ccurme	0e72ed39a0	infra: fix CI on text-splitters (#21935 )	1 week ago
Erick Friis	1b555021f7	text-splitters: release 0.2.0 (#21832 )	2 weeks ago
Erick Friis	c77d2f2b06	multiple: core 0.2 nonbreaking dep, check_diff community->langchain dep (#21646 ) 0.2 is not a breaking release for core (but it is for langchain and community) To keep the core+langchain+community packages in sync at 0.2, we will relax deps throughout the ecosystem to tolerate `langchain-core` 0.2	2 weeks ago
Lei Zhang	2cd907ad7e	text-splitters[patch]: fix MarkdownHeaderTextSplitter fails to parse headers with non-printable characters (#20645 ) Description: MarkdownHeaderTextSplitter Fails to Parse Headers with non-printable characters. more #20643 The following is the official test case. Just replacing `# Foo\n\n` with `\ufeff# Foo\n\n` will cause the test case to fail. chunk metadata is empty ```python def test_md_header_text_splitter_1() -> None: """Test markdown splitter by header: Case 1.""" markdown_document = ( "\ufeff# Foo\n\n" " ## Bar\n\n" "Hi this is Jim\n\n" "Hi this is Joe\n\n" " ## Baz\n\n" " Hi this is Molly" ) headers_to_split_on = [ ("#", "Header 1"), ("##", "Header 2"), ] markdown_splitter = MarkdownHeaderTextSplitter( headers_to_split_on=headers_to_split_on, ) output = markdown_splitter.split_text(markdown_document) expected_output = [ Document( page_content="Hi this is Jim \nHi this is Joe", metadata={"Header 1": "Foo", "Header 2": "Bar"}, ), Document( page_content="Hi this is Molly", metadata={"Header 1": "Foo", "Header 2": "Baz"}, ), ] assert output == expected_output ``` twitter: @coolbeevip Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>	1 month ago
saberuster	160bcaeb93	text-splitters[minor]: Add lua code splitting (#20421 ) - Description: Complete the support for Lua code in langchain.text_splitter module. - Dependencies: No - Twitter handle: @saberuster If no one reviews your PR within a few days, please @-mention one of baskaryan, efriis, eyurtsev, hwchase17. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 month ago
Mahdi Setayesh	c28efb878c	text-splitters[minor]: Adding a new section aware splitter to langchain (#16526 ) - Description: the layout of html pages can be variant based on the bootstrap framework or the styles of the pages. So we need to have a splitter to transform the html tags to a proper layout and then split the html content based on the provided list of tags to determine its html sections. We are using BS4 library along with xslt structure to split the html content using an section aware approach. - Dependencies: No new dependencies - Twitter handle: @m_setayesh Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` from the root of the package you've modified to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://python.langchain.com/docs/contributing/ If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. It lives in `docs/docs/integrations` directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --> --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	2 months ago
Nisarg Trivedi	1252ccce6f	text-splitters[minor]: Added Haskell support in langchain.text_splitter module (#16191 ) - Description: Haskell language support added in text_splitter module - Dependencies: No - Twitter handle: @nisargtr If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	2 months ago
Evgenii Zheltonozhskii	5b1f9c6d3a	infra: Consistent lxml requirements (#19520 ) Update the dependency for lxml to be consistent among different packages; should fix https://github.com/langchain-ai/langchain/issues/19040	2 months ago
Guangdong Liu	e5d7e455dc	splitters: Add ensure_ascii parameter (#18485 ) - Description: Add ensure_ascii parameter	2 months ago
six17	fd4f536c77	text-splitters[patch]: fix json split of RecursiveJsonSplitter (#19119 ) - Description: This modification addresses the issue of mutable default parameters in functions. In the original code, the `chunks` parameter is defaulted to a list containing an empty dictionary, which is mutable. Since default parameters in Python are evaluated only once at function definition time, modifications to the parameter would persist across future calls. By changing the default to `None` and checking/initializing within the function, a new list is created for each call, thus avoiding potential issues. --------- Co-authored-by: sixiang <sixiang@lixiang.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	2 months ago
Erick Friis	781aee0068	community, langchain, infra: revert store extended test deps outside of poetry (#19153 ) Reverts langchain-ai/langchain#18995 Because it makes installing dependencies in python 3.11 extended testing take 80 minutes	2 months ago
Erick Friis	9e569d85a4	community, langchain, infra: store extended test deps outside of poetry (#18995 ) poetry can't reliably handle resolving the number of optional "extended test" dependencies we have. If we instead just rely on pip to install extended test deps in CI, this isn't an issue.	2 months ago
Bagatur	f220af3dce	docs: text splitters readme (#18359 )	3 months ago
Bagatur	5efb5c099f	text-splitters[minor], langchain[minor], community[patch], templates, docs: langchain-text-splitters 0.0.1 (#18346 )	3 months ago

15 Commits (master)