You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

joelsprunger 3984f6604f langchain: adds recursive json splitter (#17144) ... - **Description:** This adds a recursive json splitter class to the existing text_splitters as well as unit tests - **Issue:** splitting text from structured data can cause issues if you have a large nested json object and you split it as regular text you may end up losing the structure of the json. To mitigate against this you can split the nested json into large chunks and overlap them, but this causes unnecessary text processing and there will still be times where the nested json is so big that the chunks get separated from the parent keys. As an example you wouldn't want the following to be split in half: ```shell {'val0': 'DFWeNdWhapbR', 'val1': {'val10': 'QdJo', 'val11': 'FWSDVFHClW', 'val12': 'bkVnXMMlTiQh', 'val13': 'tdDMKRrOY', 'val14': 'zybPALvL', 'val15': 'JMzGMNH', 'val16': {'val160': 'qLuLKusFw', 'val161': 'DGuotLh', 'val162': 'KztlcSBropT', -----------------------------------------------------------------------split----- 'val163': 'YlHHDrN', 'val164': 'CtzsxlGBZKf', 'val165': 'bXzhcrWLmBFp', 'val166': 'zZAqC', 'val167': 'ZtyWno', 'val168': 'nQQZRsLnaBhb', 'val169': 'gSpMbJwA'}, 'val17': 'JhgiyF', 'val18': 'aJaqjUSFFrI', 'val19': 'glqNSvoyxdg'}} ``` Any llm processing the second chunk of text may not have the context of val1, and val16 reducing accuracy. Embeddings will also lack this context and this makes retrieval less accurate. Instead you want it to be split into chunks that retain the json structure. ```shell {'val0': 'DFWeNdWhapbR', 'val1': {'val10': 'QdJo', 'val11': 'FWSDVFHClW', 'val12': 'bkVnXMMlTiQh', 'val13': 'tdDMKRrOY', 'val14': 'zybPALvL', 'val15': 'JMzGMNH', 'val16': {'val160': 'qLuLKusFw', 'val161': 'DGuotLh', 'val162': 'KztlcSBropT', 'val163': 'YlHHDrN', 'val164': 'CtzsxlGBZKf'}}} ``` and ```shell {'val1':{'val16':{ 'val165': 'bXzhcrWLmBFp', 'val166': 'zZAqC', 'val167': 'ZtyWno', 'val168': 'nQQZRsLnaBhb', 'val169': 'gSpMbJwA'}, 'val17': 'JhgiyF', 'val18': 'aJaqjUSFFrI', 'val19': 'glqNSvoyxdg'}} ``` This recursive json text splitter does this. Values that contain a list can be converted to dict first by using split(... convert_lists=True) otherwise long lists will not be split and you may end up with chunks larger than the max chunk. In my testing large json objects could be split into small chunks with ✅ Increased question answering accuracy ✅ The ability to split into smaller chunks meant retrieval queries can use fewer tokens - **Dependencies:** json import added to text_splitter.py, and random added to the unit test - **Twitter handle:** @joelsprunger --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>		8 months ago
..
cli	cli[patch]: copyright 2024 default (#17204)	8 months ago
community	community[patch]: Support SerDe transform functions in Databricks LLM (#16752)	8 months ago
core	langchain[minor], core[minor]: update json, pydantic parser. add openai-json structured output runnable (#16914)	8 months ago
experimental	community[minor]: SQLDatabase Add fetch mode `cursor`, query parameters, query by selectable, expose execution options, and documentation (#17191)	8 months ago
langchain	langchain: adds recursive json splitter (#17144)	8 months ago
partners	google-genai[patch]: added parsing of function call / response (#17245)	8 months ago