mirror of
https://github.com/hwchase17/langchain
synced 2024-11-06 03:20:49 +00:00
f4ddf64faa
- Added new document_transformer: MarkdonifyTransformer, that uses `markdonify` package with customizable options to convert HTML to Markdown. It's similar to Html2TextTransformer, but has more flexible options and also I've noticed that sometimes MarkdownifyTransformer performs better than html2text one, so that's why I use markdownify on my project. - Added docs and tests - Usage: ```python from langchain_community.document_transformers import MarkdownifyTransformer markdownify = MarkdownifyTransformer() docs_transform = markdownify.transform_documents(docs) ``` - Example of better performance on simple task, that I've noticed: ``` <html> <head><title>Reports on product movement</title></head> <body> <p data-block-key="2wst7">The reports on product movement will be useful for forming supplier orders and controlling outcomes.</p> </body> ``` **Html2TextTransformer**: ```python [Document(page_content='The reports on product movement will be useful for forming supplier orders and\ncontrolling outcomes.\n\n')] # Here we can see 'and\ncontrolling', which has extra '\n' in it ``` **MarkdownifyTranformer**: ```python [Document(page_content='Reports on product movement\n\nThe reports on product movement will be useful for forming supplier orders and controlling outcomes.')] ``` --------- Co-authored-by: Sokolov Fedor <f.sokolov@sokolov-macbook.bbrouter> Co-authored-by: Harrison Chase <hw.chase.17@gmail.com> Co-authored-by: Sokolov Fedor <f.sokolov@sokolov-macbook.local> Co-authored-by: Sokolov Fedor <f.sokolov@192.168.1.6> |
||
|---|---|---|
| .. | ||
| xsl | ||
| __init__.py | ||
| beautiful_soup_transformer.py | ||
| doctran_text_extract.py | ||
| doctran_text_qa.py | ||
| doctran_text_translate.py | ||
| embeddings_redundant_filter.py | ||
| google_translate.py | ||
| html2text.py | ||
| long_context_reorder.py | ||
| markdownify.py | ||
| nuclia_text_transform.py | ||
| openai_functions.py | ||