You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
langchain/tests/unit_tests/document_loaders
Eugene Yurtsev e46202829f
feat #4479: TextLoader auto detect encoding and improved exceptions (#4927)
# TextLoader auto detect encoding and enhanced exception handling

- Add an option to enable encoding detection on `TextLoader`. 
- The detection is done using `chardet`
- The loading is done by trying all detected encodings by order of
confidence or raise an exception otherwise.

### New Dependencies:
- `chardet`

Fixes #4479 

## Before submitting

<!-- If you're adding a new integration, include an integration test and
an example notebook showing its use! -->

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:

- @eyurtsev

---------

Co-authored-by: blob42 <spike@w530>
1 year ago
..
blob_loaders fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
loaders fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
parsers Add html parsers (#4874) 1 year ago
test_docs fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
__init__.py fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
test_base.py fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
test_bshtml.py Add html parsers (#4874) 1 year ago
test_confluence.py Add Confluence Loader unit tests (#3333) 1 year ago
test_csv_loader.py fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
test_detect_encoding.py feat #4479: TextLoader auto detect encoding and improved exceptions (#4927) 1 year ago
test_generic_loader.py Add a generic document loader (#4875) 1 year ago
test_json_loader.py fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
test_telegram.py fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
test_web_base.py fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago
test_youtube.py fix(document_loaders/telegram): fix pandas calls + add tests (#4806) 1 year ago