mirror of
https://github.com/hwchase17/langchain
synced 2024-11-10 01:10:59 +00:00
e830a4e731
- **Description:** add `remove_comments` option (default: True): do not extract html _comments_, - **Issue:** None, - **Dependencies:** None, - **Tag maintainer:** @nfcampos , - **Twitter handle:** peter_v I ran `make format`, `make lint` and `make test`. Discussion: I my use case, I prefer to not have the comments in the extracted text: * e.g. from a Google tag that is added in the html as comment * e.g. content that the authors have temporarily hidden to make it non visible to the regular reader Removing the comments makes the extracted text more alike the intended text to be seen by the reader. **Choice to make:** do we prefer to make the default for this `remove_comments` option to be True or False? I have changed it to True in a second commit, since that is how I would prefer to use it by default. Have the cleaned text (without technical Google tags etc.) and also closer to the actually visible and intended content. I am not sure what is best aligned with the conventions of langchain in general ... INITIAL VERSION (new version above): ~**Choice to make:** do we prefer to make the default for this `ignore_comments` option to be True or False? I have set it to False now to be backwards compatible. On the other hand, I would use it mostly with True. I am not sure what is best aligned with the conventions of langchain in general ...~ --------- Co-authored-by: Bagatur <baskaryan@gmail.com> |
||
|---|---|---|
| .. | ||
| agent_toolkits | ||
| callbacks | ||
| chat_loaders | ||
| chat_message_histories | ||
| chat_models | ||
| docstore | ||
| document_loaders | ||
| document_transformers | ||
| embeddings | ||
| examples | ||
| graphs | ||
| indexes | ||
| llms | ||
| retrievers | ||
| storage | ||
| tools | ||
| utilities | ||
| utils | ||
| vectorstores | ||
| __init__.py | ||
| conftest.py | ||
| test_dependencies.py | ||
| test_imports.py | ||
| test_sql_database_schema.py | ||
| test_sql_database.py | ||
| test_sqlalchemy.py | ||