langchain/docs/extras
maks-operlejn-ds 274c3dc3a8
Multilingual anonymization (#10327)
### Description

Add multiple language support to Anonymizer

PII detection in Microsoft Presidio relies on several components - in
addition to the usual pattern matching (e.g. using regex), the analyser
uses a model for Named Entity Recognition (NER) to extract entities such
as:
- `PERSON`
- `LOCATION`
- `DATE_TIME`
- `NRP`
- `ORGANIZATION`


[[Source]](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/spacy_recognizer.py)

To handle NER in specific languages, we utilize unique models from the
`spaCy` library, recognized for its extensive selection covering
multiple languages and sizes. However, it's not restrictive, allowing
for integration of alternative frameworks such as
[Stanza](https://microsoft.github.io/presidio/analyzer/nlp_engines/spacy_stanza/)
or
[transformers](https://microsoft.github.io/presidio/analyzer/nlp_engines/transformers/)
when necessary.

### Future works

- **automatic language detection** - instead of passing the language as
a parameter in `anonymizer.anonymize`, we could detect the language/s
beforehand and then use the corresponding NER model. We have discussed
this internally and @mateusz-wosinski-ds will look into a standalone
language detection tool/chain for LangChain 😄

### Twitter handle
@deepsense_ai / @MaksOpp

### Tag maintainer
@baskaryan @hwchase17 @hinthornw
2023-09-07 14:42:24 -07:00
..
_templates docs: misc modelIO fixes (#9734) 2023-09-03 20:33:20 -07:00
additional_resources Updated Additional Resources section of documentation (#10260) 2023-09-06 15:10:43 -07:00
expression_language Add LCEL cookbook examples (#10290) 2023-09-06 13:50:43 -07:00
guides Multilingual anonymization (#10327) 2023-09-07 14:42:24 -07:00
integrations Adding Self-querying for Vectara (#10332) 2023-09-07 10:24:50 -07:00
modules Adding Self-querying for Vectara (#10332) 2023-09-07 10:24:50 -07:00
use_cases Move Myscale SQL vector retrieval nb 2023-09-06 17:09:40 -07:00