mirror of
https://github.com/hwchase17/langchain
synced 2024-10-29 17:07:25 +00:00
4cc4534d81
### Description
The feature for pseudonymizing data with ability to retrieve original
text (deanonymization) has been implemented. In order to protect private
data, such as when querying external APIs (OpenAI), it is worth
pseudonymizing sensitive data to maintain full privacy. But then, after
the model response, it would be good to have the data in the original
form.
I implemented the `PresidioReversibleAnonymizer`, which consists of two
parts:
1. anonymization - it works the same way as `PresidioAnonymizer`, plus
the object itself stores a mapping of made-up values to original ones,
for example:
```
{
"PERSON": {
"<anonymized>": "<original>",
"John Doe": "Slim Shady"
},
"PHONE_NUMBER": {
"111-111-1111": "555-555-5555"
}
...
}
```
2. deanonymization - using the mapping described above, it matches fake
data with original data and then substitutes it.
Between anonymization and deanonymization user can perform different
operations, for example, passing the output to LLM.
### Future works
- **instance anonymization** - at this point, each occurrence of PII is
treated as a separate entity and separately anonymized. Therefore, two
occurrences of the name John Doe in the text will be changed to two
different names. It is therefore worth introducing support for full
instance detection, so that repeated occurrences are treated as a single
object.
- **better matching and substitution of fake values for real ones** -
currently the strategy is based on matching full strings and then
substituting them. Due to the indeterminism of language models, it may
happen that the value in the answer is slightly changed (e.g. *John Doe*
-> *John* or *Main St, New York* -> *New York*) and such a substitution
is then no longer possible. Therefore, it is worth adjusting the
matching for your needs.
- **Q&A with anonymization** - when I'm done writing all the
functionality, I thought it would be a cool resource in documentation to
write a notebook about retrieval from documents using anonymization. An
iterative process, adding new recognizers to fit the data, lessons
learned and what to look out for
### Twitter handle
@deepsense_ai / @MaksOpp
---------
Co-authored-by: MaksOpp <maks.operlejn@gmail.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
18 lines
686 B
Python
18 lines
686 B
Python
from langchain_experimental.data_anonymizer.presidio import MappingDataType
|
|
|
|
|
|
def default_matching_strategy(text: str, deanonymizer_mapping: MappingDataType) -> str:
|
|
"""
|
|
Default matching strategy for deanonymization.
|
|
It replaces all the anonymized entities with the original ones.
|
|
|
|
Args:
|
|
text: text to deanonymize
|
|
deanonymizer_mapping: mapping between anonymized entities and original ones"""
|
|
|
|
# Iterate over all the entities (PERSON, EMAIL_ADDRESS, etc.)
|
|
for entity_type in deanonymizer_mapping:
|
|
for anonymized, original in deanonymizer_mapping[entity_type].items():
|
|
text = text.replace(anonymized, original)
|
|
return text
|