### Description
Add instance anonymization - if `John Doe` will appear twice in the
text, it will be treated as the same entity.
The difference between `PresidioAnonymizer` and
`PresidioReversibleAnonymizer` is that only the second one has a
built-in memory, so it will remember anonymization mapping for multiple
texts:
```
>>> anonymizer = PresidioAnonymizer()
>>> anonymizer.anonymize("My name is John Doe. Hi John Doe!")
'My name is Noah Rhodes. Hi Noah Rhodes!'
>>> anonymizer.anonymize("My name is John Doe. Hi John Doe!")
'My name is Brett Russell. Hi Brett Russell!'
```
```
>>> anonymizer = PresidioReversibleAnonymizer()
>>> anonymizer.anonymize("My name is John Doe. Hi John Doe!")
'My name is Noah Rhodes. Hi Noah Rhodes!'
>>> anonymizer.anonymize("My name is John Doe. Hi John Doe!")
'My name is Noah Rhodes. Hi Noah Rhodes!'
```
### Twitter handle
@deepsense_ai / @MaksOpp
### Tag maintainer
@baskaryan @hwchase17 @hinthornw
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
"'My name is Laura Ruiz, call me at +1-412-982-8374x13414 or email me at javierwatkins@example.net'"
"'My name is James Martinez, call me at (576)928-1972x679 or email me at lisa44@example.com'"
]
},
"execution_count": 2,
@ -114,11 +114,11 @@
"text": [
"Dear Sir/Madam,\n",
"\n",
"We regret to inform you that Richard Fields has recently misplaced his wallet, which contains a sum of cash and his credit card bearing the number 30479847307774. \n",
"We regret to inform you that Mr. Dennis Cooper has recently misplaced his wallet. The wallet contains a sum of cash and his credit card, bearing the number 3588895295514977. \n",
"\n",
"Should you happen to come across it, we kindly request that you contact us immediately at 6439182672 or via email at frank45@example.com.\n",
"Should you happen to come across the aforementioned wallet, kindly contact us immediately at (428)451-3494x4110 or send an email to perryluke@example.com.\n",
"\n",
"Thank you for your attention to this matter.\n",
"Your prompt assistance in this matter would be greatly appreciated.\n",
"\n",
"Yours faithfully,\n",
"\n",
@ -159,7 +159,7 @@
{
"data": {
"text/plain": [
"'My name is Adrian Fleming, call me at 313-666-7440 or email me at real.slim.shady@gmail.com'"
"'My name is Shannon Steele, call me at 313-666-7440 or email me at real.slim.shady@gmail.com'"
]
},
"execution_count": 6,
@ -190,7 +190,7 @@
{
"data": {
"text/plain": [
"'My name is Justin Miller, call me at 761-824-1889 or email me at real.slim.shady@gmail.com'"
"'My name is Wesley Flores, call me at (498)576-9526 or email me at real.slim.shady@gmail.com'"
]
},
"execution_count": 7,
@ -225,7 +225,7 @@
{
"data": {
"text/plain": [
"'My name is Dr. Jennifer Baker, call me at (508)839-9329x232 or email me at ehamilton@example.com'"
"'My name is Carla Fisher, call me at 001-683-324-0721x0644 or email me at krausejeremy@example.com'"
]
},
"execution_count": 8,
@ -256,7 +256,7 @@
{
"data": {
"text/plain": [
"'My polish phone number is NRGN41434238921378'"
"'My polish phone number is QESQ21234635370499'"
]
},
"execution_count": 9,
@ -361,7 +361,7 @@
{
"data": {
"text/plain": [
"'511 622 683'"
"'665 631 080'"
]
},
"execution_count": 13,
@ -422,7 +422,7 @@
{
"data": {
"text/plain": [
"'My polish phone number is +48 734 630 977'"
"'My polish phone number is 538 521 657'"
]
},
"execution_count": 16,
@ -438,8 +438,80 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Future works\n",
"- **instance anonymization** - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object."
"## Important considerations\n",
"\n",
"### Anonymizer detection rates\n",
"\n",
"**The level of anonymization and the precision of detection are just as good as the quality of the recognizers implemented.**\n",
"\n",
"Texts from different sources and in different languages have varying characteristics, so it is necessary to test the detection precision and iteratively add recognizers and operators to achieve better and better results.\n",
"\n",
"Microsoft Presidio gives a lot of freedom to refine anonymization. The library's author has provided his [recommendations and a step-by-step guide for improving detection rates](https://github.com/microsoft/presidio/discussions/767#discussion-3567223)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Instance anonymization\n",
"\n",
"`PresidioAnonymizer` has no built-in memory. Therefore, two occurrences of the entity in the subsequent texts will be replaced with two different fake values:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My name is Robert Morales. Hi Robert Morales!\n",
"My name is Kelly Mccoy. Hi Kelly Mccoy!\n"
]
}
],
"source": [
"print(anonymizer.anonymize(\"My name is John Doe. Hi John Doe!\"))\n",
"print(anonymizer.anonymize(\"My name is John Doe. Hi John Doe!\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To preserve previous anonymization results, use `PresidioReversibleAnonymizer`, which has built-in memory:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My name is Ashley Cervantes. Hi Ashley Cervantes!\n",
"My name is Ashley Cervantes. Hi Ashley Cervantes!\n"
"We regret to inform you that Mr. Dana Rhodes has reported the loss of his wallet. The wallet contains a sum of cash and his credit card, bearing the number 4397528473885757. \n",
"We regret to inform you that Monique Turner has recently misplaced his wallet, which contains a sum of cash and his credit card with the number 213152056829866. \n",
"\n",
"If you happen to come across the aforementioned wallet, we kindly request that you contact us immediately at 258-481-7074x714 or via email at laurengoodman@example.com.\n",
"If you happen to come across this wallet, kindly contact us at (770)908-7734x2835 or send an email to barbara25@example.net.\n",
"\n",
"Your prompt assistance in this matter would be greatly appreciated.\n",
"\n",
"Yours faithfully,\n",
"Thank you for your cooperation.\n",
"\n",
"Sincerely,\n",
"[Your Name]\n"
]
}
@ -232,14 +231,13 @@
"text": [
"Dear Sir/Madam,\n",
"\n",
"We regret to inform you that Mr. Slim Shady has recently misplaced his wallet. The wallet contains a sum of cash and his credit card, bearing the number 4916 0387 9536 0861. \n",
"\n",
"If by any chance you come across the lost wallet, kindly contact us immediately at 313-666-7440 or send an email to real.slim.shady@gmail.com.\n",
"We regret to inform you that Slim Shady has recently misplaced his wallet, which contains a sum of cash and his credit card with the number 4916 0387 9536 0861. \n",
"\n",
"Your prompt assistance in this matter would be greatly appreciated.\n",
"If you happen to come across this wallet, kindly contact us at 313-666-7440 or send an email to real.slim.shady@gmail.com.\n",
"\n",
"Yours faithfully,\n",
"Thank you for your cooperation.\n",
"\n",
"Sincerely,\n",
"[Your Name]\n"
]
}
@ -356,13 +354,57 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We can save the mapping itself to a file for future use: "
"Thanks to the built-in memory, entities that have already been detected and anonymised will take the same form in subsequent processed texts, so no duplicates will exist in the mapping:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"My VISA card number is 3537672423884966 and my name is William Bowman.\n"
" \"My VISA card number is 4001 9192 5753 7193 and my name is John Doe.\"\n",
" )\n",
")\n",
"\n",
"anonymizer.deanonymizer_mapping"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can save the mapping itself to a file for future use: "
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# We can save the deanonymizer mapping as a JSON or YAML file\n",
@ -380,7 +422,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 12,
"metadata": {},
"outputs": [
{
@ -389,7 +431,7 @@
"{}"
]
},
"execution_count": 11,
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
@ -402,7 +444,7 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 13,
"metadata": {},
"outputs": [
{
@ -415,7 +457,7 @@
" '3537672423884966': '4001 9192 5753 7193'}}"
]
},
"execution_count": 12,
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
@ -432,7 +474,6 @@
"source": [
"## Future works\n",
"\n",
"- **instance anonymization** - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object.\n",
"- **better matching and substitution of fake values for real ones** - currently the strategy is based on matching full strings and then substituting them. Due to the indeterminism of language models, it may happen that the value in the answer is slightly changed (e.g. *John Doe* -> *John* or *Main St, New York* -> *New York*) and such a substitution is then no longer possible. Therefore, it is worth adjusting the matching for your needs."
maks-operlejn-ds
1 year ago
committed by
GitHub