langchain/docs/docs/guides/productionization/safety/presidio_data_anonymization/multi_language.ipynb

{
 "cells": [
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "---\n",
    "sidebar_position: 2\n",
    "title: Multi-language anonymization\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-language data anonymization with Microsoft Presidio\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/langchain-ai/langchain/blob/master/docs/docs/guides/privacy/presidio_data_anonymization/multi_language.ipynb)\n",
    "\n",
    "\n",
    "## Use case\n",
    "\n",
    "Multi-language support in data pseudonymization is essential due to differences in language structures and cultural contexts. Different languages may have varying formats for personal identifiers. For example, the structure of names, locations and dates can differ greatly between languages and regions. Furthermore, non-alphanumeric characters, accents, and the direction of writing can impact pseudonymization processes. Without multi-language support, data could remain identifiable or be misinterpreted, compromising data privacy and accuracy. Hence, it enables effective and precise pseudonymization suited for global operations.\n",
    "\n",
    "## Overview\n",
    "\n",
    "PII detection in Microsoft Presidio relies on several components - in addition to the usual pattern matching (e.g. using regex), the analyser uses a model for Named Entity Recognition (NER) to extract entities such as:\n",
    "- `PERSON`\n",
    "- `LOCATION`\n",
    "- `DATE_TIME`\n",
    "- `NRP`\n",
    "- `ORGANIZATION`\n",
    "\n",
    "[[Source]](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/spacy_recognizer.py)\n",
    "\n",
    "To handle NER in specific languages, we utilize unique models from the `spaCy` library, recognized for its extensive selection covering multiple languages and sizes. However, it's not restrictive, allowing for integration of alternative frameworks such as [Stanza](https://microsoft.github.io/presidio/analyzer/nlp_engines/spacy_stanza/) or [transformers](https://microsoft.github.io/presidio/analyzer/nlp_engines/transformers/) when necessary.\n",
    "\n",
    "\n",
    "## Quickstart\n",
    "\n"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "%pip install --upgrade --quiet  langchain langchain-openai langchain-experimental presidio-analyzer presidio-anonymizer spacy Faker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download model\n",
    "!python -m spacy download en_core_web_lg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_experimental.data_anonymizer import PresidioReversibleAnonymizer\n",
    "\n",
    "anonymizer = PresidioReversibleAnonymizer(\n",
    "    analyzed_fields=[\"PERSON\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, `PresidioAnonymizer` and `PresidioReversibleAnonymizer` use a model trained on English texts, so they handle other languages moderately well. \n",
    "\n",
    "For example, here the model did not detect the person:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Me llamo Sofía'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anonymizer.anonymize(\"Me llamo Sofía\")  # \"My name is Sofía\" in Spanish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "They may also take words from another language as actual entities. Here, both the word *'Yo'* (*'I'* in Spanish) and *Sofía* have been classified as `PERSON`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Kari Lopez soy Mary Walker'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "anonymizer.anonymize(\"Yo soy Sofía\")  # \"I am Sofía\" in Spanish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to anonymise texts from other languages, you need to download other models and add them to the anonymiser configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the models for the languages you want to use\n",
    "# ! python -m spacy download en_core_web_md\n",
    "# ! python -m spacy download es_core_news_md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp_config = {\n",
    "    \"nlp_engine_name\": \"spacy\",\n",
    "    \"models\": [\n",
    "        {\"lang_code\": \"en\", \"model_name\": \"en_core_web_md\"},\n",
    "        {\"lang_code\": \"es\", \"model_name\": \"es_core_news_md\"},\n",
    "    ],\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have therefore added a Spanish language model. Note also that we have downloaded an alternative model for English as well - in this case we have replaced the large model `en_core_web_lg` (560MB) with its smaller version `en_core_web_md` (40MB) - the size is therefore reduced by 14 times! If you care about the speed of anonymisation, it is worth considering it.\n",
    "\n",
    "All models for the different languages can be found in the [spaCy documentation](https://spacy.io/usage/models).\n",
    "\n",
    "Now pass the configuration as the `languages_config` parameter to Anonymiser. As you can see, both previous examples work flawlessly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Me llamo Christopher Smith\n",
      "Yo soy Joseph Jenkins\n"
     ]
    }
   ],
   "source": [
    "anonymizer = PresidioReversibleAnonymizer(\n",
    "    analyzed_fields=[\"PERSON\"],\n",
    "    languages_config=nlp_config,\n",
    ")\n",
    "\n",
    "print(\n",
    "    anonymizer.anonymize(\"Me llamo Sofía\", language=\"es\")\n",
    ")  # \"My name is Sofía\" in Spanish\n",
    "print(anonymizer.anonymize(\"Yo soy Sofía\", language=\"es\"))  # \"I am Sofía\" in Spanish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default, the language indicated first in the configuration will be used when anonymising text (in this case English):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My name is Shawna Bennett\n"
     ]
    }
   ],
   "source": [
    "print(anonymizer.anonymize(\"My name is John\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage with other frameworks\n",
    "\n",
    "### Language detection\n",
    "\n",
    "One of the drawbacks of the presented approach is that we have to pass the **language** of the input text directly. However, there is a remedy for that - *language detection* libraries.\n",
    "\n",
    "We recommend using one of the following frameworks:\n",
    "- fasttext (recommended)\n",
    "- langdetect\n",
    "\n",
    "From our experience *fasttext* performs a bit better, but you should verify it on your use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install necessary packages\n",
    "%pip install --upgrade --quiet  fasttext langdetect"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### langdetect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "import langdetect\n",
    "from langchain.schema import runnable\n",
    "\n",
    "\n",
    "def detect_language(text: str) -> dict:\n",
    "    language = langdetect.detect(text)\n",
    "    print(language)\n",
    "    return {\"text\": text, \"language\": language}\n",
    "\n",
    "\n",
    "chain = runnable.RunnableLambda(detect_language) | (\n",
    "    lambda x: anonymizer.anonymize(x[\"text\"], language=x[\"language\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "es\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Me llamo Michael Perez III'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke(\"Me llamo Sofía\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "en\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'My name is Ronald Bennett'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke(\"My name is John Doe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### fasttext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You need to download the fasttext model first from https://dl.fbaipublicfiles.com/fasttext/supervised-models/lid.176.ftz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning : `load_model` does not return WordVectorModel or SupervisedModel any more, but a `FastText` object which is very similar.\n"
     ]
    }
   ],
   "source": [
    "import fasttext\n",
    "\n",
    "model = fasttext.load_model(\"lid.176.ftz\")\n",
    "\n",
    "\n",
    "def detect_language(text: str) -> dict:\n",
    "    language = model.predict(text)[0][0].replace(\"__label__\", \"\")\n",
    "    print(language)\n",
    "    return {\"text\": text, \"language\": language}\n",
    "\n",
    "\n",
    "chain = runnable.RunnableLambda(detect_language) | (\n",
    "    lambda x: anonymizer.anonymize(x[\"text\"], language=x[\"language\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "es\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Yo soy Angela Werner'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke(\"Yo soy Sofía\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "en\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'My name is Carlos Newton'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke(\"My name is John Doe\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This way you only need to initialize the model with the engines corresponding to the relevant languages, but using the tool is fully automated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Advanced usage\n",
    "\n",
    "### Custom labels in NER model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It may be that the spaCy model has different class names than those supported by the Microsoft Presidio by default. Take Polish, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text: Wiktoria, Start: 12, End: 20, Label: persName\n"
     ]
    }
   ],
   "source": [
    "# ! python -m spacy download pl_core_news_md\n",
    "\n",
    "import spacy\n",
    "\n",
    "nlp = spacy.load(\"pl_core_news_md\")\n",
    "doc = nlp(\"Nazywam się Wiktoria\")  # \"My name is Wiktoria\" in Polish\n",
    "\n",
    "for ent in doc.ents:\n",
    "    print(\n",
    "        f\"Text: {ent.text}, Start: {ent.start_char}, End: {ent.end_char}, Label: {ent.label_}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The name *Victoria* was classified as `persName`, which does not correspond to the default class names `PERSON`/`PER` implemented in Microsoft Presidio (look for `CHECK_LABEL_GROUPS` in [SpacyRecognizer implementation](https://github.com/microsoft/presidio/blob/main/presidio-analyzer/presidio_analyzer/predefined_recognizers/spacy_recognizer.py)). \n",
    "\n",
    "You can find out more about custom labels in spaCy models (including your own, trained ones) in [this thread](https://github.com/microsoft/presidio/issues/851).\n",
    "\n",
    "That's why our sentence will not be anonymized:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nazywam się Wiktoria\n"
     ]
    }
   ],
   "source": [
    "nlp_config = {\n",
    "    \"nlp_engine_name\": \"spacy\",\n",
    "    \"models\": [\n",
    "        {\"lang_code\": \"en\", \"model_name\": \"en_core_web_md\"},\n",
    "        {\"lang_code\": \"es\", \"model_name\": \"es_core_news_md\"},\n",
    "        {\"lang_code\": \"pl\", \"model_name\": \"pl_core_news_md\"},\n",
    "    ],\n",
    "}\n",
    "\n",
    "anonymizer = PresidioReversibleAnonymizer(\n",
    "    analyzed_fields=[\"PERSON\", \"LOCATION\", \"DATE_TIME\"],\n",
    "    languages_config=nlp_config,\n",
    ")\n",
    "\n",
    "print(\n",
    "    anonymizer.anonymize(\"Nazywam się Wiktoria\", language=\"pl\")\n",
    ")  # \"My name is Wiktoria\" in Polish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To address this, create your own `SpacyRecognizer` with your own class mapping and add it to the anonymizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from presidio_analyzer.predefined_recognizers import SpacyRecognizer\n",
    "\n",
    "polish_check_label_groups = [\n",
    "    ({\"LOCATION\"}, {\"placeName\", \"geogName\"}),\n",
    "    ({\"PERSON\"}, {\"persName\"}),\n",
    "    ({\"DATE_TIME\"}, {\"date\", \"time\"}),\n",
    "]\n",
    "\n",
    "spacy_recognizer = SpacyRecognizer(\n",
    "    supported_language=\"pl\",\n",
    "    check_label_groups=polish_check_label_groups,\n",
    ")\n",
    "\n",
    "anonymizer.add_recognizer(spacy_recognizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now everything works smoothly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nazywam się Morgan Walters\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    anonymizer.anonymize(\"Nazywam się Wiktoria\", language=\"pl\")\n",
    ")  # \"My name is Wiktoria\" in Polish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try on more complex example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nazywam się Ernest Liu. New Taylorburgh to moje miasto rodzinne. Urodziłam się 1987-01-19\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    anonymizer.anonymize(\n",
    "        \"Nazywam się Wiktoria. Płock to moje miasto rodzinne. Urodziłam się dnia 6 kwietnia 2001 roku\",\n",
    "        language=\"pl\",\n",
    "    )\n",
    ")  # \"My name is Wiktoria. Płock is my home town. I was born on 6 April 2001\" in Polish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, thanks to class mapping, the anonymiser can cope with different types of entities. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Custom language-specific operators\n",
    "\n",
    "In the example above, the sentence has been anonymised correctly, but the fake data does not fit the Polish language at all. Custom operators can therefore be added, which will resolve the issue:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from faker import Faker\n",
    "from presidio_anonymizer.entities import OperatorConfig\n",
    "\n",
    "fake = Faker(locale=\"pl_PL\")  # Setting faker to provide Polish data\n",
    "\n",
    "new_operators = {\n",
    "    \"PERSON\": OperatorConfig(\"custom\", {\"lambda\": lambda _: fake.first_name_female()}),\n",
    "    \"LOCATION\": OperatorConfig(\"custom\", {\"lambda\": lambda _: fake.city()}),\n",
    "}\n",
    "\n",
    "anonymizer.add_operators(new_operators)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Nazywam się Marianna. Szczecin to moje miasto rodzinne. Urodziłam się 1976-11-16\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    anonymizer.anonymize(\n",
    "        \"Nazywam się Wiktoria. Płock to moje miasto rodzinne. Urodziłam się dnia 6 kwietnia 2001 roku\",\n",
    "        language=\"pl\",\n",
    "    )\n",
    ")  # \"My name is Wiktoria. Płock is my home town. I was born on 6 April 2001\" in Polish"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Limitations\n",
    "\n",
    "Remember - results are as good as your recognizers and as your NER models!\n",
    "\n",
    "Look at the example below - we downloaded the small model for Spanish (12MB) and it no longer performs as well as the medium version (40MB):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: es_core_news_sm. Result: Me llamo Sofía\n",
      "Model: es_core_news_md. Result: Me llamo Lawrence Davis\n"
     ]
    }
   ],
   "source": [
    "# ! python -m spacy download es_core_news_sm\n",
    "\n",
    "for model in [\"es_core_news_sm\", \"es_core_news_md\"]:\n",
    "    nlp_config = {\n",
    "        \"nlp_engine_name\": \"spacy\",\n",
    "        \"models\": [\n",
    "            {\"lang_code\": \"es\", \"model_name\": model},\n",
    "        ],\n",
    "    }\n",
    "\n",
    "    anonymizer = PresidioReversibleAnonymizer(\n",
    "        analyzed_fields=[\"PERSON\"],\n",
    "        languages_config=nlp_config,\n",
    "    )\n",
    "\n",
    "    print(\n",
    "        f\"Model: {model}. Result: {anonymizer.anonymize('Me llamo Sofía', language='es')}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In many cases, even the larger models from spaCy will not be sufficient - there are already other, more complex and better methods of detecting named entities, based on transformers. You can read more about this [here](https://microsoft.github.io/presidio/analyzer/nlp_engines/transformers/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}