You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
langchain/.github/workflows
maks-operlejn-ds a8f804a618
Add data anonymizer (#9863)
### Description

The feature for anonymizing data has been implemented. In order to
protect private data, such as when querying external APIs (OpenAI), it
is worth pseudonymizing sensitive data to maintain full privacy.

Anonynization consists of two steps:

1. **Identification:** Identify all data fields that contain personally
identifiable information (PII).
2. **Replacement**: Replace all PIIs with pseudo values or codes that do
not reveal any personal information about the individual but can be used
for reference. We're not using regular encryption, because the language
model won't be able to understand the meaning or context of the
encrypted data.

We use *Microsoft Presidio* together with *Faker* framework for
anonymization purposes because of the wide range of functionalities they
provide. The full implementation is available in `PresidioAnonymizer`.

### Future works

- **deanonymization** - add the ability to reverse anonymization. For
example, the workflow could look like this: `anonymize -> LLMChain ->
deanonymize`. By doing this, we will retain anonymity in requests to,
for example, OpenAI, and then be able restore the original data.
- **instance anonymization** - at this point, each occurrence of PII is
treated as a separate entity and separately anonymized. Therefore, two
occurrences of the name John Doe in the text will be changed to two
different names. It is therefore worth introducing support for full
instance detection, so that repeated occurrences are treated as a single
object.

### Twitter handle
@deepsense_ai / @MaksOpp

---------

Co-authored-by: MaksOpp <maks.operlejn@gmail.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
1 year ago
..
_lint.yml Cache poetry install + unify Python/Poetry setup for lint and test jobs. (#9625) 1 year ago
_pydantic_compatibility.yml Cache poetry install + unify Python/Poetry setup for lint and test jobs. (#9625) 1 year ago
_release.yml Use unified Python setup steps for release workflow. 1 year ago
_test.yml Cache poetry install + unify Python/Poetry setup for lint and test jobs. (#9625) 1 year ago
codespell.yml Add api cross ref linking (#8275) 1 year ago
langchain_ci.yml Cache poetry install + unify Python/Poetry setup for lint and test jobs. (#9625) 1 year ago
langchain_experimental_ci.yml Add data anonymizer (#9863) 1 year ago
langchain_experimental_release.yml Require manually triggering release workflows. (#9552) 1 year ago
langchain_release.yml Require manually triggering release workflows. (#9552) 1 year ago
scheduled_test.yml Use new Python setup approach for scheduled tests. (#9626) 1 year ago