2023-07-21 17:36:28 +00:00
|
|
|
.PHONY: all format lint test tests test_watch integration_tests docker_tests help extended_tests
|
|
|
|
|
|
|
|
# Default target executed when no arguments are given to make.
|
|
|
|
all: help
|
|
|
|
|
|
|
|
# Define a variable for the test file path.
|
|
|
|
TEST_FILE ?= tests/unit_tests/
|
|
|
|
|
|
|
|
test:
|
|
|
|
poetry run pytest $(TEST_FILE)
|
|
|
|
|
|
|
|
tests:
|
|
|
|
poetry run pytest $(TEST_FILE)
|
|
|
|
|
|
|
|
test_watch:
|
|
|
|
poetry run ptw --now . -- tests/unit_tests
|
|
|
|
|
Add data anonymizer (#9863)
### Description
The feature for anonymizing data has been implemented. In order to
protect private data, such as when querying external APIs (OpenAI), it
is worth pseudonymizing sensitive data to maintain full privacy.
Anonynization consists of two steps:
1. **Identification:** Identify all data fields that contain personally
identifiable information (PII).
2. **Replacement**: Replace all PIIs with pseudo values or codes that do
not reveal any personal information about the individual but can be used
for reference. We're not using regular encryption, because the language
model won't be able to understand the meaning or context of the
encrypted data.
We use *Microsoft Presidio* together with *Faker* framework for
anonymization purposes because of the wide range of functionalities they
provide. The full implementation is available in `PresidioAnonymizer`.
### Future works
- **deanonymization** - add the ability to reverse anonymization. For
example, the workflow could look like this: `anonymize -> LLMChain ->
deanonymize`. By doing this, we will retain anonymity in requests to,
for example, OpenAI, and then be able restore the original data.
- **instance anonymization** - at this point, each occurrence of PII is
treated as a separate entity and separately anonymized. Therefore, two
occurrences of the name John Doe in the text will be changed to two
different names. It is therefore worth introducing support for full
instance detection, so that repeated occurrences are treated as a single
object.
### Twitter handle
@deepsense_ai / @MaksOpp
---------
Co-authored-by: MaksOpp <maks.operlejn@gmail.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-30 17:39:44 +00:00
|
|
|
extended_tests:
|
|
|
|
poetry run pytest --only-extended tests/unit_tests
|
|
|
|
|
2023-10-13 16:48:24 +00:00
|
|
|
integration_tests:
|
|
|
|
poetry run pytest tests/integration_tests
|
|
|
|
|
2023-07-21 17:36:28 +00:00
|
|
|
|
|
|
|
######################
|
|
|
|
# LINTING AND FORMATTING
|
|
|
|
######################
|
|
|
|
|
|
|
|
# Define a variable for Python and notebook files.
|
|
|
|
PYTHON_FILES=.
|
|
|
|
lint format: PYTHON_FILES=.
|
2023-07-27 08:56:55 +00:00
|
|
|
lint_diff format_diff: PYTHON_FILES=$(shell git diff --relative=libs/experimental --name-only --diff-filter=d master | grep -E '\.py$$|\.ipynb$$')
|
2023-07-21 17:36:28 +00:00
|
|
|
|
|
|
|
lint lint_diff:
|
|
|
|
poetry run mypy $(PYTHON_FILES)
|
|
|
|
poetry run black $(PYTHON_FILES) --check
|
|
|
|
poetry run ruff .
|
|
|
|
|
|
|
|
format format_diff:
|
|
|
|
poetry run black $(PYTHON_FILES)
|
|
|
|
poetry run ruff --select I --fix $(PYTHON_FILES)
|
|
|
|
|
|
|
|
spell_check:
|
|
|
|
poetry run codespell --toml pyproject.toml
|
|
|
|
|
|
|
|
spell_fix:
|
|
|
|
poetry run codespell --toml pyproject.toml -w
|
|
|
|
|
|
|
|
######################
|
|
|
|
# HELP
|
|
|
|
######################
|
|
|
|
|
|
|
|
help:
|
|
|
|
@echo '----'
|
|
|
|
@echo 'format - run code formatters'
|
|
|
|
@echo 'lint - run linters'
|
|
|
|
@echo 'test - run unit tests'
|
|
|
|
@echo 'tests - run unit tests'
|
|
|
|
@echo 'test TEST_FILE=<test_file> - run all tests in file'
|
|
|
|
@echo 'test_watch - run unit tests in watch mode'
|