Commit Graph

205 Commits (5157b15446f31b3d5937cbd4990703d686e1bcb6)

Author SHA1 Message Date
Erick Friis 2b7e47a668
infra: install integration deps for test linting (#16963)
<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is
whichever of langchain, community, core, experimental, etc. is being
modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` from the root
of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in
`docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->
5 months ago
Erick Friis 13a6756067
infra: ci naming 2 (#16893) 5 months ago
Erick Friis 5b3fc86cfd
infra: ci naming (#16890)
Make it clearer how to run equivalent commands locally

Not a perfect 1:1, but will help people get started

![Screenshot 2024-02-01 at 10 53
34 AM](https://github.com/langchain-ai/langchain/assets/9557659/da271aaf-d5db-41e3-9379-cb1d8a0232c5)
5 months ago
Erick Friis 17e886388b
nomic: init pkg (#16853)
Co-authored-by: Lance Martin <lance@langchain.dev>
5 months ago
Erick Friis 0255c5808b
infra: move release workflow back (#16707) 5 months ago
Bagatur 5975bf39ec
infra: delete old CI workflows (#16680) 5 months ago
Erick Friis adc008407e
exa: init pkg (#16553) 5 months ago
Erick Friis e5878c467a
infra: scheduled testing env (#16239) 6 months ago
Erick Friis 2f348c695a
infra: add nvidia api secret to integration testing (#15972) 6 months ago
Erick Friis 50959abf0c
infra: google cse id integration test (#16238) 6 months ago
Erick Friis 92bc80483a
infra: google search api key (#16237) 6 months ago
purificant 3606c5d5e9
infra: update poetry 1.6.1 -> 1.7.1 (#15027) 6 months ago
Erick Friis 6a2889a4ec
infra: retry release if not found on test pypi (#15913)
<!-- Thank you for contributing to LangChain!

Please title your PR "<package>: <description>", where <package> is
whichever of langchain, community, core, experimental, etc. is being
modified.

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes if applicable,
  - **Dependencies:** any dependencies required for this change,
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` from the root
of the package you've modified to check this locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc: https://python.langchain.com/docs/contributing/

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in
`docs/docs/integrations` directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->
6 months ago
Erick Friis 98be1e5ed0
infra: title release action runs (#15612)
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#run-name
6 months ago
Erick Friis 1a42ad353a
infra: vertex integration test creds (#15609) 6 months ago
Erick Friis ebc75c5ca7
openai[minor]: implement langchain-openai package (#15503)
Todo

- [x] copy over integration tests
- [x] update docs with new instructions in #15513 
- [x] add linear ticket to bump core -> community, community->langchain,
and core->openai deps
- [ ] (optional): add `pip install langchain-openai` command to each
notebook using it
- [x] Update docstrings to not need `openai` install
- [x] Add serialization
- [x] deprecate old models

Contributor steps:

- [x] Add secret names to manual integrations workflow in
.github/workflows/_integration_test.yml
- [x] Add secrets to release workflow (for pre-release testing) in
.github/workflows/_release.yml

Maintainer steps (Contributors should not do these):

- [x] set up pypi and test pypi projects
- [x] add credential secrets to Github Actions
- [ ] add package to conda-forge


Functional changes to existing classes:

- now relies on openai client v1 (1.6.1) via concrete dep in
langchain-openai package

Codebase organization

- some function calling stuff moved to
`langchain_core.utils.function_calling` in order to be used in both
community and langchain-openai
6 months ago
Erick Friis 1437872df9
infra: fail check_diffs if too many files changed (#15423)
Jobs like
https://github.com/langchain-ai/langchain/actions/runs/7389187843/job/20101494206
only receive the first 300 changed files. Because of the opportunity to
miss packages, better to auto-fail and manually run.

Checking that it does what I expect in #15424
6 months ago
Bagatur 63e0cae2b1
infra: fix min deps test (#15486) 6 months ago
Bagatur 54b58c03db
infra: add minimum deps pre release check (#15485) 6 months ago
Erick Friis a8f6f33cd9
infra: remove path filter on check_diffs (#15418)
CI should run on https://github.com/langchain-ai/langchain/pull/15412

But github only checks first 300 files:
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#git-diff-comparisons

> Diffs are limited to 300 files. If there are files changed that aren't
matched in the first 300 files returned by the filter, the workflow will
not run. You may need to create more specific filters so that the
workflow will run automatically.
6 months ago
purificant 619cd3ce54
ci: upgrade actions (#15114)
This PR upgrades CI actions
[actions/setup-python](https://github.com/actions/setup-python/releases/tag/v5.0.0)
and
[google-github-actions/auth](https://github.com/google-github-actions/auth/releases/tag/v2.0.0)
6 months ago
Erick Friis 8a3360edf6
anthropic: beta messages integration (#14928) 7 months ago
Erick Friis 795cf2ddda
together: package and embedding model (#14936) 7 months ago
Bagatur a5be9f9475
mistralai: Add langchain-mistralai partner package (#14783)
Co-authored-by: Chad Phillips <chad@apartmentlines.com>
7 months ago
Erick Friis 1acc7ffa3f
infra: cut down on integration steps (#14785)
<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
7 months ago
William FH c5296fd42c
[Documentation] Updates to NVIDIA Playground/Foundation Model naming.… (#14770)
…  (#14723)

- **Description:** Minor updates per marketing requests. Namely, name
decisions (AI Foundation Models / AI Playground)
  - **Tag maintainer:** @hinthornw 

Do want to pass around the PR for a bit and ask a few more marketing
questions before merge, but just want to make sure I'm not working in a
vacuum. No major changes to code functionality intended; the PR should
be for documentation and only minor tweaks.

Note: QA model is a bit borked across staging/prod right now. Relevant
teams have been informed and are looking into it, and I'm placeholdered
the response to that of a working version in the notebook.

Co-authored-by: Vadim Kudlay <32310964+VKudlay@users.noreply.github.com>
7 months ago
Bagatur c7b5dbe8ec
infra: fix pre-release integration test and add unit test (#14742) 7 months ago
Bagatur b9975fac89
infra: add action checkout to pre-release-checks (#14732) 7 months ago
Bagatur ba897fc04c
infra: Pre-release integration tests for partner pkgs (#14687) 7 months ago
Bagatur 74211aa02e
infra: add integration test workflow (#14688) 7 months ago
William FH bc3ec78a38
[Workflows] Add nvidia-aiplay to _release.yml (#14722)
As the title says.
In the future will want to have a script to automate this
7 months ago
William FH 405d111da6
[Partner] Add langchain-google-genai package (gemini) (#14621)
Add a new ChatGoogleGenerativeAI class in a `langchain-google-genai`
package.
Still todo: add a deprecation warning in PALM

---------

Co-authored-by: Erick Friis <erick@langchain.dev>
Co-authored-by: Leonid Kuligin <lkuligin@yandex.ru>
Co-authored-by: Bagatur <baskaryan@gmail.com>
7 months ago
Erick Friis 231891706b
infra: skip extended testing for partner packages (#14630)
Tested by merging into #14627
7 months ago
Bagatur 48b7a0584d
infra: Turn release branch check back on (#14563) 7 months ago
Bagatur ed58eeb9c5
community[major], core[patch], langchain[patch], experimental[patch]: Create langchain-community (#14463)
Moved the following modules to new package langchain-community in a backwards compatible fashion:

```
mv langchain/langchain/adapters community/langchain_community
mv langchain/langchain/callbacks community/langchain_community/callbacks
mv langchain/langchain/chat_loaders community/langchain_community
mv langchain/langchain/chat_models community/langchain_community
mv langchain/langchain/document_loaders community/langchain_community
mv langchain/langchain/docstore community/langchain_community
mv langchain/langchain/document_transformers community/langchain_community
mv langchain/langchain/embeddings community/langchain_community
mv langchain/langchain/graphs community/langchain_community
mv langchain/langchain/llms community/langchain_community
mv langchain/langchain/memory/chat_message_histories community/langchain_community
mv langchain/langchain/retrievers community/langchain_community
mv langchain/langchain/storage community/langchain_community
mv langchain/langchain/tools community/langchain_community
mv langchain/langchain/utilities community/langchain_community
mv langchain/langchain/vectorstores community/langchain_community
mv langchain/langchain/agents/agent_toolkits community/langchain_community
mv langchain/langchain/cache.py community/langchain_community
mv langchain/langchain/adapters community/langchain_community
mv langchain/langchain/callbacks community/langchain_community/callbacks
mv langchain/langchain/chat_loaders community/langchain_community
mv langchain/langchain/chat_models community/langchain_community
mv langchain/langchain/document_loaders community/langchain_community
mv langchain/langchain/docstore community/langchain_community
mv langchain/langchain/document_transformers community/langchain_community
mv langchain/langchain/embeddings community/langchain_community
mv langchain/langchain/graphs community/langchain_community
mv langchain/langchain/llms community/langchain_community
mv langchain/langchain/memory/chat_message_histories community/langchain_community
mv langchain/langchain/retrievers community/langchain_community
mv langchain/langchain/storage community/langchain_community
mv langchain/langchain/tools community/langchain_community
mv langchain/langchain/utilities community/langchain_community
mv langchain/langchain/vectorstores community/langchain_community
mv langchain/langchain/agents/agent_toolkits community/langchain_community
mv langchain/langchain/cache.py community/langchain_community
```

Moved the following to core
```
mv langchain/langchain/utils/json_schema.py core/langchain_core/utils
mv langchain/langchain/utils/html.py core/langchain_core/utils
mv langchain/langchain/utils/strings.py core/langchain_core/utils
cat langchain/langchain/utils/env.py >> core/langchain_core/utils/env.py
rm langchain/langchain/utils/env.py
```

See .scripts/community_split/script_integrations.sh for all changes
7 months ago
Bagatur 300305e5e5
infra: add langchain-community release workflow (#14469) 7 months ago
Erick Friis b3f226e8f8
core[patch], langchain[patch], experimental[patch]: import CI (#14414) 7 months ago
Erick Friis 477b274a62
langchain[patch]: fix scheduled testing ci dep install (#14460) 7 months ago
Erick Friis ff0d5514c1
langchain[patch]: fix scheduled testing ci variables (#14459) 7 months ago
Erick Friis 1d725327eb
langchain[patch]: Fix scheduled testing (#14428)
- integration tests in pyproject
- integration test fixes
7 months ago
Bagatur b2280fd874
core[patch], langchain[patch]: fix required deps (#14373) 7 months ago
Bagatur ce4d81f88b
infra: ci matrix (#14306) 7 months ago
Bagatur 48fbc5513d
infra[patch], langchain[patch]: fix test deps and upper bound langchain dep on core(#13984) 7 months ago
Nuno Campos e0bcc98436
infra[patch]: Use langchain core in-tree as a dev dependency (#13957)
Using the published version means master is broken for contributors
whenever we make changes in one lib that depend on the other.
7 months ago
Bagatur bcf83988ec
Revert "INFRA: temp rm master condition (#13753)" (#13759) 7 months ago
Bagatur df471b0c0b
INFRA: temp rm master condition (#13753) 7 months ago
Bagatur b6b7654f7f
INFRA: run LC ci after core changes (#13742) 7 months ago
Bagatur c61e30632e
BUG: more core fixes (#13665)
Fix some circular deps:
- move PromptValue into top level module bc both PromptTemplates and
OutputParsers import
- move tracer context vars to `tracers.context` and import them in
functions in `callbacks.manager`
- add core import tests
8 months ago
Harrison Chase d82cbf5e76
Separate out langchain_core package (#13577)
Co-authored-by: Nuno Campos <nuno@boringbits.io>
Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Erick Friis <erick@langchain.dev>
8 months ago
Erick Friis 9545f0666d
fix cli release (#13373)
My thought is that the ==version would prevent pip from finding the
package on regular [pypi.org](http://pypi.org/), so it would look at
[test.pypi.org](http://test.pypi.org/) for that. Otherwise it'll pull
package from [pypi.org](http://pypi.org/) (e.g. sub deps)

Right now, the cli release is failing because it's going to
test.pypi.org by default, so it finds this incorrect FASTAPI package
instead of the real one: https://test.pypi.org/project/FASTAPI/
8 months ago
Predrag Gruevski 2ebd167dba
Lint Python notebooks with ruff. (#12677)
The new ruff version fixed the blocking bugs, and I was able to fairly
easily us to a passing state: ruff fixed some issues on its own, I fixed
a handful by hand, and I added a list of narrowly-targeted exclusions
for files that are currently failing ruff rules that we probably should
look into eventually.

I went pretty lenient on the docs / cookbooks rules, allowing dead code
and such things. Perhaps in the future we may want to tighten the rules
further, but this is already a good set of checks that found real issues
and will prevent them going forward.
8 months ago
Predrag Gruevski 724b92231d
Remove `black` caching config from CI lint workflow. (#12594)
To merge after #12585 is merged.
8 months ago
Predrag Gruevski 0ea837404a
Only publish to test PyPI from the `_test_release.yml` workflow. (#12668)
PyPI trusted publishing wants to know which workflow is expected to do
the publish. We always want to publish from the same workflow, so we're
making `_test_release.yml` the only workflow that publishes to Test
PyPI.
8 months ago
Predrag Gruevski 321cd44f13
Use separate jobs for building and publishing test releases. (#12671)
This follows the principle of least privilege. Our `poetry build` step
doesn't need, and shouldn't get, access to our GitHub OIDC capability.

This is the same structure as I used in the already-merged PR for
refactoring the regular PyPI release workflow: #12578.
8 months ago
Predrag Gruevski aa3f4a9bc8
Remove the CLI package's pydantic compatibility tests. (#12675)
They aren't necessary, since the CLI package doesn't have a direct
dependency on pydantic.
8 months ago
Predrag Gruevski 360cff81a3
Overwrite existing distributions when uploading to test PyPI. (#12658) 8 months ago
Predrag Gruevski 5308b836c7
Upgrade to `actions/checkout@v4` in the docs lint job. (#12581) 8 months ago
Predrag Gruevski 94f018f1ba
Support release-testing packages with dashes in their names. (#12654) 8 months ago
Predrag Gruevski 72fa5a463d
Show ruff output inline in GitHub PRs. (#12647) 8 months ago
Erick Friis e933212a3d
run poetry build in working dir (#12610)
Was failing because was trying to build from root:
https://github.com/langchain-ai/langchain/actions/runs/6700033981/job/18205251365
8 months ago
Predrag Gruevski 3c5c384f1a
Test-publish to test PyPI and separate jobs to limit permissions. (#12578)
Before making a new `langchain` release, we want to test that everything
works as expected. This PR lets us publish `langchain` to test PyPI,
then install it from there and run checks to ensure everything works
normally before publishing it "for real".

It also takes the opportunity to refactor the build process, splitting
up the build, release-creation, and PyPI upload steps into separate jobs
that do not share their elevated permissions with each other.
8 months ago
Bagatur 0b4b9e61fc
Bagatur/fix doc ci (#12529) 8 months ago
Bagatur 2424fff3f1
notebook fmt (#12498) 8 months ago
Harrison Chase 0660c06cf1
add gha for cli (#12492) 8 months ago
Erick Friis afcc12d99e
Templates CI (#12313)
Adds a `langchain-location` param to lint, so we can properly locate it.

Regular langchain and experimental lint steps are passing, so default
value seems to be working.
8 months ago
Erick Friis 4db8d82c55
CLI CI 2 (#12387)
Will run all CI because of _test change, but future PRs against CLI will
only trigger the new CLI one

Has a bunch of file changes related to formatting/linting.

No mypy yet - coming soon
8 months ago
Bagatur 7cadf00570
better lint triggering (#12376) 8 months ago
Bagatur 76230d2c08
fireworks scheduled integration tests (#12373) 8 months ago
Bagatur b10cefb160
lint fix: rm init (#12374) 8 months ago
Bagatur 2008a6438c
add experimental test release gha (#12229) 8 months ago
Bagatur 8ba97cb408
separate compile integration tests (#12171)
Co-authored-by: Predrag Gruevski <2348618+obi1kenobi@users.noreply.github.com>
8 months ago
Bagatur 963ff93476
bump 321 (#12161) 8 months ago
Predrag Gruevski 95a1b598fe
Update to `actions/checkout@v4`. (#11951)
We don't use any of the new functionality at the moment. Just making
sure we don't fall back on versions and fail to benefit from new
patches. This is an easy upgrade and it's always harder to upgrade
across multiple major versions at once.
8 months ago
Bagatur 85302a9ec1
Add CI check that integration tests compile (#12090) 9 months ago
Bagatur bd74eba152
add azure openai sched tests (#11723) 9 months ago
Bagatur eedfddac2d
Restructure docs (#11620) 9 months ago
Nuno Campos 2c11302598
Update langchain_release.yml (#11444) 9 months ago
Predrag Gruevski 88c5349196
Revert "Rm additional file check for scheduled tests (#11192)" (#11297)
This reverts commit ff90bb59bf.

Requires #11296 to merge first.
9 months ago
Predrag Gruevski 37f2f71156
Trigger Docker release workflow after new langchain release is made. (#11290)
We want to publish a new Docker image after a new langchain Python
package version is published.
9 months ago
Predrag Gruevski d21dd72d64
Upgrade CI workflows to poetry 1.6.1. (#11344) 9 months ago
Eugene Yurtsev 2343302fc6
Remove langserve from langchain repo (#11288)
LangServe has been moved to a separate repo
9 months ago
CG80499 943e4f30d8
Add scoring chain (#11123)
<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - **Description:** a description of the change, 
  - **Issue:** the issue # it fixes (if applicable),
  - **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
 -->
9 months ago
Bagatur 38d5b63a10
Bedrock scheduled tests (#11194) 9 months ago
Nuno Campos 32a8b311eb
Add base docker image and ci script for building and pushing (#10927) 9 months ago
Eugene Yurtsev aebdb1ad01
Ignore aadd (#11235) 9 months ago
Bagatur ff90bb59bf
Rm additional file check for scheduled tests (#11192)
cc @obi1kenobi Causing issues with GHA creds
https://github.com/langchain-ai/langchain/actions/runs/6342674950/job/17228926776
9 months ago
Bagatur 3508e582f1
add anthropic scheduled tests and unit tests (#11188) 9 months ago
Eugene Yurtsev 176d71dd85
LangServe: Add release workflow (#11178)
Add release workflow to langserve
9 months ago
Eugene Yurtsev b05bb9e136
LangServe (#11046)
Adds LangServe package

* Integrate Runnables with Fast API creating Server and a RemoteRunnable
client
* Support multiple runnables for a given server
* Support sync/async/batch/abatch/stream/astream/astream_log on the
client side (using async implementations on server)
* Adds validation using annotations (relying on pydantic under the hood)
-- this still has some rough edges -- e.g., open api docs do NOT
generate correctly at the moment
* Uses pydantic v1 namespace

Known issues: type translation code doesn't handle a lot of types (e.g.,
TypedDicts)

---------

Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
9 months ago
Bagatur 040d436b3f
Add vertex scheduled test (#10958) 9 months ago
Bagatur dccc20b402
add model feat table (#10921) 10 months ago
Harrison Chase 2c957de2fc
add checks on basic base modules (#10693) 10 months ago
Harrison Chase 5442d2b1fa
Harrison/stop importing from init (#10690) 10 months ago
Predrag Gruevski ccb9e3ee2d
Install dev, lint, test, typing extra deps for linting steps. (#10249)
`mypy` cannot type-check code that relies on dependencies that aren't
installed.

Eventually we'll probably want to install as many optional dependencies
as possible. However, the full "extended deps" setup for langchain
creates a 3GB cache file and takes a while to unpack and install. We'll
probably want something a bit more targeted.

This is a first step toward something better.
10 months ago
Predrag Gruevski 82d5d4d0ae
Deny creating files as a result of test runs. (#10253)
A test file was accidentally dropping a `results.json` file in the
current working directory as a result of running `make test`.

This is undesirable, since we don't want to risk accidentally adding
stray files into the repo if we run tests locally and then do `git add
.` without inspecting the file list very closely.
10 months ago
Predrag Gruevski 803be5b986
Run CI when CI infra itself has changed. (#10239)
Make sure that changes to CI infrastructure get tested on CI before
being merged.

Without this PR, changes to the poetry setup action don't trigger a CI
run and in principle could break `master` when merged.
10 months ago
maks-operlejn-ds a8f804a618
Add data anonymizer (#9863)
### Description

The feature for anonymizing data has been implemented. In order to
protect private data, such as when querying external APIs (OpenAI), it
is worth pseudonymizing sensitive data to maintain full privacy.

Anonynization consists of two steps:

1. **Identification:** Identify all data fields that contain personally
identifiable information (PII).
2. **Replacement**: Replace all PIIs with pseudo values or codes that do
not reveal any personal information about the individual but can be used
for reference. We're not using regular encryption, because the language
model won't be able to understand the meaning or context of the
encrypted data.

We use *Microsoft Presidio* together with *Faker* framework for
anonymization purposes because of the wide range of functionalities they
provide. The full implementation is available in `PresidioAnonymizer`.

### Future works

- **deanonymization** - add the ability to reverse anonymization. For
example, the workflow could look like this: `anonymize -> LLMChain ->
deanonymize`. By doing this, we will retain anonymity in requests to,
for example, OpenAI, and then be able restore the original data.
- **instance anonymization** - at this point, each occurrence of PII is
treated as a separate entity and separately anonymized. Therefore, two
occurrences of the name John Doe in the text will be changed to two
different names. It is therefore worth introducing support for full
instance detection, so that repeated occurrences are treated as a single
object.

### Twitter handle
@deepsense_ai / @MaksOpp

---------

Co-authored-by: MaksOpp <maks.operlejn@gmail.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
10 months ago
Predrag Gruevski 9aaa0fdce0 Use unified Python setup steps for release workflow. 10 months ago
Predrag Gruevski c06f34fa35
Use new Python setup approach for scheduled tests. (#9626)
Using the same new unified Python setup as the regular tests and the
lint job, as set up in #9625.
11 months ago
Predrag Gruevski 83986ea98a
Cache poetry install + unify Python/Poetry setup for lint and test jobs. (#9625)
With this PR:
- All lint and test jobs use the exact same Python + Poetry installation
approach, instead of lints doing it one way and tests doing it another
way.
- The Poetry installation itself is cached, which saves ~15s per run.
- We no longer pass shell commands as workflow arguments to a workflow
that just runs them in a shell. This makes our actions more resilient to
shell code injection.

If y'all like this approach, I can modify the scheduled tests workflow
and the release workflow to use this too.
11 months ago