12 Commits (bfd719f9d8454c69ce4fc8df2e8b4c96cb446ce3)

Author	SHA1	Message	Date
Bagatur	85302a9ec1	Add CI check that integration tests compile (#12090)	11 months ago
Predrag Gruevski	dcd0392423	Upgrade to newer black (23.10) and ruff (first 0.1.x!) versions. (#11944) ... Minor lint dependency version upgrade to pick up latest functionality. Ruff's new v0.1 version comes with lots of nice features, like fix-safety guarantees and a preview mode for not-yet-stable features: https://astral.sh/blog/ruff-v0.1.0	11 months ago
Predrag Gruevski	7c0f1bf23f	Upgrade experimental package dependencies and use Poetry 1.6.1. (#11339) ... Part of upgrading our CI to use Poetry 1.6.1.	11 months ago
Bagatur	8fafa1af91	merge	12 months ago
Predrag Gruevski	c9986bc3a9	Tweak type hints to match dependency's behavior. (#11355) ... Needs #11353 to merge first, and a new `langchain` to be published with those changes.	12 months ago
olgavrou	2dba4046fa	update experimental poetry lock	1 year ago
刘 方瑞	890ed775a3	Resolve: VectorSearch enabled SQLChain? (#10177) ... Squashed from #7454 with updated features We have separated the `SQLDatabseChain` from `VectorSQLDatabseChain` and put everything into `experimental/`. Below is the original PR message from #7454. ------- We have been working on features to fill up the gap among SQL, vector search and LLM applications. Some inspiring works like self-query retrievers for VectorStores (for example [Weaviate](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/weaviate_self_query.html) and [others](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query.html)) really turn those vector search databases into a powerful knowledge base! 🚀🚀 We are thinking if we can merge all in one, like SQL and vector search and LLMChains, making this SQL vector database memory as the only source of your data. Here are some benefits we can think of for now, maybe you have more 👀: With ALL data you have: since you store all your pasta in the database, you don't need to worry about the foreign keys or links between names from other data source. Flexible data structure: Even if you have changed your schema, for example added a table, the LLM will know how to JOIN those tables and use those as filters. SQL compatibility: We found that vector databases that supports SQL in the marketplace have similar interfaces, which means you can change your backend with no pain, just change the name of the distance function in your DB solution and you are ready to go! ### Issue resolved: - [Feature Proposal: VectorSearch enabled SQLChain?](https://github.com/hwchase17/langchain/issues/5122) ### Change made in this PR: - An improved schema handling that ignore `types.NullType` columns - A SQL output Parser interface in `SQLDatabaseChain` to enable Vector SQL capability and further more - A Retriever based on `SQLDatabaseChain` to retrieve data from the database for RetrievalQAChains and many others - Allow `SQLDatabaseChain` to retrieve data in python native format - Includes PR #6737 - Vector SQL Output Parser for `SQLDatabaseChain` and `SQLDatabaseChainRetriever` - Prompts that can implement text to VectorSQL - Corresponding unit-tests and notebook ### Twitter handle: - @MyScaleDB ### Tag Maintainer: Prompts / General: @hwchase17, @baskaryan DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev ### Dependencies: No dependency added	1 year ago
Tomaz Bratanic	db73c9d5b5	Diffbot Graph Transformer / Neo4j Graph document ingestion (#9979) ... Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
maks-operlejn-ds	a8f804a618	Add data anonymizer (#9863) ... ### Description The feature for anonymizing data has been implemented. In order to protect private data, such as when querying external APIs (OpenAI), it is worth pseudonymizing sensitive data to maintain full privacy. Anonynization consists of two steps: 1. **Identification:** Identify all data fields that contain personally identifiable information (PII). 2. **Replacement**: Replace all PIIs with pseudo values or codes that do not reveal any personal information about the individual but can be used for reference. We're not using regular encryption, because the language model won't be able to understand the meaning or context of the encrypted data. We use *Microsoft Presidio* together with *Faker* framework for anonymization purposes because of the wide range of functionalities they provide. The full implementation is available in `PresidioAnonymizer`. ### Future works - **deanonymization** - add the ability to reverse anonymization. For example, the workflow could look like this: `anonymize -> LLMChain -> deanonymize`. By doing this, we will retain anonymity in requests to, for example, OpenAI, and then be able restore the original data. - **instance anonymization** - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object. ### Twitter handle @deepsense_ai / @MaksOpp --------- Co-authored-by: MaksOpp <maks.operlejn@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Predrag Gruevski	d564ec944c	`poetry lock` the experimental package. (#9478)	1 year ago
Harrison Chase	d353d668e4	remove CVEs (#8092) ... This PR aims to move all code with CVEs into `langchain.experimental`. Note that we are NOT yet removing from the core `langchain` package - we will give people a week to migrate here. See MIGRATE.md for how to migrate Zero changes to functionality Vulnerabilities this addresses: PALChain: - https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5752409 - https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5759265 SQLDatabaseChain - https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5759268 `load_prompt` (Python files only) - https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5725807	1 year ago
Harrison Chase	da04760de1	Harrison/move experimental (#8084)	1 year ago