langchain

Commit Graph

Author	SHA1	Message	Date
maks-operlejn-ds	a8f804a618	Add data anonymizer (#9863 ) ### Description The feature for anonymizing data has been implemented. In order to protect private data, such as when querying external APIs (OpenAI), it is worth pseudonymizing sensitive data to maintain full privacy. Anonynization consists of two steps: 1. Identification: Identify all data fields that contain personally identifiable information (PII). 2. Replacement: Replace all PIIs with pseudo values or codes that do not reveal any personal information about the individual but can be used for reference. We're not using regular encryption, because the language model won't be able to understand the meaning or context of the encrypted data. We use Microsoft Presidio together with Faker framework for anonymization purposes because of the wide range of functionalities they provide. The full implementation is available in `PresidioAnonymizer`. ### Future works - deanonymization - add the ability to reverse anonymization. For example, the workflow could look like this: `anonymize -> LLMChain -> deanonymize`. By doing this, we will retain anonymity in requests to, for example, OpenAI, and then be able restore the original data. - instance anonymization - at this point, each occurrence of PII is treated as a separate entity and separately anonymized. Therefore, two occurrences of the name John Doe in the text will be changed to two different names. It is therefore worth introducing support for full instance detection, so that repeated occurrences are treated as a single object. ### Twitter handle @deepsense_ai / @MaksOpp --------- Co-authored-by: MaksOpp <maks.operlejn@gmail.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Predrag Gruevski	8dbf4cbe80	Add notice about security-sensitive experimental code to experimental README. (#9936 ) It renders like this: https://github.com/langchain-ai/langchain/tree/pg/experimental-readme/libs/experimental ![image](https://github.com/langchain-ai/langchain/assets/2348618/a5f9569d-96f6-44c6-8559-921adb3e337d)	1 year ago
Predrag Gruevski	b5cd1e0fed	Add security notices on PAL and CPAL experimental chains. (#9938 ) Clearly document that the PAL and CPAL techniques involve generating code, and that such code must be properly sandboxed and given appropriate narrowly-scoped credentials in order to ensure security. While our implementations include some mitigations, Python and SQL sandboxing is well-known to be a very hard problem and our mitigations are no replacement for proper sandboxing and permissions management. The implementation of such techniques must be performed outside the scope of the Python process where this package's code runs, so its correct setup and administration must therefore be the responsibility of the user of this code.	1 year ago
Bagatur	d6957921f0	bump 276 (#9931 )	1 year ago
maks-operlejn-ds	f327535eda	Add conftest file to langchain experimental (#9886 ) In order to use `requires` marker in langchain-experimental, there's a need for conftest.py file inside. Everything is identical to the main langchain module. Co-authored-by: maks-operlejn-ds <maks.operlejn@gmail.com>	1 year ago
Predrag Gruevski	eb3d1fa93c	Add security warning to experimental `SQLDatabaseChain` class. (#9867 ) The most reliable way to not have a chain run an undesirable SQL command is to not give it database permissions to run that command. That way the database itself performs the rule enforcement, so it's much easier to configure and use properly than anything we could add in ourselves.	1 year ago
nikhilkjha	d57d08fd01	Initial commit for comprehend moderator (#9665 ) This PR implements a custom chain that wraps Amazon Comprehend API calls. The custom chain is aimed to be used with LLM chains to provide moderation capability that let’s you detect and redact PII, Toxic and Intent content in the LLM prompt, or the LLM response. The implementation accepts a configuration object to control what checks will be performed on a LLM prompt and can be used in a variety of setups using the LangChain expression language to not only detect the configured info in chains, but also other constructs such as a retriever. The included sample notebook goes over the different configuration options and how to use it with other chains. ### Usage sample ```python from langchain_experimental.comprehend_moderation import BaseModerationActions, BaseModerationFilters moderation_config = { "filters":[ BaseModerationFilters.PII, BaseModerationFilters.TOXICITY, BaseModerationFilters.INTENT ], "pii":{ "action": BaseModerationActions.ALLOW, "threshold":0.5, "labels":["SSN"], "mask_character": "X" }, "toxicity":{ "action": BaseModerationActions.STOP, "threshold":0.5 }, "intent":{ "action": BaseModerationActions.STOP, "threshold":0.5 } } comp_moderation_with_config = AmazonComprehendModerationChain( moderation_config=moderation_config, #specify the configuration client=comprehend_client, #optionally pass the Boto3 Client verbose=True ) template = """Question: {question} Answer:""" prompt = PromptTemplate(template=template, input_variables=["question"]) responses = [ "Final Answer: A credit card number looks like 1289-2321-1123-2387. A fake SSN number looks like 323-22-9980. John Doe's phone number is (999)253-9876.", "Final Answer: This is a really shitty way of constructing a birdhouse. This is fucking insane to think that any birds would actually create their motherfucking nests here." ] llm = FakeListLLM(responses=responses) llm_chain = LLMChain(prompt=prompt, llm=llm) chain = ( prompt \| comp_moderation_with_config \| {llm_chain.input_keys[0]: lambda x: x['output'] } \| llm_chain \| { "input": lambda x: x['text'] } \| comp_moderation_with_config ) response = chain.invoke({"question": "A sample SSN number looks like this 123-456-7890. Can you give me some more samples?"}) print(response['output']) ``` ### Output ``` > Entering new AmazonComprehendModerationChain chain... Running AmazonComprehendModerationChain... Running pii validation... Found PII content..stopping.. The prompt contains PII entities and cannot be processed ``` --------- Co-authored-by: Piyush Jain <piyushjain@duck.com> Co-authored-by: Anjan Biswas <anjanavb@amazon.com> Co-authored-by: Jha <nikjha@amazon.com> Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Bagatur	9731ce5a40	bump 273 (#9751 )	1 year ago
Predrag Gruevski	d564ec944c	`poetry lock` the experimental package. (#9478 )	1 year ago
Predrag Gruevski	de1f63505b	Add `py.typed` file to `langchain-experimental`. (#9557 ) The package is linted with mypy, so its type hints are correct and should be exposed publicly. Without this file, the type hints remain private and cannot be used by downstream users of the package.	1 year ago
Predrag Gruevski	eee0d1d0dd	Update repository links in the package metadata. (#9454 )	1 year ago
Bagatur	a69d1b84f4	bump 267 (#9403 )	1 year ago
Nuno Campos	c0d67420e5	Use a submodule for pydantic v1 compat (#9371 ) <!-- Thank you for contributing to LangChain! Replace this entire comment with: - Description: a description of the change, - Issue: the issue # it fixes (if applicable), - Dependencies: any dependencies required for this change, - Tag maintainer: for a quicker response, tag the relevant maintainer (see below), - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out! Please make sure your PR is passing linting and testing before submitting. Run `make format`, `make lint` and `make test` to check this locally. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. These live is docs/extras directory. If no one reviews your PR within a few days, please @-mention one of @baskaryan, @eyurtsev, @hwchase17, @rlancemartin. -->	1 year ago
Bagatur	995ef8a7fc	unpin pydantic (#9356 )	1 year ago
Eugene Yurtsev	2673b3a314	Create pydantic v1 namespace in langchain (#9254 ) Create pydantic v1 namespace in langchain experimental	1 year ago
Bagatur	5935767056	bump lc 246, lce 9 (#9207 )	1 year ago
UmerHA	8aab39e3ce	Added SmartGPT workflow (issue #4463 ) (#4816 ) # Added SmartGPT workflow by providing SmartLLM wrapper around LLMs Edit: As @hwchase17 suggested, this should be a chain, not an LLM. I have adapted the PR. It is used like this: ``` from langchain.prompts import PromptTemplate from langchain.chains import SmartLLMChain from langchain.chat_models import ChatOpenAI hard_question = "I have a 12 liter jug and a 6 liter jug. I want to measure 6 liters. How do I do it?" hard_question_prompt = PromptTemplate.from_template(hard_question) llm = ChatOpenAI(model_name="gpt-4") prompt = PromptTemplate.from_template(hard_question) chain = SmartLLMChain(llm=llm, prompt=prompt, verbose=True) chain.run({}) ``` Original text: Added SmartLLM wrapper around LLMs to allow for SmartGPT workflow (as in https://youtu.be/wVzuvf9D9BU). SmartLLM can be used wherever LLM can be used. E.g: ``` smart_llm = SmartLLM(llm=OpenAI()) smart_llm("What would be a good company name for a company that makes colorful socks?") ``` or ``` smart_llm = SmartLLM(llm=OpenAI()) prompt = PromptTemplate( input_variables=["product"], template="What is a good name for a company that makes {product}?", ) chain = LLMChain(llm=smart_llm, prompt=prompt) chain.run("colorful socks") ``` SmartGPT consists of 3 steps: 1. Ideate - generate n possible solutions ("ideas") to user prompt 2. Critique - find flaws in every idea & select best one 3. Resolve - improve upon best idea & return it Fixes #4463 ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: - @hwchase17 - @agola11 Twitter: [@UmerHAdil](https://twitter.com/@UmerHAdil) \| Discord: RicChilligerDude#7589 --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
DJ Atha	ee52482db8	Fix issue 7445 (#7635 ) Description: updated BabyAGI examples and experimental to append the iteration to the result id to fix error storing data to vectorstore. Issue: 7445 Dependencies: no Tag maintainer: @eyurtsev This fix worked for me locally. Happy to take some feedback and iterate on a better solution. I was considering appending a uuid instead but didn't want to over complicate the example. --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Bagatur	95cf7de112	scheduled tests GHA (#8879 ) Adding scheduled daily GHA that runs marked integration tests. To start just marking some tests in test_openai	1 year ago
Harrison Chase	4d526c49ed	bump experimental to 008 (#8490 )	1 year ago
Harrison Chase	8f14ddefdf	add anthropic functions wrapper (#8475 ) a cheeky wrapper around claude that adds in function calling support (kind of, hence it going in experimental)	1 year ago
Harrison Chase	2448043b84	bump and fix (#8441 )	1 year ago
Harrison Chase	fab24457bc	remove code (#8425 )	1 year ago
Harrison Chase	3a78450883	update experimental (#8402 ) some changes were made to experimental, porting them over	1 year ago
Bagatur	61dd92f821	bump 246 (#8410 )	1 year ago
Harrison Chase	1b0bfa54cf	cr	1 year ago
Martin Krasser	93260a9922	Fix broken `make` targets `format_diff` and `lint_diff` (#8344 ) Since the refactoring into sub-projects `libs/langchain` and `libs/experimental`, the `make` targets `format_diff` and `lint_diff` do not work anymore when running `make` from these subdirectories. Reason is that ``` PYTHON_FILES=$(shell git diff --name-only --diff-filter=d master \| grep -E '\.py$$\|\.ipynb$$') ``` generates paths from the project's root directory instead of the corresponding subdirectories. This PR fixes this by adding a `--relative` command line option. - Tag maintainer: @baskaryan	1 year ago
Harrison Chase	ae78ef7fe6	bump experimental to 005 (#8339 )	1 year ago
Vadim Gubergrits	e7e5cb9d08	Tree of Thought introducing a new ToTChain. (#5167 ) # [WIP] Tree of Thought introducing a new ToTChain. This PR adds a new chain called ToTChain that implements the ["Large Language Model Guided Tree-of-Though"](https://arxiv.org/pdf/2305.08291.pdf) paper. There's a notebook example `docs/modules/chains/examples/tot.ipynb` that shows how to use it. Implements #4975 ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: - @hwchase17 - @vowelparrot --------- Co-authored-by: Vadim Gubergrits <vgubergrits@outbox.com> Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>	1 year ago
Daniel Alexander Brenot	bf1357f584	Added async support to PlanAndExecute Chain (#8239 ) - Description: Adds async support to the PlanAndExecute Chain Maintainer responsibilities: - Async: @agola11 --------- Co-authored-by: Bagatur <baskaryan@gmail.com>	1 year ago
Leonid Ganeline	ee6ff96e28	docstrings cleanup (#8311 ) - added missed docstrings - changed docstrings into consistent format @baskaryan	1 year ago
Nuno Campos	a612800ef0	Runnable single protocol (#7800 ) Objects implementing Runnable: BasePromptTemplate, LLM, ChatModel, Chain, Retriever, OutputParser - [x] Implement Runnable in base Retriever - [x] Raise TypeError in operator methods for unsupported things - [x] Implement dict which calls values in parallel and outputs dict with results - [x] Merge in `+` for prompts - [x] Confirm precedence order for operators, ideal would be `+` `\|`, https://docs.python.org/3/reference/expressions.html#operator-precedence - [x] Add support for openai functions, ie. Chat Models must return messages - [x] Implement BaseMessageChunk return type for BaseChatModel, a subclass of BaseMessage which implements __add__ to return BaseMessageChunk, concatenating all str args - [x] Update implementation of stream/astream for llm and chat models to use new `_stream`, `_astream` optional methods, with default implementation in base class `raise NotImplementedError` use https://stackoverflow.com/a/59762827 to see if it is implemented in base class - [x] Delete the IteratorCallbackHandler (leave the async one because people using) - [x] Make BaseLLMOutputParser implement Runnable, accepting either str or BaseMessage --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>	1 year ago
Bagatur	5c6dcb1960	bump 243 (#8289 )	1 year ago
Leonid Ganeline	c580c81cca	docstrings `experimental` (#7969 ) - added/changed docstring for `experimental` - added/changed docstrings for different artifacts - @baskaryan	1 year ago
Bagatur	82b8d8596c	bump lc241 exp3 (#8193 )	1 year ago
Bagatur	4928f7a9f5	undo bump (#8192 )	1 year ago
Bagatur	d5689d58ab	Bagatur/bump 241 (#8182 )	1 year ago
Harrison Chase	3caccf304c	Harrison/hugginggpt (#8162 ) Co-authored-by: Yongliang Shen <withsyl@163.com>	1 year ago
Harrison Chase	77bf75c236	bump experimental to 002 (#8150 )	1 year ago
Harrison Chase	e46126eac6	add llamaapi (#8140 )	1 year ago
Harrison Chase	9f3073d418	bump versions (#8129 )	1 year ago
Harrison Chase	aa0e69bc98	Harrison/official pre release (#8106 )	1 year ago
Harrison Chase	8dcabd9205	bump releases rc0 (#8097 )	1 year ago
Harrison Chase	d353d668e4	remove CVEs (#8092 ) This PR aims to move all code with CVEs into `langchain.experimental`. Note that we are NOT yet removing from the core `langchain` package - we will give people a week to migrate here. See MIGRATE.md for how to migrate Zero changes to functionality Vulnerabilities this addresses: PALChain: - https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5752409 - https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5759265 SQLDatabaseChain - https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5759268 `load_prompt` (Python files only) - https://security.snyk.io/vuln/SNYK-PYTHON-LANGCHAIN-5725807	1 year ago
Harrison Chase	da04760de1	Harrison/move experimental (#8084 )	1 year ago

1 2 3 4

195 Commits (f3601b0aaff4a7e9a0945a4e7bc784e44f84a481)