Following this tutoral about using OpenAI Embeddings with FAISS
https://python.langchain.com/docs/integrations/vectorstores/faiss
```python
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import FAISS
from langchain.document_loaders import TextLoader
from langchain.document_loaders import TextLoader
loader = TextLoader("../../../extras/modules/state_of_the_union.txt")
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings()
```
This works fine
```python
db = FAISS.from_documents(docs, embeddings)
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)
```
But the async version is not
```python
db = await FAISS.afrom_documents(docs, embeddings) # NotImplementedError
query = "What did the president say about Ketanji Brown Jackson"
docs = await db.asimilarity_search(query) # this will use await asyncio.get_event_loop().run_in_executor under the hood and will not call OpenAIEmbeddings.aembed_query but call OpenAIEmbeddings.embed_query
```
So this PR add async/await supports for FAISS
---------
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
- Description: adding support to Activeloop's DeepMemory feature that
boosts recall up to 25%. Added Jupyter notebook showcasing the feature
and also made index params explicit.
- Twitter handle: will really appreciate if we could announce this on
twitter.
---------
Co-authored-by: adolkhan <adilkhan.sarsen@alumni.nu.edu.kz>
Bumps
[@babel/traverse](https://github.com/babel/babel/tree/HEAD/packages/babel-traverse)
from 7.22.8 to 7.23.2.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/babel/babel/releases"><code>@babel/traverse</code>'s
releases</a>.</em></p>
<blockquote>
<h2>v7.23.2 (2023-10-11)</h2>
<p><strong>NOTE</strong>: This release also re-publishes
<code>@babel/core</code>, even if it does not appear in the linked
release commit.</p>
<p>Thanks <a
href="https://github.com/jimmydief"><code>@jimmydief</code></a> for
your first PR!</p>
<h4>🐛 Bug Fix</h4>
<ul>
<li><code>babel-traverse</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16033">#16033</a>
Only evaluate own String/Number/Math methods (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16022">#16022</a>
Rewrite <code>.tsx</code> extension when using
<code>rewriteImportExtensions</code> (<a
href="https://github.com/jimmydief"><code>@jimmydief</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16017">#16017</a>
Fix: fallback to typeof when toString is applied to incompatible object
(<a href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>,
<code>babel-runtime</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16025">#16025</a>
Avoid override mistake in namespace imports (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h4>Committers: 5</h4>
<ul>
<li>Babel Bot (<a
href="https://github.com/babel-bot"><code>@babel-bot</code></a>)</li>
<li>Huáng Jùnliàng (<a
href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
<li>James Diefenderfer (<a
href="https://github.com/jimmydief"><code>@jimmydief</code></a>)</li>
<li>Nicolò Ribaudo (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
<li><a
href="https://github.com/liuxingbaoyu"><code>@liuxingbaoyu</code></a></li>
</ul>
<h2>v7.23.1 (2023-09-25)</h2>
<p>Re-publishing <code>@babel/helpers</code> due to a publishing error
in 7.23.0.</p>
<h2>v7.23.0 (2023-09-25)</h2>
<p>Thanks <a
href="https://github.com/lorenzoferre"><code>@lorenzoferre</code></a>
and <a
href="https://github.com/RajShukla1"><code>@RajShukla1</code></a> for
your first PRs!</p>
<h4>🚀 New Feature</h4>
<ul>
<li><code>babel-plugin-proposal-import-wasm-source</code>,
<code>babel-plugin-syntax-import-source</code>,
<code>babel-plugin-transform-dynamic-import</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15870">#15870</a>
Support transforming <code>import source</code> for wasm (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-helper-module-transforms</code>,
<code>babel-helpers</code>,
<code>babel-plugin-proposal-import-defer</code>,
<code>babel-plugin-syntax-import-defer</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>,
<code>babel-runtime</code>, <code>babel-standalone</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15878">#15878</a>
Implement <code>import defer</code> proposal transform support (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-generator</code>, <code>babel-parser</code>,
<code>babel-types</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15845">#15845</a>
Implement <code>import defer</code> parsing support (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
<li><a
href="https://redirect.github.com/babel/babel/pull/15829">#15829</a> Add
parsing support for the "source phase imports" proposal (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-generator</code>,
<code>babel-helper-module-transforms</code>, <code>babel-parser</code>,
<code>babel-plugin-transform-dynamic-import</code>,
<code>babel-plugin-transform-modules-amd</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-plugin-transform-modules-systemjs</code>,
<code>babel-traverse</code>, <code>babel-types</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15682">#15682</a> Add
<code>createImportExpressions</code> parser option (<a
href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-standalone</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15671">#15671</a>
Pass through nonce to the transformed script element (<a
href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helper-function-name</code>,
<code>babel-helper-member-expression-to-functions</code>,
<code>babel-helpers</code>, <code>babel-parser</code>,
<code>babel-plugin-proposal-destructuring-private</code>,
<code>babel-plugin-proposal-optional-chaining-assign</code>,
<code>babel-plugin-syntax-optional-chaining-assign</code>,
<code>babel-plugin-transform-destructuring</code>,
<code>babel-plugin-transform-optional-chaining</code>,
<code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>,
<code>babel-runtime</code>, <code>babel-standalone</code>,
<code>babel-types</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15751">#15751</a> Add
support for optional chain in assignments (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>,
<code>babel-plugin-proposal-decorators</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15895">#15895</a>
Implement the "decorator metadata" proposal (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-traverse</code>, <code>babel-types</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15893">#15893</a> Add
<code>t.buildUndefinedNode</code> (<a
href="https://github.com/liuxingbaoyu"><code>@liuxingbaoyu</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code></li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/babel/babel/blob/main/CHANGELOG.md"><code>@babel/traverse</code>'s
changelog</a>.</em></p>
<blockquote>
<h2>v7.23.2 (2023-10-11)</h2>
<h4>🐛 Bug Fix</h4>
<ul>
<li><code>babel-traverse</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16033">#16033</a>
Only evaluate own String/Number/Math methods (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16022">#16022</a>
Rewrite <code>.tsx</code> extension when using
<code>rewriteImportExtensions</code> (<a
href="https://github.com/jimmydief"><code>@jimmydief</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16017">#16017</a>
Fix: fallback to typeof when toString is applied to incompatible object
(<a href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>,
<code>babel-runtime</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/16025">#16025</a>
Avoid override mistake in namespace imports (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h2>v7.23.0 (2023-09-25)</h2>
<h4>🚀 New Feature</h4>
<ul>
<li><code>babel-plugin-proposal-import-wasm-source</code>,
<code>babel-plugin-syntax-import-source</code>,
<code>babel-plugin-transform-dynamic-import</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15870">#15870</a>
Support transforming <code>import source</code> for wasm (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-helper-module-transforms</code>,
<code>babel-helpers</code>,
<code>babel-plugin-proposal-import-defer</code>,
<code>babel-plugin-syntax-import-defer</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>,
<code>babel-runtime</code>, <code>babel-standalone</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15878">#15878</a>
Implement <code>import defer</code> proposal transform support (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-generator</code>, <code>babel-parser</code>,
<code>babel-types</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15845">#15845</a>
Implement <code>import defer</code> parsing support (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
<li><a
href="https://redirect.github.com/babel/babel/pull/15829">#15829</a> Add
parsing support for the "source phase imports" proposal (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-generator</code>,
<code>babel-helper-module-transforms</code>, <code>babel-parser</code>,
<code>babel-plugin-transform-dynamic-import</code>,
<code>babel-plugin-transform-modules-amd</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-plugin-transform-modules-systemjs</code>,
<code>babel-traverse</code>, <code>babel-types</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15682">#15682</a> Add
<code>createImportExpressions</code> parser option (<a
href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-standalone</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15671">#15671</a>
Pass through nonce to the transformed script element (<a
href="https://github.com/JLHwung"><code>@JLHwung</code></a>)</li>
</ul>
</li>
<li><code>babel-helper-function-name</code>,
<code>babel-helper-member-expression-to-functions</code>,
<code>babel-helpers</code>, <code>babel-parser</code>,
<code>babel-plugin-proposal-destructuring-private</code>,
<code>babel-plugin-proposal-optional-chaining-assign</code>,
<code>babel-plugin-syntax-optional-chaining-assign</code>,
<code>babel-plugin-transform-destructuring</code>,
<code>babel-plugin-transform-optional-chaining</code>,
<code>babel-runtime-corejs2</code>, <code>babel-runtime-corejs3</code>,
<code>babel-runtime</code>, <code>babel-standalone</code>,
<code>babel-types</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15751">#15751</a> Add
support for optional chain in assignments (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-helpers</code>,
<code>babel-plugin-proposal-decorators</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15895">#15895</a>
Implement the "decorator metadata" proposal (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-traverse</code>, <code>babel-types</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15893">#15893</a> Add
<code>t.buildUndefinedNode</code> (<a
href="https://github.com/liuxingbaoyu"><code>@liuxingbaoyu</code></a>)</li>
</ul>
</li>
<li><code>babel-preset-typescript</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15913">#15913</a> Add
<code>rewriteImportExtensions</code> option to TS preset (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
<li><code>babel-parser</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15896">#15896</a>
Allow TS tuples to have both labeled and unlabeled elements (<a
href="https://github.com/yukukotani"><code>@yukukotani</code></a>)</li>
</ul>
</li>
</ul>
<h4>🐛 Bug Fix</h4>
<ul>
<li><code>babel-plugin-transform-block-scoping</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15962">#15962</a>
fix: <code>transform-block-scoping</code> captures the variables of the
method in the loop (<a
href="https://github.com/liuxingbaoyu"><code>@liuxingbaoyu</code></a>)</li>
</ul>
</li>
</ul>
<h4>💅 Polish</h4>
<ul>
<li><code>babel-traverse</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15797">#15797</a>
Expand evaluation of global built-ins in <code>@babel/traverse</code>
(<a
href="https://github.com/lorenzoferre"><code>@lorenzoferre</code></a>)</li>
</ul>
</li>
<li><code>babel-plugin-proposal-explicit-resource-management</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15985">#15985</a>
Improve source maps for blocks with <code>using</code> declarations (<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h4>🔬 Output optimization</h4>
<ul>
<li><code>babel-core</code>,
<code>babel-helper-module-transforms</code>,
<code>babel-plugin-transform-async-to-generator</code>,
<code>babel-plugin-transform-classes</code>,
<code>babel-plugin-transform-dynamic-import</code>,
<code>babel-plugin-transform-function-name</code>,
<code>babel-plugin-transform-modules-amd</code>,
<code>babel-plugin-transform-modules-commonjs</code>,
<code>babel-plugin-transform-modules-umd</code>,
<code>babel-plugin-transform-parameters</code>,
<code>babel-plugin-transform-react-constant-elements</code>,
<code>babel-plugin-transform-react-inline-elements</code>,
<code>babel-plugin-transform-runtime</code>,
<code>babel-plugin-transform-typescript</code>,
<code>babel-preset-env</code>
<ul>
<li><a
href="https://redirect.github.com/babel/babel/pull/15984">#15984</a>
Inline <code>exports.XXX =</code> update in simple variable declarations
(<a
href="https://github.com/nicolo-ribaudo"><code>@nicolo-ribaudo</code></a>)</li>
</ul>
</li>
</ul>
<h2>v7.22.20 (2023-09-16)</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="b4b9942a6c"><code>b4b9942</code></a>
v7.23.2</li>
<li><a
href="b13376b346"><code>b13376b</code></a>
Only evaluate own String/Number/Math methods (<a
href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/16033">#16033</a>)</li>
<li><a
href="ca58ec15cb"><code>ca58ec1</code></a>
v7.23.0</li>
<li><a
href="0f333dafcf"><code>0f333da</code></a>
Add <code>createImportExpressions</code> parser option (<a
href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/15682">#15682</a>)</li>
<li><a
href="3744545649"><code>3744545</code></a>
Fix linting</li>
<li><a
href="c7e6806e21"><code>c7e6806</code></a>
Add <code>t.buildUndefinedNode</code> (<a
href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/15893">#15893</a>)</li>
<li><a
href="38ee8b4dd6"><code>38ee8b4</code></a>
Expand evaluation of global built-ins in <code>@babel/traverse</code>
(<a
href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/15797">#15797</a>)</li>
<li><a
href="9f3dfd9021"><code>9f3dfd9</code></a>
v7.22.20</li>
<li><a
href="3ed28b29c1"><code>3ed28b2</code></a>
Fully support <code>||</code> and <code>&&</code> in
<code>pluginToggleBooleanFlag</code> (<a
href="https://github.com/babel/babel/tree/HEAD/packages/babel-traverse/issues/15961">#15961</a>)</li>
<li><a
href="77b0d73599"><code>77b0d73</code></a>
v7.22.19</li>
<li>Additional commits viewable in <a
href="https://github.com/babel/babel/commits/v7.23.2/packages/babel-traverse">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/langchain-ai/langchain/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Compare predicted json to reference. First canonicalize (sort keys, rm
whitespace separators), then return normalized string edit distance.
Not a silver bullet but maybe an easy way to capture structure
differences in a less flakey way
---------
Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
**Description**
This small change will make chunk_size a configurable parameter for
loading documents into a Supabase database.
**Issue**
https://github.com/langchain-ai/langchain/issues/11422
**Dependencies**
No chanages
**Twitter**
@ j1philli
**Reminder**
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
---------
Co-authored-by: Greg Richardson <greg.nmr@gmail.com>
This PR replaces the previous `Intent` check with the new `Prompt
Safety` check. The logic and steps to enable chain moderation via the
Amazon Comprehend service, allowing you to detect and redact PII, Toxic,
and Prompt Safety information in the LLM prompt or answer remains
unchanged.
This implementation updates the code and configuration types with
respect to `Prompt Safety`.
### Usage sample
```python
from langchain_experimental.comprehend_moderation import (BaseModerationConfig,
ModerationPromptSafetyConfig,
ModerationPiiConfig,
ModerationToxicityConfig
)
pii_config = ModerationPiiConfig(
labels=["SSN"],
redact=True,
mask_character="X"
)
toxicity_config = ModerationToxicityConfig(
threshold=0.5
)
prompt_safety_config = ModerationPromptSafetyConfig(
threshold=0.5
)
moderation_config = BaseModerationConfig(
filters=[pii_config, toxicity_config, prompt_safety_config]
)
comp_moderation_with_config = AmazonComprehendModerationChain(
moderation_config=moderation_config, #specify the configuration
client=comprehend_client, #optionally pass the Boto3 Client
verbose=True
)
template = """Question: {question}
Answer:"""
prompt = PromptTemplate(template=template, input_variables=["question"])
responses = [
"Final Answer: A credit card number looks like 1289-2321-1123-2387. A fake SSN number looks like 323-22-9980. John Doe's phone number is (999)253-9876.",
"Final Answer: This is a really shitty way of constructing a birdhouse. This is fucking insane to think that any birds would actually create their motherfucking nests here."
]
llm = FakeListLLM(responses=responses)
llm_chain = LLMChain(prompt=prompt, llm=llm)
chain = (
prompt
| comp_moderation_with_config
| {llm_chain.input_keys[0]: lambda x: x['output'] }
| llm_chain
| { "input": lambda x: x['text'] }
| comp_moderation_with_config
)
try:
response = chain.invoke({"question": "A sample SSN number looks like this 123-456-7890. Can you give me some more samples?"})
except Exception as e:
print(str(e))
else:
print(response['output'])
```
### Output
```python
> Entering new AmazonComprehendModerationChain chain...
Running AmazonComprehendModerationChain...
Running pii Validation...
Running toxicity Validation...
Running prompt safety Validation...
> Finished chain.
> Entering new AmazonComprehendModerationChain chain...
Running AmazonComprehendModerationChain...
Running pii Validation...
Running toxicity Validation...
Running prompt safety Validation...
> Finished chain.
Final Answer: A credit card number looks like 1289-2321-1123-2387. A fake SSN number looks like XXXXXXXXXXXX John Doe's phone number is (999)253-9876.
```
---------
Co-authored-by: Jha <nikjha@amazon.com>
Co-authored-by: Anjan Biswas <anjanavb@amazon.com>
Co-authored-by: Anjan Biswas <84933469+anjanvb@users.noreply.github.com>
**Description:**
This PR adds support for the [Pro version of Titan Takeoff
Server](https://docs.titanml.co/docs/category/pro-features). Users of
the Pro version will have to import the TitanTakeoffPro model, which is
different from TitanTakeoff.
**Issue:**
Also minor fixes to docs for Titan Takeoff (Community version)
**Dependencies:**
No additional dependencies
**Twitter handle:** @becoming_blake
@baskaryan @hwchase17
- **Description:** Super simple fix for colab link on
code_understanding.ipynb,
- **Issue:** not applicable
- **Dependencies:** none,
- **Tag maintainer:** ,
- **Twitter handle:** @kengoodridge
Problem statement:
In the `integrations/llms` and `integrations/chat` pages, we have a
sidebar with ToC, and we also have a ToC at the end of the page.
The ToC at the end of the page is not necessary, and it is confusing
when we mix the index page styles; moreover, it requires manual work.
So, I removed ToC at the end of the page (it was discussed with and
approved by @baskaryan)
This PR adds a data [E2B's](https://e2b.dev/) analysis/code interpreter
sandbox as a tool
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Jakub Novak <jakub@e2b.dev>
- Move Document AI provider to the Google provider page
- Change Vertex AI Matching Engine to Vector Search
- Change references from GCP to Google Cloud
- Add Gmail chat loader to Google provider page
- Change Serper page title to "Serper - Google Search API" since it is
not a Google product.
- replace `requests` package with `langchain.requests`
- add `_acall` support
- add `_stream` and `_astream`
- freshen up the documentation a bit
- update vendor doc
**Description: Allow to inject boto3 client for Cross account access
type of scenarios in using SagemakerEndpointEmbeddings and also updated
the documentation for same in the sample notebook**
**Issue:SagemakerEndpointEmbeddings cross account capability #10634
#10184**
Dependencies: None
Tag maintainer:
Twitter handle:lethargicoder
Co-authored-by: Vikram(VS) <vssht@amazon.com>
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
Adding Tavily Search API as a tool. I will be the maintainer and
assaf_elovic is the twitter handler.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
- **Description:**
- Replace Telegram with Whatsapp in whatsapp.ipynb
- Add # to mark the telegram as heading in telegram.ipynb
- **Issue:** None
- **Dependencies:** None
- **Description:** Implementing the Google Scholar Tool as requested in
PR #11505. The tool will be using the [serpapi python
package](https://serpapi.com/integrations/python#search-google-scholar).
The main idea of the tool will be to return the results from a Google
Scholar search given a query as an input to the tool.
- **Tag maintainer:** @baskaryan, @eyurtsev, @hwchase17
Replace this entire comment with:
- **Description:** Fix superfluous [Auto-fixing
parser](https://python.langchain.com/docs/modules/model_io/output_parsers/output_fixing_parser)
docs. Also switching to `langchain.pydantic_v1` from the direct
reference to `pydantic`,
- **Issue:** N/A,
- **Dependencies:** N/A,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** @dosuken123
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
Added a notebook with examples of the creation of a retriever from the
SingleStoreDB vector store, and further usage.
Co-authored-by: Volodymyr Tkachuk <vtkachuk-ua@singlestore.com>
Updated the elasticsearch self query retriever to use the match clause
for LIKE operator instead of the non-analyzed fuzzy search clause.
Other small updates include:
- fixing the stack inference integration test where the index's default
pipeline didn't use the inference pipeline created
- adding a user-agent to the old implementation to track usage
- improved the documentation for ElasticsearchStore filters
### Description:
To provide an eas llm service access methods in this pull request by
impletementing `PaiEasEndpoint` and `PaiEasChatEndpoint` classes in
`langchain.llms` and `langchain.chat_models` modules. Base on this pr,
langchain users can build up a chain to call remote eas llm service and
get the llm inference results.
### About EAS Service
EAS is a Alicloud product on Alibaba Cloud Machine Learning Platform for
AI which is short for AliCloud PAI. EAS provides model inference
deployment services for the users. We build up a llm inference services
on EAS with a general llm docker images. Therefore, end users can
quickly setup their llm remote instances to load majority of the
hugginface llm models, and serve as a backend for most of the llm apps.
### Dependencies
This pr does't involve any new dependencies.
---------
Co-authored-by: 子洪 <gaoyihong.gyh@alibaba-inc.com>
The Docs folder changed its structure, and the notebook example for
SingleStoreDChatMessageHistory has not been copied to the new place due
to a merge conflict. Adding the example to the correct place.
Co-authored-by: Volodymyr Tkachuk <vtkachuk-ua@singlestore.com>
- Update Zep Memory and Retriever docstrings
- Zep Memory Retriever: Add support for native MMR
- Add MMR example to existing ZepRetriever Notebook
@baskaryan
- Description: Considering the similarity computation method of
[BGE](https://github.com/FlagOpen/FlagEmbedding) model is cosine
similarity, set normalize_embeddings to be True.
- Tag maintainer: @baskaryan
Co-authored-by: Erick Friis <erick@langchain.dev>
Description: A large language models developed by Baichuan Intelligent
Technology，https://www.baichuan-ai.com/home
Issue: None
Dependencies: None
Tag maintainer:
Twitter handle:
The current ToC on the index page and on navbar don't match. Page titles
and Titles in ToC doesn't match
Changes:
- made ToCs equal
- made titles equal
- updated some page formattings.
**Description**
- Added the `SingleStoreDBChatMessageHistory` class that inherits
`BaseChatMessageHistory` and allows to use of a SingleStoreDB database
as a storage for chat message history.
- Added integration test to check that everything works (requires
`singlestoredb` to be installed)
- Added notebook with usage example
- Removed custom retriever for SingleStoreDB vector store (as it is
useless)
---------
Co-authored-by: Volodymyr Tkachuk <vtkachuk-ua@singlestore.com>
Fixed a typo :
"asyncrhonized" > "asynchronized"
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
Hello Folks,
Alibaba Cloud OpenSearch has released a new version of the vector
storage engine, which has significantly improved performance compared to
the previous version. At the same time, the sdk has also undergone
changes, requiring adjustments alibaba opensearch vector store code to
adapt.
This PR includes:
Adapt to the latest version of Alibaba Cloud OpenSearch API.
More comprehensive unit testing.
Improve documentation.
I have read your contributing guidelines. And I have passed the tests
below
- [x] make format
- [x] make lint
- [x] make coverage
- [x] make test
---------
Co-authored-by: zhaoshengbo <shengbo.zsb@alibaba-inc.com>
**Description:**
While working on the Docusaurus site loader #9138, I noticed some
outdated docs and tests for the Sitemap Loader.
**Issue:**
This is tangentially related to #6691 in reference to doc links. I plan
on digging in to a few of these issue when I find time next.
Related to #10800
- Errors in the Docstring of GradientLLM / Gradient.ai LLM
- Renamed the `model_id` to `model` and adapting this in all tests.
Reason to so is to be in Sync with `GradientEmbeddings` and other LLM's.
- inmproving tests so they check the headers in the sent request.
- making the aiosession a private attribute in the docs, as in the
future `pip install gradientai` will be replacing aiosession.
- adding a example how to fine-tune on the Prompt Template as suggested
in #10800
Hi,
After submitting https://github.com/langchain-ai/langchain/pull/11357,
we realized that the notebooks are moved to a new location. Sending a
new PR to update the doc.
---------
Co-authored-by: everly-studio <127131037+everly-studio@users.noreply.github.com>
- Description: Adds the ChatEverlyAI class with llama-2 7b on [EverlyAI
Hosted
Endpoints](https://everlyai.xyz/)
- It inherits from ChatOpenAI and requires openai (probably unnecessary
but it made for a quick and easy implementation)
---------
Co-authored-by: everly-studio <127131037+everly-studio@users.noreply.github.com>
Reverts langchain-ai/langchain#11714
This has linting and formatting issues, plus it's added to chat models
folder but doesn't subclass Chat Model base class
Motivation and Context
At present, the Baichuan Large Language Model is relatively popular and
efficient in performance. Due to widespread market recognition, this
model has been added to enhance the scalability of Langchain's ability
to access the big language model, so as to facilitate application access
and usage for interested users.
System Info
langchain： 0.0.295
python：3.8.3
IDE：vs code
Description
Add the following files:
1. Add baichuan_baichuaninc_endpoint.py in the
libs/langchain/langchain/chat_models
2. Modify the __init__.py file,which is located in the
libs/langchain/langchain/chat_models/__init__.py：
a. Add "from langchain.chat_models.baichuan_baichuaninc_endpoint import
BaichuanChatEndpoint"
b. Add "BaichuanChatEndpoint" In the file's __ All__ method
Your contribution
I am willing to help implement this feature and submit a PR, but I would
appreciate guidance from the maintainers or community to ensure the
changes are made correctly and in line with the project's standards and
practices.
Hi there
This PR is aim to implement chat model for Alibaba Tongyi LLM model. It
contains work below:
1.Implement ChatTongyi chat model in langchain.chat_models.tongyi. Note
this is different with tongyi llm model to another PR
https://github.com/langchain-ai/langchain/pull/10878.
For detail it implements _generate() and _stream() function in
ChatTongyi.
2. Add some examples in chat/tongyi.ipynb.
3. Add integration test in chat_models/test_tongyi.py
Note async completion for the Text API is not yet supported.
Dependencies: dashscope. It will be installed manually cause it is not
need by everyone.
**Description**
This PR adds the `ElasticsearchChatMessageHistory` implementation that
stores chat message history in the configured
[Elasticsearch](https://www.elastic.co/elasticsearch/) deployment.
```python
from langchain.memory.chat_message_histories import ElasticsearchChatMessageHistory
history = ElasticsearchChatMessageHistory(
es_url="https://my-elasticsearch-deployment-url:9200", index="chat-history-index", session_id="123"
)
history.add_ai_message("This is me, the AI")
history.add_user_message("This is me, the human")
```
**Dependencies**
- [elasticsearch client](https://elasticsearch-py.readthedocs.io/)
required
Co-authored-by: Bagatur <baskaryan@gmail.com>
Instead of accessing `langchain.debug`, `langchain.verbose`, or
`langchain.llm_cache`, please use the new getter/setter functions in
`langchain.globals`:
- `langchain.globals.set_debug()` and `langchain.globals.get_debug()`
- `langchain.globals.set_verbose()` and
`langchain.globals.get_verbose()`
- `langchain.globals.set_llm_cache()` and
`langchain.globals.get_llm_cache()`
Using the old globals directly will now raise a warning.
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
**Description:**
Add a document loader for the RSpace Electronic Lab Notebook
(www.researchspace.com), so that scientific documents and research notes
can be easily pulled into Langchain pipelines.
**Issue**
This is an new contribution, rather than an issue fix.
**Dependencies:**
There are no new required dependencies.
In order to use the loader, clients will need to install rspace_client
SDK using `pip install rspace_client`
---------
Co-authored-by: richarda23 <richard.c.adams@infinityworks.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
**Description:** Update Indexing API docs to specify vectorstores that
are compatible with the Indexing API. I add a unit test to remind
developers to update the documentation whenever they add or change a
vectorstore in a way that affects compatibility. For the unit test I
repurposed existing code from
[here](https://github.com/langchain-ai/langchain/blob/v0.0.311/libs/langchain/langchain/indexes/_api.py#L245-L257).
This is my first PR to an open source project. This is a trivially
simple PR whose main purpose is to make me more comfortable submitting
Langchain PRs. If this PR goes through I plan to submit PRs with more
substantive changes in the near future.
**Issue:** Resolves
[10482](https://github.com/langchain-ai/langchain/discussions/10482).
**Dependencies:** No new dependencies.
**Twitter handle:** None.
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
**Description:** Modify Anyscale integration to work with [Anyscale
Endpoint](https://docs.endpoints.anyscale.com/)
and it supports invoke, async invoke, stream and async invoke features
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
- **Description:** implements a retriever on top of DocAI Warehouse (to
interact with existing enterprise documents)
https://cloud.google.com/document-ai-warehouse?hl=en
- **Issue:** new functionality
@baskaryan
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Fix the documentation in
https://python.langchain.com/docs/modules/model_io/prompts/example_selectors/ngram_overlap.
It's currently declaring unrelated variables, for example, `examples`
local variable is declared twice and the first one is overwritten
immediately.
- **Issue:** N/A
- **Dependencies:** N/A
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** @dosuken123
Added demo for QA system with anonymization. It will be part of
LangChain's privacy webinar.
@hwchase17 @baskaryan @nfcampos
Twitter handle: @MaksOpp
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
**Description:** This PR adds support for ChatOpenAI models in the
Infino callback handler. In particular, this PR implements
`on_chat_model_start` callback, so that ChatOpenAI models are supported.
With this change, Infino callback handler can be used to track latency,
errors, and prompt tokens for ChatOpenAI models too (in addition to the
support for OpenAI and other non-chat models it has today). The existing
example notebook is updated to show how to use this integration as well.
cc/ @naman-modi @savannahar68
**Issue:** https://github.com/langchain-ai/langchain/issues/11607
**Dependencies:** None
**Tag maintainer:** @hwchase17
**Twitter handle:** [@vkakade](https://twitter.com/vkakade)
This PR adds support for the Azure Cosmos DB MongoDB vCore Vector Store
https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/https://learn.microsoft.com/en-us/azure/cosmos-db/mongodb/vcore/vector-search
Summary:
- **Description:** added vector store integration for Azure Cosmos DB
MongoDB vCore Vector Store,
- **Issue:** the issue # it fixes#11627,
- **Dependencies:** pymongo dependency,
- **Tag maintainer:** @hwchase17,
- **Twitter handle:** @izzyacademy
---------
Co-authored-by: Israel Ekpo <israel.ekpo@gmail.com>
Co-authored-by: Israel Ekpo <44282278+izzyacademy@users.noreply.github.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
- **Description:** This is an update to OctoAI LLM provider that adds
support for llama2 endpoints hosted on OctoAI and updates MPT-7b url
with the current one.
@baskaryan
Thanks!
---------
Co-authored-by: ML Wiz <bassemgeorgi@gmail.com>
There is some invalid link in open ai platform
[docs](https://python.langchain.com/docs/integrations/platforms/openai).
So i fixed it to valid links.
- `/docs/integrations/chat_models/openai` ->
`/docs/integrations/chat/openai`
- `/docs/integrations/chat_models/azure_openai` ->
`/docs/integrations/chat/azure_chat_openai`
Thanks! ☺️
- **Description:** This PR introduces a new LLM and Retriever API to
https://arcee.ai for the python client
- **Issue:** implements the integrations as requested in #11578 ,
- **Dependencies:** no dependencies are required,
- **Tag maintainer:** @hwchase17
- **Twitter handle:** shwooobham
**✅ `make format`, `make lint` and `make test` runs locally.**
```shell
=========== 1245 passed, 277 skipped, 20 warnings in 16.26s ===========
./scripts/check_pydantic.sh .
./scripts/check_imports.sh
poetry run ruff .
[ "." = "" ] || poetry run black . --check
All done! ✨🍰✨
1818 files would be left unchanged.
[ "." = "" ] || poetry run mypy .
Success: no issues found in 1815 source files
[ "." = "" ] || poetry run black .
All done! ✨🍰✨
1818 files left unchanged.
[ "." = "" ] || poetry run ruff --select I --fix .
poetry run codespell --toml pyproject.toml
poetry run codespell --toml pyproject.toml -w
```
**Contributions**
1. Arcee (langchain/llms), ArceeRetriever (langchain/retrievers),
ArceeWrapper (langchain/utilities)
2. docs for Arcee (llms/arcee.py) and
ArceeRetriever(retrievers/arcee.py)
3.
cc: @jacobsolawetz @ben-epstein
---------
Co-authored-by: Shubham <shubham@sORo.local>
**Description**:
- Added Momento Vector Index (MVI) as a vector store provider. This
includes an implementation with docstrings, integration tests, a
notebook, and documentation on the docs pages.
- Updated the Momento dependency in pyproject.toml and the lock file to
enable access to MVI.
- Refactored the Momento cache and chat history session store to prefer
using "MOMENTO_API_KEY" over "MOMENTO_AUTH_TOKEN" for consistency with
MVI. This change is backwards compatible with the previous "auth_token"
variable usage. Updated the code and tests accordingly.
**Dependencies**:
- Updated Momento dependency in pyproject.toml.
**Testing**:
- Run the integration tests with a Momento API key. Get one at the
[Momento Console](https://console.gomomento.com) for free. MVI is
available in AWS us-west-2 with a superuser key.
- `MOMENTO_API_KEY=<your key> poetry run pytest
tests/integration_tests/vectorstores/test_momento_vector_index.py`
**Tag maintainer:**
@eyurtsev
**Twitter handle**:
Please mention @momentohq for this addition to langchain. With the
integration of Momento Vector Index, Momento caching, and session store,
Momento provides serverless support for the core langchain data needs.
Also mention @mlonml for the integration.
**Description**
This PR adds an additional Example to the Redis integration
documentation. [The
example](https://learn.microsoft.com/azure/azure-cache-for-redis/cache-tutorial-vector-similarity)
is a step-by-step walkthrough of using Azure Cache for Redis and Azure
OpenAI for vector similarity search, using LangChain extensively
throughout.
**Issue**
Nothing specific, just adding an additional example.
**Dependencies**
None.
**Tag Maintainer**
Tagging @hwchase17 :)
- keep alias for RunnableMap
- update docs to use RunnableParallel and RunnablePassthrough.assign
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
**Description**
It is for #10423 that it will be a useful feature if we can extract
images from pdf and recognize text on them. I have implemented it with
`PyPDFLoader`, `PyPDFium2Loader`, `PyPDFDirectoryLoader`,
`PyMuPDFLoader`, `PDFMinerLoader`, and `PDFPlumberLoader`.
[RapidOCR](https://github.com/RapidAI/RapidOCR.git) is used to recognize
text on extracted images. It is time-consuming for ocr so a boolen
parameter `extract_images` is set to control whether to extract and
recognize. I have tested the time usage for each parser on my own laptop
thinkbook 14+ with AMD R7-6800H by unit test and the result is:
| extract_images | PyPDFParser | PDFMinerParser | PyMuPDFParser |
PyPDFium2Parser | PDFPlumberParser |
| ------------- | ------------- | ------------- | ------------- |
------------- | ------------- |
| False | 0.27s | 0.39s | 0.06s | 0.08s | 1.01s |
| True | 17.01s | 20.67s | 20.32s | 19,75s | 20.55s |
**Issue**
#10423
**Dependencies**
rapidocr_onnxruntime in
[RapidOCR](https://github.com/RapidAI/RapidOCR/tree/main)
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
- **Description:** This commit corrects a minor typo in the
documentation. It changes "frum" to "from" in the sentence: "The results
from search are passed back to the LLM for synthesis into an answer" in
the file `docs/extras/use_cases/more/agents/agents.ipynb`. This typo fix
enhances the clarity and accuracy of the documentation.
- **Tag maintainer:** @baskaryan
- **Description:** Just docs related to csharp code splitter
- **Issue:** It's related to a request made by @baskaryan in a comment
on my previous PR #10350
- **Dependencies:** None
- **Twitter handle:** @ather19
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
### Description
Add instance anonymization - if `John Doe` will appear twice in the
text, it will be treated as the same entity.
The difference between `PresidioAnonymizer` and
`PresidioReversibleAnonymizer` is that only the second one has a
built-in memory, so it will remember anonymization mapping for multiple
texts:
```
>>> anonymizer = PresidioAnonymizer()
>>> anonymizer.anonymize("My name is John Doe. Hi John Doe!")
'My name is Noah Rhodes. Hi Noah Rhodes!'
>>> anonymizer.anonymize("My name is John Doe. Hi John Doe!")
'My name is Brett Russell. Hi Brett Russell!'
```
```
>>> anonymizer = PresidioReversibleAnonymizer()
>>> anonymizer.anonymize("My name is John Doe. Hi John Doe!")
'My name is Noah Rhodes. Hi Noah Rhodes!'
>>> anonymizer.anonymize("My name is John Doe. Hi John Doe!")
'My name is Noah Rhodes. Hi Noah Rhodes!'
```
### Twitter handle
@deepsense_ai / @MaksOpp
### Tag maintainer
@baskaryan @hwchase17 @hinthornw
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
- Description: Google Cloud Enterprise Search was renamed to Vertex AI
Search
-
https://cloud.google.com/blog/products/ai-machine-learning/vertex-ai-search-and-conversation-is-now-generally-available
- This PR updates the documentation and Retriever class to use the new
terminology.
- Changed retriever class from `GoogleCloudEnterpriseSearchRetriever` to
`GoogleVertexAISearchRetriever`
- Updated documentation to specify that `extractive_segments` requires
the new [Enterprise
edition](https://cloud.google.com/generative-ai-app-builder/docs/about-advanced-features#enterprise-features)
to be enabled.
- Fixed spelling errors in documentation.
- Change parameter for Retriever from `search_engine_id` to
`data_store_id`
- When this retriever was originally implemented, there was no
distinction between a data store and search engine, but now these have
been split.
- Fixed an issue blocking some users where the api_endpoint can't be set
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
There are several pages in `integrations/providers/more` that belongs to
Google and AWS `integrations/providers`.
- moved content of these pages into the Google and AWS
`integrations/providers` pages
- removed these individual pages
- **Description:** add a paragraph to the GoogleDriveLoader doc on how
to bypass errors on authentication.
For some reason, specifying credential path via `credentials_path`
constructor parameter when creating `GoogleDriveLoader` makes it so that
the oAuth screen is never showing up when first using GoogleDriveLoader.
Instead, the `RefreshError: ('invalid_grant: Bad Request', {'error':
'invalid_grant', 'error_description': 'Bad Request'})` error happens.
Setting it via `os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = ...`
solves the problem. Also, `token_path` constructor parameter is
mandatory, otherwise another error happens when trying to `load()` for
the first time.
These errors are tricky and time-consuming to figure out, so I believe
it's good to mention them in the docs.
---------
Co-authored-by: Erick Friis <erick@langchain.dev>
Description: Similar in concept to the `MarkdownHeaderTextSplitter`, the
`HTMLHeaderTextSplitter` is a "structure-aware" chunker that splits text
at the element level and adds metadata for each header "relevant" to any
given chunk. It can return chunks element by element or combine elements
with the same metadata, with the objectives of (a) keeping related text
grouped (more or less) semantically and (b) preserving context-rich
information encoded in document structures. It can be used with other
text splitters as part of a chunking pipeline.
Dependency: lxml python package
Maintainer: @hwchase17
Twitter handle: @MartinZirulnik
---------
Co-authored-by: PresidioVantage <github@presidiovantage.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
- **Description:** Doc corrections and resolve notebook rendering issue
on GH
- **Issue:** N/A
- **Dependencies:** N/A
- **Tag maintainer:** @baskaryan
- **Twitter handle:** `@isaacchung1217`
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
**Description:**
Examples in the "Select by similarity" section were not really
highlighting capabilities of similarity search.
E.g. "# Input is a measurement, so should select the tall/short example"
was still outputting the "mood" example.
I tweaked the inputs a bit and fixed the examples (checking that those
are indeed what the search outputs).
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
- **Description:** Fix typo about `RetrievalQAWithSourceChain` ->
`RetrievalQAWithSourcesChain`
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
- **Description:** Adds Kotlin language to `TextSplitter`
---------
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
- **Description:** use term keyword according to the official python doc
glossary, see https://docs.python.org/3/glossary.html
- **Issue:** not applicable
- **Dependencies:** not applicable
- **Tag maintainer:** @hwchase17
- **Twitter handle:** vreyespue
continuation of PR #8550
@hwchase17 please see and merge. And also close the PR #8550.
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
Co-authored-by: Erick Friis <erick@langchain.dev>
therefor -> therefore
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/langchain-ai/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
### Description
When I was reading the document, I found that some examples had extra
spaces and violated "Unexpected spaces around keyword / parameter equals
(E251)" in pep8. I removed these extra spaces.
### Tag maintainer
@eyurtsev
### Twitter handle
[billvsme](https://twitter.com/billvsme)
### Description
renamed several repository links from `hwchase17` to `langchain-ai`.
### Why
I discovered that the README file in the devcontainer contains an old
repository name, so I took the opportunity to rename the old repository
name in all files within the repository, excluding those that do not
require changes.
### Dependencies
none
### Tag maintainer
@baskaryan
### Twitter handle
[kzk_maeda](https://twitter.com/kzk_maeda)
updated `YouTube` and `tutorial` videos with new links.
Removed couple of duplicates.
Reordered several links by view counters
Some formatting: emphasized the names of products
- updated titles and descriptions of the `integrations/memory` notebooks
into consistent and laconic format;
- removed
`docs/extras/integrations/memory/motorhead_memory_managed.ipynb` file as
a duplicate of the
`docs/extras/integrations/memory/motorhead_memory.ipynb`;
- added `integrations/providers` Integration Cards for `dynamodb`,
`motorhead`.
- updated `integrations/providers/redis.mdx` with links
- renamed several notebooks; updated `vercel.json` to reroute new names.
Enviroment -> Environment
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
- **Description:** A Document Loader for MongoDB
- **Issue:** n/a
- **Dependencies:** Motor, the async driver for MongoDB
- **Tag maintainer:** n/a
- **Twitter handle:** pigpenblue
Note that an initial mongodb document loader was created 4 months ago,
but the [PR ](https://github.com/langchain-ai/langchain/pull/4285)was
never pulled in. @leo-gan had commented on that PR, but given it is
extremely far behind the master branch and a ton has changed in
Langchain since then (including repo name and structure), I rewrote the
branch and issued a new PR with the expectation that the old one can be
closed.
Please reference that old PR for comments/context, but it can be closed
in favor of this one. Thanks!
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
Based on the customers' requests for native langchain integration,
SearchApi is ready to invest in AI and LLM space, especially in
open-source development.
- This is our initial PR and later we want to improve it based on
customers' and langchain users' feedback. Most likely changes will
affect how the final results string is being built.
- We are creating similar native integration in Python and JavaScript.
- The next plan is to integrate into Java, Ruby, Go, and others.
- Feel free to assign @SebastjanPrachovskij as a main reviewer for any
SearchApi-related searches. We will be glad to help and support
langchain development.
## Description
Expanded the upper bound for `networkx` dependency to allow installation
of latest stable version. Tested the included sample notebook with
version 3.1, and all steps ran successfully.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
- **Description:** Bedrock updated boto service name to
"bedrock-runtime" for the InvokeModel and InvokeModelWithResponseStream
APIs. This update also includes new model identifiers for Titan text,
embedding and Anthropic.
Co-authored-by: Mani Kumar Adari <maniadar@amazon.com>
Fixed Typo Error in Update get_started.mdx file by addressing a minor
typographical error.
This improvement enhances the readability and correctness of the
notebook, making it easier for users to understand and follow the
demonstration. The commit aims to maintain the quality and accuracy of
the content within the repository.
please review the change at your convenience.
@baskaryan , @hwaking
The new Fireworks and FireworksChat implementations are awesome! Added
in this PR https://github.com/langchain-ai/langchain/pull/11117 thank
you @ZixinYang
However, I think stop words were not plumbed correctly. I've made some
simple changes to do that, and also updated the notebook to be a bit
clearer with what's needed to use both new models.
---------
Co-authored-by: Taqi Jaffri <tjaffri@docugami.com>
