Commit Graph

620 Commits

Author SHA1 Message Date
Bagatur
3efab8d3df
implement vectorstores by tencent vectordb (#9989)
Hi there!
I'm excited to open this PR to add support for using 'Tencent Cloud
VectorDB' as a vector store.

Tencent Cloud VectorDB is a fully-managed, self-developed,
enterprise-level distributed database service designed for storing,
retrieving, and analyzing multi-dimensional vector data. The database
supports multiple index types and similarity calculation methods, with a
single index supporting vector scales up to 1 billion and capable of
handling millions of QPS with millisecond-level query latency. Tencent
Cloud VectorDB not only provides external knowledge bases for large
models to improve their accuracy, but also has wide applications in AI
fields such as recommendation systems, NLP services, computer vision,
and intelligent customer service.

The PR includes:
 Implementation of Vectorstore.

I have read your [contributing
guidelines](72b7d76d79/.github/CONTRIBUTING.md).
And I have passed the tests below

 make format
 make lint
 make coverage
 make test
2023-08-31 00:48:25 -07:00
Bagatur
b1644bc9ad cr 2023-08-31 00:43:34 -07:00
Cameron Vetter
e37d51cab6
fix scoring profile example (#10016)
- Description: A change in the documentation example for Azure Cognitive
Vector Search with Scoring Profile so the example works as written
  - Issue: #10015 
  - Dependencies: None
  - Tag maintainer: @baskaryan @ruoccofabrizio
  - Twitter handle: @poshporcupine
2023-08-31 00:35:06 -07:00
Hyeokjun seo
e2e05ad89e
Fix Typo : openai_api_key -> serpapi_api_key (#10020)
Fixed typo in the comments Notebook. (which says `openai_api_key` for
SerpAPI)
2023-08-31 00:33:13 -07:00
Tomaz Bratanic
f2e8399cc8
Fix link in Neo4j provider page (#10023) 2023-08-31 00:32:42 -07:00
Bagatur
7fa82900cb
guides docs nits (#10005) 2023-08-30 11:07:42 -07:00
Bagatur
2f03e71e67
rename local llm guide (#10004) 2023-08-30 10:52:46 -07:00
Bagatur
781f274d19
make privacy guide section (#10003) 2023-08-30 10:49:20 -07:00
maks-operlejn-ds
a8f804a618
Add data anonymizer (#9863)
### Description

The feature for anonymizing data has been implemented. In order to
protect private data, such as when querying external APIs (OpenAI), it
is worth pseudonymizing sensitive data to maintain full privacy.

Anonynization consists of two steps:

1. **Identification:** Identify all data fields that contain personally
identifiable information (PII).
2. **Replacement**: Replace all PIIs with pseudo values or codes that do
not reveal any personal information about the individual but can be used
for reference. We're not using regular encryption, because the language
model won't be able to understand the meaning or context of the
encrypted data.

We use *Microsoft Presidio* together with *Faker* framework for
anonymization purposes because of the wide range of functionalities they
provide. The full implementation is available in `PresidioAnonymizer`.

### Future works

- **deanonymization** - add the ability to reverse anonymization. For
example, the workflow could look like this: `anonymize -> LLMChain ->
deanonymize`. By doing this, we will retain anonymity in requests to,
for example, OpenAI, and then be able restore the original data.
- **instance anonymization** - at this point, each occurrence of PII is
treated as a separate entity and separately anonymized. Therefore, two
occurrences of the name John Doe in the text will be changed to two
different names. It is therefore worth introducing support for full
instance detection, so that repeated occurrences are treated as a single
object.

### Twitter handle
@deepsense_ai / @MaksOpp

---------

Co-authored-by: MaksOpp <maks.operlejn@gmail.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-30 10:39:44 -07:00
Bagatur
98cce7dcd3
update moderation docs (#10002) 2023-08-30 10:34:25 -07:00
Christophe Bornet
9870bfb9cd
Add bucket and object key to metadata in S3 loader (#9317)
- Description: this PR adds `s3_object_key` and `s3_bucket` to the doc
metadata when loading an S3 file. This is particularly useful when using
`S3DirectoryLoader` to remove the files from the dir once they have been
processed (getting the object keys from the metadata `source` field
seems brittle)
  - Dependencies: N/A
  - Tag maintainer: ?
  - Twitter handle: _cbornet

---------

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-30 11:03:24 -04:00
Guy Korland
24c0b01c38
Extend the FalkorDB QA demo (#9992)
- Description: Extend the FalkorDB QA demo
  - Tag maintainer: @baskaryan
2023-08-30 10:13:18 -04:00
wlleiiwang
8c4e29240c implement vectorstores by tencent vectordb 2023-08-30 16:40:58 +08:00
Leonid Ganeline
d03d6f6fd9
Merge branch 'master' into docs-tools-menu 2023-08-29 15:57:25 -07:00
Bagatur
8fb0a9594c
Add LLMonitor Callback Handler Integration - open-source observability & analytics (#9870)
Adds support for [llmonitor](https://llmonitor.com) callbacks.

It enables:
- Requests tracking / logging / analytics
- Error debugging
- Cost analytics
- User tracking

Let me know if anythings neds to be changed for merge.

Thank you!
2023-08-29 15:49:01 -07:00
leo-gan
8c1678a8c7 Updated titles, descriptions. 2023-08-29 15:42:28 -07:00
Bagatur
7bba1d911b
Fix typo in code_understanding.ipynb (#9899)
seperate -> separate
2023-08-29 15:21:32 -07:00
Bagatur
2e65434568
docs: Fix the syntax error, replace "dotenv.load_env()" with "dotenv.… (#9900)
Description: The documents incorrectly mentions "dotenv.load_env()", but
it should actually be "dotenv.load_dotenv()". You can see the screenshot
below for reference:

python-dotenv: 1.0.0


![image](https://github.com/langchain-ai/langchain/assets/2959046/94dc4b51-cc2f-412d-92e9-16b8ff0d513e)
2023-08-29 15:20:24 -07:00
Bagatur
b416f5c0c8
fix a link name format to the dependents document (#9928) 2023-08-29 15:20:06 -07:00
Bagatur
8f199239b8
docs: llms/google vertex AI example update (#9960)
Updated title, description, added sections.
2023-08-29 15:07:18 -07:00
Bagatur
2a03a0087d
docs: memory menu (#9947)
The [Memory](https://python.langchain.com/docs/modules/memory/) menu is
clogged with unnecessary wording.
I've made it more concise by simplifying titles of the example
notebooks.
As results, menu is shorter and better for comprehend.
2023-08-29 15:06:11 -07:00
Bagatur
f7cc125cac
docs: memory types menu (#9949)
The [Memory
Types](https://python.langchain.com/docs/modules/memory/types/) menu is
clogged with unnecessary wording.
I've made it more concise by simplifying titles of the example
notebooks.
As results, menu is shorter and better for comprehend.
2023-08-29 15:05:23 -07:00
Bagatur
16eb935469
Fix for similarity_search_with_score (#9903)
- Description: the implementation for similarity_search_with_score did
not actually include a score or logic to filter. Now fixed.
- Tag maintainer: @rlancemartin
- Twitter handle: @ofermend
2023-08-29 15:04:48 -07:00
Fredrik Gullberg
f69d236a4a
docs: Fix spelling mistakes in apis.ipynb (#9911)
- Description: Fix spelling mistakes in apis.ipynb
- Issue: [#9910](https://github.com/langchain-ai/langchain/issues/9910)

Co-authored-by: Fredrik Gullberg <fredrik.gullberg@klarna.com>
2023-08-29 14:53:00 -07:00
leo-gan
210de0c66b Updated title, description, added sections 2023-08-29 14:31:33 -07:00
Cameron Hutchison
bcc3463ff4
docs: Azure AD Authentication for Azure OpenAI (#9951)
# Description
This PR adds additional documentation on how to use Azure Active
Directory to authenticate to an OpenAI service within Azure. This method
of authentication allows organizations with more complex security
requirements to use Azure OpenAI.

# Issue
N/A

# Dependencies
N/A

# Twitter
https://twitter.com/CamAHutchison
2023-08-29 14:29:27 -07:00
Guy Korland
7cbe872af8
Add support for Falkordb (ex-RedisGraph) (#9821)
Replace this entire comment with:
  - Description: Add support for Falkordb (ex-RedisGraph)
  - Tag maintainer: @hwchase17
  - Twitter handle: @g_korland
2023-08-29 14:22:33 -07:00
Leonid Ganeline
393816e7bd
Merge branch 'master' into docs-memory-type-menu 2023-08-29 11:46:29 -07:00
Corvus Lee
0fb95ebe66
Docs: enrich SageMaker endpoint embeddings with docstrings and examples (#9924)
Description: added comments to address the relationship between
input/output transformations and the customised inference.py script.
2023-08-29 11:38:52 -07:00
leo-gan
d578efba35 updated notebook titles and text. 2023-08-29 11:25:53 -07:00
Leonid Ganeline
4b6e41a939
Merge branch 'master' into docs-memory-menu 2023-08-29 10:24:07 -07:00
Tomaz Bratanic
6092422e10
Add neo4j provider page (#9941) 2023-08-29 10:09:51 -07:00
leo-gan
c906041aa8 updated notebook titles and text. 2023-08-29 09:58:26 -07:00
Tomaz Bratanic
db13fba7ea
Add neo4j vector support (#9770)
Neo4j has added vector index integration just recently. To allow both
ingestion and integrating it as vector RAG applications, I wrapped it as
a vector store as the implementation is completely different from
`GraphCypherQAChain`. Here, we are not generating any Cypher statements
at query time, we are simply doing the vector similarity search using
the new vector index as if we were dealing with a vector database.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-29 07:54:20 -07:00
Tudor Golubenco
171b0b183b
Pre-release Xata version no longer required (#9915)
Tiny PR: Since we've released version 1.0.0 of the python SDK, we no
longer need to specify the pre-release version when pip installing.
2023-08-29 07:21:22 -07:00
Mike Nitsenko
c80e406e95
Cube semantic loader: allow cubes processing (#9927)
We've started to receive feedback (after launch) that using only views
is confusing.
We're considering this as a good practice, as a view serves as a
"facade" for your data - however, we decided to let users decide this on
their own.

Solves the questions from:
- https://github.com/cube-js/cube/issues/7028
- https://github.com/langchain-ai/langchain/pull/9690
2023-08-29 07:21:01 -07:00
LiaoKong
8f8455b24d fix a link name format to the dependents document 2023-08-29 21:55:05 +08:00
Ofer Mendelevitch
8b8d2a6535 fixed similarity_search_with_score to really use a score
updated unit test with a test for score threshold
Updated demo notebook
2023-08-28 22:26:55 -07:00
Ikko Eltociear Ashimine
766bbd6c6b
Fix typo in code_understanding.ipynb
seperate -> separate
2023-08-29 12:57:19 +09:00
tongtie
82a3c2a557 docs: Fix the syntax error, replace "dotenv.load_env()" with "dotenv.load_dotenv()". 2023-08-29 11:52:50 +08:00
Mazhar (Taha) Mumbaiwala
e80834d783
docs: Fix spelling mistakes in Etherscan.ipynb (#9845) 2023-08-28 19:30:00 -07:00
Philippe PRADOS
7fdb7439e0
Update google drive notebooks (#9851)
Update google drive doc loader and retriever notebooks. Show how to use with langchain-googledrive package.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-28 19:29:35 -07:00
Xiaobing Mi
5d47833ae1
Fix typo in web_scraping.ipynb (#9835) 2023-08-28 19:26:23 -07:00
Leonid Ganeline
b1bffea9c7
docs: fix for title of llm_caching nb (#9891)
Fixed title for the `extras/integrations/llms/llm_caching.ipynb`.
Existing title breaks the sorted order of items in the navbar.
Updated some formatting.
2023-08-28 18:34:04 -07:00
Leonid Ganeline
e01b00aa54
docs: ainetwork update (#9871)
* Added links to the AI Network
* Made title consistent to other tool kits
* Added `integrations/providers/` integration card page
* **No changes** in the example code!
2023-08-28 18:16:22 -07:00
Leonid Ganeline
cf122b6269
docs: Infino example fix (#9888)
- Fixed a broken link in the `integrations/providers/infino.mdx`
- Fixed a title in the `integration/collbacks/infino.ipynb` example
- Updated text format in this example.
2023-08-28 17:42:11 -07:00
William FH
b14d74dd4d
iMessage loader (#9832)
Add an iMessage chat loader
2023-08-28 13:43:59 -07:00
Lance Martin
8393ba9dab
Add instructions for GGUF (#9874)
llama.cpp migrated to GGUF model format, and new releases (e.g.,
[here](https://huggingface.co/TheBloke)) now use GGUF.
2023-08-28 12:56:46 -07:00
hughcrt
3a4d4c940c Change video width 2023-08-28 19:26:33 +02:00
hughcrt
97741d41c5 Add LLMonitorCallbackHandler 2023-08-28 19:24:50 +02:00