Commit Graph

4500 Commits

Author SHA1 Message Date
Bagatur
0c760f184c Update NucliaDB vecstore deps 2023-09-06 16:29:10 -07:00
Eric BREHAULT
19b4ecdc39
Implement NucliaDB vector store (#10236)
# Description

This pull request allows to use the
[NucliaDB](https://docs.nuclia.dev/docs/docs/nucliadb/intro) as a vector
store in LangChain.

It works with both a [local NucliaDB
instance](https://docs.nuclia.dev/docs/docs/nucliadb/deploy/basics) or
with [Nuclia Cloud](https://nuclia.cloud).

# Dependencies

It requires an up-to-date version of the `nuclia` Python package.

@rlancemartin, @eyurtsev, @hinthornw, please review it when you have a
moment :)

Note: our Twitter handler is `@NucliaAI`
2023-09-06 16:26:14 -07:00
cccs-eric
b64a443f72
Fix SQL search_path for Trino query engine (#10248)
This PR replaces the generic `SET search_path TO` statement by `USE` for
the Trino dialect since Trino does not support `SET search_path`.
Official Trino documentation can be found
[here](https://trino.io/docs/current/sql/use.html).

With this fix, the `SQLdatabase` will now be able to set the current
schema and execute queries using the Trino engine. It will use the
catalog set as default by the connection uri.
2023-09-06 16:19:37 -07:00
Bagatur
1fb7bdd595
Split sql use case docs (#10257)
Split sql use case into directory so we can add other structured data
pages
2023-09-06 16:19:21 -07:00
Bagatur
763212eafd
Add use case nb position (#10299) 2023-09-06 15:46:33 -07:00
Ikko Eltociear Ashimine
ea5d29a702
Update amazon_comprehend_chain.ipynb (#10246)
Huggingface, HuggingFace -> Hugging Face
2023-09-06 15:38:37 -07:00
Brian Antonelli
4df101cf77
Don't hardcode PGVector distance strategies (#10265)
- Description: Remove hardcoded/duplicated distance strategies in the
PGVector store.
- Issue: NA
- Dependencies: NA
- Tag maintainer: for a quicker response, tag the relevant maintainer
(see below),
- Twitter handle: @archmonkeymojo

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-09-06 15:20:44 -07:00
captivus
86cb9da735
Updated Additional Resources section of documentation (#10260)
- Description: Updated Additional Resources section of documentation and
added to YouTube videos with excellent playlist of Langchain content
from Sam Witteveen
- Issue: None -- updating documentation
- Dependencies: None
- Tag maintainer: @baskaryan
2023-09-06 15:10:43 -07:00
JaéGeR
b8669b249e
Added Hugging face inference api (#10280)
Embed documents without locally downloading the HF model


---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-09-06 14:55:48 -07:00
Ilya
6e6f15df24
Add strip text splits flag (#10295)
#10085
---------

Co-authored-by: codesee-maps[bot] <86324825+codesee-maps[bot]@users.noreply.github.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-09-06 14:06:12 -07:00
Randy
1690013711
Doc: openai_functions_agent.mdx import (#10282)
Fix the import in docmention

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-09-06 14:00:39 -07:00
William FH
13c5951e26
Add LCEL cookbook examples (#10290)
1. For passing config to runnable lambda
2. For branching and merging
2023-09-06 13:50:43 -07:00
ParamdeepSinghShorthillsAI
3cc242b591
Update rwkv.py import error (#10293)
I have updated the code to ensure consistent error handling for
ImportError. Instead of relying on ValueError as before, I've followed
the standard practice of raising ImportError while also including
detailed error messages. This modification improves code clarity and
explicitly indicates that any issues are related to module imports.
2023-09-06 13:50:21 -07:00
Pihplipe Oegr
bce38b7163
Add notebook example to use sqlite-vss as a vector store. (#10292)
Follow-up PR for https://github.com/langchain-ai/langchain/pull/10047,
simply adding a notebook quickstart example for the vector store with
SQLite, using the class SQLiteVSS.

Maintainer tag @baskaryan

Co-authored-by: Philippe Oger <philippe.oger@adevinta.com>
2023-09-06 13:46:59 -07:00
Tomaz Bratanic
db73c9d5b5
Diffbot Graph Transformer / Neo4j Graph document ingestion (#9979)
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-09-06 13:32:59 -07:00
Predrag Gruevski
ccb9e3ee2d
Install dev, lint, test, typing extra deps for linting steps. (#10249)
`mypy` cannot type-check code that relies on dependencies that aren't
installed.

Eventually we'll probably want to install as many optional dependencies
as possible. However, the full "extended deps" setup for langchain
creates a 3GB cache file and takes a while to unpack and install. We'll
probably want something a bit more targeted.

This is a first step toward something better.
2023-09-06 11:15:28 -04:00
Predrag Gruevski
82d5d4d0ae
Deny creating files as a result of test runs. (#10253)
A test file was accidentally dropping a `results.json` file in the
current working directory as a result of running `make test`.

This is undesirable, since we don't want to risk accidentally adding
stray files into the repo if we run tests locally and then do `git add
.` without inspecting the file list very closely.
2023-09-06 11:15:16 -04:00
Predrag Gruevski
8d5bf1fb20
Fix langchain lint on master. (#10289) 2023-09-06 16:01:13 +01:00
Nik
49341483da
Update Banana.dev docs to latest correct usage (#10183)
- Description: this PR updates all Banana.dev-related docs to match the
latest client usage. The code in the docs before this PR were out of
date and would never run.
- Issue: [#6404](https://github.com/langchain-ai/langchain/issues/6404)
- Dependencies: -
- Tag maintainer:  
- Twitter handle: [BananaDev_ ](https://twitter.com/BananaDev_ )
2023-09-06 07:46:17 -07:00
Bagatur
9e839d4977
bump 283 (#10287) 2023-09-06 07:33:03 -07:00
William FH
ffca5e7eea
Allow config propagation, Add default lambda name, Improve ergonomics of config passed in (#10273)
Makes it easier to do recursion using regular python compositional
patterns

```py
def lambda_decorator(func):
    """Decorate function as a RunnableLambda"""
    return runnable.RunnableLambda(func)

@lambda_decorator
def fibonacci(a, config: runnable.RunnableConfig) -> int:
    if a <= 1:
        return a
    else:
        return fibonacci.invoke(
            a - 1, config
        ) + fibonacci.invoke(a - 2, config)

fibonacci.invoke(10)
```

https://smith.langchain.com/public/cb98edb4-3a09-4798-9c22-a930037faf88/r

Also makes it more natural to do things like error handle and call other
langchain objects in ways we probably don't want to support in
`with_fallbacks()`

```py
@lambda_decorator
def handle_errors(a, config: runnable.RunnableConfig) -> int:
    try:
        return my_chain.invoke(a, config)
    except MyExceptionType as exc:
        return my_other_chain.invoke({"original": a, "error": exc}, config)
```

In this case, the next chain takes in the exception object. Maybe this
could be something we toggle in `with_fallbacks` but I fear we'll get
into uglier APIs + heavier cognitive load if we try to do too much there

---------

Co-authored-by: Nuno Campos <nuno@boringbits.io>
2023-09-06 05:54:38 -07:00
mateusz.wosinski
7b7bea5424 Fix linters, update notebook 2023-09-06 10:22:42 +02:00
Bagatur
c732d8fffd
use case docs reorder (#10074) 2023-09-05 15:11:16 -07:00
Mario Scrocca
334bd8ebbe
Fix bug in SPARQL intent selection (#8521)
- Description: Fix bug in SPARQL intent selection
- Issue: After the change in #7758 the intent is always set to "UPDATE".
Indeed, if the answer to the prompt contains only "SELECT" the
`find("SELECT")` operation returns a higher value w.r.t. `-1` returned
by `find("UPDATE")`.
- Dependencies: None,
- Tag maintainer: @baskaryan @aditya-29 
- Twitter handle: @mario_scrock
2023-09-05 14:37:02 -07:00
Predrag Gruevski
7fe8bf03a0
Final poetry action fix: manually recreate softlinks broken by caching. (#10250)
It seems the caching action was not always correctly recreating
softlinks. At first glance, the softlinks it created seemed fine, but
they didn't always work. Possibly hitting some kind of underlying bug,
but not particularly worth debugging in depth -- we can manually create
the soft links we need.
2023-09-05 15:47:58 -04:00
Predrag Gruevski
619516260d
Re-enable poetry binary caching with fix and more logging. (#10244)
- Revert "Temporarily disable step that seems to be transiently failing.
(#10234)"
- Refresh shell hashtable and show poetry/python location and version.
2023-09-05 14:03:03 -04:00
Predrag Gruevski
803be5b986
Run CI when CI infra itself has changed. (#10239)
Make sure that changes to CI infrastructure get tested on CI before
being merged.

Without this PR, changes to the poetry setup action don't trigger a CI
run and in principle could break `master` when merged.
2023-09-05 13:08:19 -04:00
Bagatur
c8d7ee62ba
bump 282 (#10233) 2023-09-05 07:58:00 -07:00
Predrag Gruevski
e34ad6fefd
Temporarily disable step that seems to be transiently failing. (#10234) 2023-09-05 10:55:47 -04:00
Nuno Campos
5d8673a3c1
Fix usage of AsyncHtmlLoader with an already running event loop (#10220) 2023-09-05 07:25:28 -07:00
vintro
ac2310a405
add NumberedListOutputParser to output_parser init (#10204)
`from langchain.output_parsers import NumberedListOutputParser` did not
work, needed to add it to the init file
2023-09-05 01:12:41 -07:00
Junlin Zhou
8b95dabfe3
update(llms/TGI): Allow None as temperature value (#10212)
Text Generation Inference's client permits the use of a None temperature
as seen
[here](033230ae66/clients/python/text_generation/client.py (L71C9-L71C20)).
While I haved dived into TGI's server code and don't know about the
implications of using None as a temperature setting, I think we should
grant users the option to pass None as a temperature parameter to TGI.
2023-09-05 01:07:57 -07:00
mateusz.wosinski
882a588264 Revert poetry files 2023-09-05 09:21:05 +02:00
William FH
be152b6a56
Better ls info (#10202) 2023-09-04 18:21:15 -07:00
Christophe Bornet
f389c4fcab
Fix S3DirectoryLoader exception (#10193)
#9304 introduced a critical bug. The S3DirectoryLoader fails completely
because boto3 checks the naming of kw arguments and one of the args is
badly named (very sorry for that)

cc @baskaryan
2023-09-04 15:59:22 -07:00
Manuel Soria
dde1992fdd
Adding custom tools to SQL Agent (#10198)
Changes in:
- `create_sql_agent` function so that user can easily add custom tools
as complement for the toolkit.
- updating **sql use case** notebook to showcase 2 examples of extra
tools.

Motivation for these changes is having the possibility of including
domain expert knowledge to the agent, which improves accuracy and
reduces time/tokens.

---------

Co-authored-by: Manuel Soria <manuel.soria@greyscaleai.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-09-04 15:28:28 -07:00
ElReyZero
5dbae94e04
OpenAIEmbeddings: Add optional an optional parameter to skip empty embeddings (#10196)
## Description

### Issue
This pull request addresses a lingering issue identified in PR #7070. In
that previous pull request, an attempt was made to address the problem
of empty embeddings when using the `OpenAIEmbeddings` class. While PR
#7070 introduced a mechanism to retry requests for embeddings, it didn't
fully resolve the issue as empty embeddings still occasionally
persisted.

### Problem
In certain specific use cases, empty embeddings can be encountered when
requesting data from the OpenAI API. In some cases, these empty
embeddings can be skipped or removed without affecting the functionality
of the application. However, they might not always be resolved through
retries, and their presence can adversely affect the functionality of
applications relying on the `OpenAIEmbeddings` class.

### Solution
To provide a more robust solution for handling empty embeddings, we
propose the introduction of an optional parameter, `skip_empty`, in the
`OpenAIEmbeddings` class. When set to `True`, this parameter will enable
the behavior of automatically skipping empty embeddings, ensuring that
problematic empty embeddings do not disrupt the processing flow. The
developer will be able to optionally toggle this behavior if needed
without disrupting the application flow.

## Changes Made
- Added an optional parameter, `skip_empty`, to the `OpenAIEmbeddings`
class.
- When `skip_empty` is set to `True`, empty embeddings are automatically
skipped without causing errors or disruptions.

### Example Usage
```python
from openai.embeddings import OpenAIEmbeddings

# Initialize the OpenAIEmbeddings class with skip_empty=True
embeddings = OpenAIEmbeddings(api_key="your_api_key", skip_empty=True)

# Request embeddings, empty embeddings are automatically skipped. docs is a variable containing the already splitted text.
results = embeddings.embed_documents(docs)

# Process results without interruption from empty embeddings
```
2023-09-04 14:10:36 -07:00
Lance Martin
8998060d85
Update docs w/ prompt hub (#10197)
Small updates to docs
2023-09-04 14:09:08 -07:00
Bagatur
a94dc6ee44
model garden nit (#10194) 2023-09-04 11:42:35 -07:00
Louis
bb8c095127
Add 'download_dir' argument to VLLM (#9754)
- Description:
Add a 'download_dir' argument to VLLM model (to change the cache
download directotu when retrieving a model from HF hub)
- Issue:
On some remote machine, I want the cache dir to be in a volume where I
have space (models are heavy nowadays). Sometimes the default HF cache
dir might not be what we want.
- Dependencies:
None

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-09-04 10:53:48 -07:00
Aashish Saini
8bba69ffd0
Fixed some grammatical typos in doc files (#10191)
Fixed some grammatical typos in doc files
CC: @baskaryan, @eyurtsev, @rlancemartin.

---------

Co-authored-by: Aashish Saini <141953346+AashishSainiShorthillsAI@users.noreply.github.com>
Co-authored-by: AryamanJaiswalShorthillsAI <142397527+AryamanJaiswalShorthillsAI@users.noreply.github.com>
Co-authored-by: Adarsh Shrivastav <142413097+AdarshKumarShorthillsAI@users.noreply.github.com>
Co-authored-by: Vishal <141389263+VishalYadavShorthillsAI@users.noreply.github.com>
Co-authored-by: ChetnaGuptaShorthillsAI <142381084+ChetnaGuptaShorthillsAI@users.noreply.github.com>
Co-authored-by: PankajKumarShorthillsAI <142473460+PankajKumarShorthillsAI@users.noreply.github.com>
Co-authored-by: AbhishekYadavShorthillsAI <142393903+AbhishekYadavShorthillsAI@users.noreply.github.com>
Co-authored-by: AmitSinghShorthillsAI <142410046+AmitSinghShorthillsAI@users.noreply.github.com>
Co-authored-by: Md Nazish Arman <142379599+MdNazishArmanShorthillsAI@users.noreply.github.com>
Co-authored-by: KamalSharmaShorthillsAI <142474019+KamalSharmaShorthillsAI@users.noreply.github.com>
Co-authored-by: Lakshya <lakshyagupta87@yahoo.com>
Co-authored-by: Aayush <142384656+AayushShorthillsAI@users.noreply.github.com>
Co-authored-by: AnujMauryaShorthillsAI <142393269+AnujMauryaShorthillsAI@users.noreply.github.com>
2023-09-04 10:48:08 -07:00
Bagatur
098b4aa465
bump 281 (#10189) 2023-09-04 08:51:50 -07:00
Aashish Saini
699f58fb83
Fixed Import Error type (#10168)
I have restructured the code to ensure uniform handling of ImportError.
In place of previously used ValueError, I've adopted the standard
practice of raising ImportError with explanatory messages. This
modification enhances code readability and clarifies that any problems
stem from module importation.

---------

Co-authored-by: Aashish Saini <141953346+AashishSainiShorthillsAI@users.noreply.github.com>
Co-authored-by: AryamanJaiswalShorthillsAI <142397527+AryamanJaiswalShorthillsAI@users.noreply.github.com>
Co-authored-by: Adarsh Shrivastav <142413097+AdarshKumarShorthillsAI@users.noreply.github.com>
Co-authored-by: Vishal <141389263+VishalYadavShorthillsAI@users.noreply.github.com>
Co-authored-by: ChetnaGuptaShorthillsAI <142381084+ChetnaGuptaShorthillsAI@users.noreply.github.com>
Co-authored-by: PankajKumarShorthillsAI <142473460+PankajKumarShorthillsAI@users.noreply.github.com>
Co-authored-by: AbhishekYadavShorthillsAI <142393903+AbhishekYadavShorthillsAI@users.noreply.github.com>
Co-authored-by: AmitSinghShorthillsAI <142410046+AmitSinghShorthillsAI@users.noreply.github.com>
Co-authored-by: Aayush <142384656+AayushShorthillsAI@users.noreply.github.com>
Co-authored-by: AnujMauryaShorthillsAI <142393269+AnujMauryaShorthillsAI@users.noreply.github.com>
2023-09-04 08:43:28 -07:00
刘 方瑞
de9e545542
MyScale hot fix on type check (#10180)
Previous PR #9353 has incomplete type checks and deprecation warnings.
This PR will fix those type check and add deprecation warning to myscale
vectorstore
2023-09-04 08:40:58 -07:00
JunXiang
cb928ed3d5
Fix: the duplicate characters wrong results when using pdfplumber loader (#10165)
(Reopen PR #7706, hope this problem can fix.)

When using `pdfplumber`, some documents may be parsed incorrectly,
resulting in **duplicated characters**.

Taking the
[linked](https://bruusgaard.no/wp-content/uploads/2021/05/Datasheet1000-series.pdf)
document as an example:

## Before
```python
from langchain.document_loaders import PDFPlumberLoader

pdf_file = 'file.pdf'
loader = PDFPlumberLoader(pdf_file)
docs = loader.load()
print(docs[0].page_content)
```

Results:
```
11000000 SSeerriieess
PPoorrttaabbllee ssiinnggllee ggaass ddeetteeccttoorrss ffoorr HHyyddrrooggeenn aanndd CCoommbbuussttiibbllee ggaasseess
TThhee RRiikkeenn KKeeiikkii GGPP--11000000 iiss aa ccoommppaacctt aanndd
lliigghhttwweeiigghhtt ggaass ddeetteeccttoorr wwiitthh hhiigghh sseennssiittiivviittyy ffoorr
tthhee ddeetteeccttiioonn ooff hhyyddrrooccaarrbboonnss.. TThhee mmeeaassuurreemmeenntt
iiss ppeerrffoorrmmeedd ffoorr tthhiiss ppuurrppoossee bbyy mmeeaannss ooff ccaattaallyyttiicc
sseennssoorr.. TThhee GGPP--11000000 hhaass aa bbuuiilltt--iinn ppuummpp wwiitthh
ppuummpp bboooosstteerr ffuunnccttiioonn aanndd aa ddiirreecctt sseelleeccttiioonn ffrroomm
aa lliisstt ooff 2255 hhyyddrrooccaarrbboonnss ffoorr eexxaacctt aalliiggnnmmeenntt ooff tthhee
ttaarrggeett ggaass -- OOnnllyy ccaalliibbrraattiioonn oonn CCHH iiss nneecceessssaarryy..
44
FFeeaattuurreess
TThhee RRiikkeenn KKeeiikkii 110000vvvvttaabbllee ssiinnggllee HHyyddrrooggeenn aanndd
CCoommbbuussttiibbllee ggaass ddeetteeccttoorrss..
TThheerree aarree 33 ssttaannddaarrdd mmooddeellss::
GGPP--11000000:: 00--1100%%LLEELL // 00--110000%%LLEELL ›› LLEELL ddeetteeccttoorr
NNCC--11000000:: 00--11000000ppppmm // 00--1100000000ppppmm ›› PPPPMM
ddeetteeccttoorr
DDiirreecctt rreeaaddiinngg ooff tthhee ccoonncceennttrraattiioonn vvaalluueess ooff
ccoommbbuussttiibbllee ggaasseess ooff 2255 ggaasseess ((55 NNPP--11000000))..
EEaassyy ooppeerraattiioonn ffeeaattuurree ooff cchhaannggiinngg tthhee ggaass nnaammee
ddiissppllaayy wwiitthh 11 sswwiittcchh bbuuttttoonn..
LLoonngg ddiissttaannccee ddrraawwiinngg ppoossssiibbllee wwiitthh tthhee ppuummpp
bboooosstteerr ffuunnccttiioonn..
VVaarriioouuss ccoommbbuussttiibbllee ggaasseess ccaann bbee mmeeaassuurreedd bbyy tthhee
ppppmm oorrddeerr wwiitthh NNCC--11000000..
www.bruusgaard.no postmaster@bruusgaard.no +47 67 54 93 30 Rev: 446-2
```

We can see that there are a large number of duplicated characters in the
text, which can cause issues in subsequent applications.

## After

Therefore, based on the
[solution](https://github.com/jsvine/pdfplumber/issues/71) provided by
the `pdfplumber` source project. I added the `"dedupe_chars()"` method
to address this problem. (Just pass the parameter `dedupe` to `True`)

```python
from langchain.document_loaders import PDFPlumberLoader

pdf_file = 'file.pdf'
loader = PDFPlumberLoader(pdf_file, dedupe=True)
docs = loader.load()
print(docs[0].page_content)
```

Results:

```
1000 Series
Portable single gas detectors for Hydrogen and Combustible gases
The Riken Keiki GP-1000 is a compact and
lightweight gas detector with high sensitivity for
the detection of hydrocarbons. The measurement
is performed for this purpose by means of catalytic
sensor. The GP-1000 has a built-in pump with
pump booster function and a direct selection from
a list of 25 hydrocarbons for exact alignment of the
target gas - Only calibration on CH is necessary.
4
Features
The Riken Keiki 100vvtable single Hydrogen and
Combustible gas detectors.
There are 3 standard models:
GP-1000: 0-10%LEL / 0-100%LEL › LEL detector
NC-1000: 0-1000ppm / 0-10000ppm › PPM
detector
Direct reading of the concentration values of
combustible gases of 25 gases (5 NP-1000).
Easy operation feature of changing the gas name
display with 1 switch button.
Long distance drawing possible with the pump
booster function.
Various combustible gases can be measured by the
ppm order with NC-1000.
www.bruusgaard.no postmaster@bruusgaard.no +47 67 54 93 30 Rev: 446-2
```

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-09-04 08:37:00 -07:00
mateusz.wosinski
1b7caa1a29 PR comments 2023-09-04 15:32:08 +02:00
mateusz.wosinski
e9abe176bc Update dependencies 2023-09-04 15:32:08 +02:00
mateusz.wosinski
6b9529e11a Update notebook 2023-09-04 15:23:24 +02:00
mateusz.wosinski
c6149aacef Fix linters 2023-09-04 15:23:24 +02:00
mateusz.wosinski
800fe4a73f Integration with eleven labs 2023-09-04 15:23:24 +02:00