Commit Graph

532 Commits

Author SHA1 Message Date
Nuno Campos
93bbf67afc WIP
Add test

Add test

Lint
2023-08-23 19:48:33 +01:00
Nuno Campos
c184be5511 Use a shared executor for all parallel calls 2023-08-23 19:48:33 +01:00
Nuno Campos
db4b256a28 Add error for batch of 0 2023-08-23 19:39:46 +01:00
Nuno Campos
3458489936 Lint 2023-08-23 19:39:46 +01:00
Nuno Campos
e420bf22b6 Lint 2023-08-23 19:39:46 +01:00
Nuno Campos
cc83f54694 L:int 2023-08-23 19:39:46 +01:00
Nuno Campos
d414d47c78 Use a shared executor for all parallel calls 2023-08-23 19:39:46 +01:00
Bagatur
a40c12bb88
Update the nlpcloud connector after some changes on the NLP Cloud API (#9586)
- Description: remove some text generation deprecated parameters and
update the embeddings doc,
- Tag maintainer: @rlancemartin
2023-08-23 11:35:08 -07:00
Bagatur
e2e582f1f6
Fixed source key name for docugami loader (#8598)
The Docugami loader was not returning the source metadata key. This was
triggering this exception when used with retrievers, per
https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/schema/prompt_template.py#L193C1-L195C41

The fix is simple and just updates the metadata key name for the
document each chunk is sourced from, from "name" to "source" as
expected.

I tested by running the python notebook that has an end to end scenario
in it.

Tagging DataLoader maintainers @rlancemartin @eyurtsev
2023-08-23 11:24:55 -07:00
karynzv
5508baf1eb
Add CrateDB prompt (#9657)
Adds a prompt template for the CrateDB SQL dialect.
2023-08-23 13:33:37 -04:00
Bagatur
a8e8a31b41 Merge branch 'master' into bagatur/locals_in_config 2023-08-23 10:26:11 -07:00
Bagatur
ef2500584c fmt 2023-08-23 10:15:45 -07:00
Zizhong Zhang
8a03836160
docs: fix PromptGuard docs (#9659)
Fix PromptGuard docs. Noticed several trivial issues on the docs when
integrating the new class.
cc @baskaryan
2023-08-23 10:04:53 -07:00
Guy Korland
39a5d02225
Cleanup of ruff warnings use isinstance() instead of type() (#9655)
Minor cosmetic PR just cleanup of `ruff` warnings use `isinstance()`
instead of `type()`
2023-08-23 07:14:31 -07:00
Joseph McElroy
2a06e7b216
ElasticsearchStore: improve error logging for adding documents (#9648)
Not obvious what the error is when you cannot index. This pr adds the
ability to log the first errors reason, to help the user diagnose the
issue.

Also added some more documentation for when you want to use the
vectorstore with an embedding model deployed in elasticsearch.

Credit: @elastic and @phoey1
2023-08-23 07:04:09 -07:00
Julien Salinas
f1072cc31f
Merge branch 'master' into master 2023-08-23 14:42:40 +02:00
Jun Liu
b379c5f9c8
Fixed the error on ConfluenceLoader when content_format=VIEW and keep_markdown_format=True (#9633)
- Description: a description of the change

when I set `content_format=ContentFormat.VIEW` and
`keep_markdown_format=True` on ConfluenceLoader, it shows the following
error:
```
langchain/document_loaders/confluence.py", line 459, in process_page
    page["body"]["storage"]["value"], heading_style="ATX"
KeyError: 'storage'
```
The reason is because the content format was set to `view` but it was
still trying to get the content from `page["body"]["storage"]["value"]`.

Also added the other content formats which are supported by Atlassian
API

https://stackoverflow.com/questions/34353955/confluence-rest-api-expanding-page-body-when-retrieving-page-by-title/34363386#34363386

  - Issue: the issue # it fixes (if applicable),

Not applicable.

  - Dependencies: any dependencies required for this change,

Added optional dependency `markdownify` if anyone wants to extract in
markdown format.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-22 21:00:15 -07:00
Gabriel Fu
b2d9970fc1
Allow specifying dtype in langchain.llms.VLLM (#9635)
- Description: add `dtype` argument for VLLM 
  - Issue: #9593 
  - Dependencies: none
  - Tag maintainer: @hwchase17, @baskaryan
2023-08-22 20:21:56 -07:00
anifort
900c1f3e8d
Add support for structured data sources with google enterprise search (#9037)
<!-- Thank you for contributing to LangChain!

Replace this comment with:
- Description: Added the capability to handles structured data from
google enterprise search,
- Issue: Retriever failed when underline search engine was integrated
with structured data,
  - Dependencies: google-api-core
  - Tag maintainer: @jarokaz
  - Twitter handle: anifort

Please make sure you're PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the
same people again.

See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->

---------

Co-authored-by: Christos Aniftos <aniftos@google.com>
Co-authored-by: Holt Skinner <13262395+holtskinner@users.noreply.github.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-22 23:18:10 -04:00
Harrison Chase
02545a54b3
python repl improvement for csv agent (#9618) 2023-08-22 17:06:18 -07:00
Erick Friis
fc64e6349e
Hub stub updates (#9577)
Updates the hub stubs to not fail when no api key is found. For
supporting singleton tenants and default values from sdk 0.1.6.

Also adds the ability to define is_public and description for backup
repo creation on push.
2023-08-22 16:05:41 -07:00
Kim Minjong
ca8232a3c1
Update BaseChatModel.astream to respect generation_info (#9430)
Currently, generation_info is not respected by only reflecting messages
in chunks. Change it to add generations so that generation chunks are
merged properly.

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-08-22 15:18:24 -07:00
Bagatur
81163e3c0c
parent retriever nit (#9570)
if ids are nullable seems like they should have default val None.
mirrors VectorStore interface as well. cc @mcantillon21 @jacoblee93
2023-08-22 14:58:16 -04:00
Myeongseop Kim
f1e602996a
import tqdm.auto instead of tqdm tqdm for OpenAIEmbeddings (#9584)
- Description: current code does not work very well on jupyter notebook,
so I changed the code so that it imports `tqdm.auto` instead.
  - Issue: #9582 
  - Dependencies: N/A
  - Tag maintainer: @hwchase17, @baskaryan
  - Twitter handle: N/A

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-22 14:54:07 -04:00
Predrag Gruevski
d564ec944c
poetry lock the experimental package. (#9478) 2023-08-22 14:09:35 -04:00
Predrag Gruevski
65e893b9cd
poetry lock on langchain. (#9476) 2023-08-22 14:09:23 -04:00
Predrag Gruevski
3c7cc4d440
Test experimental package with langchain on master branch. (#9621)
It's possible that langchain-experimental works fine with the latest
*published* langchain, but is broken with the langchain on `master`.
Unfortunately, you can see this is currently the case — this is why this
PR also includes a minor fix for the `langchain` package itself.

We want to catch situations like that *before* releasing a new
langchain, hence this test.
2023-08-22 13:35:21 -04:00
Eugene Yurtsev
3408810748
Add batch util (#9620)
Add `batch` utility to langchain
2023-08-22 12:31:18 -04:00
Bagatur
2b663089b5
bump 271 (#9615) 2023-08-22 08:10:22 -07:00
klae01
b868ef23bc
Add AINetwork blockchain toolkit integration (#9527)
# Description
This PR introduces a new toolkit for interacting with the AINetwork
blockchain. The toolkit provides a set of tools for performing various
operations on the AINetwork blockchain, such as transferring AIN,
reading and writing values to the blockchain database, managing apps,
setting rules and owners.

# Dependencies
[ain-py](https://github.com/ainblockchain/ain-py) >= 1.0.2

# Misc
The example notebook
(langchain/docs/extras/integrations/toolkits/ainetwork.ipynb) is in the
PR

---------

Co-authored-by: kriii <kriii@users.noreply.github.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-22 08:03:33 -07:00
Bagatur
e99ef12cb1
Bagatur/litellm model name (#9613)
Co-authored-by: ishaan-jaff <ishaanjaffer0324@gmail.com>
2023-08-22 07:44:00 -07:00
Harrison Chase
1720e99397
add variables for field names (#9563) 2023-08-22 07:43:21 -07:00
Anthony Mahanna
dfb9ff1079
bugfix: ArangoDB Empty Schema Case (#9574)
- Introduces a conditional in `ArangoGraph.generate_schema()` to exclude
empty ArangoDB Collections from the schema
- Add empty collection test case

Issue: N/A
Dependencies: None
2023-08-22 07:41:06 -07:00
Philippe PRADOS
d4c49b16e4
Fix ChatMessageHistory (#9594)
The initialization of the array of ChatMessageHistory is buggy.
The list is shared with all instances.
2023-08-22 07:36:36 -07:00
toddkim95
fba29f203a
Add to support polars (#9610)
### Description
Polars is a DataFrame interface on top of an OLAP Query Engine
implemented in Rust.
Polars is faster to read than pandas, so I'm looking forward to seeing
it added to the document loader.

### Dependencies
polars (https://pola-rs.github.io/polars-book/user-guide/)

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-22 07:36:24 -07:00
Aashish Saini
3c4f32c8b8
Replacing Exception type from ValueError to ImportError (#9588)
I have restructured the code to ensure uniform handling of ImportError.
In place of previously used ValueError, I've adopted the standard
practice of raising ImportError with explanatory messages. This
modification enhances code readability and clarifies that any problems
stem from module importation.

@eyurtsev , @baskaryan 

Thanks
2023-08-22 07:34:05 -07:00
Julien Salinas
033b874701 Remove some deprecated text generation parameters. 2023-08-22 09:26:37 +02:00
Bagatur
4e7e6bfe0a revert 2023-08-21 18:01:49 -07:00
Bagatur
a9bf409a09 param 2023-08-21 17:37:07 -07:00
Bagatur
fa478638a9 Merge branch 'master' into bagatur/locals_in_config 2023-08-21 17:31:39 -07:00
Bagatur
182b059bf4 param 2023-08-21 17:31:38 -07:00
Bagatur
04f2d69b83
improve confluence doc loader param validation (#9568) 2023-08-21 15:02:36 -07:00
Zizhong Zhang
00eff8c4a7
feat: Add PromptGuard integration (#9481)
Add PromptGuard integration
-------
There are two approaches to integrate PromptGuard with a LangChain
application.

1. PromptGuardLLMWrapper
2. functions that can be used in LangChain expression.

-----
- Dependencies
`promptguard` python package, which is a runtime requirement if you'd
try out the demo.

- @baskaryan @hwchase17 Thanks for the ideas and suggestions along the
development process.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 14:59:36 -07:00
Sathindu
652c542b2f
fix: Imports for the ConfluenceLoader:process_page (#9432)
### Description
When we're loading documents using `ConfluenceLoader`:`load` function
and, if both `include_comments=True` and `keep_markdown_format=True`,
we're getting an error saying `NameError: free variable 'BeautifulSoup'
referenced before assignment in enclosing scope`.
    
    loader = ConfluenceLoader(url="URI", token="TOKEN")
    documents = loader.load(
        space_key="SPACE", 
        include_comments=True, 
        keep_markdown_format=True, 
    )

This happens because previous imports only consider the
`keep_markdown_format` parameter, however to include the comments, it's
using `BeautifulSoup`

Now it's fixed to handle all four scenarios considering both
`include_comments` and `keep_markdown_format`.

### Twitter
`@SathinduGA`

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 13:44:52 -07:00
Mike Salvatore
7c0b1b8171
Add session to ConfluenceLoader.__init__() (#9437)
- Description: Allows the user of `ConfluenceLoader` to pass a
`requests.Session` object in lieu of an authentication mechanism
- Issue: None
- Dependencies: None
- Tag maintainer: @hwchase17
2023-08-21 13:18:35 -07:00
Kim Minjong
3d1095218c
Update ChatOpenAI._astream to respect finish_reason (#9431)
Currently, ChatOpenAI._astream does not reflect finish_reason to
generation_info. Change it to reflect that.
2023-08-21 12:56:42 -07:00
Matthew Zeiler
949b2cf177
Improvements to the Clarifai integration (#9290)
- Improved docs
- Improved performance in multiple ways through batching, threading,
etc.
 - fixed error message 
 - Added support for metadata filtering during similarity search.

@baskaryan PTAL
2023-08-21 12:53:36 -07:00
ricki-epsilla
66a47d9a61
add Epsilla vectorstore (#9239)
[Epsilla](https://github.com/epsilla-cloud/vectordb) vectordb is an
open-source vector database that leverages the advanced academic
parallel graph traversal techniques for vector indexing.
This PR adds basic integration with
[pyepsilla](https://github.com/epsilla-cloud/epsilla-python-client)(Epsilla
vectordb python client) as a vectorstore.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 12:51:15 -07:00
Bagatur
dda5b1e370
Bagatur/doc loader confluence (#9524)
Co-authored-by: chanjetsdp <chanjetsdp@chanjet.com>
2023-08-21 12:40:44 -07:00
Predrag Gruevski
de1f63505b
Add py.typed file to langchain-experimental. (#9557)
The package is linted with mypy, so its type hints are correct and
should be exposed publicly. Without this file, the type hints remain
private and cannot be used by downstream users of the package.
2023-08-21 15:37:16 -04:00