Commit Graph

809 Commits

Author SHA1 Message Date
Bagatur
16eb935469
Fix for similarity_search_with_score (#9903)
- Description: the implementation for similarity_search_with_score did
not actually include a score or logic to filter. Now fixed.
- Tag maintainer: @rlancemartin
- Twitter handle: @ofermend
2023-08-29 15:04:48 -07:00
Bagatur
c70bb0ec28
Activeloopai runtime arg (#9961) 2023-08-29 15:01:46 -07:00
Bagatur
0f85671630 fmt 2023-08-29 14:55:25 -07:00
Bagatur
78c014399f fmt 2023-08-29 14:53:15 -07:00
Eugene Yurtsev
5cce6529a4
Speed up openai tests (#9943)
Saves ~8-10 seconds from total unit tests times

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-29 14:30:41 -07:00
Guy Korland
7cbe872af8
Add support for Falkordb (ex-RedisGraph) (#9821)
Replace this entire comment with:
  - Description: Add support for Falkordb (ex-RedisGraph)
  - Tag maintainer: @hwchase17
  - Twitter handle: @g_korland
2023-08-29 14:22:33 -07:00
Bagatur
9f2d908316 cr 2023-08-29 14:16:48 -07:00
Bagatur
3c1547925a fix 2023-08-29 14:02:13 -07:00
William FH
fbd792ac7c
Fix import (#9945) 2023-08-29 12:38:42 -07:00
Zizhong Zhang
8bd7a9d18e
feat: PromptGuard takes a list of str (#9948)
Recently we made the decision that PromptGuard takes a list of strings
instead of a string.
@ggroode implemented the integration change.

---------

Co-authored-by: ggroode <ggroode@berkeley.edu>
Co-authored-by: ggroode <46691276+ggroode@users.noreply.github.com>
2023-08-29 12:22:30 -07:00
Predrag Gruevski
8dbf4cbe80
Add notice about security-sensitive experimental code to experimental README. (#9936)
It renders like this:
https://github.com/langchain-ai/langchain/tree/pg/experimental-readme/libs/experimental


![image](https://github.com/langchain-ai/langchain/assets/2348618/a5f9569d-96f6-44c6-8559-921adb3e337d)
2023-08-29 14:21:30 -04:00
Predrag Gruevski
b5cd1e0fed
Add security notices on PAL and CPAL experimental chains. (#9938)
Clearly document that the PAL and CPAL techniques involve generating
code, and that such code must be properly sandboxed and given
appropriate narrowly-scoped credentials in order to ensure security.

While our implementations include some mitigations, Python and SQL
sandboxing is well-known to be a very hard problem and our mitigations
are no replacement for proper sandboxing and permissions management. The
implementation of such techniques must be performed outside the scope of
the Python process where this package's code runs, so its correct setup
and administration must therefore be the responsibility of the user of
this code.
2023-08-29 13:51:56 -04:00
Jan-Luca Barthel
f5faac8859
addition of cosine distance function for faiss (#9939)
- Description: added the _cosine_relevance_score_fn to
_select_relevance_score_fn of faiss.py to enable the use of cosine
distance for similarity for this vector store and to comply with the
Error Message, that implies, that cosine should be a valid distance
strategy
- Issue: no relevant Issue found, but needed this function myself and
tested it in a private repo
  - Dependencies: none
2023-08-29 10:29:51 -07:00
Eugene Yurtsev
880bf06290 x 2023-08-29 11:15:41 -04:00
Eugene Yurtsev
9efc29e3d1 x 2023-08-29 11:13:42 -04:00
Bagatur
d6957921f0
bump 276 (#9931) 2023-08-29 08:00:38 -07:00
Tomaz Bratanic
db13fba7ea
Add neo4j vector support (#9770)
Neo4j has added vector index integration just recently. To allow both
ingestion and integrating it as vector RAG applications, I wrapped it as
a vector store as the implementation is completely different from
`GraphCypherQAChain`. Here, we are not generating any Cypher statements
at query time, we are simply doing the vector similarity search using
the new vector index as if we were dealing with a vector database.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-29 07:54:20 -07:00
Bagatur
49ebbe4bcd
fix pydantic import (#9930) 2023-08-29 07:53:01 -07:00
Mike Nitsenko
c80e406e95
Cube semantic loader: allow cubes processing (#9927)
We've started to receive feedback (after launch) that using only views
is confusing.
We're considering this as a good practice, as a view serves as a
"facade" for your data - however, we decided to let users decide this on
their own.

Solves the questions from:
- https://github.com/cube-js/cube/issues/7028
- https://github.com/langchain-ai/langchain/pull/9690
2023-08-29 07:21:01 -07:00
Nikhil Suresh
dd10cf945c fixed minor linting issues 2023-08-29 14:15:59 +00:00
olgavrou
48aaa27bf7 update score to take entire response object to make it easier for user 2023-08-29 08:46:55 -04:00
olgavrou
c4ccaebbbb activate and deactivate scorer 2023-08-29 08:37:59 -04:00
olgavrou
7eaaad51de cb defaults and some fixes 2023-08-29 07:42:45 -04:00
olgavrou
5727148f2b make sure test don't try to download sentence transformer models 2023-08-29 07:09:58 -04:00
olgavrou
5de212d907
Merge branch 'langchain-ai:master' into master 2023-08-29 05:58:22 -04:00
olgavrou
4e6e03ef50 fix mypy complaint 2023-08-29 05:51:52 -04:00
olgavrou
d50c0f139d re order imports 2023-08-29 05:46:56 -04:00
olgavrou
758225dc17 include type 2023-08-29 05:44:09 -04:00
olgavrou
44485c2b26 make input arg type more explicit 2023-08-29 05:42:45 -04:00
olgavrou
8d10a52525 fix linting complaints 2023-08-29 05:36:45 -04:00
olgavrou
b3c0728de2 fix mypy errors in tests 2023-08-29 05:28:43 -04:00
olgavrou
0b8691c6e5 fix all mypy errors and some renaming and refactoring 2023-08-29 05:19:19 -04:00
olgavrou
a11ad11d06 fix all mypy errors 2023-08-29 03:59:01 -04:00
adilkhan
bbae8cb88f Added runtime argument 2023-08-29 12:12:49 +06:00
Ofer Mendelevitch
4454204455 reformat black 2023-08-28 23:04:57 -07:00
Ofer Mendelevitch
318a21e267 fixed typo in spelling 2023-08-28 23:01:11 -07:00
hughcrt
e71f4760db Change multiline comment width 2023-08-29 07:55:10 +02:00
Ofer Mendelevitch
a5450be32e fixed lint 2023-08-28 22:31:39 -07:00
Ofer Mendelevitch
8b8d2a6535 fixed similarity_search_with_score to really use a score
updated unit test with a test for score threshold
Updated demo notebook
2023-08-28 22:26:55 -07:00
hughcrt
7979cef06a Replace | by Union 2023-08-29 06:22:50 +02:00
Nikhil Suresh
23ef836b48 matches colon and any number of white spaces after colon 2023-08-29 04:18:33 +00:00
Nikhil Suresh
64eb5a6082 removed unnecessary white space in regex that breaks qa with sources chain 2023-08-29 03:54:38 +00:00
Nikhil Suresh
8a4670e127 updated formatting changes 2023-08-29 03:54:38 +00:00
Nikhil Suresh
b1f649bca5 fixed issue with white space and added unit tests 2023-08-29 03:54:38 +00:00
Nikhil Suresh
6d3485e798 fixed regex to match sources for all cases, also includes source 2023-08-29 03:54:25 +00:00
Predrag Gruevski
47499c6db4
Avoid type: ignore suppression by adding mypy type hint. (#9881)
Mypy was not able to determine a good type for `type_to_loader_dict`,
since the values in the dict are functions whose return types are
related to each other in a complex way. One can see this by adding a
line like `reveal_type(type_to_loader_dict)` and running mypy, which
will get mypy to show what type it has inferred for that value.

Adding an explicit type hint to help out mypy avoids the need for a mypy
suppression and allows the code to type-check cleanly.
2023-08-28 17:53:33 -07:00
maks-operlejn-ds
f327535eda
Add conftest file to langchain experimental (#9886)
In order to use `requires` marker in langchain-experimental, there's a
need for *conftest.py* file inside. Everything is identical to the main
langchain module.

Co-authored-by: maks-operlejn-ds <maks.operlejn@gmail.com>
2023-08-28 17:52:16 -07:00
William FH
907c57e324
Add collect_runs callback (#9885) 2023-08-28 15:30:41 -07:00
William FH
3103f07e03
Use existing required args obj if specified (#9883)
We always overwrote the required args but we infer them by default.
Doing it only the old way makes it so the llm guesses even if an arg is
optional (e.g., for uuids)
2023-08-28 14:40:22 -07:00
William FH
b14d74dd4d
iMessage loader (#9832)
Add an iMessage chat loader
2023-08-28 13:43:59 -07:00
Predrag Gruevski
eb3d1fa93c
Add security warning to experimental SQLDatabaseChain class. (#9867)
The most reliable way to not have a chain run an undesirable SQL command
is to not give it database permissions to run that command. That way the
database itself performs the rule enforcement, so it's much easier to
configure and use properly than anything we could add in ourselves.
2023-08-28 13:53:27 -04:00
hughcrt
97741d41c5 Add LLMonitorCallbackHandler 2023-08-28 19:24:50 +02:00
eryk-dsai
7f5713b80a
feat: grammar-based sampling in llama-cpp (#9712)
## Description 

The following PR enables the [grammar-based
sampling](https://github.com/ggerganov/llama.cpp/tree/master/grammars)
in llama-cpp LLM.

In short, loading file with formal grammar definition will constrain
model outputs. For instance, one can force the model to generate valid
JSON or generate only python lists.

In the follow-up PR we will add:
* docs with some description why it is cool and how it works
* maybe some code sample for some task such as in llama repo

---------

Co-authored-by: Lance Martin <lance@langchain.dev>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-28 09:52:55 -07:00
William FH
cb642ef658
Return feedback (#9629)
Return the feedback values in an eval run result

Also made a helper method to display as a dataframe but it may be
overkill
2023-08-28 09:15:05 -07:00
Bagatur
5e2d0cf54e
bump 275 (#9860) 2023-08-28 07:27:07 -07:00
Leonid Kuligin
00baddf34c fixed enterprise search returning an empty array 2023-08-28 15:38:56 +02:00
Eugene Yurtsev
5edf819524
Qdrant Client: Expose instance for creating client (#9706)
Expose classmethods to convenient initialize the vectostore.

The purpose of this PR is to make it easy for users to initialize an
empty vectorstore that's properly pre-configured without having to index
documents into it via `from_documents`.

This will make it easier for users to rely on the following indexing
code: https://github.com/langchain-ai/langchain/pull/9614
to help manage data in the qdrant vectorstore.
2023-08-28 09:30:59 -04:00
olgavrou
dd6fff1c62 no errors in pick best chain 2023-08-28 08:13:23 -04:00
olgavrou
6a1102d4c0 mypy fixes and formatting 2023-08-28 06:58:33 -04:00
olgavrou
7725192a0d update deps for vw 2023-08-28 04:58:55 -04:00
olgavrou
2bfa73257f sync from upstream master 2023-08-28 04:15:57 -04:00
Harrison Chase
610f46d83a
accept openai terms (#9826) 2023-08-27 17:18:24 -07:00
Harrison Chase
c1badc1fa2
add gmail loader (#9810) 2023-08-27 17:18:09 -07:00
Bagatur
0d01cede03
bump 274 (#9805) 2023-08-26 12:16:26 -07:00
Nikhil Suresh
0da5803f5a
fixed regex to match sources for all cases, also includes source (#9775)
- Description: Updated the regex to handle all the different cases for
string matching (SOURCES, sources, Sources),
  - Issue: https://github.com/langchain-ai/langchain/issues/9774
  - Dependencies: N/A
2023-08-25 18:10:33 -07:00
Sam Partee
a28eea5767
Redis metadata filtering and specification, index customization (#8612)
### Description

The previous Redis implementation did not allow for the user to specify
the index configuration (i.e. changing the underlying algorithm) or add
additional metadata to use for querying (i.e. hybrid or "filtered"
search).

This PR introduces the ability to specify custom index attributes and
metadata attributes as well as use that metadata in filtered queries.
Overall, more structure was introduced to the Redis implementation that
should allow for easier maintainability moving forward.

# New Features

The following features are now available with the Redis integration into
Langchain

## Index schema generation

The schema for the index will now be automatically generated if not
specified by the user. For example, the data above has the multiple
metadata categories. The the following example

```python

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores.redis import Redis

embeddings = OpenAIEmbeddings()


rds, keys = Redis.from_texts_return_keys(
    texts,
    embeddings,
    metadatas=metadata,
    redis_url="redis://localhost:6379",
    index_name="users"
)
```

Loading the data in through this and the other ``from_documents`` and
``from_texts`` methods will now generate index schema in Redis like the
following.

view index schema with the ``redisvl`` tool. [link](redisvl.com)

```bash
$ rvl index info -i users
```


Index Information:
| Index Name | Storage Type | Prefixes | Index Options | Indexing |

|--------------|----------------|---------------|-----------------|------------|
| users | HASH | ['doc:users'] | [] | 0 |
Index Fields:
| Name | Attribute | Type | Field Option | Option Value |

|----------------|----------------|---------|----------------|----------------|
| user | user | TEXT | WEIGHT | 1 |
| job | job | TEXT | WEIGHT | 1 |
| credit_score | credit_score | TEXT | WEIGHT | 1 |
| content | content | TEXT | WEIGHT | 1 |
| age | age | NUMERIC | | |
| content_vector | content_vector | VECTOR | | |


### Custom Metadata specification

The metadata schema generation has the following rules
1. All text fields are indexed as text fields.
2. All numeric fields are index as numeric fields.

If you would like to have a text field as a tag field, users can specify
overrides like the following for the example data

```python

# this can also be a path to a yaml file
index_schema = {
    "text": [{"name": "user"}, {"name": "job"}],
    "tag": [{"name": "credit_score"}],
    "numeric": [{"name": "age"}],
}

rds, keys = Redis.from_texts_return_keys(
    texts,
    embeddings,
    metadatas=metadata,
    redis_url="redis://localhost:6379",
    index_name="users"
)
```
This will change the index specification to 

Index Information:
| Index Name | Storage Type | Prefixes | Index Options | Indexing |

|--------------|----------------|----------------|-----------------|------------|
| users2 | HASH | ['doc:users2'] | [] | 0 |
Index Fields:
| Name | Attribute | Type | Field Option | Option Value |

|----------------|----------------|---------|----------------|----------------|
| user | user | TEXT | WEIGHT | 1 |
| job | job | TEXT | WEIGHT | 1 |
| content | content | TEXT | WEIGHT | 1 |
| credit_score | credit_score | TAG | SEPARATOR | , |
| age | age | NUMERIC | | |
| content_vector | content_vector | VECTOR | | |


and throw a warning to the user (log output) that the generated schema
does not match the specified schema.

```text
index_schema does not match generated schema from metadata.
index_schema: {'text': [{'name': 'user'}, {'name': 'job'}], 'tag': [{'name': 'credit_score'}], 'numeric': [{'name': 'age'}]}
generated_schema: {'text': [{'name': 'user'}, {'name': 'job'}, {'name': 'credit_score'}], 'numeric': [{'name': 'age'}]}
```

As long as this is on purpose,  this is fine.

The schema can be defined as a yaml file or a dictionary

```yaml

text:
  - name: user
  - name: job
tag:
  - name: credit_score
numeric:
  - name: age

```

and you pass in a path like

```python
rds, keys = Redis.from_texts_return_keys(
    texts,
    embeddings,
    metadatas=metadata,
    redis_url="redis://localhost:6379",
    index_name="users3",
    index_schema=Path("sample1.yml").resolve()
)
```

Which will create the same schema as defined in the dictionary example


Index Information:
| Index Name | Storage Type | Prefixes | Index Options | Indexing |

|--------------|----------------|----------------|-----------------|------------|
| users3 | HASH | ['doc:users3'] | [] | 0 |
Index Fields:
| Name | Attribute | Type | Field Option | Option Value |

|----------------|----------------|---------|----------------|----------------|
| user | user | TEXT | WEIGHT | 1 |
| job | job | TEXT | WEIGHT | 1 |
| content | content | TEXT | WEIGHT | 1 |
| credit_score | credit_score | TAG | SEPARATOR | , |
| age | age | NUMERIC | | |
| content_vector | content_vector | VECTOR | | |



### Custom Vector Indexing Schema

Users with large use cases may want to change how they formulate the
vector index created by Langchain

To utilize all the features of Redis for vector database use cases like
this, you can now do the following to pass in index attribute modifiers
like changing the indexing algorithm to HNSW.

```python
vector_schema = {
    "algorithm": "HNSW"
}

rds, keys = Redis.from_texts_return_keys(
    texts,
    embeddings,
    metadatas=metadata,
    redis_url="redis://localhost:6379",
    index_name="users3",
    vector_schema=vector_schema
)

```

A more complex example may look like

```python
vector_schema = {
    "algorithm": "HNSW",
    "ef_construction": 200,
    "ef_runtime": 20
}

rds, keys = Redis.from_texts_return_keys(
    texts,
    embeddings,
    metadatas=metadata,
    redis_url="redis://localhost:6379",
    index_name="users3",
    vector_schema=vector_schema
)
```

All names correspond to the arguments you would set if using Redis-py or
RedisVL. (put in doc link later)


### Better Querying

Both vector queries and Range (limit) queries are now available and
metadata is returned by default. The outputs are shown.

```python
>>> query = "foo"
>>> results = rds.similarity_search(query, k=1)
>>> print(results)
[Document(page_content='foo', metadata={'user': 'derrick', 'job': 'doctor', 'credit_score': 'low', 'age': '14', 'id': 'doc:users:657a47d7db8b447e88598b83da879b9d', 'score': '7.15255737305e-07'})]

>>> results = rds.similarity_search_with_score(query, k=1, return_metadata=False)
>>> print(results) # no metadata, but with scores
[(Document(page_content='foo', metadata={}), 7.15255737305e-07)]

>>> results = rds.similarity_search_limit_score(query, k=6, score_threshold=0.0001)
>>> print(len(results)) # range query (only above threshold even if k is higher)
4
```

### Custom metadata filtering

A big advantage of Redis in this space is being able to do filtering on
data stored alongside the vector itself. With the example above, the
following is now possible in langchain. The equivalence operators are
overridden to describe a new expression language that mimic that of
[redisvl](redisvl.com). This allows for arbitrarily long sequences of
filters that resemble SQL commands that can be used directly with vector
queries and range queries.

There are two interfaces by which to do so and both are shown. 

```python

>>> from langchain.vectorstores.redis import RedisFilter, RedisNum, RedisText

>>> age_filter = RedisFilter.num("age") > 18
>>> age_filter = RedisNum("age") > 18 # equivalent
>>> results = rds.similarity_search(query, filter=age_filter)
>>> print(len(results))
3

>>> job_filter = RedisFilter.text("job") == "engineer" 
>>> job_filter = RedisText("job") == "engineer" # equivalent
>>> results = rds.similarity_search(query, filter=job_filter)
>>> print(len(results))
2

# fuzzy match text search
>>> job_filter = RedisFilter.text("job") % "eng*"
>>> results = rds.similarity_search(query, filter=job_filter)
>>> print(len(results))
2


# combined filters (AND)
>>> combined = age_filter & job_filter
>>> results = rds.similarity_search(query, filter=combined)
>>> print(len(results))
1

# combined filters (OR)
>>> combined = age_filter | job_filter
>>> results = rds.similarity_search(query, filter=combined)
>>> print(len(results))
4
```

All the above filter results can be checked against the data above.


### Other

  - Issue: #3967 
  - Dependencies: No added dependencies
  - Tag maintainer: @hwchase17 @baskaryan @rlancemartin 
  - Twitter handle: @sampartee

---------

Co-authored-by: Naresh Rangan <naresh.rangan0@walmart.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-25 17:22:50 -07:00
nikhilkjha
d57d08fd01
Initial commit for comprehend moderator (#9665)
This PR implements a custom chain that wraps Amazon Comprehend API
calls. The custom chain is aimed to be used with LLM chains to provide
moderation capability that let’s you detect and redact PII, Toxic and
Intent content in the LLM prompt, or the LLM response. The
implementation accepts a configuration object to control what checks
will be performed on a LLM prompt and can be used in a variety of setups
using the LangChain expression language to not only detect the
configured info in chains, but also other constructs such as a
retriever.
The included sample notebook goes over the different configuration
options and how to use it with other chains.

###  Usage sample
```python
from langchain_experimental.comprehend_moderation import BaseModerationActions, BaseModerationFilters

moderation_config = { 
        "filters":[ 
                BaseModerationFilters.PII, 
                BaseModerationFilters.TOXICITY,
                BaseModerationFilters.INTENT
        ],
        "pii":{ 
                "action": BaseModerationActions.ALLOW, 
                "threshold":0.5, 
                "labels":["SSN"],
                "mask_character": "X"
        },
        "toxicity":{ 
                "action": BaseModerationActions.STOP, 
                "threshold":0.5
        },
        "intent":{ 
                "action": BaseModerationActions.STOP, 
                "threshold":0.5
        }
}

comp_moderation_with_config = AmazonComprehendModerationChain(
    moderation_config=moderation_config, #specify the configuration
    client=comprehend_client,            #optionally pass the Boto3 Client
    verbose=True
)

template = """Question: {question}

Answer:"""

prompt = PromptTemplate(template=template, input_variables=["question"])

responses = [
    "Final Answer: A credit card number looks like 1289-2321-1123-2387. A fake SSN number looks like 323-22-9980. John Doe's phone number is (999)253-9876.", 
    "Final Answer: This is a really shitty way of constructing a birdhouse. This is fucking insane to think that any birds would actually create their motherfucking nests here."
]
llm = FakeListLLM(responses=responses)

llm_chain = LLMChain(prompt=prompt, llm=llm)

chain = ( 
    prompt 
    | comp_moderation_with_config 
    | {llm_chain.input_keys[0]: lambda x: x['output'] }  
    | llm_chain 
    | { "input": lambda x: x['text'] } 
    | comp_moderation_with_config 
)

response = chain.invoke({"question": "A sample SSN number looks like this 123-456-7890. Can you give me some more samples?"})

print(response['output'])


```
### Output
```
> Entering new AmazonComprehendModerationChain chain...
Running AmazonComprehendModerationChain...
Running pii validation...
Found PII content..stopping..
The prompt contains PII entities and cannot be processed
```

---------

Co-authored-by: Piyush Jain <piyushjain@duck.com>
Co-authored-by: Anjan Biswas <anjanavb@amazon.com>
Co-authored-by: Jha <nikjha@amazon.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-25 15:11:27 -07:00
William FH
1960ac8d25
token chunks (#9739)
Co-authored-by: Andrew <abatutin@gmail.com>
2023-08-25 12:52:07 -07:00
Bagatur
9731ce5a40
bump 273 (#9751) 2023-08-25 03:05:04 -07:00
Fabrizio Ruocco
cacaf487c3
Azure Cognitive Search - update sdk b8, mod user agent, search with scores (#9191)
Description: Update Azure Cognitive Search SDK to version b8 (breaking
change)
Customizable User Agent.
Implemented Similarity search with scores 

@baskaryan

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-25 02:34:09 -07:00
Sergey Kozlov
135cb86215
Fix QuestionListOutputParser (#9738)
This PR fixes `QuestionListOutputParser` text splitting.

`QuestionListOutputParser` incorrectly splits numbered list text into
lines. If text doesn't end with `\n` , the regex doesn't capture the
last item. So it always returns `n - 1` items, and
`WebResearchRetriever.llm_chain` generates less queries than requested
in the search prompt.

How to reproduce:

```python
from langchain.retrievers.web_research import QuestionListOutputParser

parser = QuestionListOutputParser()

good = parser.parse(
    """1. This is line one.
    2. This is line two.
    """  # <-- !
)

bad = parser.parse(
    """1. This is line one.
    2. This is line two."""    # <-- No new line.
)

assert good.lines == ['1. This is line one.\n', '2. This is line two.\n'], good.lines
assert bad.lines == ['1. This is line one.\n', '2. This is line two.'], bad.lines
```

NOTE: Last item will not contain a line break but this seems ok because
the items are stripped in the
`WebResearchRetriever.clean_search_query()`.
2023-08-25 01:47:17 -07:00
Jurik-001
d04fe0d3ea
remove Value error "pyspark is not installed. Please install it with `pip i… (#9723)
Description: You cannot execute spark_sql with versions prior to 3.4 due
to the introduction of pyspark.errors in version 3.4.
And if you are below you get 3.4 "pyspark is not installed. Please
install it with pip nstall pyspark" which is not helpful. Also if you
not have pyspark installed you get already the error in init. I would
return all errors. But if you have a different idea feel free to
comment.

Issue: None
Dependencies: None
Maintainer:

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-24 22:18:55 -07:00
Margaret Qian
30151c99c7
Update Mosaic endpoint input/output api (#7391)
As noted in prior PRs (https://github.com/hwchase17/langchain/pull/6060,
https://github.com/hwchase17/langchain/pull/7348), the input/output
format has changed a few times as we've stabilized our inference API.
This PR updates the API to the latest stable version as indicated in our
docs: https://docs.mosaicml.com/en/latest/inference.html

The input format looks like this:

`{"inputs": [<prompt>]}
`

The output format looks like this:
`
{"outputs": [<output_text>]}
`
---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-24 22:13:17 -07:00
Leonid Kuligin
87da56fb1e
Added a pdf parser based on DocAI (#9579)
#9578

---------

Co-authored-by: Leonid Kuligin <kuligin@google.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-24 21:44:49 -07:00
Naama Magami
adb21782b8
Add del vector pgvector + adding modification time to confluence and google drive docs (#9604)
Description:
- adding implementation of delete for pgvector
- adding modification time in docs metadata for confluence and google
drive.

Issue:
https://github.com/langchain-ai/langchain/issues/9312

Tag maintainer: @baskaryan, @eyurtsev, @hwchase17, @rlancemartin.

---------

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-24 21:09:30 -07:00
Erick Friis
3e5cda3405
Hub Push Ergonomics (#9731)
Improves the hub pushing experience, returning a url instead of just a
commit hash.

Requires hub sdk 0.1.8
2023-08-24 17:41:54 -07:00
Tudor Golubenco
dc30edf51c
Xata as a chat message memory store (#9719)
This adds Xata as a memory store also to the python version of
LangChain, similar to the [one for
LangChain.js](https://github.com/hwchase17/langchainjs/pull/2217).

I have added a Jupyter Notebook with a simple and a more complex example
using an agent.

To run the integration test, you need to execute something like:

```
XATA_API_KEY='xau_...' XATA_DB_URL="https://demo-uni3q8.eu-west-1.xata.sh/db/langchain"  poetry run pytest tests/integration_tests/memory/test_xata.py
```

Where `langchain` is the database you create in Xata.
2023-08-24 17:37:46 -07:00
William FH
dff00ea91e
Chat Loaders (#9708)
Still working out interface/notebooks + need discord data dump to test
out things other than copy+paste

Update:
- Going to remove the 'user_id' arg in the loaders themselves and just
standardize on putting the "sender" arg in the extra kwargs. Then can
provide a utility function to map these to ai and human messages
- Going to move the discord one into just a notebook since I don't have
a good dump to test on and copy+paste maybe isn't the greatest thing to
support in v0
- Need to do more testing on slack since it seems the dump only includes
channels and NOT 1 on 1 convos
-

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-08-24 17:23:27 -07:00
Bagatur
0f48e6c36e
fix integration deps (#9722) 2023-08-24 15:06:53 -07:00
Bagatur
a0800c9f15
rm google api core and add more dependency testing (#9721) 2023-08-24 14:20:58 -07:00
Andrew White
2bcf581a23
Added search parameters to qdrant max_marginal_relevance_search (#7745)
Adds the qdrant search filter/params to the
`max_marginal_relevance_search` method, which is present on others. I
did not add `offset` for pagination, because it's behavior would be
ambiguous in this setting (since we fetch extra and down-select).

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Kacper Łukawski <lukawski.kacper@gmail.com>
2023-08-24 14:11:30 -07:00
Tomaz Bratanic
dacf96895a
Add the option to use separate LLMs for GraphCypherQA chain (#9689)
The Graph Chains are different in the way that it uses two LLMChains
instead of one like the retrievalQA chains. Therefore, sometimes you
want to use different LLM to generate the database query and to generate
the final answer.

This feature would make it more convenient to use different LLMs in the
same chain.

I have also renamed the Graph DB QA Chain to Neo4j DB QA Chain in the
documentation only as it is used only for Neo4j. The naming was
ambigious as it was the first graphQA chain added and wasn't sure how do
you want to spin it.
2023-08-24 11:50:38 -07:00
Bagatur
f5ea725796
bump 272 (#9704) 2023-08-24 07:46:15 -07:00
Patrick Loeber
6bedfdf25a
Fix docs for AssemblyAIAudioTranscriptLoader (shorter import path) (#9687)
Uses the shorter import path

`from langchain.document_loaders import` instead of the full path
`from langchain.document_loaders.assemblyai`

Applies those changes to the docs and the unit test.

See #9667 that adds this new loader.
2023-08-24 07:24:53 -07:00
Nuno Campos
78ffcdd9a9 Lint 2023-08-24 16:09:38 +02:00
Nuno Campos
20d2c0571c Do not share executors between parent and child tasks 2023-08-24 16:05:10 +02:00
Harrison Chase
9963b32e59
Harrison/multi vector (#9700) 2023-08-24 06:42:42 -07:00
Leonid Ganeline
c19888c12c
docstrings: vectorstores consistency (#9349)
 
- updated the top-level descriptions to a consistent format;
- changed several `ValueError` to `ImportError` in the import cases;
- changed the format of several internal functions from "name" to
"_name". So, these functions are not shown in the Top-level API
Reference page (with lists of classes/functions)
2023-08-23 23:17:05 -07:00
Kim Minjong
d0ff0db698
Update ChatOpenAI._stream to respect finish_reason (#9672)
Currently, ChatOpenAI._stream does not reflect finish_reason to
generation_info. Change it to reflect that.

Same patch as https://github.com/langchain-ai/langchain/pull/9431 , but
also applies to _stream.
2023-08-23 22:58:14 -07:00
Patrick Loeber
5990651070
Add new document_loader: AssemblyAIAudioTranscriptLoader (#9667)
This PR adds a new document loader `AssemblyAIAudioTranscriptLoader`
that allows to transcribe audio files with the [AssemblyAI
API](https://www.assemblyai.com) and loads the transcribed text into
documents.

- Add new document_loader with class `AssemblyAIAudioTranscriptLoader`
- Add optional dependency `assemblyai`
- Add unit tests (using a Mock client)
- Add docs notebook

This is the equivalent to the JS integration already available in
LangChain.js. See the [LangChain JS docs AssemblyAI
page](https://js.langchain.com/docs/modules/data_connection/document_loaders/integrations/web_loaders/assemblyai_audio_transcription).

At its simplest, you can use the loader to get a transcript back from an
audio file like this:

```python
from langchain.document_loaders.assemblyai import AssemblyAIAudioTranscriptLoader

loader =  AssemblyAIAudioTranscriptLoader(file_path="./testfile.mp3")
docs = loader.load()
```

To use it, it needs the `assemblyai` python package installed, and the
environment variable `ASSEMBLYAI_API_KEY` set with your API key.
Alternatively, the API key can also be passed as an argument.

Twitter handles to shout out if so kindly 🙇
[@AssemblyAI](https://twitter.com/AssemblyAI) and
[@patloeber](https://twitter.com/patloeber)

---------

Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-23 22:51:19 -07:00
Eugene Yurtsev
9e1dbd4b49 x 2023-08-23 22:51:49 -04:00
Eugene Yurtsev
b88dfcb42a
Add indexing support (#9614)
This PR introduces a persistence layer to help with indexing workflows
into
vectostores.

The indexing code helps users to:

1. Avoid writing duplicated content into the vectostore
2. Avoid over-writing content if it's unchanged

Importantly, this keeps on working even if the content being written is
derived
via a set of transformations from some source content (e.g., indexing
children
documents that were derived from parent documents by chunking.)

The two main components are:

1. Persistence layer that keeps track of which keys were updated and
when.
Keeping track of the timestamp of updates, allows to clean up old
content
   safely, and with minimal complexity.
2. HashedDocument which is used to hash the contents (including
metadata) of
   the documents. We rely on the hashes for identifying duplicates.


The indexing code works with **ANY** document loader. To add
transformations
to the documents, users for now can add a custom document loader
that composes an existing loader together with document transformers.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-23 21:41:38 -04:00
刘 方瑞
c215481531
Update default index type and metric type for MyScale vector store (#9353)
We update the default index type from `IVFFLAT` to `MSTG`, a new vector
type developed by MyScale.
2023-08-23 18:26:29 -07:00
Joshua Sundance Bailey
a9c86774da
Anthropic: Allow the use of kwargs consistent with ChatOpenAI. (#9515)
- Description: ~~Creates a new root_validator in `_AnthropicCommon` that
allows the use of `model_name` and `max_tokens` keyword arguments.~~
Adds pydantic field aliases to support `model_name` and `max_tokens` as
keyword arguments. Ultimately, this makes `ChatAnthropic` more
consistent with `ChatOpenAI`, making the two classes more
interchangeable for the developer.
  - Issue: https://github.com/langchain-ai/langchain/issues/9510

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-23 18:23:21 -07:00
Bagatur
342087bdfa
fix integration test imports (#9669) 2023-08-23 16:47:01 -07:00
Keras Conv3d
cbaea8d63b
tair fix distance_type error, and add hybrid search (#9531)
- fix: distance_type error, 
- feature: Tair add hybrid search

---------

Co-authored-by: thw <hanwen.thw@alibaba-inc.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-23 16:38:31 -07:00
Eugene Yurtsev
cd81e8a8f2
Add exclude to GenericLoader.from_file_system (#9539)
support exclude param in GenericLoader.from_filesystem

---------

Co-authored-by: Kyle Pancamo <50267605+KylePancamo@users.noreply.github.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-23 16:09:10 -07:00
Jacob Lee
278ef0bdcf
Adds ChatOllama (#9628)
@rlancemartin

---------

Co-authored-by: Adilkhan Sarsen <54854336+adolkhan@users.noreply.github.com>
Co-authored-by: Kim Minjong <make.dirty.code@gmail.com>
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
Co-authored-by: Lance Martin <lance@langchain.dev>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-23 13:02:26 -07:00
Nuno Campos
20ce283fa7 Format 2023-08-23 20:03:35 +01:00
Nuno Campos
6424b3cde0 Add another test 2023-08-23 20:02:35 +01:00
William FH
da18e177f1 Update libs/langchain/langchain/schema/runnable/base.py
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-23 20:00:16 +01:00
Nuno Campos
c326751085 Lint 2023-08-23 20:00:16 +01:00
Nuno Campos
6d19709b65 RunnableLambda, if func returns a Runnable, run it 2023-08-23 20:00:16 +01:00
Nuno Campos
677da6a0fd Add support for async funcs in RunnableSequence 2023-08-23 19:54:48 +01:00
Nuno Campos
1751fe114d Add one more test 2023-08-23 19:52:13 +01:00
Nuno Campos
882b97cfd2 Lint 2023-08-23 19:50:20 +01:00
Nuno Campos
3ddabe8b2c Code review 2023-08-23 19:48:33 +01:00
Nuno Campos
fdcd50aab4 Extend test 2023-08-23 19:48:33 +01:00
Nuno Campos
9777c2801d Update method and docstring 2023-08-23 19:48:33 +01:00
Nuno Campos
93bbf67afc WIP
Add test

Add test

Lint
2023-08-23 19:48:33 +01:00
Nuno Campos
c184be5511 Use a shared executor for all parallel calls 2023-08-23 19:48:33 +01:00
Nuno Campos
db4b256a28 Add error for batch of 0 2023-08-23 19:39:46 +01:00
Nuno Campos
3458489936 Lint 2023-08-23 19:39:46 +01:00
Nuno Campos
e420bf22b6 Lint 2023-08-23 19:39:46 +01:00
Nuno Campos
cc83f54694 L:int 2023-08-23 19:39:46 +01:00
Nuno Campos
d414d47c78 Use a shared executor for all parallel calls 2023-08-23 19:39:46 +01:00
Bagatur
a40c12bb88
Update the nlpcloud connector after some changes on the NLP Cloud API (#9586)
- Description: remove some text generation deprecated parameters and
update the embeddings doc,
- Tag maintainer: @rlancemartin
2023-08-23 11:35:08 -07:00
Bagatur
e2e582f1f6
Fixed source key name for docugami loader (#8598)
The Docugami loader was not returning the source metadata key. This was
triggering this exception when used with retrievers, per
https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/schema/prompt_template.py#L193C1-L195C41

The fix is simple and just updates the metadata key name for the
document each chunk is sourced from, from "name" to "source" as
expected.

I tested by running the python notebook that has an end to end scenario
in it.

Tagging DataLoader maintainers @rlancemartin @eyurtsev
2023-08-23 11:24:55 -07:00
karynzv
5508baf1eb
Add CrateDB prompt (#9657)
Adds a prompt template for the CrateDB SQL dialect.
2023-08-23 13:33:37 -04:00
Bagatur
a8e8a31b41 Merge branch 'master' into bagatur/locals_in_config 2023-08-23 10:26:11 -07:00
Bagatur
ef2500584c fmt 2023-08-23 10:15:45 -07:00
Zizhong Zhang
8a03836160
docs: fix PromptGuard docs (#9659)
Fix PromptGuard docs. Noticed several trivial issues on the docs when
integrating the new class.
cc @baskaryan
2023-08-23 10:04:53 -07:00
Guy Korland
39a5d02225
Cleanup of ruff warnings use isinstance() instead of type() (#9655)
Minor cosmetic PR just cleanup of `ruff` warnings use `isinstance()`
instead of `type()`
2023-08-23 07:14:31 -07:00
Joseph McElroy
2a06e7b216
ElasticsearchStore: improve error logging for adding documents (#9648)
Not obvious what the error is when you cannot index. This pr adds the
ability to log the first errors reason, to help the user diagnose the
issue.

Also added some more documentation for when you want to use the
vectorstore with an embedding model deployed in elasticsearch.

Credit: @elastic and @phoey1
2023-08-23 07:04:09 -07:00
Julien Salinas
f1072cc31f
Merge branch 'master' into master 2023-08-23 14:42:40 +02:00
Jun Liu
b379c5f9c8
Fixed the error on ConfluenceLoader when content_format=VIEW and keep_markdown_format=True (#9633)
- Description: a description of the change

when I set `content_format=ContentFormat.VIEW` and
`keep_markdown_format=True` on ConfluenceLoader, it shows the following
error:
```
langchain/document_loaders/confluence.py", line 459, in process_page
    page["body"]["storage"]["value"], heading_style="ATX"
KeyError: 'storage'
```
The reason is because the content format was set to `view` but it was
still trying to get the content from `page["body"]["storage"]["value"]`.

Also added the other content formats which are supported by Atlassian
API

https://stackoverflow.com/questions/34353955/confluence-rest-api-expanding-page-body-when-retrieving-page-by-title/34363386#34363386

  - Issue: the issue # it fixes (if applicable),

Not applicable.

  - Dependencies: any dependencies required for this change,

Added optional dependency `markdownify` if anyone wants to extract in
markdown format.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-22 21:00:15 -07:00
Gabriel Fu
b2d9970fc1
Allow specifying dtype in langchain.llms.VLLM (#9635)
- Description: add `dtype` argument for VLLM 
  - Issue: #9593 
  - Dependencies: none
  - Tag maintainer: @hwchase17, @baskaryan
2023-08-22 20:21:56 -07:00
anifort
900c1f3e8d
Add support for structured data sources with google enterprise search (#9037)
<!-- Thank you for contributing to LangChain!

Replace this comment with:
- Description: Added the capability to handles structured data from
google enterprise search,
- Issue: Retriever failed when underline search engine was integrated
with structured data,
  - Dependencies: google-api-core
  - Tag maintainer: @jarokaz
  - Twitter handle: anifort

Please make sure you're PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
  2. an example notebook showing its use.

Maintainer responsibilities:
  - General / Misc / if you don't know who to tag: @baskaryan
  - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
  - Models / Prompts: @hwchase17, @baskaryan
  - Memory: @hwchase17
  - Agents / Tools / Toolkits: @hinthornw
  - Tracing / Callbacks: @agola11
  - Async: @agola11

If no one reviews your PR within a few days, feel free to @-mention the
same people again.

See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
 -->

---------

Co-authored-by: Christos Aniftos <aniftos@google.com>
Co-authored-by: Holt Skinner <13262395+holtskinner@users.noreply.github.com>
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-22 23:18:10 -04:00
Harrison Chase
02545a54b3
python repl improvement for csv agent (#9618) 2023-08-22 17:06:18 -07:00
Erick Friis
fc64e6349e
Hub stub updates (#9577)
Updates the hub stubs to not fail when no api key is found. For
supporting singleton tenants and default values from sdk 0.1.6.

Also adds the ability to define is_public and description for backup
repo creation on push.
2023-08-22 16:05:41 -07:00
Kim Minjong
ca8232a3c1
Update BaseChatModel.astream to respect generation_info (#9430)
Currently, generation_info is not respected by only reflecting messages
in chunks. Change it to add generations so that generation chunks are
merged properly.

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-08-22 15:18:24 -07:00
Bagatur
81163e3c0c
parent retriever nit (#9570)
if ids are nullable seems like they should have default val None.
mirrors VectorStore interface as well. cc @mcantillon21 @jacoblee93
2023-08-22 14:58:16 -04:00
Myeongseop Kim
f1e602996a
import tqdm.auto instead of tqdm tqdm for OpenAIEmbeddings (#9584)
- Description: current code does not work very well on jupyter notebook,
so I changed the code so that it imports `tqdm.auto` instead.
  - Issue: #9582 
  - Dependencies: N/A
  - Tag maintainer: @hwchase17, @baskaryan
  - Twitter handle: N/A

Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-08-22 14:54:07 -04:00
Predrag Gruevski
d564ec944c
poetry lock the experimental package. (#9478) 2023-08-22 14:09:35 -04:00
Predrag Gruevski
65e893b9cd
poetry lock on langchain. (#9476) 2023-08-22 14:09:23 -04:00
Predrag Gruevski
3c7cc4d440
Test experimental package with langchain on master branch. (#9621)
It's possible that langchain-experimental works fine with the latest
*published* langchain, but is broken with the langchain on `master`.
Unfortunately, you can see this is currently the case — this is why this
PR also includes a minor fix for the `langchain` package itself.

We want to catch situations like that *before* releasing a new
langchain, hence this test.
2023-08-22 13:35:21 -04:00
Eugene Yurtsev
3408810748
Add batch util (#9620)
Add `batch` utility to langchain
2023-08-22 12:31:18 -04:00
Bagatur
2b663089b5
bump 271 (#9615) 2023-08-22 08:10:22 -07:00
klae01
b868ef23bc
Add AINetwork blockchain toolkit integration (#9527)
# Description
This PR introduces a new toolkit for interacting with the AINetwork
blockchain. The toolkit provides a set of tools for performing various
operations on the AINetwork blockchain, such as transferring AIN,
reading and writing values to the blockchain database, managing apps,
setting rules and owners.

# Dependencies
[ain-py](https://github.com/ainblockchain/ain-py) >= 1.0.2

# Misc
The example notebook
(langchain/docs/extras/integrations/toolkits/ainetwork.ipynb) is in the
PR

---------

Co-authored-by: kriii <kriii@users.noreply.github.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-22 08:03:33 -07:00
Bagatur
e99ef12cb1
Bagatur/litellm model name (#9613)
Co-authored-by: ishaan-jaff <ishaanjaffer0324@gmail.com>
2023-08-22 07:44:00 -07:00
Harrison Chase
1720e99397
add variables for field names (#9563) 2023-08-22 07:43:21 -07:00
Anthony Mahanna
dfb9ff1079
bugfix: ArangoDB Empty Schema Case (#9574)
- Introduces a conditional in `ArangoGraph.generate_schema()` to exclude
empty ArangoDB Collections from the schema
- Add empty collection test case

Issue: N/A
Dependencies: None
2023-08-22 07:41:06 -07:00
Philippe PRADOS
d4c49b16e4
Fix ChatMessageHistory (#9594)
The initialization of the array of ChatMessageHistory is buggy.
The list is shared with all instances.
2023-08-22 07:36:36 -07:00
toddkim95
fba29f203a
Add to support polars (#9610)
### Description
Polars is a DataFrame interface on top of an OLAP Query Engine
implemented in Rust.
Polars is faster to read than pandas, so I'm looking forward to seeing
it added to the document loader.

### Dependencies
polars (https://pola-rs.github.io/polars-book/user-guide/)

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-22 07:36:24 -07:00
Aashish Saini
3c4f32c8b8
Replacing Exception type from ValueError to ImportError (#9588)
I have restructured the code to ensure uniform handling of ImportError.
In place of previously used ValueError, I've adopted the standard
practice of raising ImportError with explanatory messages. This
modification enhances code readability and clarifies that any problems
stem from module importation.

@eyurtsev , @baskaryan 

Thanks
2023-08-22 07:34:05 -07:00
Julien Salinas
033b874701 Remove some deprecated text generation parameters. 2023-08-22 09:26:37 +02:00
Bagatur
4e7e6bfe0a revert 2023-08-21 18:01:49 -07:00
Bagatur
a9bf409a09 param 2023-08-21 17:37:07 -07:00
Bagatur
fa478638a9 Merge branch 'master' into bagatur/locals_in_config 2023-08-21 17:31:39 -07:00
Bagatur
182b059bf4 param 2023-08-21 17:31:38 -07:00
Bagatur
04f2d69b83
improve confluence doc loader param validation (#9568) 2023-08-21 15:02:36 -07:00
Zizhong Zhang
00eff8c4a7
feat: Add PromptGuard integration (#9481)
Add PromptGuard integration
-------
There are two approaches to integrate PromptGuard with a LangChain
application.

1. PromptGuardLLMWrapper
2. functions that can be used in LangChain expression.

-----
- Dependencies
`promptguard` python package, which is a runtime requirement if you'd
try out the demo.

- @baskaryan @hwchase17 Thanks for the ideas and suggestions along the
development process.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 14:59:36 -07:00
Sathindu
652c542b2f
fix: Imports for the ConfluenceLoader:process_page (#9432)
### Description
When we're loading documents using `ConfluenceLoader`:`load` function
and, if both `include_comments=True` and `keep_markdown_format=True`,
we're getting an error saying `NameError: free variable 'BeautifulSoup'
referenced before assignment in enclosing scope`.
    
    loader = ConfluenceLoader(url="URI", token="TOKEN")
    documents = loader.load(
        space_key="SPACE", 
        include_comments=True, 
        keep_markdown_format=True, 
    )

This happens because previous imports only consider the
`keep_markdown_format` parameter, however to include the comments, it's
using `BeautifulSoup`

Now it's fixed to handle all four scenarios considering both
`include_comments` and `keep_markdown_format`.

### Twitter
`@SathinduGA`

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 13:44:52 -07:00
Mike Salvatore
7c0b1b8171
Add session to ConfluenceLoader.__init__() (#9437)
- Description: Allows the user of `ConfluenceLoader` to pass a
`requests.Session` object in lieu of an authentication mechanism
- Issue: None
- Dependencies: None
- Tag maintainer: @hwchase17
2023-08-21 13:18:35 -07:00
Kim Minjong
3d1095218c
Update ChatOpenAI._astream to respect finish_reason (#9431)
Currently, ChatOpenAI._astream does not reflect finish_reason to
generation_info. Change it to reflect that.
2023-08-21 12:56:42 -07:00
Matthew Zeiler
949b2cf177
Improvements to the Clarifai integration (#9290)
- Improved docs
- Improved performance in multiple ways through batching, threading,
etc.
 - fixed error message 
 - Added support for metadata filtering during similarity search.

@baskaryan PTAL
2023-08-21 12:53:36 -07:00
ricki-epsilla
66a47d9a61
add Epsilla vectorstore (#9239)
[Epsilla](https://github.com/epsilla-cloud/vectordb) vectordb is an
open-source vector database that leverages the advanced academic
parallel graph traversal techniques for vector indexing.
This PR adds basic integration with
[pyepsilla](https://github.com/epsilla-cloud/epsilla-python-client)(Epsilla
vectordb python client) as a vectorstore.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 12:51:15 -07:00
Bagatur
dda5b1e370
Bagatur/doc loader confluence (#9524)
Co-authored-by: chanjetsdp <chanjetsdp@chanjet.com>
2023-08-21 12:40:44 -07:00
Predrag Gruevski
de1f63505b
Add py.typed file to langchain-experimental. (#9557)
The package is linted with mypy, so its type hints are correct and
should be exposed publicly. Without this file, the type hints remain
private and cannot be used by downstream users of the package.
2023-08-21 15:37:16 -04:00
Raynor Chavez
973866c894
fix: Updated marqo integration for marqo version 1.0.0+ (#9521)
- Description: Updated marqo integration to use tensor_fields instead of
non_tensor_fields. Upgraded marqo version to 1.2.4
  - Dependencies: marqo 1.2.4

---------

Co-authored-by: Raynor Kirkson E. Chavez <raynor.chavez@192.168.254.171>
Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 10:43:15 -07:00
Bagatur
c7a5bb6031
bump 270 (#9549) 2023-08-21 10:18:46 -07:00
Nuno Campos
28e1ee4891
Nc/small fixes 21aug (#9542)
<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
- Tag maintainer: for a quicker response, tag the relevant maintainer
(see below),
- Twitter handle: we announce bigger features on Twitter. If your PR
gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. These live is docs/extras
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
2023-08-21 18:01:20 +01:00
Bagatur
d11841d760
bump 269 (#9487) 2023-08-21 08:34:16 -07:00
axiangcoding
05aa02005b
feat(llms): support ERNIE Embedding-V1 (#9370)
- Description: support [ERNIE
Embedding-V1](https://cloud.baidu.com/doc/WENXINWORKSHOP/s/alj562vvu),
which is part of ERNIE ecology
- Issue: None
- Dependencies: None
- Tag maintainer: @baskaryan

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 07:52:25 -07:00
José Ferraz Neto
f116e10d53
Add SharePoint Loader (#4284)
- Added a loader (`SharePointLoader`) that can pull documents (`pdf`,
`docx`, `doc`) from the [SharePoint Document
Library](https://support.microsoft.com/en-us/office/what-is-a-document-library-3b5976dd-65cf-4c9e-bf5a-713c10ca2872).
- Added a Base Loader (`O365BaseLoader`) to be used for all Loaders that
use [O365](https://github.com/O365/python-o365) Package
- Code refactoring on `OneDriveLoader` to use the new `O365BaseLoader`.

---------

Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-08-21 07:49:07 -07:00
Utku Ege Tuluk
bb4f7936f9
feat(llms): add streaming support to textgen (#9295)
- Description: Added streaming support to the textgen component in the
llms module.
  - Dependencies: websocket-client = "^1.6.1"
2023-08-21 07:39:14 -07:00
Eugene Yurtsev
02c5c13a6e
Fast linters go first (#9501)
Proposal to reverse the order of linters based on the principle of
running the
fast ones first.
2023-08-21 00:20:54 -07:00
Ofer Mendelevitch
a758496236
Fixed issue with metadata in query (#9500)
- Description: Changed metadata retrieval so that it combines Vectara
doc level and part level metadata
  - Tag maintainer: @rlancemartin
  - Twitter handle: @ofermend
2023-08-20 16:00:14 -07:00
Eugene Yurtsev
e51bccdb28
Add strict flag to the JSON parser (#9471)
This updates the default configuration since I think it's almost always
what we want to happen. But we should evaluate whether there are any issues.
2023-08-19 22:02:12 -04:00
Taqi Jaffri
5cd244e9b7 CR feedback 2023-08-19 13:48:15 -07:00
Predrag Gruevski
be9bc62f8b
Fix bash test regex for Linux under WSL2. (#9475)
It fails with `Permission denied` and not `not found`. Both seem
reasonable.
2023-08-19 09:27:14 -04:00
Lorenzo
5b3dbf12a5
Uniform valid suffixes and clarify exceptions (#9463)
**Description**:
- Uniformed the current valid suffixes (file formats) for loading agents
from hubs and files (to better handle future additions);
 - Clarified exception messages (also in unit test).
2023-08-18 21:35:53 -07:00
Brendan Collins
9f545825b7
Added Geometry Validation, Geometry Metadata, and WKT instead of Python str() to GeoDataFrame Loader (#9466)
@rlancemartin The current implementation within `Geopandas.GeoDataFrame`
loader uses the python builtin `str()` function on the input geometries.
While this looks very close to WKT (Well known text), Python's str
function doesn't guarantee that.

In the interest of interop., I've changed to the of use `wkt` property
on the Shapely geometries for generating the text representation of the
geometries.

Also, included here:
- validation of the input `page_content_column` as being a GeoSeries.
- geometry `crs` (Coordinate Reference System) / bounds
(xmin/ymin/xmax/ymax) added to Document metadata. Having the CRS is
critical... having the bounds is just helpful!

I think there is a larger question of "Should the geometry live in the
`page_content`, or should the record be better summarized and tuck the
geom into metadata?" ...something for another day and another PR.
2023-08-18 21:35:39 -07:00
Kacper Łukawski
616e728ef9
Enhance qdrant vs using async embed documents (#9462)
This is an extension of #8104. I updated some of the signatures so all
the tests pass.

@danhnn I couldn't commit to your PR, so I created a new one. Thanks for
your contribution!

@baskaryan Could you please merge it?

---------

Co-authored-by: Danh Nguyen <dnncntt@gmail.com>
2023-08-18 18:59:48 -07:00
Matt Robinson
83d2a871eb
fix: apply unstructured preprocess functions (#9473)
### Summary

Fixes a bug from #7850 where post processing functions in Unstructured
loaders were not apply. Adds a assertion to the test to verify the post
processing function was applied and also updates the explanation in the
example notebook.
2023-08-18 18:54:28 -07:00
William FH
292ae8468e
Let you specify run id in trace as chain group (#9484)
I think we'll deprecate this soon anyway but still nice to be able to
fetch the run id
2023-08-18 17:21:53 -07:00
Predrag Gruevski
df8e35fd81
Remove incorrect ABC from two Elasticsearch classes. (#9470)
Neither is an ABC because their own example code instantiates them directly.
2023-08-18 15:01:02 -04:00
Predrag Gruevski
82f28ca9ef
ChatPromptTemplate is not an ABC, it's instantiated directly. (#9468)
Its own `__add__` method constructs `ChatPromptTemplate` objects
directly, it cannot be abstract.

Found while debugging something else with @nfcampos.
2023-08-18 14:37:10 -04:00
vamseeyarla
82fb56b79c
Issue 9401 - SequentialChain runs the same callbacks over and over in async mode (#9452)
Issue: https://github.com/langchain-ai/langchain/issues/9401

In the Async mode, SequentialChain implementation seems to run the same
callbacks over and over since it is re-using the same callbacks object.

Langchain version: 0.0.264, master

The implementation of this aysnc route differs from the sync route and
sync approach follows the right pattern of generating a new callbacks
object instead of re-using the old one and thus avoiding the cascading
run of callbacks at each step.

Async mode:
```
        _run_manager = run_manager or AsyncCallbackManagerForChainRun.get_noop_manager()
        callbacks = _run_manager.get_child()
        ...
        for i, chain in enumerate(self.chains):
            _input = await chain.arun(_input, callbacks=callbacks)
            ...
```

Regular mode:
```
        _run_manager = run_manager or CallbackManagerForChainRun.get_noop_manager()
        for i, chain in enumerate(self.chains):
            _input = chain.run(_input, callbacks=_run_manager.get_child(f"step_{i+1}"))
            ...
```

Notice how we are reusing the callbacks object in the Async code which
will have a cascading effect as we run through the chain. It runs the
same callbacks over and over resulting in issues.

Solution:
Define the async function in the same pattern as the regular one and
added tests.
---------

Co-authored-by: vamsee_yarlagadda <vamsee.y@airbnb.com>
2023-08-18 11:26:12 -07:00
William FH
c29fbede59
Wfh/rm num repetitions (#9425)
Makes it hard to do test run comparison views and we'd probably want to
just run multiple runs right now
2023-08-18 10:08:39 -07:00
Predrag Gruevski
eee0d1d0dd
Update repository links in the package metadata. (#9454) 2023-08-18 12:55:43 -04:00
Bagatur
50b8f4dcc7
bump 268 (#9455) 2023-08-18 08:46:39 -07:00
Nuno Campos
354c42afd2 Lint 2023-08-18 15:30:30 +01:00
Nuno Campos
4452314aab Merge branch 'master' into bagatur/locals_in_config 2023-08-18 15:23:05 +01:00
Nuno Campos
d5eb228874
Add kwargs to all other optional runnable methods (#9439)
<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
- Tag maintainer: for a quicker response, tag the relevant maintainer
(see below),
- Twitter handle: we announce bigger features on Twitter. If your PR
gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. These live is docs/extras
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
2023-08-18 15:04:26 +01:00
Leonid Ganeline
a3dd4dcadf
📖 docstrings retrievers consistency (#9422)
📜 
- updated the top-level descriptions to a consistent format;
- changed the format of several 100% internal functions from "name" to
"_name". So, these functions are not shown in the Top-level API
Reference page (with lists of classes/functions)
2023-08-18 09:20:39 -04:00
Nuno Campos
9417961b17
Add lock on tee peer cleanup (#9446)
<!-- Thank you for contributing to LangChain!

Replace this entire comment with:
  - Description: a description of the change, 
  - Issue: the issue # it fixes (if applicable),
  - Dependencies: any dependencies required for this change,
- Tag maintainer: for a quicker response, tag the relevant maintainer
(see below),
- Twitter handle: we announce bigger features on Twitter. If your PR
gets announced and you'd like a mention, we'll gladly shout you out!

Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.

See contribution guidelines for more information on how to write/run
tests, lint, etc:

https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md

If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. These live is docs/extras
directory.

If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
 -->
2023-08-18 14:20:09 +01:00
olgavrou
c9e9c0eeae add sentence transformers to extended test deps 2023-08-18 07:56:20 -04:00
olgavrou
44badd0707 add dependency requirements to test file 2023-08-18 07:19:56 -04:00
olgavrou
e276ae2616 linting and formatting 2023-08-18 07:12:39 -04:00
olgavrou
5aafb3bc46 resolving linting and formatting errors 2023-08-18 07:09:30 -04:00
Nuno Campos
d3f10d2f4f Update test 2023-08-18 11:36:16 +01:00
Nuno Campos
6ae58da668 Assign defaults in batch calls 2023-08-18 10:53:10 +01:00
olgavrou
a2f807e055 make vw dependency optional 2023-08-18 05:51:26 -04:00
olgavrou
1ae5a9c7a3 fix lock, imports, deps, test w deps, typo, formatting 2023-08-18 05:45:21 -04:00
olgavrou
e9423300d9
Merge pull request #1 from VowpalWabbit/add_rl_chain
Initial commit of rl_chain code
2023-08-22 09:18:23 -04:00
Nuno Campos
ddcb4ff5fb Li t 2023-08-18 10:30:42 +01:00
Nuno Campos
1baedc4e18 Move patch_config 2023-08-18 10:28:39 +01:00
Nuno Campos
46f3850794 Lint 2023-08-18 10:25:41 +01:00
Nuno Campos
24a197f96a Merge branch 'master' into bagatur/locals_in_config 2023-08-18 10:12:10 +01:00