- **Description:**
be able to use langchain with other version than tiktoken 0.3.3 i.e
0.5.1
- **Issue:**
cannot installed the conda-forge version since it applied all optional
dependency:
https://github.com/conda-forge/langchain-feedstock/pull/85
replace "^0.3.2" by "">=0.3.2,<0.6.0" and "^3.9" by python=">=3.9"
Tested with python 3.10, langchain=0.0.288 and tiktoken==0.5.0
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
## Description
As of now, when instantiating and during inference, `LlamaCppEmbeddings`
outputs (a lot of) verbose when controlled from Langchain binding - it
is a bit annoying when computing the embeddings of long documents, for
instance.
This PR adds `verbose` for `LlamaCppEmbeddings` objects to be able
**not** to print the verbose of the model to `stderr`. It is natively
supported by `llama-cpp-python` and directly passed to the library – the
PR is hence very small.
The value of `verbose` is `True` by default, following the way it is
defined in [`LlamaCpp` (`llamacpp.py`
#L136-L137)](c87e9fb2ce/libs/langchain/langchain/llms/llamacpp.py (L136-L137))
## Issue
_No issue linked_
## Dependencies
_No additional dependency needed_
## To see it in action
```python
from langchain.embeddings import LlamaCppEmbeddings
MODEL_PATH = "<path_to_gguf_file>"
if __name__ == "__main__":
llm_embeddings = LlamaCppEmbeddings(
model_path=MODEL_PATH,
n_gpu_layers=1,
n_batch=512,
n_ctx=2048,
f16_kv=True,
verbose=False,
)
```
Co-authored-by: Bagatur <baskaryan@gmail.com>
# Description
Adds logic for NotionDBLoader to correctly populate `last_edited_time`
and `created_time` fields from [page
properties](https://developers.notion.com/reference/page#property-value-object).
There are no relevant tests for this code to be updated.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Based on the customers' requests for native langchain integration,
SearchApi is ready to invest in AI and LLM space, especially in
open-source development.
- This is our initial PR and later we want to improve it based on
customers' and langchain users' feedback. Most likely changes will
affect how the final results string is being built.
- We are creating similar native integration in Python and JavaScript.
- The next plan is to integrate into Java, Ruby, Go, and others.
- Feel free to assign @SebastjanPrachovskij as a main reviewer for any
SearchApi-related searches. We will be glad to help and support
langchain development.
- **Description:**
- Make running integration test for opensearch easy
- Provide a way to use different text for embedding: refer to #11002 for
more of the use case and design decision.
- **Issue:** N/A
- **Dependencies:** None other than the existing ones.
Both black and mypy expect a list of files or directories as input.
As-is the Makefile computes a list files changed relative to the last
commit; these are passed to black and mypy in the `format_diff` and
`lint_diff` targets. This is done by way of the Makefile variable
`PYTHON_FILES`. This is to save time by skipping running mypy and black
over the whole source tree.
When no changes have been made, this variable is empty, so the call to
black (and mypy) lacks input files. The call exits with error causing
the Makefile target to error out with:
```bash
$ make format_diff
poetry run black
Usage: black [OPTIONS] SRC ...
One of 'SRC' or 'code' is required.
make: *** [format_diff] Error 1
```
This is unexpected and undesirable, as the naive caller (that's me! 😄 )
will think something else is wrong. This commit smooths over this by
short circuiting when `PYTHON_FILES` is empty.
- **Description:** The types of 'destination_chains' and 'default_chain'
in 'MultiPromptChain' were changed from 'LLMChain' to 'Chain'. and
removed variables declared overlapping with the parent class
- **Issue:** When a class that inherits only Chain and not LLMChain,
such as 'SequentialChain' or 'RetrievalQA', is entered in
'destination_chains' and 'default_chain', a pydantic validation error is
raised.
- - codes
```
retrieval_chain = ConversationalRetrievalChain(
retriever=doc_retriever,
combine_docs_chain=combine_docs_chain,
question_generator=question_gen_chain,
)
destination_chains = {
'retrieval': retrieval_chain,
}
main_chain = MultiPromptChain(
router_chain=router_chain,
destination_chains=destination_chains,
default_chain=default_chain,
verbose=True,
)
```
✅ `make format`, `make lint` and `make test`
## Description
Expanded the upper bound for `networkx` dependency to allow installation
of latest stable version. Tested the included sample notebook with
version 3.1, and all steps ran successfully.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Adds support for the `$vectorSearch` operator for
MongoDBAtlasVectorSearch, which was announced at .Local London
(September 26th, 2023). This change maintains breaks compatibility
support for the existing `$search` operator used by the original
integration (https://github.com/langchain-ai/langchain/pull/5338) due to
incompatibilities in the Atlas search implementations.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
We noticed that as we have been moving developers to the new
`ElasticsearchStore` implementation, we want to keep the
ElasticVectorSearch class still available as developers transition
slowly to the new store.
To speed up this process, I updated the blurb giving them a better
recommendation of why they should use ElasticsearchStore.
Description: Add "source" metadata to OutlookMessageLoader
This pull request adds the "source" metadata to the OutlookMessageLoader
class in the load method. The "source" metadata is required when
indexing with RecordManager in order to sync the index documents with a
source.
Issue: None
Dependencies: None
Twitter handle: @ATelders
Co-authored-by: Arthur Telders <arthur.telders@roquette.com>
- **Description:** Bedrock updated boto service name to
"bedrock-runtime" for the InvokeModel and InvokeModelWithResponseStream
APIs. This update also includes new model identifiers for Titan text,
embedding and Anthropic.
Co-authored-by: Mani Kumar Adari <maniadar@amazon.com>
The key of stopping strings used in text-generation-webui api is
[`stopping_strings`](https://github.com/oobabooga/text-generation-webui/blob/main/api-examples/api-example.py#L51),
not `stop`.
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
- **Description:** Changed data type from `text` to `json` in xata for
improved performance. Also corrected the `additionalKwargs` key in the
`messages()` function to `additional_kwargs` to adhere to `BaseMessage`
requirements.
- **Issue:** The Chathisroty.messages() will return {} of
`additional_kwargs`, as the name is wrong for `additionalKwargs` .
- **Dependencies:** N/A
- **Tag maintainer:** N/A
- **Twitter handle:** N/A
My PR is passing linting and testing before submitting.
This adds `input_schema` and `output_schema` properties to all
runnables, which are Pydantic models for the input and output types
respectively. These are inferred from the structure of the Runnable as
much as possible, the only manual typing needed is
- optionally add type hints to lambdas (which get translated to
input/output schemas)
- optionally add type hint to RunnablePassthrough
These schemas can then be used to create JSON Schema descriptions of
input and output types, see the tests
- [x] Ensure no InputType and OutputType in our classes use abstract
base classes (replace with union of subclasses)
- [x] Implement in BaseChain and LLMChain
- [x] Implement in RunnableBranch
- [x] Implement in RunnableBinding, RunnableMap, RunnablePassthrough,
RunnableEach, RunnableRouter
- [x] Implement in LLM, Prompt, Chat Model, Output Parser, Retriever
- [x] Implement in RunnableLambda from function signature
- [x] Implement in Tool
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
Adds LangServe package
* Integrate Runnables with Fast API creating Server and a RemoteRunnable
client
* Support multiple runnables for a given server
* Support sync/async/batch/abatch/stream/astream/astream_log on the
client side (using async implementations on server)
* Adds validation using annotations (relying on pydantic under the hood)
-- this still has some rough edges -- e.g., open api docs do NOT
generate correctly at the moment
* Uses pydantic v1 namespace
Known issues: type translation code doesn't handle a lot of types (e.g.,
TypedDicts)
---------
Co-authored-by: Bagatur <22008038+baskaryan@users.noreply.github.com>
The current behaviour just calls the handler without awaiting the
coroutine, which results in exceptions/warnings, and obviously doesn't
actually execute whatever the callback handler does
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
- **Description:** Prompt wrapping requirements have been implemented on
the service side of AWS Bedrock for the Anthropic Claude models to
provide parity between Anthropic's offering and Bedrock's offering. This
overnight change broke most existing implementations of Claude, Bedrock
and Langchain. This PR just steals the the Anthropic LLM implementation
to enforce alias/role wrapping and implements it in the existing
mechanism for building the request body. This has also been tested to
fix the chat_model implementation as well. Happy to answer any further
questions or make changes where necessary to get things patched and up
to PyPi ASAP, TY.
- **Issue:** No issue opened at the moment, though will update when
these roll in.
- **Dependencies:** None
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
Co-authored-by: Bagatur <baskaryan@gmail.com>
### Description:
NotionDB supports a number of common property types. I have found three
common types that are not included in notiondb loader. When programs
loaded them with notiondb, which will cause some metadata information
not to be passed to langchain. Therefore, I added three common types:
- date
- created_time
- last_edit_time.
### Issue:
no
### Dependencies:
No dependencies added :)
### Tag maintainer:
@rlancemartin, @eyurtsev
### Twitter handle:
@BJTUTC
Reverts langchain-ai/langchain#8610
this is actually an oversight - this merges all dfs into one df. we DO
NOT want to do this - the idea is we work and manipulate multiple dfs
This removes the use of the intermediate df list and directly
concatenates the dataframes if path is a list of strings. The pd.concat
function combines the dataframes efficiently, making it faster and more
memory-efficient compared to appending dataframes to a list.
<!-- Thank you for contributing to LangChain!
Replace this comment with:
- Description: a description of the change,
- Issue: the issue # it fixes (if applicable),
- Dependencies: any dependencies required for this change,
- Tag maintainer: for a quicker response, tag the relevant maintainer
(see below),
- Twitter handle: we announce bigger features on Twitter. If your PR
gets announced and you'd like a mention, we'll gladly shout you out!
Please make sure you're PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use.
Maintainer responsibilities:
- General / Misc / if you don't know who to tag: @baskaryan
- DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev
- Models / Prompts: @hwchase17, @baskaryan
- Memory: @hwchase17
- Agents / Tools / Toolkits: @hinthornw
- Tracing / Callbacks: @agola11
- Async: @agola11
If no one reviews your PR within a few days, feel free to @-mention the
same people again.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
-->
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
- Description: this PR adds the support for arxiv identifier of the
ArxivAPIWrapper. I modified the `run()` and `load()` functions in
`arxiv.py`, using regex to recognize if the query is in the form of
arxiv identifier (see
[https://info.arxiv.org/help/find/index.html](https://info.arxiv.org/help/find/index.html)).
If so, it will directly search the paper corresponding to the arxiv
identifier. I also modified and added tests in `test_arxiv.py`.
- Issue: #9047
- Dependencies: N/A
- Tag maintainer: N/A
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
The new Fireworks and FireworksChat implementations are awesome! Added
in this PR https://github.com/langchain-ai/langchain/pull/11117 thank
you @ZixinYang
However, I think stop words were not plumbed correctly. I've made some
simple changes to do that, and also updated the notebook to be a bit
clearer with what's needed to use both new models.
---------
Co-authored-by: Taqi Jaffri <tjaffri@docugami.com>
**Description:**
As long as `enforce_stop_tokens` returns a first occurrence, we can
speed up the execution by setting the optional `maxsplit` parameter to
1.
Tag maintainer:
@agola11
@hwchase17
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
**Description:** New metadata fields were added to
`unstructured==0.10.15`, and our hosted api has been updated to reflect
this. When users call `partition_via_api` with an older version of the
library, they'll hit a parsing error related to the new fields.
Description
* Refactor Fireworks within Langchain LLMs.
* Remove FireworksChat within Langchain LLMs.
* Add ChatFireworks (which uses chat completion api) to Langchain chat
models.
* Users have to install `fireworks-ai` and register an api key to use
the api.
Issue - Not applicable
Dependencies - None
Tag maintainer - @rlancemartin @baskaryan
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:**: Adds LLM as a judge as an eval chain
- **Tag maintainer:** @hwchase17
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
---------
Co-authored-by: William FH <13333726+hinthornw@users.noreply.github.com>
This enables bulk args like `chunk_size` to be passed down from the
ingest methods (from_text, from_documents) to be passed down to the bulk
API.
This helps alleviate issues where bulk importing a large amount of
documents into Elasticsearch was resulting in a timeout.
Contribution Shoutout
- @elastic
- [x] Updated Integration tests
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Sometimes you don't want the LLM to be aware of the whole graph schema,
and want it to ignore parts of the graph when it is constructing Cypher
statements.
- **Description**: Adding retrievers for [kay.ai](https://kay.ai) and
SEC filings powered by Kay and Cybersyn. Kay provides context as a
service: it's an API built for RAG.
- **Issue**: N/A
- **Dependencies**: Just added a dep to the
[kay](https://pypi.org/project/kay/) package
- **Tag maintainer**: @baskaryan @hwchase17 Discussed in slack
- **Twtter handle:** [@vishalrohra_](https://twitter.com/vishalrohra_)
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
The huggingface pipeline in langchain (used for locally hosted models)
does not support batching. If you send in a batch of prompts, it just
processes them serially using the base implementation of _generate:
https://github.com/docugami/langchain/blob/master/libs/langchain/langchain/llms/base.py#L1004C2-L1004C29
This PR adds support for batching in this pipeline, so that GPUs can be
fully saturated. I updated the accompanying notebook to show GPU batch
inference.
---------
Co-authored-by: Taqi Jaffri <tjaffri@docugami.com>
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
Closes#8842
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- Description: a description of the change,
- Issue: the issue # it fixes (if applicable),
- Dependencies: any dependencies required for this change,
- Tag maintainer: for a quicker response, tag the relevant maintainer
(see below),
- Twitter handle: we announce bigger features on Twitter. If your PR
gets announced and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. These live is docs/extras
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17, @rlancemartin.
-->
- Description: fix `ChatMessageChunk` concat error
- Issue: #10173
- Dependencies: None
- Tag maintainer: @baskaryan, @eyurtsev, @rlancemartin
- Twitter handle: None
---------
Co-authored-by: wangshuai.scotty <wangshuai.scotty@bytedance.com>
Co-authored-by: Nuno Campos <nuno@boringbits.io>
### Description
This PR makes the following changes to OpenSearch:
1. Pass optional ids with `from_texts`
2. Pass an optional index name with `add_texts` and `search` instead of
using the same index name that was used during `from_texts`
### Issue
https://github.com/langchain-ai/langchain/issues/10967
### Maintainers
@rlancemartin, @eyurtsev, @navneet1v
Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
LLMRails Embedding Integration
This PR provides integration with LLMRails. Implemented here are:
langchain/embeddings/llm_rails.py
docs/extras/integrations/text_embedding/llm_rails.ipynb
Hi @hwchase17 after adding our vectorstore integration to langchain with
confirmation of you and @baskaryan, now we want to add our embedding
integration
---------
Co-authored-by: Anar Aliyev <aaliyev@mgmt.cloudnet.services>
Co-authored-by: Bagatur <baskaryan@gmail.com>
Adds support for gradient.ai's embedding model.
This will remain a Draft, as the code will likely be refactored with the
`pip install gradientai` python sdk.
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
- **Description:** a fix for `index`.
- **Issue:** Not applicable.
- **Dependencies:** None
- **Tag maintainer:**
- **Twitter handle:** richarddwang
# Problem
Replication code
```python
from pprint import pprint
from langchain.embeddings import OpenAIEmbeddings
from langchain.indexes import SQLRecordManager, index
from langchain.schema import Document
from langchain.vectorstores import Qdrant
from langchain_setup.qdrant import pprint_qdrant_documents, create_inmemory_empty_qdrant
# Documents
metadata1 = {"source": "fullhell.alchemist"}
doc1_1 = Document(page_content="1-1 I have a dog~", metadata=metadata1)
doc1_2 = Document(page_content="1-2 I have a daugter~", metadata=metadata1)
doc1_3 = Document(page_content="1-3 Ahh! O..Oniichan", metadata=metadata1)
doc2 = Document(page_content="2 Lancer died again.", metadata={"source": "fate.docx"})
# Create empty vectorstore
collection_name = "secret_of_D_disk"
vectorstore: Qdrant = create_inmemory_empty_qdrant()
# Create record Manager
import tempfile
from pathlib import Path
record_manager = SQLRecordManager(
namespace="qdrant/{collection_name}",
db_url=f"sqlite:///{Path(tempfile.gettempdir())/collection_name}.sql",
)
record_manager.create_schema() # 必須
sync_result = index(
[doc1_1, doc1_2, doc1_2, doc2],
record_manager,
vectorstore,
cleanup="full",
source_id_key="source",
)
print(sync_result, end="\n\n")
pprint_qdrant_documents(vectorstore)
```
<details>
<summary>Code of helper functions `pprint_qdrant_documents` and
`create_inmemory_empty_qdrant`</summary>
```python
def create_inmemory_empty_qdrant(**from_texts_kwargs):
# Qdrant requires vector size, which can be only know after applying embedder
vectorstore = Qdrant.from_texts(["dummy"], location=":memory:", embedding=OpenAIEmbeddings(), **from_texts_kwargs)
dummy_document_id = vectorstore.client.scroll(vectorstore.collection_name)[0][0].id
vectorstore.delete([dummy_document_id])
return vectorstore
def pprint_qdrant_documents(vectorstore, limit: int = 100, **scroll_kwargs):
document_ids, documents = [], []
for record in vectorstore.client.scroll(
vectorstore.collection_name, limit=100, **scroll_kwargs
)[0]:
document_ids.append(record.id)
documents.append(
Document(
page_content=record.payload["page_content"],
metadata=record.payload["metadata"] or {},
)
)
pprint_documents(documents, document_ids=document_ids)
def pprint_document(document: Document = None, document_id=None, return_string=False):
displayed_text = ""
if document_id:
displayed_text += f"Document {document_id}:\n\n"
displayed_text += f"{document.page_content}\n\n"
metadata_text = pformat(document.metadata, indent=1)
if "\n" in metadata_text:
displayed_text += f"Metadata:\n{metadata_text}"
else:
displayed_text += f"Metadata:{metadata_text}"
if return_string:
return displayed_text
else:
print(displayed_text)
def pprint_documents(documents, document_ids=None):
if not document_ids:
document_ids = [i + 1 for i in range(len(documents))]
displayed_texts = []
for document_id, document in zip(document_ids, documents):
displayed_text = pprint_document(
document_id=document_id, document=document, return_string=True
)
displayed_texts.append(displayed_text)
print(f"\n{'-' * 100}\n".join(displayed_texts))
```
</details>
You will get
```
{'num_added': 3, 'num_updated': 0, 'num_skipped': 0, 'num_deleted': 0}
Document 1b19816e-b802-53c0-ad60-5ff9d9b9b911:
1-2 I have a daugter~
Metadata:{'source': 'fullhell.alchemist'}
----------------------------------------------------------------------------------------------------
Document 3362f9bc-991a-5dd5-b465-c564786ce19c:
1-1 I have a dog~
Metadata:{'source': 'fullhell.alchemist'}
----------------------------------------------------------------------------------------------------
Document a4d50169-2fda-5339-a196-249b5f54a0de:
1-2 I have a daugter~
Metadata:{'source': 'fullhell.alchemist'}
```
This is not correct. We should be able to expect that the vectorsotre
now includes doc1_1, doc1_2, and doc2, but not doc1_1, doc1_2, and
doc1_2.
# Reason
In `index`, the original code is
```python
uids = []
docs_to_index = []
for doc, hashed_doc, doc_exists in zip(doc_batch, hashed_docs, exists_batch):
if doc_exists:
# Must be updated to refresh timestamp.
record_manager.update([hashed_doc.uid], time_at_least=index_start_dt)
num_skipped += 1
continue
uids.append(hashed_doc.uid)
docs_to_index.append(doc)
```
In the aforementioned example, `len(doc_batch) == 4`, but
`len(hashed_docs) == len(exists_batch) == 3`. This is because the
deduplication of input documents [doc1_1, doc1_2, doc1_2, doc2] is
[doc1_1, doc1_2, doc2]. So `index` insert doc1_1, doc1_2, doc1_2 with
the uid of doc1_1, doc1_2, doc2.
---------
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
This PR makes `ChatAnthropic.anthropic_api_key` a `pydantic.SecretStr`
to avoid inadvertently exposing API keys when the `ChatAnthropic` object
is represented as a str.
**Description**
Fixes broken link to `CONTRIBUTING.md` in `libs/langchain/README.md`.
Because`libs/langchain/README.md` was copied from the top level README,
and because the README contains a link to `.github/CONTRIBUTING.md`, the
copied README's link relative path must be updated. This commit fixes
that link.
**Description:**
Default refine template does not actually use the refine template
defined above, it uses a string with the variable name.
@baskaryan, @eyurtsev, @hwchase17
- chat vertex async
- vertex stream
- vertex full generation info
- vertex use server-side stopping
- model garden async
- update docs for all the above
in follow up will add
[] chat vertex full generation info
[] chat vertex retries
[] scheduled tests
**Description:**
This commit adds a vector store for the Postgres-based vector database
(`TimescaleVector`).
Timescale Vector(https://www.timescale.com/ai) is PostgreSQL++ for AI
applications. It enables you to efficiently store and query billions of
vector embeddings in `PostgreSQL`:
- Enhances `pgvector` with faster and more accurate similarity search on
1B+ vectors via DiskANN inspired indexing algorithm.
- Enables fast time-based vector search via automatic time-based
partitioning and indexing.
- Provides a familiar SQL interface for querying vector embeddings and
relational data.
Timescale Vector scales with you from POC to production:
- Simplifies operations by enabling you to store relational metadata,
vector embeddings, and time-series data in a single database.
- Benefits from rock-solid PostgreSQL foundation with enterprise-grade
feature liked streaming backups and replication, high-availability and
row-level security.
- Enables a worry-free experience with enterprise-grade security and
compliance.
Timescale Vector is available on Timescale, the cloud PostgreSQL
platform. (There is no self-hosted version at this time.) LangChain
users get a 90-day free trial for Timescale Vector.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Co-authored-by: Avthar Sewrathan <avthar@timescale.com>
- **Description:** This PR implements a new LLM API to
https://gradient.ai
- **Issue:** Feature request for LLM #10745
- **Dependencies**: No additional dependencies are introduced.
- **Tag maintainer:** I am opening this PR for visibility, once ready
for review I'll tag.
- ```make format && make lint && make test``` is running.
- added a `integration` and `mock unit` test.
Co-authored-by: michaelfeil <me@michaelfeil.eu>
Co-authored-by: Bagatur <baskaryan@gmail.com>
We are introducing the py integration to Javelin AI Gateway
www.getjavelin.io. Javelin is an enterprise-scale fast llm router &
gateway. Could you please review and let us know if there is anything
missing.
Javelin AI Gateway wraps Embedding, Chat and Completion LLMs. Uses
javelin_sdk under the covers (pip install javelin_sdk).
Author: Sharath Rajasekar, Twitter: @sharathr, @javelinai
Thanks!!
### Description
- Add support for streaming with `Bedrock` LLM and `BedrockChat` Chat
Model.
- Bedrock as of now supports streaming for the `anthropic.claude-*` and
`amazon.titan-*` models only, hence support for those have been built.
- Also increased the default `max_token_to_sample` for Bedrock
`anthropic` model provider to `256` from `50` to keep in line with the
`Anthropic` defaults.
- Added examples for streaming responses to the bedrock example
notebooks.
**_NOTE:_**: This PR fixes the issues mentioned in #9897 and makes that
PR redundant.
- **Description:** QianfanEndpoint bugs for SystemMessages. When the
`SystemMessage` is input as the messages to
`chat_models.QianfanEndpoint`. A `TypeError` will be raised.
- **Issue:** #10643
- **Dependencies:**
- **Tag maintainer:** @baskaryan
- **Twitter handle:** no
This PR addresses the limitation of Azure OpenAI embeddings, which can
handle at maximum 16 texts in a batch. This can be solved setting
`chunk_size=16`. However, I'd love to have this automated, not to force
the user to figure where the issue comes from and how to solve it.
Closes#4575.
@baskaryan
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
**Description:** Possible to filter with substrings in
similarity_search_with_score, for example: filter={'user_id':
{'substring': 'user'}}
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
**Description:**
changed return parameter of YouTubeSearchTool
1. changed the returning links of youtube videos by adding prefix
"https://www.youtube.com", now this will return the exact links to the
videos
2. updated the returning type from 'string' to 'list', which will be
more suited for further processings
**Issue:**
Fixes#10742
**Dependencies:**
None
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** changed return parameter of YouTubeSearchTool
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** None
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
**Description:** This PR adds HTTP PUT support for the langchain openapi
agent toolkit by leveraging existing structure and HTTP put request
wrapper. The PUT method is almost identical to HTTP POST but should be
idempotent and therefore tighter than POST which is not idempotent. Some
APIs may consider to use PUT instead of POST which is unfortunately not
supported with the current toolkit yet.
### Description
Implements synthetic data generation with the fields and preferences
given by the user. Adds showcase notebook.
Corresponding prompt was proposed for langchain-hub.
### Example
```
output = chain({"fields": {"colors": ["blue", "yellow"]}, "preferences": {"style": "Make it in a style of a weather forecast."}})
print(output)
# {'fields': {'colors': ['blue', 'yellow']},
'preferences': {'style': 'Make it in a style of a weather forecast.'},
'text': "Good morning! Today's weather forecast brings a beautiful combination of colors to the sky, with hues of blue and yellow gently blending together like a mesmerizing painting."}
```
### Twitter handle
@deepsense_ai @matt_wosinski
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
**Description:** upgrade the `dataclasses_json` dependency to its latest
version ([no real breaking
change](https://github.com/lidatong/dataclasses-json/releases/tag/v0.6.0)
if used correctly), while allowing previous version to not break other
users' setup
**Issue:** I need to use the latest version of that dependency in my
project, but `langchain` prevents it.
Note: it looks like running `poetry lock --no-update` did some changes
to the lockfiles as it was the first time it was with the
`macosx_11_0_arm64` architecture 🤷
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
**Description**
Adds new output parser, this time enabling the output of LLM to be of an
XML format. Seems to be particularly useful together with Claude model.
Addresses [issue
9820](https://github.com/langchain-ai/langchain/issues/9820).
**Twitter handle**
@deepsense_ai @matt_wosinski
using sample:
```
endpoint_url = API URL
ChatGLM_llm = ChatGLM(
endpoint_url=endpoint_url,
api_key=Your API Key by ChatGLM
)
print(ChatGLM_llm("hello"))
```
```
model = ChatChatGLM(
chatglm_api_key="api_key",
chatglm_api_base="api_base_url",
model_name="model_name"
)
chain = LLMChain(llm=model)
```
Description: The call of ChatGLM has been adapted.
Issue: The call of ChatGLM has been adapted.
Dependencies: Need python package `zhipuai` and `aiostream`
Tag maintainer: @baskaryan
Twitter handle: None
I remove the compatibility test for pydantic version 2, because pydantic
v2 can't not pickle classmethod,but BaseModel use @root_validator is a
classmethod decorator.
---------
Co-authored-by: Bagatur <baskaryan@gmail.com>
Description:
If metadata field returned in results, previous behavior unchanged. If
metadata field does not exist in results, expand metadata to any fields
returned outside of content field.
There's precedence for this as well, see the retriever:
https://github.com/langchain-ai/langchain/blob/master/libs/langchain/langchain/retrievers/azure_cognitive_search.py#L96C46-L96C46
Issue:
#9765 - Ameliorates hard-coding in case you already indexed to cognitive
search without a metadata field but rather placed metadata in separate
fields.
@hwchase17
## Description
This PR updates the `NeptuneGraph` class to start using the boto API for
connecting to the Neptune service. With boto integration, the graph
class now supports authenticating requests using Sigv4; this is
encapsulated with the boto API, and users only have to ensure they have
the correct AWS credentials setup in their workspace to work with the
graph class.
This PR also introduces a conditional prompt that uses a simpler prompt
when using the `Anthropic` model provider. A simpler prompt have seemed
to work better for generating cypher queries in our testing.
**Note**: This version will require boto3 version 1.28.38 or greater to
work.
**Description:**
This commit enriches the `WeaviateHybridSearchRetriever` class by
introducing a new parameter, `hybrid_search_kwargs`, within the
`_get_relevant_documents` method. This parameter accommodates arbitrary
keyword arguments (`**kwargs`) which can be channeled to the inherited
public method, `get_relevant_documents`, originating from the
`BaseRetriever` class.
This modification facilitates more intricate querying capabilities,
allowing users to convey supplementary arguments to the `.with_hybrid()`
method. This expansion not only makes it possible to perform a more
nuanced search targeting specific properties but also grants the ability
to boost the weight of searched properties, to carry out a search with a
custom vector, and to apply the Fusion ranking method. The documentation
has been updated accordingly to delineate these new possibilities in
detail.
In light of the layered approach in which this search operates,
initiating with `query.get()` and then transitioning to
`.with_hybrid()`, several advantageous opportunities are unlocked for
the hybrid component that were previously unattainable.
Here’s a representative example showcasing a query structure that was
formerly unfeasible:
[Specific Properties
Only](https://weaviate.io/developers/weaviate/search/hybrid#selected-properties-only)
"The example below illustrates a BM25 search targeting the keyword
'food' exclusively within the 'question' property, integrated with
vector search results corresponding to 'food'."
```python
response = (
client.query
.get("JeopardyQuestion", ["question", "answer"])
.with_hybrid(
query="food",
properties=["question"], # Will now be possible moving forward
alpha=0.25
)
.with_limit(3)
.do()
)
```
This functionality is now accessible through my alterations, by
conveying `hybrid_search_kwargs={"properties": ["question", "answer"]}`
as an argument to
`WeaviateHybridSearchRetriever.get_relevant_documents()`. For example:
```python
import os
from weaviate import Client
from langchain.retrievers import WeaviateHybridSearchRetriever
client = Client(
url=os.getenv("WEAVIATE_CLIENT_URL"),
additional_headers={
"X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY"),
"Authorization": f"Bearer {os.getenv('WEAVIATE_API_KEY')}",
},
)
index_name = "Document"
text_key = "content"
attributes = ["title", "summary", "header", "url"]
retriever = ExtendedWeaviateHybridSearchRetriever(
client=client,
index_name=index_name,
text_key=text_key,
attributes=attributes,
)
# Warning: to utilize properties in this way, each use property must also be in the list `attributes + [text_key]`.
hybrid_search_kwargs = {"properties": ["summary^2", "content"]}
query_text = "Some Query Text"
relevant_docs = retriever.get_relevant_documents(
query=query_text,
hybrid_search_kwargs=hybrid_search_kwargs
)
```
In my experience working with the `weaviate-client` library, I have
found that these supplementary options stand as vital tools for
refining/finetuning searches, notably within multifaceted datasets. As a
final note, this implementation supports both backwards and forward
(within reason) compatiblity. It accommodates any future additional
parameters Weaviate may add to `.with_hybrid()`, without necessitating
further alterations.
**Additional Documentation:**
For a more comprehensive understanding and to explore a myriad of useful
options that are now accessible, please refer to the Weaviate
documentation:
- [Fusion Ranking
Method](https://weaviate.io/developers/weaviate/search/hybrid#fusion-ranking-method)
- [Selected Properties
Only](https://weaviate.io/developers/weaviate/search/hybrid#selected-properties-only)
- [Weight Boost Searched
Properties](https://weaviate.io/developers/weaviate/search/hybrid#weight-boost-searched-properties)
- [With a Custom
Vector](https://weaviate.io/developers/weaviate/search/hybrid#with-a-custom-vector)
**Tag Maintainer:**
@hwchase17 - I have tagged you based on your frequent contributions to
the pertinent file, `/retrievers/weaviate_hybrid_search.py`. My
apologies if this was not the appropriate choice.
Thank you for considering my contribution, I look forward to your
feedback, and to future collaboration.
I was trying to use web loaders on some spanish documentation (e.g.
[this site](https://www.fromdoppler.com/es/mailing-tendencias/), but the
auto-encoding introduced in
https://github.com/langchain-ai/langchain/pull/3602 was detected as
"MacRoman" instead of the (correct) "UTF-8".
To address this, I've added the ability to disable the auto-encoding, as
well as the ability to explicitly tell the loader what encoding to use.
- **Description:** Makes auto-setting the encoding optional in
`WebBaseLoader`, and introduces an `encoding` option to explicitly set
it.
- **Dependencies:** N/A
- **Tag maintainer:** @hwchase17
- **Twitter handle:** @czue
**Description:**
Pinecone hybrid search is now limited to default namespace. There is no
option for the user to provide a namespace to partition an index, which
is one of the most important features of pinecone.
**Resource:**
https://docs.pinecone.io/docs/namespaces
---------
Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
- **Description:** Updating URL in Context Callback Docstrings and
update metadata key Context CallbackHandler uses to send model names.
- **Issue:** The URL in ContextCallbackHandler is out of date. Model
data being sent to Context should be under the "model" key and not
"llm_model". This allows Context to do more sophisticated analysis.
- **Dependencies:** None
Tagging @agamble.
- This pr adds `llm_kwargs` to the initialization of Xinference LLMs
(integrated in #8171 ).
- With this enhancement, users can not only provide `generate_configs`
when calling the llms for generation but also during the initialization
process. This allows users to include custom configurations when
utilizing LangChain features like LLMChain.
- It also fixes some format issues for the docstrings.
Hello @hwchase17
**Issue**:
The class WebResearchRetriever accept only
RecursiveCharacterTextSplitter, but never uses a specification of this
class. I propose to change the type to TextSplitter. Then, the lint can
accept all subtypes.
- tools invoked in async methods would not work due to missing await
- RunnableSequence.stream() was creating an extra root run by mistake,
and it can simplified due to existence of default implementation for
.transform()
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
**Description:** Renamed argument `database` in
`SQLDatabaseSequentialChain.from_llm()` to `db`,
I realize it's tiny and a bit of a nitpick but for consistency with
SQLDatabaseChain (and all the others actually) I thought it should be
renamed. Also got me while working and using it today.
✔️ Please make sure your PR is passing linting and
testing before submitting. Run `make format`, `make lint` and `make
test` to check this locally.
This PR is a documentation fix.
Description:
* fixes imports in the code samples in the docstrings of
`create_openai_fn_chain` and `create_structured_output_chain`
* fixes imports in
`docs/extras/modules/chains/how_to/openai_functions.ipynb`
* removes unused imports from the notebook
Issues:
* the docstrings use `from pydantic_v1 import BaseModel, Field` which
this PR changes to `from langchain.pydantic_v1 import BaseModel, Field`
* importing `pydantic` instead of `langchain.pydantic_v1` leads to
errors later in the notebook
<!-- Thank you for contributing to LangChain!
Replace this entire comment with:
- **Description:** a description of the change,
- **Issue:** the issue # it fixes (if applicable),
- **Dependencies:** any dependencies required for this change,
- **Tag maintainer:** for a quicker response, tag the relevant
maintainer (see below),
- **Twitter handle:** we announce bigger features on Twitter. If your PR
gets announced, and you'd like a mention, we'll gladly shout you out!
Please make sure your PR is passing linting and testing before
submitting. Run `make format`, `make lint` and `make test` to check this
locally.
See contribution guidelines for more information on how to write/run
tests, lint, etc:
https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md
If you're adding a new integration, please include:
1. a test for the integration, preferably unit tests that do not rely on
network access,
2. an example notebook showing its use. It lives in `docs/extras`
directory.
If no one reviews your PR within a few days, please @-mention one of
@baskaryan, @eyurtsev, @hwchase17.
-->
