Archives/langchain
2
0
Fork 1
mirror of https://github.com/hwchase17/langchain synced 2024-11-18 09:25:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
233b52735e
langchain/pyproject.toml

351 lines
11 KiB
TOML
Raw Normal View History

chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
[tool.poetry]
name = "langchain"
bump version to 190 (#5704)
2023-06-05 03:04:50 +00:00
version = "0.0.190"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
description = "Building applications with LLMs through composability"
authors = []
license = "MIT"
readme = "README.md"
repository = "https://www.github.com/hwchase17/langchain"
add tracing support to langchain (#741) * add implementations of `BaseCallbackHandler` to support tracing: `SharedTracer` which is thread-safe and `Tracer` which is not and is meant to be used locally. * Tracers persist runs to locally running `langchain-server` Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-01-27 01:38:13 +00:00
[tool.poetry.scripts]
langchain-server = "langchain.server:main"
Add Server Command (#4695) Add Support for `langchain server {start|stop}` commands, with support for using ngrok to tunnel to a remote notebook
2023-05-16 00:44:30 +00:00
langchain = "langchain.cli.main:main"
add tracing support to langchain (#741) * add implementations of `BaseCallbackHandler` to support tracing: `SharedTracer` which is thread-safe and `Tracer` which is not and is meant to be used locally. * Tracers persist runs to locally running `langchain-server` Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-01-27 01:38:13 +00:00
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
[tool.poetry.dependencies]
python = ">=3.8.1,<4.0"
pydantic = "^1"
Vwp/sqlalchemy (#4145) Bump threshold to 1.4 from 1.3. Change import to be compatible Resolves #4142 and #4129 --------- Co-authored-by: ndaugreal <ndaugreal@gmail.com> Co-authored-by: Jeremy Lopez <lopez86@users.noreply.github.com>
2023-05-05 03:46:38 +00:00
SQLAlchemy = ">=1.4,<3"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
requests = "^2"
Loosen PyYAML dependency (#1698) Hitting some dependency issues relating to this strict pinning. Unsure of the knock-on effects, but wanted to propose this loosening down a couple of versions.
2023-03-17 00:05:36 +00:00
PyYAML = ">=5.4.1"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
numpy = "^1"
bump version to 151 (#3658)
2023-04-27 16:02:39 +00:00
azure-core = {version = "^1.26.4", optional=true}
Harrison/site map tqdm (#3184) Co-authored-by: Tianyi Pan <60060750+tipani86@users.noreply.github.com> Co-authored-by: Tianyi Pan <tianyi.pan@clobotics.com>
2023-04-20 03:48:47 +00:00
tqdm = {version = ">=4.48.0", optional = true}
Harrison/openapi parser (#2461) Co-authored-by: William FH <13333726+hinthornw@users.noreply.github.com>
2023-04-06 05:19:09 +00:00
openapi-schema-pydantic = "^1.2"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
faiss-cpu = {version = "^1", optional = true}
wikipedia = {version = "^1", optional = true}
elasticsearch = {version = "^8", optional = true}
Add Support for OpenSearch Vector database (#1191) ### Description This PR adds a wrapper which adds support for the OpenSearch vector database. Using opensearch-py client we are ingesting the embeddings of given text into opensearch cluster using Bulk API. We can perform the `similarity_search` on the index using the 3 popular searching methods of OpenSearch k-NN plugin: - `Approximate k-NN Search` use approximate nearest neighbor (ANN) algorithms from the [nmslib](https://github.com/nmslib/nmslib), [faiss](https://github.com/facebookresearch/faiss), and [Lucene](https://lucene.apache.org/) libraries to power k-NN search. - `Script Scoring` extends OpenSearch’s script scoring functionality to execute a brute force, exact k-NN search. - `Painless Scripting` adds the distance functions as painless extensions that can be used in more complex combinations. Also, supports brute force, exact k-NN search like Script Scoring. ### Issues Resolved https://github.com/hwchase17/langchain/issues/1054 --------- Signed-off-by: Naveen Tatikonda <navtat@amazon.com>
2023-02-21 02:39:34 +00:00
opensearch-py = {version = "^2.0.0", optional = true}
Add Redis cache implementation (#397) I'm using a hash function for the key just to make sure its length doesn't get out of hand, otherwise the implementation is quite similar.
2022-12-22 17:31:27 +00:00
redis = {version = "^4", optional = true}
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
manifest-ml = {version = "^0.0.1", optional = true}
spacy = {version = "^3", optional = true}
nltk = {version = "^3", optional = true}
transformers = {version = "^4", optional = true}
LLMRequestsChain (#267)
2022-12-07 05:55:02 +00:00
beautifulsoup4 = {version = "^4", optional = true}
torch 2 support (#2865) Lang-chain seems to work with torch 2
2023-04-14 04:38:49 +00:00
torch = {version = ">=1,<3", optional = true}
Harrison/jinja formatter (#385) Co-authored-by: Benjamin <BenderV@users.noreply.github.com>
2022-12-19 21:40:39 +00:00
jinja2 = {version = "^3", optional = true}
Fix tiktoken version (#1882) Fix https://github.com/hwchase17/langchain/issues/1881 This issue occurs when using `'gpt-3.5-turbo'` with `VectorDBQAWithSourcesChain`
2023-03-23 02:39:57 +00:00
tiktoken = {version = "^0.3.2", optional = true, python="^3.9"}
Harrison/improve data augmented generation docs (#390) Co-authored-by: cameronccohen <cameron.c.cohen@gmail.com> Co-authored-by: Cameron Cohen <cameron.cohen@quantco.com>
2022-12-21 03:24:08 +00:00
pinecone-client = {version = "^2", optional = true}
Harrison/pinecone hybrid update (#2742) Co-authored-by: acatav <39461369+acatav@users.noreply.github.com> Co-authored-by: Amnon Catav <catav.amnon1@gmail.com>
2023-04-12 04:32:17 +00:00
pinecone-text = {version = "^0.4.2", optional = true}
adding MongoDBAtlasVectorSearch (#5338) # Add MongoDBAtlasVectorSearch for the python library Fixes #5337 --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-30 14:59:01 +00:00
pymongo = {version = "^4.3.3", optional = true}
Harrison/myscale (#3352) Co-authored-by: Fangrui Liu <fangruil@moqi.ai> Co-authored-by: 刘 方瑞 <fangrui.liu@outlook.com> Co-authored-by: Fangrui.Liu <fangrui.liu@ubc.ca>
2023-04-22 16:17:38 +00:00
clickhouse-connect = {version="^0.5.14", optional=true}
Harrison/improve data augmented generation docs (#390) Co-authored-by: cameronccohen <cameron.c.cohen@gmail.com> Co-authored-by: Cameron Cohen <cameron.cohen@quantco.com>
2022-12-21 03:24:08 +00:00
weaviate-client = {version = "^3", optional = true}
Harrison/new search engine (#477) Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2022-12-30 13:06:57 +00:00
google-api-python-client = {version = "2.70.0", optional = true}
Ability to specify credentials wihen using Google BigQuery as a data loader (#5466) # Adds ability to specify credentials when using Google BigQuery as a data loader Fixes #5465 . Adds ability to set credentials which must be of the `google.auth.credentials.Credentials` type. This argument is optional and will default to `None. Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-30 23:25:22 +00:00
google-auth = {version = "^2.18.1", optional = true}
Harrison/wolfram alpha (#579) Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2023-01-11 13:52:19 +00:00
wolframalpha = {version = "5.0.0", optional = true}
Fix ChatAnthropic stop_sequences error (#2919) (#2920) Note to self: Always run integration tests, even on "that last minute change you thought would be safe" :) --------- Co-authored-by: Mike Lambert <mike.lambert@anthropic.com>
2023-04-15 00:22:01 +00:00
anthropic = {version = "^0.2.6", optional = true}
Upgrade Qdrant to 1.1.2 (#2554) This is a minor upgrade for Qdrant. We made a small bugfix in the local mode, so it might also be good to upgrade Qdrant for LangChain users.
2023-04-07 19:24:32 +00:00
qdrant-client = {version = "^1.1.2", optional = true, python = ">=3.8.1,<3.12"}
add tracing support to langchain (#741) * add implementations of `BaseCallbackHandler` to support tracing: `SharedTracer` which is thread-safe and `Tracer` which is not and is meant to be used locally. * Tracers persist runs to locally running `langchain-server` Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-01-27 01:38:13 +00:00
dataclasses-json = "^0.5.7"
Harrison/tf embeddings (#817) Co-authored-by: Ryohei Kuroki <10434946+yakigac@users.noreply.github.com>
2023-01-31 08:00:08 +00:00
tensorflow-text = {version = "^2.11.0", optional = true, python = "^3.10, <3.12"}
Add retry to OpenAI llm (#849) add ability to retry when certain exceptions are raised by `openai.Completions.create` Test plan: ran all OpenAI integration tests.
2023-02-03 03:56:26 +00:00
tenacity = "^8.1.0"
Harrison/deps (#963) Co-authored-by: Jon Luo <20971593+jzluo@users.noreply.github.com> Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MBP.attlocal.net>
2023-02-10 07:19:19 +00:00
cohere = {version = "^3", optional = true}
openai = {version = "^0", optional = true}
nlpcloud = {version = "^1", optional = true}
Harrison/atlas db (#1315) Co-authored-by: Brandon Duderstadt <brandonduderstadt@gmail.com>
2023-02-27 06:11:38 +00:00
nomic = {version = "^1.0.43", optional = true}
Harrison/deps (#963) Co-authored-by: Jon Luo <20971593+jzluo@users.noreply.github.com> Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MBP.attlocal.net>
2023-02-10 07:19:19 +00:00
huggingface_hub = {version = "^0", optional = true}
Harrison/jina (#2043) Co-authored-by: numb3r3 <wangfelix87@gmail.com> Co-authored-by: felix-wang <35718120+numb3r3@users.noreply.github.com>
2023-03-28 15:16:17 +00:00
jina = {version = "^3.14", optional = true}
Harrison/deps (#963) Co-authored-by: Jon Luo <20971593+jzluo@users.noreply.github.com> Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MBP.attlocal.net>
2023-02-10 07:19:19 +00:00
google-search-results = {version = "^2", optional = true}
sentence-transformers = {version = "^2", optional = true}
Add asyncio support for LLM (OpenAI), Chain (LLMChain, LLMMathChain), and Agent (#841) Supporting asyncio in langchain primitives allows for users to run them concurrently and creates more seamless integration with asyncio-supported frameworks (FastAPI, etc.) Summary of changes: **LLM** * Add `agenerate` and `_agenerate` * Implement in OpenAI by leveraging `client.Completions.acreate` **Chain** * Add `arun`, `acall`, `_acall` * Implement them in `LLMChain` and `LLMMathChain` for now **Agent** * Refactor and leverage async chain and llm methods * Add ability for `Tools` to contain async coroutine * Implement async SerpaPI `arun` Create demo notebook. Open questions: * Should all the async stuff go in separate classes? I've seen both patterns (keeping the same class and having async and sync methods vs. having class separation)
2023-02-08 05:21:57 +00:00
aiohttp = "^3.8.3"
Harrison/arxiv tool (#3186) Co-authored-by: leo-gan <leo.gan.57@gmail.com>
2023-04-19 23:53:34 +00:00
arxiv = {version = "^1.4", optional = true}
Harrison/format agent instructions (#973) Co-authored-by: Andrew White <white.d.andrew@gmail.com> Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MBP.attlocal.net> Co-authored-by: Peng Qu <82029664+pengqu123@users.noreply.github.com>
2023-02-10 18:07:26 +00:00
pypdf = {version = "^3.4.0", optional = true}
Harrion/kg (#1016) Co-authored-by: William FH <13333726+hinthornw@users.noreply.github.com>
2023-02-13 07:01:26 +00:00
networkx = {version="^2.6.3", optional = true}
fix imports (#1288)
2023-02-25 16:48:02 +00:00
aleph-alpha-client = {version="^2.15.0", optional = true}
Deep Lake mini upgrades (#3375) Improvements * set default num_workers for ingestion to 0 * upgraded notebooks for avoiding dataset creation ambiguity * added `force_delete_dataset_by_path` * bumped deeplake to 3.3.0 * creds arg passing to deeplake object that would allow custom S3 Notes * please double check if poetry is not messed up (thanks!) Asks * Would be great to create a shared slack channel for quick questions --------- Co-authored-by: Davit Buniatyan <d@activeloop.ai>
2023-04-24 04:23:54 +00:00
deeplake = {version = "^3.3.0", optional = true}
Harrison/pgvector (#1679) Co-authored-by: Aman Kumar <krsingh.aman@gmail.com>
2023-03-15 04:13:58 +00:00
pgvector = {version = "^0.1.6", optional = true}
psycopg2-binary = {version = "^2.9.5", optional = true}
Add OpenWeatherMap API Tool (#2083) Added tool for OpenWeatherMap API
2023-03-28 19:02:14 +00:00
pyowm = {version = "^3.3.0", optional = true}
Feature: AgentExecutor execution time limit (#2399) `AgentExecutor` already has support for limiting the number of iterations. But the amount of time taken for each iteration can vary quite a bit, so it is difficult to place limits on the execution time. This PR adds a new field `max_execution_time` to the `AgentExecutor` model. When called asynchronously, the agent loop is wrapped in an `asyncio.timeout()` context which triggers the early stopping response if the time limit is reached. When called synchronously, the agent loop checks for both the max_iteration limit and the time limit after each iteration. When used asynchronously `max_execution_time` gives really tight control over the max time for an execution chain. When used synchronously, the chain can unfortunately exceed max_execution_time, but it still gives more control than trying to estimate the number of max_iterations needed to cap the execution time. --------- Co-authored-by: Zachary Jones <zjones@zetaglobal.com>
2023-04-06 19:54:32 +00:00
async-timeout = {version = "^4.0.0", python = "<3.11"}
Harrison/power bi (#3205) Co-authored-by: Eduard van Valkenburg <eavanvalkenburg@users.noreply.github.com>
2023-04-22 15:24:48 +00:00
azure-identity = {version = "^1.12.0", optional=true}
Harrison/gpt cache (#2744) Co-authored-by: SimFG <bang.fu@zilliz.com>
2023-04-12 21:16:58 +00:00
gptcache = {version = ">=0.1.7", optional = true}
Harrison/confluent loader (#2994) Co-authored-by: Justin Flick <Justinjayflick@gmail.com>
2023-04-18 03:23:45 +00:00
atlassian-python-api = {version = "^3.36.0", optional=true}
pytesseract = {version = "^0.3.10", optional=true}
html2text = {version="^2020.1.16", optional=true}
Remove pythonrepl from LLM-MathChain (#2943) Use numexpr evaluate instead of the python REPL to avoid malicious code injection. Tested against the (limited) math dataset and got the same score as before. For more permissive tools (like the REPL tool itself), other approaches ought to be provided (some combination of Sanitizer + Restricted python + unprivileged-docker + ...), but for a calculator tool, only mathematical expressions should be permitted. See https://github.com/hwchase17/langchain/issues/814
2023-04-16 15:50:32 +00:00
numexpr = "^2.8.4"
Harrison/ddg (#3206) Co-authored-by: itai <itai.marks@gmail.com> Co-authored-by: Itai Marks <itaim@users.noreply.github.com> Co-authored-by: Tianyi Pan <60060750+tipani86@users.noreply.github.com> Co-authored-by: Tianyi Pan <tianyi.pan@clobotics.com> Co-authored-by: Adilzhan Ismailov <13088690+aismlv@users.noreply.github.com> Co-authored-by: Justin Flick <Justinjayflick@gmail.com> Co-authored-by: Justin Flick <jflick@homesite.com>
2023-04-20 04:32:26 +00:00
duckduckgo-search = {version="^2.8.6", optional=true}
Azure CosmosDB memory (#3434) Still needs docs, otherwise works.
2023-04-25 05:15:12 +00:00
azure-cosmos = {version="^4.4.0b1", optional=true}
Self-query with generic query constructor (#3607) Alternate implementation of #3452 that relies on a generic query constructor chain and language and then has vector store-specific translation layer. Still refactoring and updating examples but general structure is there and seems to work s well as #3452 on exampels --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-04-27 15:36:00 +00:00
lark = {version="^1.1.5", optional=true}
Harrison/lancedb (#3634) Co-authored-by: Minh Le <minhle@canva.com>
2023-04-27 15:14:36 +00:00
lancedb = {version = "^0.1", optional = true}
Remove Pexpect Dependency (#3667) Resolves #3664 Next PR will be to clean up CI to catch this earlier. Triaging this, it looks like it wasn't caught because pexpect is a `poetry` dependency. --------- Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-04-27 18:39:01 +00:00
pexpect = {version = "^4.8.0", optional = true}
Harrison/vespa (#3761) Co-authored-by: Lester Solbakken <lesters@users.noreply.github.com>
2023-04-29 02:48:43 +00:00
pyvespa = {version = "^0.33.0", optional = true}
Harrison/one drive loader (#4081) Co-authored-by: José Ferraz Neto <netoferraz@gmail.com>
2023-05-04 05:55:34 +00:00
O365 = {version = "^2.0.26", optional = true}
JSON loader (#4067) This implements a loader of text passages in JSON format. The `jq` syntax is used to define a schema for accessing the relevant contents from the JSON file. This requires dependency on the `jq` package: https://pypi.org/project/jq/. --------- Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
2023-05-05 21:48:13 +00:00
jq = {version = "^1.4.1", optional = true}
Add Steamship Image Generation Tool (#4580) Co-authored-by: Enias Cailliau <enias@steamship.com>
2023-05-12 17:35:01 +00:00
steamship = {version = "^2.16.9", optional = true}
Add workflow for testing with all deps (#4410) # Add action to test with all dependencies installed PR adds a custom action for setting up poetry that allows specifying a cache key: https://github.com/actions/setup-python/issues/505#issuecomment-1273013236 This makes it possible to run 2 types of unit tests: (1) unit tests with only core dependencies (2) unit tests with extended dependencies (e.g., those that rely on an optional pdf parsing library) As part of this PR, we're moving some pdf parsing tests into the unit-tests section and making sure that these unit tests get executed when running with extended dependencies.
2023-05-10 13:35:07 +00:00
pdfminer-six = {version = "^20221105", optional = true}
fix: revert docarray explicit transitive dependencies and use extras instead (#5015) tldr: The docarray [integration PR](https://github.com/hwchase17/langchain/pull/4483) introduced a pinned dependency to protobuf. This is a docarray dependency, not a langchain dependency. Since this is handled by the docarray dependencies, it is unnecessary here. Further, as a pinned dependency, this quickly leads to incompatibilities with application code that consumes the library. Much less with a heavily used library like protobuf. Detail: as we see in the [docarray integration](https://github.com/hwchase17/langchain/pull/4483/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R81-R83), the transitive dependencies of docarray were also listed as langchain dependencies. This is unnecessary as the docarray project has an appropriate [extras](https://github.com/docarray/docarray/blob/a01a05542d17264b8a164bec783633658deeedb8/pyproject.toml#L70). The docarray project also does not require this _pinned_ version of protobuf, rather [a minimum version](https://github.com/docarray/docarray/blob/a01a05542d17264b8a164bec783633658deeedb8/pyproject.toml#L41). So this pinned version was likely in error. To fix this, this PR reverts the explicit hnswlib and protobuf dependencies and adds the hnswlib extras install for docarray (which installs hnswlib and protobuf, as originally intended). Because version `0.32.0` of the docarray hnswlib extras added protobuf, we bump the docarray dependency from `^0.31.0` to `^0.32.0`. # revert docarray explicit transitive dependencies and use extras instead ## Who can review? @dev2049 -- reviewed the original PR @eyurtsev -- bumped the pinned protobuf dependency a few days ago --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 16:48:09 +00:00
docarray = {version="^0.32.0", extras=["hnswlib"], optional=true}
Docugami DataLoader (#4727) ### Adds a document loader for Docugami Specifically: 1. Adds a data loader that talks to the [Docugami](http://docugami.com) API to download processed documents as semantic XML 2. Parses the semantic XML into chunks, with additional metadata capturing chunk semantics 3. Adds a detailed notebook showing how you can use additional metadata returned by Docugami for techniques like the [self-querying retriever](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query_retriever.html) 4. Adds an integration test, and related documentation Here is an example of a result that is not possible without the capabilities added by Docugami (from the notebook): <img width="1585" alt="image" src="https://github.com/hwchase17/langchain/assets/749277/bb6c1ce3-13dc-4349-a53b-de16681fdd5b"> --------- Co-authored-by: Taqi Jaffri <tjaffri@docugami.com> Co-authored-by: Taqi Jaffri <tjaffri@gmail.com>
2023-05-15 14:53:00 +00:00
lxml = {version = "^4.9.2", optional = true}
Clean up tests for pdf parsers (#4595) # Organize tests for pdf parsers Clean up tests for pdf parsers, remove duplicate tests, convert to unit tests.
2023-05-15 18:21:05 +00:00
pymupdf = {version = "^1.22.3", optional = true}
pypdfium2 = {version = "^4.10.0", optional = true}
[Feature] Add GraphQL Query Tool (#4409) # Add GraphQL Query Support This PR introduces a GraphQL API Wrapper tool that allows LLM agents to query GraphQL databases. The tool utilizes the httpx and gql Python packages to interact with GraphQL APIs and provides a simple interface for running queries with LLM agents. @vowelparrot --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-15 21:06:12 +00:00
gql = {version = "^3.4.1", optional = true}
fix(document_loaders/telegram): fix pandas calls + add tests (#4806) # Fix Telegram API loader + add tests. I was testing this integration and it was broken with next error: ```python message_threads = loader._get_message_threads(df) KeyError: False ``` Also, this particular loader didn't have any tests / related group in poetry, so I added those as well. @hwchase17 / @eyurtsev please take a look on this fix PR. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-16 21:35:25 +00:00
pandas = {version = "^2.0.1", optional = true}
telethon = {version = "^1.28.5", optional = true}
Harrison/neo4j (#5078) Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 14:31:48 +00:00
neo4j = {version = "^5.8.1", optional = true}
Update psychicapi version (#5471) Update [psychicapi](https://pypi.org/project/psychicapi/) python package dependency to the latest version 0.5. The newest python package version addresses breaking changes in the Psychic http api.
2023-05-30 22:55:22 +00:00
psychicapi = {version = "^0.5", optional = true}
Zep Hybrid Search (#5742) Zep now supports persisting custom metadata with messages and hybrid search across both message embeddings and structured metadata. This PR implements custom metadata and enhancements to the `ZepChatMessageHistory` and `ZepRetriever` classes to implement this support. Tag maintainers/contributors who might be interested: VectorStores / Retrievers / Memory - @dev2049 --------- Co-authored-by: Daniel Chalef <daniel.chalef@private.org>
2023-06-05 19:59:28 +00:00
zep-python = {version=">=0.31", optional=true}
WhyLabs callback (#4906) # Add a WhyLabs callback handler * Adds a simple WhyLabsCallbackHandler * Add required dependencies as optional * protect against missing modules with imports * Add docs/ecosystem basic example based on initial prototype from @andrewelizondo > this integration gathers privacy preserving telemetry on text with whylogs and sends stastical profiles to WhyLabs platform to monitoring these metrics over time. For more information on what WhyLabs is see: https://whylabs.ai After you run the notebook (if you have env variables set for the API Keys, org_id and dataset_id) you get something like this in WhyLabs: ![Screenshot (443)](https://github.com/hwchase17/langchain/assets/88007022/6bdb3e1c-4243-4ae8-b974-23a8bb12edac) Co-authored-by: Andre Elizondo <andre@whylabs.ai> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-23 03:29:47 +00:00
langkit = {version = ">=0.0.1.dev3, <0.1.0", optional = true}
feat #4479: TextLoader auto detect encoding and improved exceptions (#4927) # TextLoader auto detect encoding and enhanced exception handling - Add an option to enable encoding detection on `TextLoader`. - The detection is done using `chardet` - The loading is done by trying all detected encodings by order of confidence or raise an exception otherwise. ### New Dependencies: - `chardet` Fixes #4479 ## Before submitting <!-- If you're adding a new integration, include an integration test and an example notebook showing its use! --> ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: - @eyurtsev --------- Co-authored-by: blob42 <spike@w530>
2023-05-18 13:55:14 +00:00
chardet = {version="^5.1.0", optional=true}
Fix graphql tool (#4984) Fix construction and add unit test.
2023-05-19 22:27:50 +00:00
requests-toolbelt = {version = "^1.0.0", optional = true}
Add OpenLM LLM multi-provider (#4993) OpenLM is a zero-dependency OpenAI-compatible LLM provider that can call different inference endpoints directly via HTTP. It implements the OpenAI Completion class so that it can be used as a drop-in replacement for the OpenAI API. This changeset utilizes BaseOpenAI for minimal added code. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-23 01:09:53 +00:00
openlm = {version = "^0.0.5", optional = true}
tfidf retriever (#5114) Co-authored-by: vempaliakhil96 <vempaliakhil96@gmail.com>
2023-05-24 17:02:09 +00:00
scikit-learn = {version = "^1.2.2", optional = true}
Add AzureCognitiveServicesToolkit to call Azure Cognitive Services API (#5012) # Add AzureCognitiveServicesToolkit to call Azure Cognitive Services API: achieve some multimodal capabilities This PR adds a toolkit named AzureCognitiveServicesToolkit which bundles the following tools: - AzureCogsImageAnalysisTool: calls Azure Cognitive Services image analysis API to extract caption, objects, tags, and text from images. - AzureCogsFormRecognizerTool: calls Azure Cognitive Services form recognizer API to extract text, tables, and key-value pairs from documents. - AzureCogsSpeech2TextTool: calls Azure Cognitive Services speech to text API to transcribe speech to text. - AzureCogsText2SpeechTool: calls Azure Cognitive Services text to speech API to synthesize text to speech. This toolkit can be used to process image, document, and audio inputs. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-23 13:45:48 +00:00
azure-ai-formrecognizer = {version = "^3.2.1", optional = true}
azure-ai-vision = {version = "^0.11.1b1", optional = true}
azure-cognitiveservices-speech = {version = "^1.28.0", optional = true}
New Trello document loader (#4767) # Added New Trello loader class and documentation Simple Loader on top of py-trello wrapper. With a board name you can pull cards and to do some field parameter tweaks on load operation. I included documentation and examples. Included unit test cases using patch and a fixture for py-trello client class. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-30 02:47:56 +00:00
py-trello = {version = "^0.19.0", optional = true}
feat: add Momento as a standard cache and chat message history provider (#5221) # Add Momento as a standard cache and chat message history provider This PR adds Momento as a standard caching provider. Implements the interface, adds integration tests, and documentation. We also add Momento as a chat history message provider along with integration tests, and documentation. [Momento](https://www.gomomento.com/) is a fully serverless cache. Similar to S3 or DynamoDB, it requires zero configuration, infrastructure management, and is instantly available. Users sign up for free and get 50GB of data in/out for free every month. ## Before submitting ✅ We have added documentation, notebooks, and integration tests demonstrating usage. Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-26 02:13:21 +00:00
momento = {version = "^1.5.0", optional = true}
Bibtex integration for document loader and retriever (#5137) # Bibtex integration Wrap bibtexparser to retrieve a list of docs from a bibtex file. * Get the metadata from the bibtex entries * `page_content` get from the local pdf referenced in the `file` field of the bibtex entry using `pymupdf` * If no valid pdf file, `page_content` set to the `abstract` field of the bibtex entry * Support Zotero flavour using regex to get the file path * Added usage example in `docs/modules/indexes/document_loaders/examples/bibtex.ipynb` --------- Co-authored-by: Sébastien M. Popoff <sebastien.popoff@espci.fr> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-25 07:21:31 +00:00
bibtexparser = {version = "^1.4.0", optional = true}
Harrison/spark reader (#5405) Co-authored-by: Rithwik Ediga Lakhamsani <rithwik.ediga@databricks.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-30 03:23:17 +00:00
pyspark = {version = "^3.4.0", optional = true}
feat: Support for `Tigris` Vector Database for vector search (#5703) ### Changes - New vector store integration - [Tigris](https://tigrisdata.com) - Adds [tigrisdb](https://pypi.org/project/tigrisdb/) optional dependency - Example notebook demonstrating usage Fixes #5535 Closes tigrisdata/tigris-client-python#40 #### Twitter handles We'd love a shoutout on our [@TigrisData](https://twitter.com/TigrisData) and [@adilansari](https://twitter.com/adilansari) twitter handles #### Who can review? @dev2049 --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-06-06 03:39:16 +00:00
tigrisdb = {version = "^1.0.0b6", optional = true}
feat #4479: TextLoader auto detect encoding and improved exceptions (#4927) # TextLoader auto detect encoding and enhanced exception handling - Add an option to enable encoding detection on `TextLoader`. - The detection is done using `chardet` - The loading is done by trying all detected encodings by order of confidence or raise an exception otherwise. ### New Dependencies: - `chardet` Fixes #4479 ## Before submitting <!-- If you're adding a new integration, include an integration test and an example notebook showing its use! --> ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: - @eyurtsev --------- Co-authored-by: blob42 <spike@w530>
2023-05-18 13:55:14 +00:00
Feature: linkcheck-action (#534) (#542) - Add support for local build and linkchecking of docs - Add GitHub Action to automatically check links before prior to publication - Minor reformat of Contributing readme - Fix existing broken links Co-authored-by: Hunter Gerlach <hunter@huntergerlach.com> Co-authored-by: Hunter Gerlach <HunterGerlach@users.noreply.github.com> Co-authored-by: Hunter Gerlach <hunter@huntergerlach.com>
2023-01-05 05:39:50 +00:00
[tool.poetry.group.docs.dependencies]
autodoc_pydantic = "^1.8.0"
myst_parser = "^0.18.1"
nbsphinx = "^0.8.9"
sphinx = "^4.5.0"
sphinx-autobuild = "^2021.3.14"
sphinx_book_theme = "^0.3.3"
sphinx_rtd_theme = "^1.0.0"
sphinx-typlog-theme = "^0.8.0"
sphinx-panels = "^0.6.0"
toml = "^0.10.2"
myst-nb = "^0.17.1"
linkchecker = "^10.2.1"
Add copy button to sphinx notebooks (#1622) This adds a copy button at the top right corner of all notebook cells in sphinx notebooks.
2023-03-13 04:15:07 +00:00
sphinx-copybutton = "^0.5.1"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
[tool.poetry.group.test.dependencies]
Catch changes to test group (#4802) # Catch changes to test group Add test to catch changes to test group.
2023-05-16 18:48:56 +00:00
# The only dependencies that should be added are
# dependencies used for running tests (e.g., pytest, freezegun, response).
# Any dependencies that do not meet that criteria will be removed.
fix: some imports for integration tests (#2612) Add more missed imports for integration tests. Bump `pytest` to the current latest version. Fix `tests/integration_tests/vectorstores/test_elasticsearch.py` to update its cassette(easy fix). Related PR: https://github.com/hwchase17/langchain/pull/2560
2023-04-12 03:45:36 +00:00
pytest = "^7.3.0"
unit test / code coverage improvements (#322) This PR has two contributions: 1. Add test for when stop token is found in middle of text 2. Add code coverage tooling and instructions - Add pytest-cov via poetry - Add necessary config files - Add new make instruction for `coverage` - Update README with coverage guidance - Update minor README formatting/spelling Co-authored-by: Hunter Gerlach <hunter@huntergerlach.com>
2022-12-13 13:48:53 +00:00
pytest-cov = "^4.0.0"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
pytest-dotenv = "^0.5.2"
Bump duckdb-engine to 0.7.0 (#1726) Resolves https://github.com/hwchase17/langchain/issues/1272 Resolves https://github.com/hwchase17/langchain/issues/1578
2023-03-17 04:55:35 +00:00
duckdb-engine = "^0.7.0"
Add watch mode for test runner (#453)
2022-12-28 22:13:08 +00:00
pytest-watcher = "^0.2.6"
add tracing support to langchain (#741) * add implementations of `BaseCallbackHandler` to support tracing: `SharedTracer` which is thread-safe and `Tracer` which is not and is meant to be used locally. * Tracers persist runs to locally running `langchain-server` Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-01-27 01:38:13 +00:00
freezegun = "^1.2.2"
Centralize logic for loading from LangChainHub, add ability to pin dependencies (#805) It's generally considered to be a good practice to pin dependencies to prevent surprise breakages when a new version of a dependency is released. This commit adds the ability to pin dependencies when loading from LangChainHub. Centralizing this logic and using urllib fixes an issue identified by some windows users highlighted in this video - https://youtu.be/aJ6IQUh8MLQ?t=537
2023-01-30 22:52:17 +00:00
responses = "^0.22.0"
Add asyncio support for LLM (OpenAI), Chain (LLMChain, LLMMathChain), and Agent (#841) Supporting asyncio in langchain primitives allows for users to run them concurrently and creates more seamless integration with asyncio-supported frameworks (FastAPI, etc.) Summary of changes: **LLM** * Add `agenerate` and `_agenerate` * Implement in OpenAI by leveraging `client.Completions.acreate` **Chain** * Add `arun`, `acall`, `_acall` * Implement them in `LLMChain` and `LLMMathChain` for now **Agent** * Refactor and leverage async chain and llm methods * Add ability for `Tools` to contain async coroutine * Implement async SerpaPI `arun` Create demo notebook. Open questions: * Should all the async stuff go in separate classes? I've seen both patterns (keeping the same class and having async and sync methods vs. having class separation)
2023-02-08 05:21:57 +00:00
pytest-asyncio = "^0.20.3"
Add query parsing unit tests (#3672)
2023-04-27 20:42:12 +00:00
lark = "^1.1.5"
Harrison/csv loader (#3771) Co-authored-by: mrT23 <tal.r@codium.ai>
2023-04-29 04:54:24 +00:00
pytest-mock = "^3.10.0"
Block sockets for unit-tests (#4803) # Block usage of sockets during unit tests Catch any tests that attempt to use the network.
2023-05-16 18:41:24 +00:00
pytest-socket = "^0.6.0"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
fix: elasticsearch (#2402) - Create a new docker-compose file to start an Elasticsearch instance for integration tests. - Add new tests to `test_elasticsearch.py` to verify Elasticsearch functionality. - Include an optional group `test_integration` in the `pyproject.toml` file. This group should contain dependencies for integration tests and can be installed using the command `poetry install --with test_integration`. Any new dependencies should be added by running `poetry add some_new_deps --group "test_integration" ` Note: New tests running in live mode, which involve end-to-end testing of the OpenAI API. In the future, adding `pytest-vcr` to record and replay all API requests would be a nice feature for testing process.More info: https://pytest-vcr.readthedocs.io/en/latest/ Fixes https://github.com/hwchase17/langchain/issues/2386
2023-04-05 13:51:32 +00:00
[tool.poetry.group.test_integration]
optional = true
[tool.poetry.group.test_integration.dependencies]
Add instructions to pyproject.toml (#5138) # Add instructions to pyproject.toml * Add instructions to pyproject.toml about how to handle optional dependencies. ## Before submitting ## Who can review? --------- Co-authored-by: Davis Chase <130488702+dev2049@users.noreply.github.com> Co-authored-by: Zander Chase <130414180+vowelparrot@users.noreply.github.com>
2023-05-26 17:29:07 +00:00
# Do not add dependencies in the test_integration group
# Instead:
# 1. Add an optional dependency to the main group
# poetry add --optional [package name]
# 2. Add the package name to the extended_testing extra (find it below)
# 3. Relock the poetry file
# poetry lock --no-update
# 4. Favor unit tests not integration tests.
# Use the @pytest.mark.requires(pkg_name) decorator in unit_tests.
# Your tests should not rely on network access, as it prevents other
# developers from being able to easily run them.
# Instead write unit tests that use the `responses` library or mock.patch with
# fixtures. Keep the fixtures minimal.
# See CONTRIBUTING.md for more instructions on working with optional dependencies.
# https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md#working-with-optional-dependencies
feat: add pytest-vcr for recording HTTP interactions in integration tests (#2445) Using `pytest-vcr` in integration tests has several benefits. Firstly, it removes the need to mock external services, as VCR records and replays HTTP interactions on the fly. Secondly, it simplifies the integration test setup by eliminating the need to set up and tear down external services in some cases. Finally, it allows for more reliable and deterministic integration tests by ensuring that HTTP interactions are always replayed with the same response. Overall, `pytest-vcr` is a valuable tool for simplifying integration test setup and improving their reliability This commit adds the `pytest-vcr` package as a dependency for integration tests in the `pyproject.toml` file. It also introduces two new fixtures in `tests/integration_tests/conftest.py` files for managing cassette directories and VCR configurations. In addition, the `tests/integration_tests/vectorstores/test_elasticsearch.py` file has been updated to use the `@pytest.mark.vcr` decorator for recording and replaying HTTP interactions. Finally, this commit removes the `documents` fixture from the `test_elasticsearch.py` file and replaces it with a new fixture defined in `tests/integration_tests/vectorstores/conftest.py` that yields a list of documents to use in any other tests. This also includes my second attempt to fix issue : https://github.com/hwchase17/langchain/issues/2386 Maybe related https://github.com/hwchase17/langchain/issues/2484
2023-04-07 14:28:57 +00:00
pytest-vcr = "^1.0.2"
wrapt = "^1.15.0"
fix: some imports for integration tests (#2612) Add more missed imports for integration tests. Bump `pytest` to the current latest version. Fix `tests/integration_tests/vectorstores/test_elasticsearch.py` to update its cassette(easy fix). Related PR: https://github.com/hwchase17/langchain/pull/2560
2023-04-12 03:45:36 +00:00
openai = "^0.27.4"
elasticsearch = {extras = ["async"], version = "^8.6.2"}
fix: missed deps integrations tests (#2560) Almost all integration tests have failed, but we haven't encountered any import errors yet. Some tests failed due to lazy import issues. It doesn't seem like a problem to resolve some of these errors in the next PR. I have a headache from resolving conflicts with `deeplake` and `boto3`, so I will temporarily comment out `boto3`. fix https://github.com/hwchase17/langchain/issues/2426
2023-04-08 03:43:53 +00:00
redis = "^4.5.4"
pinecone-client = "^2.2.1"
Harrison/pinecone hybrid update (#2742) Co-authored-by: acatav <39461369+acatav@users.noreply.github.com> Co-authored-by: Amnon Catav <catav.amnon1@gmail.com>
2023-04-12 04:32:17 +00:00
pinecone-text = "^0.4.2"
adding MongoDBAtlasVectorSearch (#5338) # Add MongoDBAtlasVectorSearch for the python library Fixes #5337 --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-30 14:59:01 +00:00
pymongo = "^4.3.3"
Harrison/myscale (#3352) Co-authored-by: Fangrui Liu <fangruil@moqi.ai> Co-authored-by: 刘 方瑞 <fangrui.liu@outlook.com> Co-authored-by: Fangrui.Liu <fangrui.liu@ubc.ca>
2023-04-22 16:17:38 +00:00
clickhouse-connect = "^0.5.14"
fix: missed deps integrations tests (#2560) Almost all integration tests have failed, but we haven't encountered any import errors yet. Some tests failed due to lazy import issues. It doesn't seem like a problem to resolve some of these errors in the next PR. I have a headache from resolving conflicts with `deeplake` and `boto3`, so I will temporarily comment out `boto3`. fix https://github.com/hwchase17/langchain/issues/2426
2023-04-08 03:43:53 +00:00
pgvector = "^0.1.6"
transformers = "^4.27.4"
pandas = "^2.0.0"
deeplake = "^3.2.21"
Harrison/weaviate fixes (#2872) Co-authored-by: cs0lar <cristiano.solarino@gmail.com> Co-authored-by: cs0lar <cristiano.solarino@brightminded.com>
2023-04-14 05:37:34 +00:00
weaviate-client = "^3.15.5"
Harrison/pinecone hybrid update (#2742) Co-authored-by: acatav <39461369+acatav@users.noreply.github.com> Co-authored-by: Amnon Catav <catav.amnon1@gmail.com>
2023-04-12 04:32:17 +00:00
torch = "^1.0.0"
fix: some imports for integration tests (#2612) Add more missed imports for integration tests. Bump `pytest` to the current latest version. Fix `tests/integration_tests/vectorstores/test_elasticsearch.py` to update its cassette(easy fix). Related PR: https://github.com/hwchase17/langchain/pull/2560
2023-04-12 03:45:36 +00:00
chromadb = "^0.3.21"
tiktoken = "^0.3.3"
feat: improve pinecone tests (#2806) Improve the integration tests for Pinecone by adding an `.env.example` file for local testing. Additionally, add some dev dependencies specifically for integration tests. This change also helps me understand how Pinecone deals with certain things, see related issues https://github.com/hwchase17/langchain/issues/2484 https://github.com/hwchase17/langchain/issues/2816
2023-04-14 04:49:31 +00:00
python-dotenv = "^1.0.0"
Add Sentence Transformers Embeddings (#3409) Add embeddings based on the sentence transformers library. Add a notebook and integration tests. Co-authored-by: khimaros <me@khimaros.com>
2023-04-24 01:25:20 +00:00
sentence-transformers = "^2"
feat: improve pinecone tests (#2806) Improve the integration tests for Pinecone by adding an `.env.example` file for local testing. Additionally, add some dev dependencies specifically for integration tests. This change also helps me understand how Pinecone deals with certain things, see related issues https://github.com/hwchase17/langchain/issues/2484 https://github.com/hwchase17/langchain/issues/2816
2023-04-14 04:49:31 +00:00
gptcache = "^0.1.9"
promptlayer = "^0.1.80"
Harrison/tair (#3770) Co-authored-by: Seth Huang <848849+seth-hg@users.noreply.github.com>
2023-04-29 04:25:33 +00:00
tair = "^1.3.3"
added `Wikipedia` retriever (#4302) - added `Wikipedia` retriever. It is effectively a wrapper for `WikipediaAPIWrapper`. It wrapps load() into get_relevant_documents() - sorted `__all__` in the `retrievers/__init__` - added integration tests for the WikipediaRetriever - added an example (as Jupyter notebook) for the WikipediaRetriever
2023-05-09 17:08:39 +00:00
wikipedia = "^1"
Cassandra support for chat history (#4378) (#4764) # Cassandra support for chat history ### Description - Store chat messages in cassandra ### Dependency - cassandra-driver - Python Module ## Before submitting - Added Integration Test ## Who can review? @hwchase17 @agola11 # Your PR Title (What it does) <!-- Thank you for contributing to LangChain! Your PR will appear in our next release under the title you set. Please make sure it highlights your valuable contribution. Replace this with a description of the change, the issue it fixes (if applicable), and relevant context. List any dependencies required for this change. After you're done, someone will review your PR. They may suggest improvements. If no one reviews your PR within a few days, feel free to @-mention the same people again, as notifications can get lost. --> <!-- Remove if not applicable --> Fixes # (issue) ## Before submitting <!-- If you're adding a new integration, include an integration test and an example notebook showing its use! --> ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: <!-- For a quicker response, figure out the right person to tag with @ @hwchase17 - project lead Tracing / Callbacks - @agola11 Async - @agola11 DataLoaders - @eyurtsev Models - @hwchase17 - @agola11 Agents / Tools / Toolkits - @vowelparrot VectorStores / Retrievers / Memory - @dev2049 --> Co-authored-by: Jinto Jose <129657162+jj701@users.noreply.github.com>
2023-05-16 06:43:09 +00:00
cassandra-driver = "^3.27.0"
Add `arxiv` retriever (#4538)
2023-05-12 05:48:38 +00:00
arxiv = "^1.4"
Add Mastodon toots loader (#5036) # Add Mastodon toots loader. Loader works either with public toots, or Mastodon app credentials. Toot text and user info is loaded. I've also added integration test for this new loader as it works with public data, and a notebook with example output run now. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 23:43:07 +00:00
mastodon-py = "^1.8.1"
feat: add Momento as a standard cache and chat message history provider (#5221) # Add Momento as a standard cache and chat message history provider This PR adds Momento as a standard caching provider. Implements the interface, adds integration tests, and documentation. We also add Momento as a chat history message provider along with integration tests, and documentation. [Momento](https://www.gomomento.com/) is a fully serverless cache. Similar to S3 or DynamoDB, it requires zero configuration, infrastructure management, and is instantly available. Users sign up for free and get 50GB of data in/out for free every month. ## Before submitting ✅ We have added documentation, notebooks, and integration tests demonstrating usage. Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-26 02:13:21 +00:00
momento = "^1.5.0"
Add instructions to pyproject.toml (#5138) # Add instructions to pyproject.toml * Add instructions to pyproject.toml about how to handle optional dependencies. ## Before submitting ## Who can review? --------- Co-authored-by: Davis Chase <130488702+dev2049@users.noreply.github.com> Co-authored-by: Zander Chase <130414180+vowelparrot@users.noreply.github.com>
2023-05-26 17:29:07 +00:00
# Please do not add any dependencies in the test_integration group
# See instructions above ^^
fix: elasticsearch (#2402) - Create a new docker-compose file to start an Elasticsearch instance for integration tests. - Add new tests to `test_elasticsearch.py` to verify Elasticsearch functionality. - Include an optional group `test_integration` in the `pyproject.toml` file. This group should contain dependencies for integration tests and can be installed using the command `poetry install --with test_integration`. Any new dependencies should be added by running `poetry add some_new_deps --group "test_integration" ` Note: New tests running in live mode, which involve end-to-end testing of the OpenAI API. In the future, adding `pytest-vcr` to record and replay all API requests would be a nice feature for testing process.More info: https://pytest-vcr.readthedocs.io/en/latest/ Fixes https://github.com/hwchase17/langchain/issues/2386
2023-04-05 13:51:32 +00:00
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
[tool.poetry.group.lint.dependencies]
ruff ruff (#1203)
2023-02-25 16:59:52 +00:00
ruff = "^0.0.249"
hotfix for api logging (#262)
2022-12-05 05:12:05 +00:00
types-toml = "^0.10.8.1"
add tracing support to langchain (#741) * add implementations of `BaseCallbackHandler` to support tracing: `SharedTracer` which is thread-safe and `Tracer` which is not and is meant to be used locally. * Tracers persist runs to locally running `langchain-server` Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-01-27 01:38:13 +00:00
types-redis = "^4.3.21.6"
bump version to 0082 (#980) Co-authored-by: Harrison Chase <harrisonchase@Harrisons-MacBook-Pro.local>
2023-02-10 19:38:24 +00:00
black = "^23.1.0"
feat #4479: TextLoader auto detect encoding and improved exceptions (#4927) # TextLoader auto detect encoding and enhanced exception handling - Add an option to enable encoding detection on `TextLoader`. - The detection is done using `chardet` - The loading is done by trying all detected encodings by order of confidence or raise an exception otherwise. ### New Dependencies: - `chardet` Fixes #4479 ## Before submitting <!-- If you're adding a new integration, include an integration test and an example notebook showing its use! --> ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: - @eyurtsev --------- Co-authored-by: blob42 <spike@w530>
2023-05-18 13:55:14 +00:00
types-chardet = "^5.0.4.6"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
[tool.poetry.group.typing.dependencies]
mypy = "^0.991"
types-pyyaml = "^6.0.12.2"
types-requests = "^2.28.11.5"
[tool.poetry.group.dev]
optional = true
[tool.poetry.group.dev.dependencies]
jupyter = "^1.0.0"
playwright = "^1.28.0"
fix: testing in Windows and add missing dev dependency (#2340) This changes addresses two issues. First, we add `setuptools` to the dev dependencies in order to debug tests locally with an IDE, especially with PyCharm. All dependencies dev dependencies should be installed with `poetry install --extras "dev"`. Second, we use PurePosixPath instead of Path for URL paths to fix issues with testing in Windows. This ensures that forward slashes are used as the path separator regardless of the operating system. Closes https://github.com/hwchase17/langchain/issues/2334
2023-04-03 21:11:18 +00:00
setuptools = "^67.6.1"
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
[tool.poetry.extras]
Add OpenLM LLM multi-provider (#4993) OpenLM is a zero-dependency OpenAI-compatible LLM provider that can call different inference endpoints directly via HTTP. It implements the OpenAI Completion class so that it can be used as a drop-in replacement for the OpenAI API. This changeset utilizes BaseOpenAI for minimal added code. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-23 01:09:53 +00:00
llms = ["anthropic", "cohere", "openai", "openlm", "nlpcloud", "huggingface_hub", "manifest-ml", "torch", "transformers"]
Qdrant update to 1.1.1 & docs polishing (#2388) This PR updates Qdrant to 1.1.1 and introduces local mode, so there is no need to spin up the Qdrant server. By that occasion, the Qdrant example notebooks also got updated, covering more cases and answering some commonly asked questions. All the Qdrant's integration tests were switched to local mode, so no Docker container is required to launch them.
2023-04-04 13:48:21 +00:00
qdrant = ["qdrant-client"]
[pyproject.toml] add `tiktoken` when install `langchain[openai]` (#4514) # Add `tiktoken` as dependency when installed as `langchain[openai]` Fixes #4513 (issue) ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: @vowelparrot <!-- For a quicker response, figure out the right person to tag with @ @hwchase17 - project lead Tracing / Callbacks - @agola11 Async - @agola11 DataLoaders - @eyurtsev Models - @hwchase17 - @agola11 Agents / Tools / Toolkits - @vowelparrot VectorStores / Retrievers / Memory - @dev2049 -->
2023-05-11 19:21:06 +00:00
openai = ["openai", "tiktoken"]
feat #4479: TextLoader auto detect encoding and improved exceptions (#4927) # TextLoader auto detect encoding and enhanced exception handling - Add an option to enable encoding detection on `TextLoader`. - The detection is done using `chardet` - The loading is done by trying all detected encodings by order of confidence or raise an exception otherwise. ### New Dependencies: - `chardet` Fixes #4479 ## Before submitting <!-- If you're adding a new integration, include an integration test and an example notebook showing its use! --> ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: - @eyurtsev --------- Co-authored-by: blob42 <spike@w530>
2023-05-18 13:55:14 +00:00
text_helpers = ["chardet"]
Qdrant update to 1.1.1 & docs polishing (#2388) This PR updates Qdrant to 1.1.1 and introduces local mode, so there is no need to spin up the Qdrant server. By that occasion, the Qdrant example notebooks also got updated, covering more cases and answering some commonly asked questions. All the Qdrant's integration tests were switched to local mode, so no Docker container is required to launch them.
2023-04-04 13:48:21 +00:00
cohere = ["cohere"]
fix: revert docarray explicit transitive dependencies and use extras instead (#5015) tldr: The docarray [integration PR](https://github.com/hwchase17/langchain/pull/4483) introduced a pinned dependency to protobuf. This is a docarray dependency, not a langchain dependency. Since this is handled by the docarray dependencies, it is unnecessary here. Further, as a pinned dependency, this quickly leads to incompatibilities with application code that consumes the library. Much less with a heavily used library like protobuf. Detail: as we see in the [docarray integration](https://github.com/hwchase17/langchain/pull/4483/files#diff-50c86b7ed8ac2cf95bd48334961bf0530cdc77b5a56f852c5c61b89d735fd711R81-R83), the transitive dependencies of docarray were also listed as langchain dependencies. This is unnecessary as the docarray project has an appropriate [extras](https://github.com/docarray/docarray/blob/a01a05542d17264b8a164bec783633658deeedb8/pyproject.toml#L70). The docarray project also does not require this _pinned_ version of protobuf, rather [a minimum version](https://github.com/docarray/docarray/blob/a01a05542d17264b8a164bec783633658deeedb8/pyproject.toml#L41). So this pinned version was likely in error. To fix this, this PR reverts the explicit hnswlib and protobuf dependencies and adds the hnswlib extras install for docarray (which installs hnswlib and protobuf, as originally intended). Because version `0.32.0` of the docarray hnswlib extras added protobuf, we bump the docarray dependency from `^0.31.0` to `^0.32.0`. # revert docarray explicit transitive dependencies and use extras instead ## Who can review? @dev2049 -- reviewed the original PR @eyurtsev -- bumped the pinned protobuf dependency a few days ago --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 16:48:09 +00:00
docarray = ["docarray"]
Add Sentence Transformers Embeddings (#3409) Add embeddings based on the sentence transformers library. Add a notebook and integration tests. Co-authored-by: khimaros <me@khimaros.com>
2023-04-24 01:25:20 +00:00
embeddings = ["sentence-transformers"]
Add AzureCognitiveServicesToolkit to call Azure Cognitive Services API (#5012) # Add AzureCognitiveServicesToolkit to call Azure Cognitive Services API: achieve some multimodal capabilities This PR adds a toolkit named AzureCognitiveServicesToolkit which bundles the following tools: - AzureCogsImageAnalysisTool: calls Azure Cognitive Services image analysis API to extract caption, objects, tags, and text from images. - AzureCogsFormRecognizerTool: calls Azure Cognitive Services form recognizer API to extract text, tables, and key-value pairs from documents. - AzureCogsSpeech2TextTool: calls Azure Cognitive Services speech to text API to transcribe speech to text. - AzureCogsText2SpeechTool: calls Azure Cognitive Services text to speech API to synthesize text to speech. This toolkit can be used to process image, document, and audio inputs. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-23 13:45:48 +00:00
azure = ["azure-identity", "azure-cosmos", "openai", "azure-core", "azure-ai-formrecognizer", "azure-ai-vision", "azure-cognitiveservices-speech"]
Harrison/neo4j (#5078) Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 14:31:48 +00:00
all = [
"anthropic",
"cohere",
"openai",
"nlpcloud",
"huggingface_hub",
"jina",
"manifest-ml",
"elasticsearch",
"opensearch-py",
"google-search-results",
"faiss-cpu",
"sentence-transformers",
"transformers",
"spacy",
"nltk",
"wikipedia",
"beautifulsoup4",
"tiktoken",
"torch",
"jinja2",
"pinecone-client",
"pinecone-text",
adding MongoDBAtlasVectorSearch (#5338) # Add MongoDBAtlasVectorSearch for the python library Fixes #5337 --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-30 14:59:01 +00:00
"pymongo",
Harrison/neo4j (#5078) Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 14:31:48 +00:00
"weaviate-client",
"redis",
"google-api-python-client",
Ability to specify credentials wihen using Google BigQuery as a data loader (#5466) # Adds ability to specify credentials when using Google BigQuery as a data loader Fixes #5465 . Adds ability to set credentials which must be of the `google.auth.credentials.Credentials` type. This argument is optional and will default to `None. Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-30 23:25:22 +00:00
"google-auth",
Harrison/neo4j (#5078) Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 14:31:48 +00:00
"wolframalpha",
"qdrant-client",
"tensorflow-text",
"pypdf",
"networkx",
"nomic",
"aleph-alpha-client",
"deeplake",
"pgvector",
"psycopg2-binary",
"pyowm",
"pytesseract",
"html2text",
"atlassian-python-api",
"gptcache",
"duckduckgo-search",
"arxiv",
"azure-identity",
"clickhouse-connect",
"azure-cosmos",
"lancedb",
WhyLabs callback (#4906) # Add a WhyLabs callback handler * Adds a simple WhyLabsCallbackHandler * Add required dependencies as optional * protect against missing modules with imports * Add docs/ecosystem basic example based on initial prototype from @andrewelizondo > this integration gathers privacy preserving telemetry on text with whylogs and sends stastical profiles to WhyLabs platform to monitoring these metrics over time. For more information on what WhyLabs is see: https://whylabs.ai After you run the notebook (if you have env variables set for the API Keys, org_id and dataset_id) you get something like this in WhyLabs: ![Screenshot (443)](https://github.com/hwchase17/langchain/assets/88007022/6bdb3e1c-4243-4ae8-b974-23a8bb12edac) Co-authored-by: Andre Elizondo <andre@whylabs.ai> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-23 03:29:47 +00:00
"langkit",
Harrison/neo4j (#5078) Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 14:31:48 +00:00
"lark",
"pexpect",
"pyvespa",
"O365",
"jq",
"docarray",
"steamship",
"pdfminer-six",
"lxml",
"requests-toolbelt",
"neo4j",
Add AzureCognitiveServicesToolkit to call Azure Cognitive Services API (#5012) # Add AzureCognitiveServicesToolkit to call Azure Cognitive Services API: achieve some multimodal capabilities This PR adds a toolkit named AzureCognitiveServicesToolkit which bundles the following tools: - AzureCogsImageAnalysisTool: calls Azure Cognitive Services image analysis API to extract caption, objects, tags, and text from images. - AzureCogsFormRecognizerTool: calls Azure Cognitive Services form recognizer API to extract text, tables, and key-value pairs from documents. - AzureCogsSpeech2TextTool: calls Azure Cognitive Services speech to text API to transcribe speech to text. - AzureCogsText2SpeechTool: calls Azure Cognitive Services text to speech API to synthesize text to speech. This toolkit can be used to process image, document, and audio inputs. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-23 13:45:48 +00:00
"openlm",
"azure-ai-formrecognizer",
"azure-ai-vision",
"azure-cognitiveservices-speech",
feat: Support for `Tigris` Vector Database for vector search (#5703) ### Changes - New vector store integration - [Tigris](https://tigrisdata.com) - Adds [tigrisdb](https://pypi.org/project/tigrisdb/) optional dependency - Example notebook demonstrating usage Fixes #5535 Closes tigrisdata/tigris-client-python#40 #### Twitter handles We'd love a shoutout on our [@TigrisData](https://twitter.com/TigrisData) and [@adilansari](https://twitter.com/adilansari) twitter handles #### Who can review? @dev2049 --------- Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-06-06 03:39:16 +00:00
"momento",
"tigrisdb"
Harrison/neo4j (#5078) Co-authored-by: Tomaz Bratanic <bratanic.tomaz@gmail.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-22 14:31:48 +00:00
]
feature/4493 Improve Evernote Document Loader (#4577) # Improve Evernote Document Loader When exporting from Evernote you may export more than one note. Currently the Evernote loader concatenates the content of all notes in the export into a single document and only attaches the name of the export file as metadata on the document. This change ensures that each note is loaded as an independent document and all available metadata on the note e.g. author, title, created, updated are added as metadata on each document. It also uses an existing optional dependency of `html2text` instead of `pypandoc` to remove the need to download the pandoc application via `download_pandoc()` to be able to use the `pypandoc` python bindings. Fixes #4493 Co-authored-by: Mike McGarry <mike.mcgarry@finbourne.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-19 21:28:17 +00:00
Add workflow for testing with all deps (#4410) # Add action to test with all dependencies installed PR adds a custom action for setting up poetry that allows specifying a cache key: https://github.com/actions/setup-python/issues/505#issuecomment-1273013236 This makes it possible to run 2 types of unit tests: (1) unit tests with only core dependencies (2) unit tests with extended dependencies (e.g., those that rely on an optional pdf parsing library) As part of this PR, we're moving some pdf parsing tests into the unit-tests section and making sure that these unit tests get executed when running with extended dependencies.
2023-05-10 13:35:07 +00:00
# An extra used to be able to add extended testing.
Docugami DataLoader (#4727) ### Adds a document loader for Docugami Specifically: 1. Adds a data loader that talks to the [Docugami](http://docugami.com) API to download processed documents as semantic XML 2. Parses the semantic XML into chunks, with additional metadata capturing chunk semantics 3. Adds a detailed notebook showing how you can use additional metadata returned by Docugami for techniques like the [self-querying retriever](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query_retriever.html) 4. Adds an integration test, and related documentation Here is an example of a result that is not possible without the capabilities added by Docugami (from the notebook): <img width="1585" alt="image" src="https://github.com/hwchase17/langchain/assets/749277/bb6c1ce3-13dc-4349-a53b-de16681fdd5b"> --------- Co-authored-by: Taqi Jaffri <tjaffri@docugami.com> Co-authored-by: Taqi Jaffri <tjaffri@gmail.com>
2023-05-15 14:53:00 +00:00
# Please use new-line on formatting to make it easier to add new packages without
# merge-conflicts
Turn on extended tests (#4588) # Turn on strict extended tests This PR turns on strict testing for extended tests.
2023-05-12 18:50:08 +00:00
extended_testing = [
Add beautiful soup 4 to extended testing extra (#4869) # Add bs4 to extended testing extra Updating extended testing extra in preparation for more refactors.
2023-05-17 18:11:26 +00:00
"beautifulsoup4",
Bibtex integration for document loader and retriever (#5137) # Bibtex integration Wrap bibtexparser to retrieve a list of docs from a bibtex file. * Get the metadata from the bibtex entries * `page_content` get from the local pdf referenced in the `file` field of the bibtex entry using `pymupdf` * If no valid pdf file, `page_content` set to the `abstract` field of the bibtex entry * Support Zotero flavour using regex to get the file path * Added usage example in `docs/modules/indexes/document_loaders/examples/bibtex.ipynb` --------- Co-authored-by: Sébastien M. Popoff <sebastien.popoff@espci.fr> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-25 07:21:31 +00:00
"bibtexparser",
feat #4479: TextLoader auto detect encoding and improved exceptions (#4927) # TextLoader auto detect encoding and enhanced exception handling - Add an option to enable encoding detection on `TextLoader`. - The detection is done using `chardet` - The loading is done by trying all detected encodings by order of confidence or raise an exception otherwise. ### New Dependencies: - `chardet` Fixes #4479 ## Before submitting <!-- If you're adding a new integration, include an integration test and an example notebook showing its use! --> ## Who can review? Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested: - @eyurtsev --------- Co-authored-by: blob42 <spike@w530>
2023-05-18 13:55:14 +00:00
"chardet",
Docugami DataLoader (#4727) ### Adds a document loader for Docugami Specifically: 1. Adds a data loader that talks to the [Docugami](http://docugami.com) API to download processed documents as semantic XML 2. Parses the semantic XML into chunks, with additional metadata capturing chunk semantics 3. Adds a detailed notebook showing how you can use additional metadata returned by Docugami for techniques like the [self-querying retriever](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query_retriever.html) 4. Adds an integration test, and related documentation Here is an example of a result that is not possible without the capabilities added by Docugami (from the notebook): <img width="1585" alt="image" src="https://github.com/hwchase17/langchain/assets/749277/bb6c1ce3-13dc-4349-a53b-de16681fdd5b"> --------- Co-authored-by: Taqi Jaffri <tjaffri@docugami.com> Co-authored-by: Taqi Jaffri <tjaffri@gmail.com>
2023-05-15 14:53:00 +00:00
"jq",
"pdfminer.six",
"pypdf",
Clean up tests for pdf parsers (#4595) # Organize tests for pdf parsers Clean up tests for pdf parsers, remove duplicate tests, convert to unit tests.
2023-05-15 18:21:05 +00:00
"pymupdf",
"pypdfium2",
Docugami DataLoader (#4727) ### Adds a document loader for Docugami Specifically: 1. Adds a data loader that talks to the [Docugami](http://docugami.com) API to download processed documents as semantic XML 2. Parses the semantic XML into chunks, with additional metadata capturing chunk semantics 3. Adds a detailed notebook showing how you can use additional metadata returned by Docugami for techniques like the [self-querying retriever](https://python.langchain.com/en/latest/modules/indexes/retrievers/examples/self_query_retriever.html) 4. Adds an integration test, and related documentation Here is an example of a result that is not possible without the capabilities added by Docugami (from the notebook): <img width="1585" alt="image" src="https://github.com/hwchase17/langchain/assets/749277/bb6c1ce3-13dc-4349-a53b-de16681fdd5b"> --------- Co-authored-by: Taqi Jaffri <tjaffri@docugami.com> Co-authored-by: Taqi Jaffri <tjaffri@gmail.com>
2023-05-15 14:53:00 +00:00
"tqdm",
"lxml",
Add Confluence Loader unit tests (#3333) Adds some basic unit tests for the ConfluenceLoader that can be extended later. Ports this [PR from llama-hub](https://github.com/emptycrown/llama-hub/pull/208) and adapts it to `langchain`. @Jflick58 and @zywilliamli adding you here as potential reviewers --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-16 22:17:07 +00:00
"atlassian-python-api",
"beautifulsoup4",
fix(document_loaders/telegram): fix pandas calls + add tests (#4806) # Fix Telegram API loader + add tests. I was testing this integration and it was broken with next error: ```python message_threads = loader._get_message_threads(df) KeyError: False ``` Also, this particular loader didn't have any tests / related group in poetry, so I added those as well. @hwchase17 / @eyurtsev please take a look on this fix PR. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-16 21:35:25 +00:00
"pandas",
"telethon",
Harrison/psychic (#5063) Co-authored-by: Ayan Bandyopadhyay <ayanb9440@gmail.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-21 16:13:20 +00:00
"psychicapi",
feature/4493 Improve Evernote Document Loader (#4577) # Improve Evernote Document Loader When exporting from Evernote you may export more than one note. Currently the Evernote loader concatenates the content of all notes in the export into a single document and only attaches the name of the export file as metadata on the document. This change ensures that each note is loaded as an independent document and all available metadata on the note e.g. author, title, created, updated are added as metadata on each document. It also uses an existing optional dependency of `html2text` instead of `pypandoc` to remove the need to download the pandoc application via `download_pandoc()` to be able to use the `pypandoc` python bindings. Fixes #4493 Co-authored-by: Mike McGarry <mike.mcgarry@finbourne.com> Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-19 21:28:17 +00:00
"zep-python",
Fix graphql tool (#4984) Fix construction and add unit test.
2023-05-19 22:27:50 +00:00
"gql",
"requests_toolbelt",
tfidf retriever (#5114) Co-authored-by: vempaliakhil96 <vempaliakhil96@gmail.com>
2023-05-24 17:02:09 +00:00
"html2text",
New Trello document loader (#4767) # Added New Trello loader class and documentation Simple Loader on top of py-trello wrapper. With a board name you can pull cards and to do some field parameter tweaks on load operation. I included documentation and examples. Included unit test cases using patch and a fixture for py-trello client class. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-30 02:47:56 +00:00
"py-trello",
tfidf retriever (#5114) Co-authored-by: vempaliakhil96 <vempaliakhil96@gmail.com>
2023-05-24 17:02:09 +00:00
"scikit-learn",
Feature: Qdrant filters supports (#5446) # Support Qdrant filters Qdrant has an [extensive filtering system](https://qdrant.tech/documentation/concepts/filtering/) with rich type support. This PR makes it possible to use the filters in Langchain by passing an additional param to both the `similarity_search_with_score` and `similarity_search` methods. ## Who can review? @dev2049 @hwchase17 --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-31 09:26:16 +00:00
"pyspark"
Turn on extended tests (#4588) # Turn on strict extended tests This PR turns on strict testing for extended tests.
2023-05-12 18:50:08 +00:00
]
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
ruff ruff (#1203)
2023-02-25 16:59:52 +00:00
[tool.ruff]
select = [
"E", # pycodestyle
"F", # pyflakes
"I", # isort
]
Add PythonLoader which auto-detects encoding of Python files (#3311) This PR contributes a `PythonLoader`, which inherits from `TextLoader` but detects and sets the encoding automatically.
2023-04-21 17:47:57 +00:00
exclude = [
"tests/integration_tests/examples/non-utf8-encoding.py",
]
initial commit
2022-10-24 21:51:15 +00:00
[tool.mypy]
ignore_missing_imports = "True"
disallow_untyped_defs = "True"
exclude = ["notebooks"]
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
chore: move coverage config to pyproject (#694) This PR aims to move the contents of `.coveragerc` to `pyproject.toml` to make the overall file structure more minimal.
2023-01-22 22:48:20 +00:00
[tool.coverage.run]
omit = [
"tests/*",
]
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
[build-system]
Use a consistent poetry version everywhere (#3250) Fixes the discrepancy of poetry version in Dockerfile and the GAs
2023-04-25 01:19:51 +00:00
requires = ["poetry-core>=1.0.0"]
chore: use poetry as dependency manager (#242) * Adopts [Poetry](https://python-poetry.org/) as a dependency manager * Introduces dependency version requirements * Deprecates Python 3.7 support **TODO** - [x] Update developer guide - [x] Add back `playwright`, `manifest-ml`, and `jupyter` to dependency group **Not Doing => Fast Follow** - Investigate single source for version, perhaps relying on GitHub tags and [tackling this issue](https://github.com/hwchase17/langchain/issues/26)
2022-12-04 00:42:59 +00:00
build-backend = "poetry.core.masonry.api"
Add progress bar to filesystemblob loader, update pytest config for unit tests (#4212) This PR adds: * Option to show a tqdm progress bar when using the file system blob loader * Update pytest run configuration to be stricter * Adding a new marker that checks that required pkgs exist
2023-05-08 20:15:09 +00:00
[tool.pytest.ini_options]
# --strict-markers will raise errors on unknown marks.
# https://docs.pytest.org/en/7.1.x/how-to/mark.html#raising-errors-on-unknown-marks
#
# https://docs.pytest.org/en/7.1.x/reference/reference.html
# --strict-config any warnings encountered while parsing the `pytest`
# section of the configuration file raise errors.
addopts = "--strict-markers --strict-config --durations=5"
# Registering custom markers.
# https://docs.pytest.org/en/7.1.x/example/markers.html#registering-markers
markers = [
"requires: mark tests as requiring a specific library"
]
Reference in New Issue Copy Permalink