Commit Graph

2240 Commits

Author SHA1 Message Date
Yuekai Zhang
1ed4228822
Fix bilibili (#4860)
# Fix bilibili api import error

bilibili-api package is depracated and there is no sync module.

<!--
Thank you for contributing to LangChain! Your PR will appear in our next
release under the title you set. Please make sure it highlights your
valuable contribution.

Replace this with a description of the change, the issue it fixes (if
applicable), and relevant context. List any dependencies required for
this change.

After you're done, someone will review your PR. They may suggest
improvements. If no one reviews your PR within a few days, feel free to
@-mention the same people again, as notifications can get lost.
-->

<!-- Remove if not applicable -->

Fixes #2673 #2724 

## Before submitting

<!-- If you're adding a new integration, include an integration test and
an example notebook showing its use! -->

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
@vowelparrot  @liaokongVFX 

<!-- For a quicker response, figure out the right person to tag with @

        @hwchase17 - project lead

        Tracing / Callbacks
        - @agola11

        Async
        - @agola11

        DataLoaders
        - @eyurtsev

        Models
        - @hwchase17
        - @agola11

        Agents / Tools / Toolkits
        - @vowelparrot
        
        VectorStores / Retrievers / Memory
        - @dev2049
        
 -->
2023-05-18 09:56:51 -04:00
Eugene Yurtsev
e46202829f
feat #4479: TextLoader auto detect encoding and improved exceptions (#4927)
# TextLoader auto detect encoding and enhanced exception handling

- Add an option to enable encoding detection on `TextLoader`. 
- The detection is done using `chardet`
- The loading is done by trying all detected encodings by order of
confidence or raise an exception otherwise.

### New Dependencies:
- `chardet`

Fixes #4479 

## Before submitting

<!-- If you're adding a new integration, include an integration test and
an example notebook showing its use! -->

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:

- @eyurtsev

---------

Co-authored-by: blob42 <spike@w530>
2023-05-18 09:55:14 -04:00
张城铭
8c28ad6dac
API update: Engines -> Models (#4915)
# API update: Engines -> Models

see: https://community.openai.com/t/api-update-engines-models/18597

Co-authored-by: assert <zhangchengming@kkguan.com>
2023-05-18 09:54:42 -04:00
Eugene Yurtsev
c06a47a691
Load specific file types from Google Drive (issue #4878) (#4926)
# Load specific file types from Google Drive (issue #4878)
Add the possibility to define what file types you want to load from
Google Drive.
 
```
 loader = GoogleDriveLoader(
    folder_id="1yucgL9WGgWZdM1TOuKkeghlPizuzMYb5",
    file_types=["document", "pdf"]
    recursive=False
)
```

Fixes ##4878

## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
DataLoaders
- @eyurtsev

Twitter: [@UmerHAdil](https://twitter.com/@UmerHAdil) | Discord:
RicChilligerDude#7589

---------

Co-authored-by: UmerHA <40663591+UmerHA@users.noreply.github.com>
2023-05-18 09:27:53 -04:00
Harrison Chase
dfbf45f028
bump version to 173 (#4910) 2023-05-17 23:36:45 -07:00
Harrison Chase
b8d48939a2
Harrison/unified objectives (#4905)
Co-authored-by: Matthias Samwald <samwald@gmx.at>
2023-05-17 23:03:57 -07:00
Harrison Chase
9165267f8a
Harrison/improved retry tool (#4842) 2023-05-17 21:41:01 -07:00
Harrison Chase
ba023d53ca
Harrison/faiss norm (#4903)
Co-authored-by: Jiaxin Shan <seedjeffwan@gmail.com>
2023-05-17 21:40:49 -07:00
Harrison Chase
9e2227ba11
Harrison/serper api bug (#4902)
Co-authored-by: Jerry Luan <xmaswillyou@gmail.com>
2023-05-17 21:40:39 -07:00
Leonid Ganeline
c998569c8f
docs: text splitters improvements (#4490)
#docs: text splitters improvements

Changes are only in the Jupyter notebooks.
- added links to the source packages and a short description of these
packages
- removed " Text Splitters" suffixes from the TOC elements (they made
the list of the text splitters messy)
- moved text splitters, based on the length function into a separate
list. They can be mixed with any classes from the "Text Splitters", so
it is a different classification.

## Who can review?
        @hwchase17 - project lead
        @eyurtsev
        @vowelparrot

NOTE: please, check out the results of the `Python code` text splitter
example (text_splitters/examples/python.ipynb). It looks suboptimal.
2023-05-17 21:33:34 -07:00
Steve Kim
613bf9b514
Update getting_started.md (#4482)
# Added another helpful way for developers who want to set OpenAI API
Key dynamically

Previous methods like exporting environment variables are good for
project-wide settings.
But many use cases need to assign API keys dynamically, recently.

```python
from langchain.llms import OpenAI
llm = OpenAI(openai_api_key="OPENAI_API_KEY")
```

## Before submitting
```bash
export OPENAI_API_KEY="..."
```
Or,
```python
import os
os.environ["OPENAI_API_KEY"] = "..."
```

<hr>

Thank you.
Cheers,
Bongsang
2023-05-17 21:32:25 -07:00
Ismael G Serrano
41e2394c9c
Fix AzureOpenAI embeddings documentation example. model -> deployment (#4389)
# Documentation for Azure OpenAI embeddings model

- OPENAI_API_VERSION environment variable is needed for the endpoint
- The constructor does not work with model, it works with deployment.

I fixed it in the notebook.

(This is my first contribution)

## Who can review?

@hwchase17 
@agola

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
2023-05-17 21:05:53 -07:00
Davis Chase
a4ac006658
Update gallery (#4873) 2023-05-17 20:59:41 -07:00
Davis Chase
8966f61ca5
Zep memory (#4898)
Co-authored-by: Daniel Chalef <daniel.chalef@private.org>
Co-authored-by: Daniel Chalef <131175+danielchalef@users.noreply.github.com>
2023-05-17 20:01:01 -07:00
Davis Chase
e28bdf4453
Cadlabs/python tool sanitization (#4754)
Co-authored-by: BenSchZA <BenSchZA@users.noreply.github.com>
2023-05-17 19:46:12 -07:00
Eugene Yurtsev
0dc304ca80
Add html parsers (#4874)
# Add bs4 html parser

* Some minor refactors
* Extract the bs4 html parsing code from the bs html loader
* Move some tests from integration tests to unit tests
2023-05-17 22:39:11 -04:00
Eugene Yurtsev
8e41143bf5
Add a generic document loader (#4875)
# Add generic document loader

* This PR adds a generic document loader which can assemble a loader
from a blob loader and a parser
* Adds a registry for parsers
* Populate registry with a default mimetype based parser


## Expected changes

- Parsing involves loading content via IO so can be sped up via:
  * Threading in sync
  * Async  
- The actual parsing logic may be computatinoally involved: may need to
figure out to add multi-processing support
- May want to add suffix based parser since suffixes are easier to
specify in comparison to mime types

## Before submitting

No notebooks yet, we first need to get a few of the basic parsers up
(prior to advertising the interface)
2023-05-17 22:38:55 -04:00
Davis Chase
df0c33a005
Faiss no avx2 (#4895)
Co-authored-by: Ali Mirlou <alimirlou@gmail.com>
2023-05-17 19:18:57 -07:00
Emil Ahlbäck
5c9205d5f4
ConversationalChatAgent: Allow customizing TEMPLATE_TOOL_RESPONSE (#2361)
It's currently not possible to change the `TEMPLATE_TOOL_RESPONSE`
prompt for ConversationalChatAgent, this PR changes that.

---------

Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-17 17:23:08 -07:00
Zander Chase
1ff7c958b0
Bold Crumbs (#4876) 2023-05-17 22:50:35 +00:00
Alexander Miasoiedov (Myasoedov)
4c3ab55e94
feat(Add FastAPI + Vercel deployment option): (#4520)
# Update deployments doc with langcorn API server

API server example 

```python
from fastapi import FastAPI

from langcorn import create_service

app: FastAPI = create_service(
    "examples.ex1:chain",
    "examples.ex2:chain",
    "examples.ex3:chain",
    "examples.ex4:sequential_chain",
    "examples.ex5:conversation",
    "examples.ex6:conversation_with_summary",
)

```
More examples: https://github.com/msoedov/langcorn/tree/main/examples

Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-17 15:50:25 -07:00
Taqi Jaffri
ef8b5f64bc
Tiny code review and docs fix for Docugami DataLoader (#4877)
# Docs and code review fixes for Docugami DataLoader

1. I noticed a couple of hyperlinks that are not loading in the
langchain docs (I guess need explicit anchor tags). Added those.
2. In code review @eyurtsev had a
[suggestion](https://github.com/hwchase17/langchain/pull/4727#discussion_r1194069347)
to allow string paths. Turns out just updating the type works (I tested
locally with string paths).

# Pre-submission checks
I ran `make lint` and `make tests` successfully.

---------

Co-authored-by: Taqi Jaffri <tjaffri@docugami.com>
2023-05-17 15:31:43 -07:00
C.J. Jameson
d6e0b9a43d
fix homepage typo (#4883)
# Fix Homepage Typo

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested... not sure
2023-05-17 15:30:23 -07:00
Leonid Ganeline
b96ab4b763
docs retriever improvements (#4430)
# Docs: improvements in the `retrievers/examples/` notebooks

Its primary purpose is to make the Jupyter notebook examples
**consistent** and more suitable for first-time viewers.
- add links to the integration source (if applicable) with a short
description of this source;
- removed `_retriever` suffix from the file names (where it existed) for
consistency;
- removed ` retriever` from the notebook title (where it existed) for
consistency;
- added code to install necessary Python package(s);
- added code to set up the necessary API Key.
- very small fixes in notebooks from other folders (for consistency):
  - docs/modules/indexes/vectorstores/examples/elasticsearch.ipynb
  - docs/modules/indexes/vectorstores/examples/pinecone.ipynb
  - docs/modules/models/llms/integrations/cohere.ipynb
- fixed misspelling in langchain/retrievers/time_weighted_retriever.py
comment (sorry, about this change in a .py file )

## Who can review
@dev2049
2023-05-17 15:29:22 -07:00
Justin Levi Winter
0147f845f1
Update getting_started.ipynb (#4850)
minor grammer issue
2023-05-17 13:19:14 -07:00
Yong Fu
3e12f0957a
Remove unused variables in Milvus vectorstore (#4868)
# Remove unused variables in Milvus vectorstore
This PR simply removes a variable unused in Milvus. The variable looks
like a copy-paste from other functions in Milvus but it is really
unnecessary.
2023-05-17 12:00:37 -07:00
Eugene Yurtsev
c5ab9782c6
Add beautiful soup 4 to extended testing extra (#4869)
# Add bs4 to extended testing extra

Updating extended testing extra in preparation for more refactors.
2023-05-17 14:11:26 -04:00
Ryan Culligan
6a9cdc43f5
Fix TypeError in Vectorstore Redis class methods (#4857)
# Fix TypeError in Vectorstore Redis class methods

This change resolves a TypeError that was raised when invoking the
`from_texts_return_keys` method from the `from_texts` method in the
`Redis` class. The error was due to the `cls` argument being passed
explicitly, which led to it being provided twice since it's also
implicitly passed in class methods. No relevant tests were added as the
issue appeared to be better suited for linters to catch proactively.

Changes:
- Removed `cls=cls` from the call to `from_texts_return_keys` in the
`from_texts` method.

Related to:
https://github.com/hwchase17/langchain/pull/4653
2023-05-17 10:48:09 -07:00
Eugene Yurtsev
2d20a1196e
Hugging Face Loader: Add lazy load (#4799)
# Add lazy load to HF datasets loader

Unfortunately, there are no tests as far as i can tell. Verified code manually.
2023-05-17 12:04:23 -04:00
Davis Chase
a63ab7ded1
bump 172 (#4864) 2023-05-17 08:54:39 -07:00
yujiosaka
2f8eb95a91
Remove unnecessary comment (#4845)
# Remove unnecessary comment

Remove unnecessary comment accidentally included in #4800

## Before submitting

- no test
- no document

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
2023-05-17 11:53:03 -04:00
UmerHA
e257380deb
Typos (#4851)
# Fixed typos (issues #4818 & #4668 & more typos)
- At some places, it said `model = ChatOpenAI(model='gpt-3.5-turbo')`
but should be `model = ChatOpenAI(model_name='gpt-3.5-turbo')`
- Fixes some other typos

Fixes #4818, #4668

## Who can review?
Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:
        Models
        - @hwchase17
        - @agola11
        Agents / Tools / Toolkits
        - @vowelparrot
2023-05-17 11:52:22 -04:00
Zander Chase
8dcad0f272
Add Support for Flexible Input Format for LLM and Chat Model Runs (#4805)
Previously, the client expected a strict 'prompt' or 'messages' format
and wouldn't permit running a chat model or llm on prompts or messages
(respectively).

Since many datasets may want to specify custom key: string , relax this
requirement.
Also, add support for running a chat model on raw prompts and LLM on
chat messages through their respective fallbacks.
2023-05-17 14:24:17 +00:00
Zander Chase
a47c62fcba
Add dev option (#4828)
enable running
```
langchain plus start --dev
```

To use the RC iamges instead
2023-05-17 14:09:25 +00:00
Harrison Chase
720ac49f42
2markdown loader (#4796)
Co-authored-by: Eugene Yurtsev <eyurtsev@gmail.com>
2023-05-16 23:42:53 -07:00
Ankush Gola
aa73a888fa
Some notebook and client fixes (add retries, clean up docs, etc) (#4820)
# Your PR Title (What it does)

<!--
Thank you for contributing to LangChain! Your PR will appear in our next
release under the title you set. Please make sure it highlights your
valuable contribution.

Replace this with a description of the change, the issue it fixes (if
applicable), and relevant context. List any dependencies required for
this change.

After you're done, someone will review your PR. They may suggest
improvements. If no one reviews your PR within a few days, feel free to
@-mention the same people again, as notifications can get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting

<!-- If you're adding a new integration, include an integration test and
an example notebook showing its use! -->

## Who can review?

Community members can review the PR once tests pass. Tag
maintainers/contributors who might be interested:

<!-- For a quicker response, figure out the right person to tag with @

        @hwchase17 - project lead

        Tracing / Callbacks
        - @agola11

        Async
        - @agola11

        DataLoaders
        - @eyurtsev

        Models
        - @hwchase17
        - @agola11

        Agents / Tools / Toolkits
        - @vowelparrot
        
        VectorStores / Retrievers / Memory
        - @dev2049
        
 -->
2023-05-16 20:23:00 -07:00
Davis Chase
0a591da6db
Add weaviate by_text (#4824)
Thanks @ZouhairElhadi! Made small change

Closes #4742

---------

Co-authored-by: Zouhair Elhadi <zouhair11elhadi@gmail.com>
Co-authored-by: ZouhairElhadi <87149442+ZouhairElhadi@users.noreply.github.com>
2023-05-16 19:43:15 -07:00
Zander Chase
d1b6839d97
Retry session and tenant (#4822) 2023-05-17 01:54:40 +00:00
Nguyen Trung Duc (john)
49e4aaf673
Fix subclassing OpenAIEmbeddings (#4500)
# Fix subclassing OpenAIEmbeddings

Fixes #4498 

## Before submitting

- Problem: Due to annotated type `Tuple[()]`.
- Fix: Change the annotated type to "Iterable[str]". Even though
tiktoken use
[Collection[str]](095924e02c/tiktoken/core.py (L80))
type annotation, but pydantic doesn't support Collection type, and
[Iterable](https://docs.pydantic.dev/latest/usage/types/#typing-iterables)
is the closest to Collection.
2023-05-16 18:35:19 -07:00
Harrison Chase
08df80bed6
console callback verbose (#4696)
add verbose callback

Co-authored-by: vowelparrot <130414180+vowelparrot@users.noreply.github.com>
2023-05-17 01:28:43 +00:00
David Peterson
d5d4c0a172
Update summarize.ipynb (#4529)
# Update order in which tasks are stated (logically correct)

Fixes the order in which steps are placed under titles.

@vowelparrot
2023-05-16 18:14:00 -07:00
Django
bcffc704c1
fix: agenerate miss run_manager args in llm.py (#4566)
# fix: agenerate miss run_manager args in llm.py

<!--
Thank you for contributing to LangChain! Your PR will appear in our next
release under the title you set. Please make sure it highlights your
valuable contribution.

Replace this with a description of the change, the issue it fixes (if
applicable), and relevant context. List any dependencies required for
this change.

After you're done, someone will review your PR. They may suggest
improvements. If no one reviews your PR within a few days, feel free to
@-mention the same people again, as notifications can get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
fix: agenerate miss run_manager args in llm.py


<!-- For a quicker response, figure out the right person to tag with @

        @hwchase17 - project lead

        Tracing / Callbacks
        - @agola11

        Async
        - @agola11

        DataLoaders
        - @eyurtsev

        Models
        - @hwchase17
        - @agola11

        Agents / Tools / Toolkits
        - @vowelparrot
        
        VectorStores / Retrievers / Memory
        - @dev2049
        
 -->
2023-05-16 17:37:56 -07:00
Brendan Mannix
4e56d3119c
update qdrant docs to reflect the proper way to initialize Qdrant() constructor (#4596)
# update qdrant docs to reflect the proper way to initialize Qdrant()
constructor

The [Qdrant
docs](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/qdrant.html)
still contain an old reference for passing an `embedding_function` into
the constructor. This is no longer supported.

This PR updates the docs to reflect the proper way to initialize
`Qdrant()`

Old:
![Screenshot 2023-05-12 at 3 06 33
PM](https://github.com/hwchase17/langchain/assets/1552962/dd4063d2-2a07-4340-91bb-e305f7215ddd)

New:
![Screenshot 2023-05-12 at 3 21 09
PM](https://github.com/hwchase17/langchain/assets/1552962/aebc3f63-1a8b-4ca3-93c0-a2ce30dcd282)
2023-05-16 17:30:38 -07:00
Sean Morgan
5372a06a8c
DOC: Fix SageMaker example (#4598)
# Fix SageMaker example typing

Since https://github.com/hwchase17/langchain/pull/3249 a new type
`LLMContentHandler` is enforced for SageMaker Endpoints

Fixes #4168
2023-05-16 17:28:16 -07:00
Steve Kim
e90654f39b
Added cleaning up the downloaded PDF files (#4601)
ArxivAPIWrapper searches and downloads PDFs to get related information.
But I found that it doesn't delete the downloaded file. The reason why
this is a problem is that a lot of PDF files remain on the server. For
example, one size is about 28M.
So, I added a delete line because it's too big to maintain on the
server.

# Clean up downloaded PDF files
- Changes: Added new line to delete downloaded file
- Background: To get the information on arXiv's paper, ArxivAPIWrapper
class downloads a PDF.
It's a natural approach, but the wrapper retains a lot of PDF files on
the server.
- Problem: One size of PDFs is about 28M. It's too big to maintain on a
small server like AWS.
- Dependency: import os

Thank you.

---------

Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-16 17:26:56 -07:00
Quinn
6fbd5e837f
Update planner_prompt.py, change usery to user (#4623)
# Fix misspell in planner_prompt.py

before

```
Usery query: I want to buy a couch
```

after

```
User query: I want to buy a couch
```
2023-05-16 17:24:27 -07:00
Tony Zhang
432421ffa5
[Fix][GenerativeAgent] Get the memory importance score from regex matched group (#4636)
# Get the memory importance score from regex matched group

In `GenerativeAgentMemory`, the `_score_memory_importance()` will make a
prompt to get a rating score. The prompt is:
```
        prompt = PromptTemplate.from_template(
            "On the scale of 1 to 10, where 1 is purely mundane"
            + " (e.g., brushing teeth, making bed) and 10 is"
            + " extremely poignant (e.g., a break up, college"
            + " acceptance), rate the likely poignancy of the"
            + " following piece of memory. Respond with a single integer."
            + "\nMemory: {memory_content}"
            + "\nRating: "
        )
```
For some LLM, it will respond with, for example, `Rating: 8`. Thus we
might want to get the score from the matched regex group.
2023-05-16 16:59:50 -07:00
Daniel Maturana
be405ac139
Query_constructor.base.py function _get_prompt() not including passed examples. (#4680)
The function _get_prompt() was returning the DEFAULT_EXAMPLES even if
some custom examples were given. The return FewShotPromptTemplate was
returnong DEFAULT_EXAMPLES and not examples
2023-05-16 16:31:10 -07:00
Anam Hira
3af448d72e
Update huggingface_tools.ipynb (#4700) 2023-05-16 16:28:27 -07:00
rajib
e28f4a5f39
changed cohere.py to update the default model of embedding (#4709)
# The cohere embedding model do not use large, small. It is deprecated.
Changed the modules default model

Fixes #4694


Co-authored-by: rajib76 <rajib76@yahoo.com>
Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
2023-05-16 16:27:23 -07:00