mirror of
https://github.com/hwchase17/langchain
synced 2024-10-31 15:20:26 +00:00
082976d8d0
This link for the notebook of OpenLLM is not migrated to the new format Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com> <!-- Thank you for contributing to LangChain! Replace this comment with: - Description: a description of the change, - Issue: the issue # it fixes (if applicable), - Dependencies: any dependencies required for this change, - Tag maintainer: for a quicker response, tag the relevant maintainer (see below), - Twitter handle: we announce bigger features on Twitter. If your PR gets announced and you'd like a mention, we'll gladly shout you out! If you're adding a new integration, please include: 1. a test for the integration, preferably unit tests that do not rely on network access, 2. an example notebook showing its use. Maintainer responsibilities: - General / Misc / if you don't know who to tag: @dev2049 - DataLoaders / VectorStores / Retrievers: @rlancemartin, @eyurtsev - Models / Prompts: @hwchase17, @dev2049 - Memory: @hwchase17 - Agents / Tools / Toolkits: @vowelparrot - Tracing / Callbacks: @agola11 - Async: @agola11 If no one reviews your PR within a few days, feel free to @-mention the same people again. See contribution guidelines for more information on how to write/run tests, lint, etc: https://github.com/hwchase17/langchain/blob/master/.github/CONTRIBUTING.md --> Signed-off-by: Aaron <29749331+aarnphm@users.noreply.github.com>
71 lines
1.7 KiB
Plaintext
71 lines
1.7 KiB
Plaintext
# OpenLLM
|
|
|
|
This page demonstrates how to use [OpenLLM](https://github.com/bentoml/OpenLLM)
|
|
with LangChain.
|
|
|
|
`OpenLLM` is an open platform for operating large language models (LLMs) in
|
|
production. It enables developers to easily run inference with any open-source
|
|
LLMs, deploy to the cloud or on-premises, and build powerful AI apps.
|
|
|
|
## Installation and Setup
|
|
|
|
Install the OpenLLM package via PyPI:
|
|
|
|
```bash
|
|
pip install openllm
|
|
```
|
|
|
|
## LLM
|
|
|
|
OpenLLM supports a wide range of open-source LLMs as well as serving users' own
|
|
fine-tuned LLMs. Use `openllm model` command to see all available models that
|
|
are pre-optimized for OpenLLM.
|
|
|
|
## Wrappers
|
|
|
|
There is a OpenLLM Wrapper which supports loading LLM in-process or accessing a
|
|
remote OpenLLM server:
|
|
|
|
```python
|
|
from langchain.llms import OpenLLM
|
|
```
|
|
|
|
### Wrapper for OpenLLM server
|
|
|
|
This wrapper supports connecting to an OpenLLM server via HTTP or gRPC. The
|
|
OpenLLM server can run either locally or on the cloud.
|
|
|
|
To try it out locally, start an OpenLLM server:
|
|
|
|
```bash
|
|
openllm start flan-t5
|
|
```
|
|
|
|
Wrapper usage:
|
|
|
|
```python
|
|
from langchain.llms import OpenLLM
|
|
|
|
llm = OpenLLM(server_url='http://localhost:3000')
|
|
|
|
llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")
|
|
```
|
|
|
|
### Wrapper for Local Inference
|
|
|
|
You can also use the OpenLLM wrapper to load LLM in current Python process for
|
|
running inference.
|
|
|
|
```python
|
|
from langchain.llms import OpenLLM
|
|
|
|
llm = OpenLLM(model_name="dolly-v2", model_id='databricks/dolly-v2-7b')
|
|
|
|
llm("What is the difference between a duck and a goose? And why there are so many Goose in Canada?")
|
|
```
|
|
|
|
### Usage
|
|
|
|
For a more detailed walkthrough of the OpenLLM Wrapper, see the
|
|
[example notebook](/docs/modules/model_io/models/llms/integrations/openllm.html)
|