mirror of https://github.com/hwchase17/langchain
community: refactor Baseten integration with new API endpoints & docs (#15017)
- **Description:** In response to user feedback, this PR refactors the Baseten integration with updated model endpoints, as well as updates relevant documentation. This PR has been tested by end users in production and works as expected. - **Issue:** N/A - **Dependencies:** This PR actually removes the dependency on the `baseten` package! - **Twitter handle:** https://twitter.com/basetencopull/15080/head
Philip Kiely - Baseten
6 months ago
committed by
GitHub
parent
3fc1b3553b
commit
6342da333a
@ -1,25 +1,71 @@
|
||||
# Baseten
|
||||
|
||||
Learn how to use LangChain with models deployed on Baseten.
|
||||
[Baseten](https://baseten.co) provides all the infrastructure you need to deploy and serve ML models performantly, scalably, and cost-efficiently.
|
||||
|
||||
## Installation and setup
|
||||
As a model inference platform, Baseten is a `Provider` in the LangChain ecosystem. The Baseten integration currently implements a single `Component`, LLMs, but more are planned!
|
||||
|
||||
- Create a [Baseten](https://baseten.co) account and [API key](https://docs.baseten.co/settings/api-keys).
|
||||
- Install the Baseten Python client with `pip install baseten`
|
||||
- Use your API key to authenticate with `baseten login`
|
||||
Baseten lets you run both open source models like Llama 2 or Mistral and run proprietary or fine-tuned models on dedicated GPUs. If you're used to a provider like OpenAI, using Baseten has a few differences:
|
||||
|
||||
## Invoking a model
|
||||
* Rather than paying per token, you pay per minute of GPU used.
|
||||
* Every model on Baseten uses [Truss](https://truss.baseten.co/welcome), our open-source model packaging framework, for maximum customizability.
|
||||
* While we have some [OpenAI ChatCompletions-compatible models](https://docs.baseten.co/api-reference/openai), you can define your own I/O spec with Truss.
|
||||
|
||||
Baseten integrates with LangChain through the LLM module, which provides a standardized and interoperable interface for models that are deployed on your Baseten workspace.
|
||||
You can learn more about Baseten in [our docs](https://docs.baseten.co/) or read on for LangChain-specific info.
|
||||
|
||||
You can deploy foundation models like WizardLM and Alpaca with one click from the [Baseten model library](https://app.baseten.co/explore/) or if you have your own model, [deploy it with this tutorial](https://docs.baseten.co/deploying-models/deploy).
|
||||
## Setup: LangChain + Baseten
|
||||
|
||||
In this example, we'll work with WizardLM. [Deploy WizardLM here](https://app.baseten.co/explore/wizardlm) and follow along with the deployed [model's version ID](https://docs.baseten.co/managing-models/manage).
|
||||
You'll need two things to use Baseten models with LangChain:
|
||||
|
||||
- A [Baseten account](https://baseten.co)
|
||||
- An [API key](https://docs.baseten.co/observability/api-keys)
|
||||
|
||||
Export your API key to your as an environment variable called `BASETEN_API_KEY`.
|
||||
|
||||
```sh
|
||||
export BASETEN_API_KEY="paste_your_api_key_here"
|
||||
```
|
||||
|
||||
## Component guide: LLMs
|
||||
|
||||
Baseten integrates with LangChain through the [LLM component](https://python.langchain.com/docs/integrations/llms/baseten), which provides a standardized and interoperable interface for models that are deployed on your Baseten workspace.
|
||||
|
||||
You can deploy foundation models like Mistral and Llama 2 with one click from the [Baseten model library](https://app.baseten.co/explore/) or if you have your own model, [deploy it with Truss](https://truss.baseten.co/welcome).
|
||||
|
||||
In this example, we'll work with Mistral 7B. [Deploy Mistral 7B here](https://app.baseten.co/explore/mistral_7b_instruct) and follow along with the deployed model's ID, found in the model dashboard.
|
||||
|
||||
To use this module, you must:
|
||||
|
||||
* Export your Baseten API key as the environment variable BASETEN_API_KEY
|
||||
* Get the model ID for your model from your Baseten dashboard
|
||||
* Identify the model deployment ("production" for all model library models)
|
||||
|
||||
[Learn more](https://docs.baseten.co/deploy/lifecycle) about model IDs and deployments.
|
||||
|
||||
Production deployment (standard for model library models)
|
||||
|
||||
```python
|
||||
from langchain.llms import Baseten
|
||||
from langchain_community.llms import Baseten
|
||||
|
||||
mistral = Baseten(model="MODEL_ID", deployment="production")
|
||||
mistral("What is the Mistral wind?")
|
||||
```
|
||||
|
||||
wizardlm = Baseten(model="MODEL_VERSION_ID", verbose=True)
|
||||
Development deployment
|
||||
|
||||
wizardlm("What is the difference between a Wizard and a Sorcerer?")
|
||||
```python
|
||||
from langchain_community.llms import Baseten
|
||||
|
||||
mistral = Baseten(model="MODEL_ID", deployment="development")
|
||||
mistral("What is the Mistral wind?")
|
||||
```
|
||||
|
||||
Other published deployment
|
||||
|
||||
```python
|
||||
from langchain_community.llms import Baseten
|
||||
|
||||
mistral = Baseten(model="MODEL_ID", deployment="DEPLOYMENT_ID")
|
||||
mistral("What is the Mistral wind?")
|
||||
```
|
||||
|
||||
Streaming LLM output, chat completions, embeddings models, and more are all supported on the Baseten platform and coming soon to our LangChain integration. Contact us at [support@baseten.co](mailto:support@baseten.co) with any questions about using Baseten with LangChain.
|
||||
|
||||
Loading…
Reference in New Issue