langchain/docs/extras/integrations/text_embedding/xinference.ipynb

145 lines
3.3 KiB
Plaintext
Raw Normal View History

FEAT: Integrate Xinference LLMs and Embeddings (#8171) - [Xorbits Inference(Xinference)](https://github.com/xorbitsai/inference) is a powerful and versatile library designed to serve language, speech recognition, and multimodal models. Xinference supports a variety of GGML-compatible models including chatglm, whisper, and vicuna, and utilizes heterogeneous hardware and a distributed architecture for seamless cross-device and cross-server model deployment. - This PR integrates Xinference models and Xinference embeddings into LangChain. - Dependencies: To install the depenedencies for this integration, run `pip install "xinference[all]"` - Example Usage: To start a local instance of Xinference, run `xinference`. To deploy Xinference in a distributed cluster, first start an Xinference supervisor using `xinference-supervisor`: `xinference-supervisor -H "${supervisor_host}"` Then, start the Xinference workers using `xinference-worker` on each server you want to run them on. `xinference-worker -e "http://${supervisor_host}:9997"` To use Xinference with LangChain, you also need to launch a model. You can use command line interface (CLI) to do so. Fo example: `xinference launch -n vicuna-v1.3 -f ggmlv3 -q q4_0`. This launches a model named vicuna-v1.3 with `model_format="ggmlv3"` and `quantization="q4_0"`. A model UID is returned for you to use. Now you can use Xinference with LangChain: ```python from langchain.llms import Xinference llm = Xinference( server_url="http://0.0.0.0:9997", # suppose the supervisor_host is "0.0.0.0" model_uid = {model_uid} # model UID returned from launching a model ) llm( prompt="Q: where can we visit in the capital of France? A:", generate_config={"max_tokens": 1024}, ) ``` You can also use RESTful client to launch a model: ```python from xinference.client import RESTfulClient client = RESTfulClient("http://0.0.0.0:9997") model_uid = client.launch_model(model_name="vicuna-v1.3", model_size_in_billions=7, quantization="q4_0") ``` The following code block demonstrates how to use Xinference embeddings with LangChain: ```python from langchain.embeddings import XinferenceEmbeddings xinference = XinferenceEmbeddings( server_url="http://0.0.0.0:9997", model_uid = model_uid ) ``` ```python query_result = xinference.embed_query("This is a test query") ``` ```python doc_result = xinference.embed_documents(["text A", "text B"]) ``` Xinference is still under rapid development. Feel free to [join our Slack community](https://xorbitsio.slack.com/join/shared_invite/zt-1z3zsm9ep-87yI9YZ_B79HLB2ccTq4WA) to get the latest updates! - Request for review: @hwchase17, @baskaryan - Twitter handle: https://twitter.com/Xorbitsio --------- Co-authored-by: Bagatur <baskaryan@gmail.com>
2023-07-28 04:23:19 +00:00
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Xorbits inference (Xinference)\n",
"\n",
"This notebook goes over how to use Xinference embeddings within LangChain"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installation\n",
"\n",
"Install `Xinference` through PyPI:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%pip install \"xinference[all]\""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Deploy Xinference Locally or in a Distributed Cluster.\n",
"\n",
"For local deployment, run `xinference`. \n",
"\n",
"To deploy Xinference in a cluster, first start an Xinference supervisor using the `xinference-supervisor`. You can also use the option -p to specify the port and -H to specify the host. The default port is 9997.\n",
"\n",
"Then, start the Xinference workers using `xinference-worker` on each server you want to run them on. \n",
"\n",
"You can consult the README file from [Xinference](https://github.com/xorbitsai/inference) for more information.\n",
"\n",
"## Wrapper\n",
"\n",
"To use Xinference with LangChain, you need to first launch a model. You can use command line interface (CLI) to do so:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Model uid: 915845ee-2a04-11ee-8ed4-d29396a3f064\n"
]
}
],
"source": [
"!xinference launch -n vicuna-v1.3 -f ggmlv3 -q q4_0"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A model UID is returned for you to use. Now you can use Xinference embeddings with LangChain:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [],
"source": [
"from langchain.embeddings import XinferenceEmbeddings\n",
"\n",
"xinference = XinferenceEmbeddings(\n",
" server_url=\"http://0.0.0.0:9997\",\n",
" model_uid = \"915845ee-2a04-11ee-8ed4-d29396a3f064\"\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
"query_result = xinference.embed_query(\"This is a test query\")"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"doc_result = xinference.embed_documents([\"text A\", \"text B\"])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Lastly, terminate the model when you do not need to use it:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
"!xinference terminate --model-uid \"915845ee-2a04-11ee-8ed4-d29396a3f064\""
]
}
],
"metadata": {
"kernelspec": {
"display_name": "base",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
},
"orig_nbformat": 4
},
"nbformat": 4,
"nbformat_minor": 2
}