forked from Archives/langchain
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
275 lines
8.1 KiB
Plaintext
275 lines
8.1 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6eaf7e66-f49c-42da-8d11-22ea13bef718",
|
|
"metadata": {},
|
|
"source": [
|
|
"# How to stream LLM and Chat Model responses\n",
|
|
"\n",
|
|
"LangChain provides streaming support for LLMs. Currently, we support streaming for the `OpenAI`, `ChatOpenAI`. and `Anthropic` implementations, but streaming support for other LLM implementations is on the roadmap. To utilize streaming, use a [`CallbackHandler`](https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/base.py) that implements `on_llm_new_token`. In this example, we are using [`StreamingStdOutCallbackHandler`]()."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 1,
|
|
"id": "4ac0ff54-540a-4f2b-8d9a-b590fec7fe07",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.llms import OpenAI, Anthropic\n",
|
|
"from langchain.chat_models import ChatOpenAI\n",
|
|
"from langchain.callbacks.base import CallbackManager\n",
|
|
"from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
|
|
"from langchain.schema import HumanMessage"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "77f60a4b-f786-41f2-972e-e5bb8a48dcd5",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"Verse 1\n",
|
|
"I'm sippin' on sparkling water,\n",
|
|
"It's so refreshing and light,\n",
|
|
"It's the perfect way to quench my thirst\n",
|
|
"On a hot summer night.\n",
|
|
"\n",
|
|
"Chorus\n",
|
|
"Sparkling water, sparkling water,\n",
|
|
"It's the best way to stay hydrated,\n",
|
|
"It's so crisp and so clean,\n",
|
|
"It's the perfect way to stay refreshed.\n",
|
|
"\n",
|
|
"Verse 2\n",
|
|
"I'm sippin' on sparkling water,\n",
|
|
"It's so bubbly and bright,\n",
|
|
"It's the perfect way to cool me down\n",
|
|
"On a hot summer night.\n",
|
|
"\n",
|
|
"Chorus\n",
|
|
"Sparkling water, sparkling water,\n",
|
|
"It's the best way to stay hydrated,\n",
|
|
"It's so crisp and so clean,\n",
|
|
"It's the perfect way to stay refreshed.\n",
|
|
"\n",
|
|
"Verse 3\n",
|
|
"I'm sippin' on sparkling water,\n",
|
|
"It's so light and so clear,\n",
|
|
"It's the perfect way to keep me cool\n",
|
|
"On a hot summer night.\n",
|
|
"\n",
|
|
"Chorus\n",
|
|
"Sparkling water, sparkling water,\n",
|
|
"It's the best way to stay hydrated,\n",
|
|
"It's so crisp and so clean,\n",
|
|
"It's the perfect way to stay refreshed."
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"llm = OpenAI(streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]), verbose=True, temperature=0)\n",
|
|
"resp = llm(\"Write me a song about sparkling water.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "61fb6de7-c6c8-48d0-a48e-1204c027a23c",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"We still have access to the end `LLMResult` if using `generate`. However, `token_usage` is not currently supported for streaming."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 4,
|
|
"id": "a35373f1-9ee6-4753-a343-5aee749b8527",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"Q: What did the fish say when it hit the wall?\n",
|
|
"A: Dam!"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"LLMResult(generations=[[Generation(text='\\n\\nQ: What did the fish say when it hit the wall?\\nA: Dam!', generation_info={'finish_reason': None, 'logprobs': None})]], llm_output={'token_usage': {}, 'model_name': 'text-davinci-003'})"
|
|
]
|
|
},
|
|
"execution_count": 4,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"llm.generate([\"Tell me a joke.\"])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a93a4d61-0476-49db-8321-7de92bd74059",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here's an example with the `ChatOpenAI` chat model implementation:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "22665f16-e05b-473c-a4bd-ad75744ea024",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"Verse 1:\n",
|
|
"Bubbles rising to the top\n",
|
|
"A refreshing drink that never stops\n",
|
|
"Clear and crisp, it's oh so pure\n",
|
|
"Sparkling water, I can't ignore\n",
|
|
"\n",
|
|
"Chorus:\n",
|
|
"Sparkling water, oh how you shine\n",
|
|
"A taste so clean, it's simply divine\n",
|
|
"You quench my thirst, you make me feel alive\n",
|
|
"Sparkling water, you're my favorite vibe\n",
|
|
"\n",
|
|
"Verse 2:\n",
|
|
"No sugar, no calories, just H2O\n",
|
|
"A drink that's good for me, don't you know\n",
|
|
"With lemon or lime, you're even better\n",
|
|
"Sparkling water, you're my forever\n",
|
|
"\n",
|
|
"Chorus:\n",
|
|
"Sparkling water, oh how you shine\n",
|
|
"A taste so clean, it's simply divine\n",
|
|
"You quench my thirst, you make me feel alive\n",
|
|
"Sparkling water, you're my favorite vibe\n",
|
|
"\n",
|
|
"Bridge:\n",
|
|
"You're my go-to drink, day or night\n",
|
|
"You make me feel so light\n",
|
|
"I'll never give you up, you're my true love\n",
|
|
"Sparkling water, you're sent from above\n",
|
|
"\n",
|
|
"Chorus:\n",
|
|
"Sparkling water, oh how you shine\n",
|
|
"A taste so clean, it's simply divine\n",
|
|
"You quench my thirst, you make me feel alive\n",
|
|
"Sparkling water, you're my favorite vibe\n",
|
|
"\n",
|
|
"Outro:\n",
|
|
"Sparkling water, you're the one for me\n",
|
|
"I'll never let you go, can't you see\n",
|
|
"You're my drink of choice, forevermore\n",
|
|
"Sparkling water, I adore."
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"chat = ChatOpenAI(streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]), verbose=True, temperature=0)\n",
|
|
"resp = chat([HumanMessage(content=\"Write me a song about sparkling water.\")])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "909ae48b-0f07-4990-bbff-e627f706c93e",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here is an example with the `Anthropic` LLM implementation, which uses their `claude` model."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "eadae4ba-9f21-4ec8-845d-dd43b0edc2dc",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"Sparkling water, bubbles so bright,\n",
|
|
"\n",
|
|
"Fizzing and popping in the light.\n",
|
|
"\n",
|
|
"No sugar or calories, a healthy delight,\n",
|
|
"\n",
|
|
"Sparkling water, refreshing and light.\n",
|
|
"\n",
|
|
"Carbonation that tickles the tongue,\n",
|
|
"\n",
|
|
"In flavors of lemon and lime unsung.\n",
|
|
"\n",
|
|
"Sparkling water, a drink quite all right,\n",
|
|
"\n",
|
|
"Bubbles sparkling in the light."
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"'\\nSparkling water, bubbles so bright,\\n\\nFizzing and popping in the light.\\n\\nNo sugar or calories, a healthy delight,\\n\\nSparkling water, refreshing and light.\\n\\nCarbonation that tickles the tongue,\\n\\nIn flavors of lemon and lime unsung.\\n\\nSparkling water, a drink quite all right,\\n\\nBubbles sparkling in the light.'"
|
|
]
|
|
},
|
|
"execution_count": 3,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"llm = Anthropic(streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]), verbose=True, temperature=0)\n",
|
|
"llm(\"Write me a song about sparkling water.\")"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|