forked from Archives/langchain
222 lines
6.2 KiB
Plaintext
222 lines
6.2 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6eaf7e66-f49c-42da-8d11-22ea13bef718",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Streaming with LLMs\n",
|
|
"\n",
|
|
"LangChain provides streaming support for LLMs. Currently, we only support streaming for the `OpenAI` and `ChatOpenAI` LLM implementation, but streaming support for other LLM implementations is on the roadmap. To utilize streaming, use a [`CallbackHandler`](https://github.com/hwchase17/langchain/blob/master/langchain/callbacks/base.py) that implements `on_llm_new_token`. In this example, we are using [`StreamingStdOutCallbackHandler`]()."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 2,
|
|
"id": "4ac0ff54-540a-4f2b-8d9a-b590fec7fe07",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"Verse 1\n",
|
|
"I'm sippin' on sparkling water,\n",
|
|
"It's so refreshing and light,\n",
|
|
"It's the perfect way to quench my thirst\n",
|
|
"On a hot summer night.\n",
|
|
"\n",
|
|
"Chorus\n",
|
|
"Sparkling water, sparkling water,\n",
|
|
"It's the best way to stay hydrated,\n",
|
|
"It's so crisp and so clean,\n",
|
|
"It's the perfect way to stay refreshed.\n",
|
|
"\n",
|
|
"Verse 2\n",
|
|
"I'm sippin' on sparkling water,\n",
|
|
"It's so bubbly and bright,\n",
|
|
"It's the perfect way to cool me down\n",
|
|
"On a hot summer night.\n",
|
|
"\n",
|
|
"Chorus\n",
|
|
"Sparkling water, sparkling water,\n",
|
|
"It's the best way to stay hydrated,\n",
|
|
"It's so crisp and so clean,\n",
|
|
"It's the perfect way to stay refreshed.\n",
|
|
"\n",
|
|
"Verse 3\n",
|
|
"I'm sippin' on sparkling water,\n",
|
|
"It's so light and so clear,\n",
|
|
"It's the perfect way to keep me cool\n",
|
|
"On a hot summer night.\n",
|
|
"\n",
|
|
"Chorus\n",
|
|
"Sparkling water, sparkling water,\n",
|
|
"It's the best way to stay hydrated,\n",
|
|
"It's so crisp and so clean,\n",
|
|
"It's the perfect way to stay refreshed."
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"from langchain.llms import OpenAI\n",
|
|
"from langchain.chat_models import ChatOpenAI\n",
|
|
"from langchain.callbacks.base import CallbackManager\n",
|
|
"from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
|
|
"from langchain.schema import HumanMessage\n",
|
|
"\n",
|
|
"\n",
|
|
"llm = OpenAI(streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]), verbose=True, temperature=0)\n",
|
|
"resp = llm(\"Write me a song about sparkling water.\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "61fb6de7-c6c8-48d0-a48e-1204c027a23c",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"source": [
|
|
"We still have access to the end `LLMResult` if using `generate`. However, `token_usage` is not currently supported for streaming."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 6,
|
|
"id": "a35373f1-9ee6-4753-a343-5aee749b8527",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"Q: What did the fish say when it hit the wall?\n",
|
|
"A: Dam!"
|
|
]
|
|
},
|
|
{
|
|
"data": {
|
|
"text/plain": [
|
|
"LLMResult(generations=[[Generation(text='\\n\\nQ: What did the fish say when it hit the wall?\\nA: Dam!', generation_info={'finish_reason': None, 'logprobs': None})]], llm_output={'token_usage': {}})"
|
|
]
|
|
},
|
|
"execution_count": 6,
|
|
"metadata": {},
|
|
"output_type": "execute_result"
|
|
}
|
|
],
|
|
"source": [
|
|
"llm.generate([\"Tell me a joke.\"])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a93a4d61-0476-49db-8321-7de92bd74059",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here's an example with `ChatOpenAI`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": 3,
|
|
"id": "22665f16-e05b-473c-a4bd-ad75744ea024",
|
|
"metadata": {
|
|
"tags": []
|
|
},
|
|
"outputs": [
|
|
{
|
|
"name": "stdout",
|
|
"output_type": "stream",
|
|
"text": [
|
|
"\n",
|
|
"\n",
|
|
"Verse 1:\n",
|
|
"Bubbles rising to the top\n",
|
|
"A refreshing drink that never stops\n",
|
|
"Clear and crisp, it's pure delight\n",
|
|
"A taste that's sure to excite\n",
|
|
"\n",
|
|
"Chorus:\n",
|
|
"Sparkling water, oh so fine\n",
|
|
"A drink that's always on my mind\n",
|
|
"With every sip, I feel alive\n",
|
|
"Sparkling water, you're my vibe\n",
|
|
"\n",
|
|
"Verse 2:\n",
|
|
"No sugar, no calories, just pure bliss\n",
|
|
"A drink that's hard to resist\n",
|
|
"It's the perfect way to quench my thirst\n",
|
|
"A drink that always comes first\n",
|
|
"\n",
|
|
"Chorus:\n",
|
|
"Sparkling water, oh so fine\n",
|
|
"A drink that's always on my mind\n",
|
|
"With every sip, I feel alive\n",
|
|
"Sparkling water, you're my vibe\n",
|
|
"\n",
|
|
"Bridge:\n",
|
|
"From the mountains to the sea\n",
|
|
"Sparkling water, you're the key\n",
|
|
"To a healthy life, a happy soul\n",
|
|
"A drink that makes me feel whole\n",
|
|
"\n",
|
|
"Chorus:\n",
|
|
"Sparkling water, oh so fine\n",
|
|
"A drink that's always on my mind\n",
|
|
"With every sip, I feel alive\n",
|
|
"Sparkling water, you're my vibe\n",
|
|
"\n",
|
|
"Outro:\n",
|
|
"Sparkling water, you're the one\n",
|
|
"A drink that's always so much fun\n",
|
|
"I'll never let you go, my friend\n",
|
|
"Sparkling"
|
|
]
|
|
}
|
|
],
|
|
"source": [
|
|
"chat = ChatOpenAI(streaming=True, callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]), verbose=True, temperature=0)\n",
|
|
"resp = chat([HumanMessage(content=\"Write me a song about sparkling water.\")])"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "eadae4ba-9f21-4ec8-845d-dd43b0edc2dc",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": []
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.10.9"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|