Archives
/
openai-cookbook
mirror of https://github.com/openai/openai-cookbook
2
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
main
revert-1014-origin/jbeutler/embeddings-sdk-update
charu/assistants_streaming
ankur/fix-pinecone-embeddings-notebook
katia/pdf-analysis
james/sdk_updates
james/quick_fix_notebook
katia/graphdb
colin/testing_sql_generation
jhills20-patch-1
james/making_slides
colin/guardrails
james/even_more_sdk_updates
feature/rz/UpdateToUsePythonSDK
ilan/assistants_overview_bugs
ilan/assistants_api_overview
applied-research/custom-tokenizer-model
ilan/structured_output
shyamal-chained-func-spec
chained-func-calling
cathykc/add-py-file
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'main'
${ noResults }
openai-cookbook/examples/How_to_stream_completions.i...

664 lines
32 KiB
Plaintext
Raw Permalink Blame History

{
"cells": [
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"# How to stream completions\n",
"\n",
"By default, when you request a completion from the OpenAI, the entire completion is generated before being sent back in a single response.\n",
"\n",
"If you're generating long completions, waiting for the response can take many seconds.\n",
"\n",
"To get responses sooner, you can 'stream' the completion as it's being generated. This allows you to start printing or processing the beginning of the completion before the full completion is finished.\n",
"\n",
"To stream completions, set `stream=True` when calling the chat completions or completions endpoints. This will return an object that streams back the response as [data-only server-sent events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). Extract chunks from the `delta` field rather than the `message` field.\n",
"\n",
"## Downsides\n",
"\n",
"Note that using `stream=True` in a production application makes it more difficult to moderate the content of the completions, as partial completions may be more difficult to evaluate. This may have implications for [approved usage](https://beta.openai.com/docs/usage-guidelines).\n",
"\n",
"## Example code\n",
"\n",
"Below, this notebook shows:\n",
"1. What a typical chat completion response looks like\n",
"2. What a streaming chat completion response looks like\n",
"3. How much time is saved by streaming a chat completion\n",
"4. How to get token usage data for streamed chat completion response"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"# !pip install openai"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# imports\n",
"import time # for measuring time duration of API calls\n",
"from openai import OpenAI\n",
"import os\n",
"client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### 1. What a typical chat completion response looks like\n",
"\n",
"With a typical ChatCompletions API call, the response is first computed and then returned all at once."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Full response received 5.27 seconds after request\n",
"Full response received:\n",
"ChatCompletion(id='chatcmpl-8ZB8ywkV5DuuJO7xktqUcNYfG8j6I', choices=[Choice(finish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(content='1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100.', role='assistant', function_call=None, tool_calls=None))], created=1703395008, model='gpt-3.5-turbo-0613', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=299, prompt_tokens=36, total_tokens=335))\n"
]
}
],
"source": [
"# Example of an OpenAI ChatCompletion request\n",
"# https://platform.openai.com/docs/guides/text-generation/chat-completions-api\n",
"\n",
"# record the time before the request is sent\n",
"start_time = time.time()\n",
"\n",
"# send a ChatCompletion request to count to 100\n",
"response = client.chat.completions.create(\n",
" model='gpt-3.5-turbo',\n",
" messages=[\n",
" {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}\n",
" ],\n",
" temperature=0,\n",
")\n",
"# calculate the time it took to receive the response\n",
"response_time = time.time() - start_time\n",
"\n",
"# print the time delay and text received\n",
"print(f\"Full response received {response_time:.2f} seconds after request\")\n",
"print(f\"Full response received:\\n{response}\")\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"The reply can be extracted with `response.choices[0].message`.\n",
"\n",
"The content of the reply can be extracted with `response.choices[0].message.content`."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Extracted reply: \n",
"ChatCompletionMessage(content='1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100.', role='assistant', function_call=None, tool_calls=None)\n",
"Extracted content: \n",
"1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100.\n"
]
}
],
"source": [
"reply = response.choices[0].message\n",
"print(f\"Extracted reply: \\n{reply}\")\n",
"\n",
"reply_content = response.choices[0].message.content\n",
"print(f\"Extracted content: \\n{reply_content}\")\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2. How to stream a chat completion\n",
"\n",
"With a streaming API call, the response is sent back incrementally in chunks via an [event stream](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events#event_stream_format). In Python, you can iterate over these events with a `for` loop.\n",
"\n",
"Let's see what it looks like:"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"ChatCompletionChunk(id='chatcmpl-8ZB9m2Ubv8FJs3CIb84WvYwqZCHST', choices=[Choice(delta=ChoiceDelta(content='', function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1703395058, model='gpt-3.5-turbo-0613', object='chat.completion.chunk', system_fingerprint=None)\n",
"\n",
"****************\n",
"ChatCompletionChunk(id='chatcmpl-8ZB9m2Ubv8FJs3CIb84WvYwqZCHST', choices=[Choice(delta=ChoiceDelta(content='2', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1703395058, model='gpt-3.5-turbo-0613', object='chat.completion.chunk', system_fingerprint=None)\n",
"2\n",
"****************\n",
"ChatCompletionChunk(id='chatcmpl-8ZB9m2Ubv8FJs3CIb84WvYwqZCHST', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None)], created=1703395058, model='gpt-3.5-turbo-0613', object='chat.completion.chunk', system_fingerprint=None)\n",
"None\n",
"****************\n"
]
}
],
"source": [
"# Example of an OpenAI ChatCompletion request with stream=True\n",
"# https://platform.openai.com/docs/api-reference/streaming#chat/create-stream\n",
"\n",
"# a ChatCompletion request\n",
"response = client.chat.completions.create(\n",
" model='gpt-3.5-turbo',\n",
" messages=[\n",
" {'role': 'user', 'content': \"What's 1+1? Answer in one word.\"}\n",
" ],\n",
" temperature=0,\n",
" stream=True # this time, we set stream=True\n",
")\n",
"\n",
"for chunk in response:\n",
" print(chunk)\n",
" print(chunk.choices[0].delta.content)\n",
" print(\"****************\")"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"As you can see above, streaming responses have a `delta` field rather than a `message` field. `delta` can hold things like:\n",
"- a role token (e.g., `{\"role\": \"assistant\"}`)\n",
"- a content token (e.g., `{\"content\": \"\\n\\n\"}`)\n",
"- nothing (e.g., `{}`), when the stream is over"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### 3. How much time is saved by streaming a chat completion\n",
"\n",
"Now let's ask `gpt-3.5-turbo` to count to 100 again, and see how long it takes."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Message received 0.31 seconds after request: \n",
"Message received 0.31 seconds after request: 1\n",
"Message received 0.34 seconds after request: ,\n",
"Message received 0.34 seconds after request: \n",
"Message received 0.34 seconds after request: 2\n",
"Message received 0.39 seconds after request: ,\n",
"Message received 0.39 seconds after request: \n",
"Message received 0.39 seconds after request: 3\n",
"Message received 0.42 seconds after request: ,\n",
"Message received 0.42 seconds after request: \n",
"Message received 0.42 seconds after request: 4\n",
"Message received 0.47 seconds after request: ,\n",
"Message received 0.47 seconds after request: \n",
"Message received 0.47 seconds after request: 5\n",
"Message received 0.51 seconds after request: ,\n",
"Message received 0.51 seconds after request: \n",
"Message received 0.51 seconds after request: 6\n",
"Message received 0.55 seconds after request: ,\n",
"Message received 0.55 seconds after request: \n",
"Message received 0.55 seconds after request: 7\n",
"Message received 0.59 seconds after request: ,\n",
"Message received 0.59 seconds after request: \n",
"Message received 0.59 seconds after request: 8\n",
"Message received 0.63 seconds after request: ,\n",
"Message received 0.63 seconds after request: \n",
"Message received 0.63 seconds after request: 9\n",
"Message received 0.67 seconds after request: ,\n",
"Message received 0.67 seconds after request: \n",
"Message received 0.67 seconds after request: 10\n",
"Message received 0.71 seconds after request: ,\n",
"Message received 0.71 seconds after request: \n",
"Message received 0.71 seconds after request: 11\n",
"Message received 0.75 seconds after request: ,\n",
"Message received 0.75 seconds after request: \n",
"Message received 0.75 seconds after request: 12\n",
"Message received 0.98 seconds after request: ,\n",
"Message received 0.98 seconds after request: \n",
"Message received 0.98 seconds after request: 13\n",
"Message received 1.02 seconds after request: ,\n",
"Message received 1.02 seconds after request: \n",
"Message received 1.02 seconds after request: 14\n",
"Message received 1.04 seconds after request: ,\n",
"Message received 1.04 seconds after request: \n",
"Message received 1.04 seconds after request: 15\n",
"Message received 1.08 seconds after request: ,\n",
"Message received 1.08 seconds after request: \n",
"Message received 1.08 seconds after request: 16\n",
"Message received 1.12 seconds after request: ,\n",
"Message received 1.12 seconds after request: \n",
"Message received 1.12 seconds after request: 17\n",
"Message received 1.16 seconds after request: ,\n",
"Message received 1.16 seconds after request: \n",
"Message received 1.16 seconds after request: 18\n",
"Message received 1.19 seconds after request: ,\n",
"Message received 1.19 seconds after request: \n",
"Message received 1.19 seconds after request: 19\n",
"Message received 1.23 seconds after request: ,\n",
"Message received 1.23 seconds after request: \n",
"Message received 1.23 seconds after request: 20\n",
"Message received 1.27 seconds after request: ,\n",
"Message received 1.27 seconds after request: \n",
"Message received 1.27 seconds after request: 21\n",
"Message received 1.31 seconds after request: ,\n",
"Message received 1.31 seconds after request: \n",
"Message received 1.31 seconds after request: 22\n",
"Message received 1.35 seconds after request: ,\n",
"Message received 1.35 seconds after request: \n",
"Message received 1.35 seconds after request: 23\n",
"Message received 1.39 seconds after request: ,\n",
"Message received 1.39 seconds after request: \n",
"Message received 1.39 seconds after request: 24\n",
"Message received 1.43 seconds after request: ,\n",
"Message received 1.43 seconds after request: \n",
"Message received 1.43 seconds after request: 25\n",
"Message received 1.47 seconds after request: ,\n",
"Message received 1.47 seconds after request: \n",
"Message received 1.47 seconds after request: 26\n",
"Message received 1.51 seconds after request: ,\n",
"Message received 1.51 seconds after request: \n",
"Message received 1.51 seconds after request: 27\n",
"Message received 1.55 seconds after request: ,\n",
"Message received 1.55 seconds after request: \n",
"Message received 1.55 seconds after request: 28\n",
"Message received 1.59 seconds after request: ,\n",
"Message received 1.59 seconds after request: \n",
"Message received 1.59 seconds after request: 29\n",
"Message received 1.59 seconds after request: ,\n",
"Message received 1.59 seconds after request: \n",
"Message received 1.59 seconds after request: 30\n",
"Message received 1.59 seconds after request: ,\n",
"Message received 1.59 seconds after request: \n",
"Message received 1.59 seconds after request: 31\n",
"Message received 1.59 seconds after request: ,\n",
"Message received 1.59 seconds after request: \n",
"Message received 1.60 seconds after request: 32\n",
"Message received 1.60 seconds after request: ,\n",
"Message received 1.60 seconds after request: \n",
"Message received 1.60 seconds after request: 33\n",
"Message received 1.60 seconds after request: ,\n",
"Message received 1.60 seconds after request: \n",
"Message received 1.67 seconds after request: 34\n",
"Message received 1.67 seconds after request: ,\n",
"Message received 1.67 seconds after request: \n",
"Message received 1.68 seconds after request: 35\n",
"Message received 1.68 seconds after request: ,\n",
"Message received 1.68 seconds after request: \n",
"Message received 1.86 seconds after request: 36\n",
"Message received 1.86 seconds after request: ,\n",
"Message received 1.86 seconds after request: \n",
"Message received 1.90 seconds after request: 37\n",
"Message received 1.90 seconds after request: ,\n",
"Message received 1.90 seconds after request: \n",
"Message received 1.94 seconds after request: 38\n",
"Message received 1.94 seconds after request: ,\n",
"Message received 1.94 seconds after request: \n",
"Message received 1.98 seconds after request: 39\n",
"Message received 1.98 seconds after request: ,\n",
"Message received 1.98 seconds after request: \n",
"Message received 2.05 seconds after request: 40\n",
"Message received 2.05 seconds after request: ,\n",
"Message received 2.05 seconds after request: \n",
"Message received 2.09 seconds after request: 41\n",
"Message received 2.09 seconds after request: ,\n",
"Message received 2.09 seconds after request: \n",
"Message received 2.14 seconds after request: 42\n",
"Message received 2.14 seconds after request: ,\n",
"Message received 2.14 seconds after request: \n",
"Message received 2.14 seconds after request: 43\n",
"Message received 2.14 seconds after request: ,\n",
"Message received 2.14 seconds after request: \n",
"Message received 2.14 seconds after request: 44\n",
"Message received 2.14 seconds after request: ,\n",
"Message received 2.14 seconds after request: \n",
"Message received 2.14 seconds after request: 45\n",
"Message received 2.14 seconds after request: ,\n",
"Message received 2.14 seconds after request: \n",
"Message received 2.15 seconds after request: 46\n",
"Message received 2.15 seconds after request: ,\n",
"Message received 2.15 seconds after request: \n",
"Message received 2.30 seconds after request: 47\n",
"Message received 2.30 seconds after request: ,\n",
"Message received 2.30 seconds after request: \n",
"Message received 2.30 seconds after request: 48\n",
"Message received 2.30 seconds after request: ,\n",
"Message received 2.30 seconds after request: \n",
"Message received 2.30 seconds after request: 49\n",
"Message received 2.30 seconds after request: ,\n",
"Message received 2.30 seconds after request: \n",
"Message received 2.31 seconds after request: 50\n",
"Message received 2.31 seconds after request: ,\n",
"Message received 2.31 seconds after request: \n",
"Message received 2.39 seconds after request: 51\n",
"Message received 2.39 seconds after request: ,\n",
"Message received 2.39 seconds after request: \n",
"Message received 2.40 seconds after request: 52\n",
"Message received 2.40 seconds after request: ,\n",
"Message received 2.40 seconds after request: \n",
"Message received 2.48 seconds after request: 53\n",
"Message received 2.48 seconds after request: ,\n",
"Message received 2.48 seconds after request: \n",
"Message received 2.49 seconds after request: 54\n",
"Message received 2.49 seconds after request: ,\n",
"Message received 2.49 seconds after request: \n",
"Message received 2.68 seconds after request: 55\n",
"Message received 2.68 seconds after request: ,\n",
"Message received 2.68 seconds after request: \n",
"Message received 2.72 seconds after request: 56\n",
"Message received 2.72 seconds after request: ,\n",
"Message received 2.72 seconds after request: \n",
"Message received 2.77 seconds after request: 57\n",
"Message received 2.77 seconds after request: ,\n",
"Message received 2.77 seconds after request: \n",
"Message received 2.80 seconds after request: 58\n",
"Message received 2.80 seconds after request: ,\n",
"Message received 2.80 seconds after request: \n",
"Message received 2.85 seconds after request: 59\n",
"Message received 2.85 seconds after request: ,\n",
"Message received 2.85 seconds after request: \n",
"Message received 2.88 seconds after request: 60\n",
"Message received 2.88 seconds after request: ,\n",
"Message received 2.88 seconds after request: \n",
"Message received 2.88 seconds after request: 61\n",
"Message received 2.88 seconds after request: ,\n",
"Message received 2.88 seconds after request: \n",
"Message received 2.89 seconds after request: 62\n",
"Message received 2.89 seconds after request: ,\n",
"Message received 2.89 seconds after request: \n",
"Message received 2.89 seconds after request: 63\n",
"Message received 2.89 seconds after request: ,\n",
"Message received 2.89 seconds after request: \n",
"Message received 2.92 seconds after request: 64\n",
"Message received 2.92 seconds after request: ,\n",
"Message received 2.92 seconds after request: \n",
"Message received 3.37 seconds after request: 65\n",
"Message received 3.37 seconds after request: ,\n",
"Message received 3.37 seconds after request: \n",
"Message received 3.38 seconds after request: 66\n",
"Message received 3.38 seconds after request: ,\n",
"Message received 3.38 seconds after request: \n",
"Message received 3.38 seconds after request: 67\n",
"Message received 3.38 seconds after request: ,\n",
"Message received 3.38 seconds after request: \n",
"Message received 3.38 seconds after request: 68\n",
"Message received 3.38 seconds after request: ,\n",
"Message received 3.38 seconds after request: \n",
"Message received 3.42 seconds after request: 69\n",
"Message received 3.42 seconds after request: ,\n",
"Message received 3.42 seconds after request: \n",
"Message received 3.43 seconds after request: 70\n",
"Message received 3.43 seconds after request: ,\n",
"Message received 3.43 seconds after request: \n",
"Message received 3.46 seconds after request: 71\n",
"Message received 3.46 seconds after request: ,\n",
"Message received 3.46 seconds after request: \n",
"Message received 3.47 seconds after request: 72\n",
"Message received 3.47 seconds after request: ,\n",
"Message received 3.47 seconds after request: \n",
"Message received 3.50 seconds after request: 73\n",
"Message received 3.50 seconds after request: ,\n",
"Message received 3.50 seconds after request: \n",
"Message received 3.51 seconds after request: 74\n",
"Message received 3.51 seconds after request: ,\n",
"Message received 3.51 seconds after request: \n",
"Message received 3.52 seconds after request: 75\n",
"Message received 3.52 seconds after request: ,\n",
"Message received 3.52 seconds after request: \n",
"Message received 3.54 seconds after request: 76\n",
"Message received 3.54 seconds after request: ,\n",
"Message received 3.54 seconds after request: \n",
"Message received 3.56 seconds after request: 77\n",
"Message received 3.56 seconds after request: ,\n",
"Message received 3.56 seconds after request: \n",
"Message received 3.59 seconds after request: 78\n",
"Message received 3.59 seconds after request: ,\n",
"Message received 3.59 seconds after request: \n",
"Message received 3.59 seconds after request: 79\n",
"Message received 3.59 seconds after request: ,\n",
"Message received 3.59 seconds after request: \n",
"Message received 3.59 seconds after request: 80\n",
"Message received 3.59 seconds after request: ,\n",
"Message received 3.59 seconds after request: \n",
"Message received 3.61 seconds after request: 81\n",
"Message received 3.61 seconds after request: ,\n",
"Message received 3.61 seconds after request: \n",
"Message received 3.65 seconds after request: 82\n",
"Message received 3.65 seconds after request: ,\n",
"Message received 3.65 seconds after request: \n",
"Message received 3.85 seconds after request: 83\n",
"Message received 3.85 seconds after request: ,\n",
"Message received 3.85 seconds after request: \n",
"Message received 3.90 seconds after request: 84\n",
"Message received 3.90 seconds after request: ,\n",
"Message received 3.90 seconds after request: \n",
"Message received 3.95 seconds after request: 85\n",
"Message received 3.95 seconds after request: ,\n",
"Message received 3.95 seconds after request: \n",
"Message received 4.00 seconds after request: 86\n",
"Message received 4.00 seconds after request: ,\n",
"Message received 4.00 seconds after request: \n",
"Message received 4.04 seconds after request: 87\n",
"Message received 4.04 seconds after request: ,\n",
"Message received 4.04 seconds after request: \n",
"Message received 4.08 seconds after request: 88\n",
"Message received 4.08 seconds after request: ,\n",
"Message received 4.08 seconds after request: \n",
"Message received 4.12 seconds after request: 89\n",
"Message received 4.12 seconds after request: ,\n",
"Message received 4.12 seconds after request: \n",
"Message received 4.18 seconds after request: 90\n",
"Message received 4.18 seconds after request: ,\n",
"Message received 4.18 seconds after request: \n",
"Message received 4.18 seconds after request: 91\n",
"Message received 4.18 seconds after request: ,\n",
"Message received 4.18 seconds after request: \n",
"Message received 4.18 seconds after request: 92\n",
"Message received 4.18 seconds after request: ,\n",
"Message received 4.18 seconds after request: \n",
"Message received 4.19 seconds after request: 93\n",
"Message received 4.19 seconds after request: ,\n",
"Message received 4.19 seconds after request: \n",
"Message received 4.20 seconds after request: 94\n",
"Message received 4.20 seconds after request: ,\n",
"Message received 4.20 seconds after request: \n",
"Message received 4.23 seconds after request: 95\n",
"Message received 4.23 seconds after request: ,\n",
"Message received 4.23 seconds after request: \n",
"Message received 4.27 seconds after request: 96\n",
"Message received 4.27 seconds after request: ,\n",
"Message received 4.27 seconds after request: \n",
"Message received 4.39 seconds after request: 97\n",
"Message received 4.39 seconds after request: ,\n",
"Message received 4.39 seconds after request: \n",
"Message received 4.39 seconds after request: 98\n",
"Message received 4.39 seconds after request: ,\n",
"Message received 4.39 seconds after request: \n",
"Message received 4.41 seconds after request: 99\n",
"Message received 4.41 seconds after request: ,\n",
"Message received 4.41 seconds after request: \n",
"Message received 4.41 seconds after request: 100\n",
"Message received 4.41 seconds after request: .\n",
"Message received 4.41 seconds after request: None\n",
"Full response received 4.41 seconds after request\n",
"Full conversation received: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100.\n"
]
}
],
"source": [
"# Example of an OpenAI ChatCompletion request with stream=True\n",
"# https://platform.openai.com/docs/api-reference/streaming#chat/create-stream\n",
"\n",
"# record the time before the request is sent\n",
"start_time = time.time()\n",
"\n",
"# send a ChatCompletion request to count to 100\n",
"response = client.chat.completions.create(\n",
" model='gpt-3.5-turbo',\n",
" messages=[\n",
" {'role': 'user', 'content': 'Count to 100, with a comma between each number and no newlines. E.g., 1, 2, 3, ...'}\n",
" ],\n",
" temperature=0,\n",
" stream=True # again, we set stream=True\n",
")\n",
"# create variables to collect the stream of chunks\n",
"collected_chunks = []\n",
"collected_messages = []\n",
"# iterate through the stream of events\n",
"for chunk in response:\n",
" chunk_time = time.time() - start_time # calculate the time delay of the chunk\n",
" collected_chunks.append(chunk) # save the event response\n",
" chunk_message = chunk.choices[0].delta.content # extract the message\n",
" collected_messages.append(chunk_message) # save the message\n",
" print(f\"Message received {chunk_time:.2f} seconds after request: {chunk_message}\") # print the delay and text\n",
"\n",
"# print the time delay and text received\n",
"print(f\"Full response received {chunk_time:.2f} seconds after request\")\n",
"# clean None in collected_messages\n",
"collected_messages = [m for m in collected_messages if m is not None]\n",
"full_reply_content = ''.join(collected_messages)\n",
"print(f\"Full conversation received: {full_reply_content}\")\n"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Time comparison\n",
"\n",
"In the example above, both requests took about 4 to 5 seconds to fully complete. Request times will vary depending on load and other stochastic factors.\n",
"\n",
"However, with the streaming request, we received the first token after 0.1 seconds, and subsequent tokens every ~0.01-0.02 seconds."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 4. How to get token usage data for streamed chat completion response\n",
"\n",
"You can get token usage statistics for your streamed response by setting `stream_options={\"include_usage\": True}`. When you do so, an extra chunk will be streamed as the final chunk. You can access the usage data for the entire request via the `usage` field on this chunk. A few important notes when you set `stream_options={\"include_usage\": True}`:\n",
"* The value for the `usage` field on all chunks except for the last one will be null.\n",
"* The `usage` field on the last chunk contains token usage statistics for the entire request.\n",
"* The `choices` field on the last chunk will always be an empty array `[]`.\n",
"\n",
"Let's see how it works using the example in 2."
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"choices: [Choice(delta=ChoiceDelta(content='', function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=None)]\n",
"usage: None\n",
"****************\n",
"choices: [Choice(delta=ChoiceDelta(content='2', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)]\n",
"usage: None\n",
"****************\n",
"choices: [Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None)]\n",
"usage: None\n",
"****************\n",
"choices: []\n",
"usage: CompletionUsage(completion_tokens=1, prompt_tokens=19, total_tokens=20)\n",
"****************\n"
]
}
],
"source": [
"# Example of an OpenAI ChatCompletion request with stream=True and stream_options={\"include_usage\": True}\n",
"\n",
"# a ChatCompletion request\n",
"response = client.chat.completions.create(\n",
" model='gpt-3.5-turbo',\n",
" messages=[\n",
" {'role': 'user', 'content': \"What's 1+1? Answer in one word.\"}\n",
" ],\n",
" temperature=0,\n",
" stream=True,\n",
" stream_options={\"include_usage\": True}, # retrieving token usage for stream response\n",
")\n",
"\n",
"for chunk in response:\n",
" print(f\"choices: {chunk.choices}\\nusage: {chunk.usage}\")\n",
" print(\"****************\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3.9.9 ('openai')",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.3"
},
"orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "365536dcbde60510dc9073d6b991cd35db2d9bac356a11f5b64279a5e6708b97"
}
}
},
"nbformat": 4,
"nbformat_minor": 2
}
Reference in New Issue View Git Blame Copy Permalink