openai-cookbook/examples/Deterministic_outputs_with_the_seed_parameter.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"cell_id":"67bb097e130b41099c9d257dc06a4054","deepnote_cell_type":"markdown"},"source":["# How to make your completions outputs consistent with the new seed parameter\n","\n","**TLDR**: Developers can now specify `seed` parameter in the Chat Completion request for consistent completions. We always include a `system_fingerprint` in the response that helps developers understand changes in our system that will affect determinism.\n","\n","### Context\n","\n","Determinism has always been a big request from user communities when using our APIs. For instance, when granted the capability of getting deterministic numerical result, users can unlock quite a bit of use cases that’s sensitive to numerical changes.\n","\n","#### Model level features for consistent outputs\n","\n","The Chat Completions and Completions APIs are non-deterministic by default (which means model outputs may differ from request to request), but now offer some control towards deterministic outputs using a few model level controls.\n","\n","This can unlock consistent completions which enables full control on the model behaviors for anything built on top of the APIs, and quite useful for reproducing results and testing so you know get peace of mind from knowing exactly what you’d get.\n","\n","#### Implementing consistent outputs\n","\n","To receive _mostly_ deterministic outputs across API calls:\n","\n","- Set the `seed` parameter to any integer of your choice, but use the same value across requests. For example, `12345`.\n","- Set all other parameters (prompt, temperature, top_p, etc.) to the same values across requests.\n","- In the response, check the `system_fingerprint` field. The system fingerprint is an identifier for the current combination of model weights, infrastructure, and other configuration options used by OpenAI servers to generate the completion. It changes whenever you change request parameters, or OpenAI updates numerical configuration of the infrastructure serving our models (which may happen a few times a year).\n","\n","If the `seed`, request parameters, and `system_fingerprint` all match across your requests, then model outputs will mostly be identical. There is a small chance that responses differ even when request parameters and `system_fingerprint` match, due to the inherent non-determinism of computers.\n"]},{"cell_type":"markdown","metadata":{"cell_id":"f49611fa59af4303883d76c491095fea","deepnote_cell_type":"markdown"},"source":["### Model level controls for consistent outputs - `seed` and `system_fingerprint`\n","\n","##### `seed`\n","\n","If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed, and you should refer to the `system_fingerprint` response parameter to monitor changes in the backend.\n","\n","##### `system_fingerprint`\n","\n","This fingerprint represents the backend configuration that the model runs with. It can be used in conjunction with the seed request parameter to understand when backend changes have been made that might impact determinism.This is the indicator on whether users should expect \"almost always the same result\".\n"]},{"cell_type":"markdown","metadata":{"cell_id":"cc6cd37b9a2243aaa4688ef8832512eb","deepnote_cell_type":"markdown"},"source":["## Example: Generating a consistent short story with a fixed seed\n","\n","In this example, we will demonstrate how to generate a consistent short story using a fixed seed. This can be particularly useful in scenarios where you need to reproduce the same results for testing, debugging, or for applications that require consistent outputs.\n"]},{"cell_type":"markdown","metadata":{},"source":["### Python SDK\n","\n","> **Note**\n","> Switch to latest version of the SDK (`1.3.3` at time of writing)."]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!pip install --upgrade openai # Switch to the latest version of OpenAI (1.3.3 at time of writing)"]},{"cell_type":"code","execution_count":null,"metadata":{"cell_id":"48fd2d4c95ad465090ef97254a4a10d2","deepnote_cell_type":"code"},"outputs":[],"source":["import asyncio\n","import openai\n","import pprint\n","import difflib\n","from IPython.display import display, HTML\n","\n","GPT_MODEL = \"gpt-3.5-turbo-1106\""]},{"cell_type":"code","execution_count":null,"metadata":{"cell_id":"e54e0958be3746d39b6e4c16c59b395a","deepnote_cell_type":"code","deepnote_to_be_reexecuted":false,"execution_millis":5,"execution_start":1699034108287,"source_hash":null},"outputs":[],"source":["async def get_chat_response(system_message: str, user_request: str, seed: int = None):\n","    try:\n","        messages = [\n","            {\"role\": \"system\", \"content\": system_message},\n","            {\"role\": \"user\", \"content\": user_request},\n","        ]\n","\n","        response = openai.chat.completions.create(\n","            model=GPT_MODEL,\n","            messages=messages,\n","            seed=seed,\n","            max_tokens=200,\n","            temperature=0.7,\n","        )\n","\n","        response_content = response.choices[0].message.content\n","        system_fingerprint = response.system_fingerprint\n","        prompt_tokens = response.usage.prompt_tokens\n","        completion_tokens = (\n","            response.usage.total_tokens - response.usage.prompt_tokens\n","        )\n","\n","        table = f\"\"\"\n","        <table>\n","        <tr><th>Response</th><td>{response_content}</td></tr>\n","        <tr><th>System Fingerprint</th><td>{system_fingerprint}</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>{prompt_tokens}</td></tr>\n","        <tr><th>Number of completion tokens</th><td>{completion_tokens}</td></tr>\n","        </table>\n","        \"\"\"\n","        display(HTML(table))\n","\n","        return response_content\n","    except Exception as e:\n","        print(f\"An error occurred: {e}\")\n","        return None\n","\n","\n","# This function compares two responses and displays the differences in a table.\n","# Deletions are highlighted in red and additions are highlighted in green.\n","# If no differences are found, it prints \"No differences found.\"\n","\n","\n","def compare_responses(previous_response: str, response: str):\n","    d = difflib.Differ()\n","    diff = d.compare(previous_response.splitlines(), response.splitlines())\n","\n","    diff_table = \"<table>\"\n","    diff_exists = False\n","\n","    for line in diff:\n","        if line.startswith(\"- \"):\n","            diff_table += f\"<tr style='color: red;'><td>{line}</td></tr>\"\n","            diff_exists = True\n","        elif line.startswith(\"+ \"):\n","            diff_table += f\"<tr style='color: green;'><td>{line}</td></tr>\"\n","            diff_exists = True\n","        else:\n","            diff_table += f\"<tr><td>{line}</td></tr>\"\n","\n","    diff_table += \"</table>\"\n","\n","    if diff_exists:\n","        display(HTML(diff_table))\n","    else:\n","        print(\"No differences found.\")"]},{"cell_type":"markdown","metadata":{"cell_id":"dfa39a438aa948cc910a46254df937af","deepnote_cell_type":"text-cell-p","formattedRanges":[]},"source":["First, let's try generating a short story about \"a journey to Mars\" without the `seed` parameter. This is the default behavior:\n"]},{"cell_type":"code","execution_count":null,"metadata":{"cell_id":"9d09f63309c449e4929364caccfd7065","deepnote_cell_type":"code","deepnote_to_be_reexecuted":false,"execution_millis":964,"execution_start":1699034108745,"source_hash":null},"outputs":[{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>In the year 2050, a team of courageous astronauts embarked on a groundbreaking mission to Mars. The journey was filled with uncertainty and danger, but the crew was undeterred by the challenges that lay ahead.\n","\n","As their spacecraft hurtled through the vast expanse of space, the astronauts marveled at the beauty of the stars and the distant planets. They passed the time by conducting experiments, training for the mission ahead, and bonding with one another.\n","\n","After months of travel, the red planet finally came into view. The crew prepared for the landing, their hearts pounding with a mix of excitement and nervous anticipation. As the spacecraft touched down on the Martian surface, cheers erupted in the control room back on Earth.\n","\n","The astronauts stepped out onto the alien terrain, taking in the breathtaking landscape of rusty red rocks and dusty plains. They set up their base camp and began their scientific research, collecting samples and conducting experiments to better understand the planet's composition and potential for sustaining life.\n","\n","Despite the challenges of living</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_fefa7b2153</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>31</td></tr>\n","        <tr><th>Number of completion tokens</th><td>200</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>In the year 2050, a team of astronauts set out on a groundbreaking mission to Mars. The journey was long and arduous, but the crew was determined to make history. As they approached the red planet, they marveled at its otherworldly beauty and the sense of awe and wonder filled their hearts.\n","\n","Upon landing, the astronauts began to explore the alien landscape, conducting scientific experiments and collecting samples. They were amazed by the vast canyons, towering mountains, and the eerie silence that surrounded them. Each step they took was a giant leap for humankind, and they felt a profound sense of accomplishment.\n","\n","As they prepared to return to Earth, the astronauts reflected on the significance of their journey. They knew that their discoveries would pave the way for future generations to explore and inhabit Mars. With their mission complete, they boarded their spacecraft and set their sights on the distant blue planet in the sky, knowing that they had left their mark on the history of space exploration.</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_fefa7b2153</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>31</td></tr>\n","        <tr><th>Number of completion tokens</th><td>198</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["<table><tr style='color: red;'><td>- In the year 2050, a team of courageous astronauts embarked on a groundbreaking mission to Mars. The journey was filled with uncertainty and danger, but the crew was undeterred by the challenges that lay ahead.</td></tr><tr style='color: green;'><td>+ In the year 2050, a team of astronauts set out on a groundbreaking mission to Mars. The journey was long and arduous, but the crew was determined to make history. As they approached the red planet, they marveled at its otherworldly beauty and the sense of awe and wonder filled their hearts.</td></tr><tr><td>  </td></tr><tr style='color: red;'><td>- As their spacecraft hurtled through the vast expanse of space, the astronauts marveled at the beauty of the stars and the distant planets. They passed the time by conducting experiments, training for the mission ahead, and bonding with one another.</td></tr><tr style='color: green;'><td>+ Upon landing, the astronauts began to explore the alien landscape, conducting scientific experiments and collecting samples. They were amazed by the vast canyons, towering mountains, and the eerie silence that surrounded them. Each step they took was a giant leap for humankind, and they felt a profound sense of accomplishment.</td></tr><tr><td>  </td></tr><tr style='color: green;'><td>+ As they prepared to return to Earth, the astronauts reflected on the significance of their journey. They knew that their discoveries would pave the way for future generations to explore and inhabit Mars. With their mission complete, they boarded their spacecraft and set their sights on the distant blue planet in the sky, knowing that they had left their mark on the history of space exploration.</td></tr><tr style='color: red;'><td>- After months of travel, the red planet finally came into view. The crew prepared for the landing, their hearts pounding with a mix of excitement and nervous anticipation. As the spacecraft touched down on the Martian surface, cheers erupted in the control room back on Earth.</td></tr><tr style='color: red;'><td>- </td></tr><tr style='color: red;'><td>- The astronauts stepped out onto the alien terrain, taking in the breathtaking landscape of rusty red rocks and dusty plains. They set up their base camp and began their scientific research, collecting samples and conducting experiments to better understand the planet's composition and potential for sustaining life.</td></tr><tr style='color: red;'><td>- </td></tr><tr style='color: red;'><td>- Despite the challenges of living</td></tr></table>"],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"}],"source":["topic = \"a journey to Mars\"\n","system_message = \"You are a helpful assistant that generates short stories.\"\n","user_request = f\"Generate a short story about {topic}.\"\n","\n","previous_response = await get_chat_response(\n","    system_message=system_message, user_request=user_request\n",")\n","\n","response = await get_chat_response(\n","    system_message=system_message, user_request=user_request\n",")\n","\n","# The function compare_responses is then called with the two responses as arguments.\n","# This function will compare the two responses and display the differences in a table.\n","# If no differences are found, it will print \"No differences found.\"\n","compare_responses(previous_response, response)"]},{"cell_type":"markdown","metadata":{"cell_id":"e7eaf30e13ac4841b11dcffc505379c1","deepnote_cell_type":"markdown"},"source":["Now, let's try to generate the short story with the same topic (a journey to Mars) with a constant `seed` of 123 and compare the responses and `system_fingerprint`.\n"]},{"cell_type":"code","execution_count":null,"metadata":{"cell_id":"a5754b8ef4074cf7adb479d44bebd97b","deepnote_cell_type":"code"},"outputs":[{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>In the not-so-distant future, a team of brave astronauts embarked on a groundbreaking journey to Mars. The spacecraft, named \"Odyssey,\" soared through the vast expanse of space, leaving Earth behind as they ventured toward the mysterious red planet.\n","\n","As the crew navigated through the cosmos, they encountered a series of challenges and obstacles, from intense solar flares to treacherous asteroid fields. However, their unwavering determination and spirit of camaraderie propelled them forward, overcoming each hurdle with courage and resilience.\n","\n","Upon reaching Mars, the astronauts were greeted by a breathtaking landscape of rust-colored deserts and towering canyons. They marveled at the alien terrain, conducting scientific experiments and collecting samples to better understand the planet's enigmatic history.\n","\n","Amidst their exploration, the crew faced unexpected setbacks, including a sudden dust storm that threatened their safety. Yet, they stood united, devising ingenious solutions and supporting each other through the adversity.\n","\n","After a successful mission on Mars, the</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_fefa7b2153</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>31</td></tr>\n","        <tr><th>Number of completion tokens</th><td>200</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>In the not-so-distant future, a team of brave astronauts embarked on a groundbreaking journey to Mars. The spacecraft, named \"Odyssey,\" soared through the vast expanse of space, leaving Earth behind as they ventured toward the mysterious red planet.\n","\n","As the crew navigated through the cosmos, they encountered a series of challenges and obstacles, from intense solar flares to treacherous asteroid fields. However, their unwavering determination and spirit of camaraderie propelled them forward, overcoming each hurdle with courage and resilience.\n","\n","Upon reaching Mars, the astronauts were greeted by a breathtaking landscape of rust-colored deserts and towering canyons. They marveled at the alien terrain, conducting scientific experiments and collecting samples to better understand the planet's enigmatic history.\n","\n","Amidst their exploration, the crew faced unexpected setbacks, including a sudden dust storm that threatened their safety. Yet, they stood united, devising ingenious solutions and supporting each other through the adversity.\n","\n","After a successful mission on Mars, the</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_fefa7b2153</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>31</td></tr>\n","        <tr><th>Number of completion tokens</th><td>200</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["No differences found.\n"]}],"source":["SEED = 123\n","response = await get_chat_response(\n","    system_message=system_message, seed=SEED, user_request=user_request\n",")\n","previous_response = response\n","response = await get_chat_response(\n","    system_message=system_message, seed=SEED, user_request=user_request\n",")\n","\n","compare_responses(previous_response, response)"]},{"cell_type":"markdown","metadata":{"cell_id":"f6c8ae9a6e29451baaeb52b7203fbea8","deepnote_cell_type":"markdown"},"source":["## Conclusion\n","\n","We demonstrated how to use a fixed integer `seed` to generate consistent outputs from our model.This is particularly useful in scenarios where reproducibility is important. However, it's important to note that while the `seed` ensures consistency, it does not guarantee the quality of the output. For instance, in the example provided, we used the same seed to generate a short story about a journey to Mars. Despite querying the model multiple times, the output remained consistent, demonstrating the effectiveness of using this model level control for reproducibility. Another great extension of this could be to use consistent `seed` when benchmarking/evaluating the performance of different prompts or models, to ensure that each version is evaluated under the same conditions, making the comparisons fair and the results reliable.\n"]},{"cell_type":"markdown","metadata":{"created_in_deepnote_cell":true,"deepnote_cell_type":"markdown"},"source":["<a style='text-decoration:none;line-height:16px;display:flex;color:#5B5B62;padding:10px;justify-content:end;' href='https://deepnote.com?utm_source=created-in-deepnote-cell&projectId=baca1bec-71a4-4a27-a9d1-210d39960b44' target=\"_blank\">\n","<img alt='Created in deepnote.com' style='display:inline;max-height:16px;margin:0px;margin-right:7.5px;' src='data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB3aWR0aD0iODBweCIgaGVpZ2h0PSI4MHB4IiB2aWV3Qm94PSIwIDAgODAgODAiIHZlcnNpb249IjEuMSIgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzIwMDAvc3ZnIiB4bWxuczp4bGluaz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94bGluayI+CiAgICA8IS0tIEdlbmVyYXRvcjogU2tldGNoIDU0LjEgKDc2NDkwKSAtIGh0dHBzOi8vc2tldGNoYXBwLmNvbSAtLT4KICAgIDx0aXRsZT5Hcm91cCAzPC90aXRsZT4KICAgIDxkZXNjPkNyZWF0ZWQgd2l0aCBTa2V0Y2guPC9kZXNjPgogICAgPGcgaWQ9IkxhbmRpbmciIHN0cm9rZT0ibm9uZSIgc3Ryb2tlLXdpZHRoPSIxIiBmaWxsPSJub25lIiBmaWxsLXJ1bGU9ImV2ZW5vZGQiPgogICAgICAgIDxnIGlkPSJBcnRib2FyZCIgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTEyMzUuMDAwMDAwLCAtNzkuMDAwMDAwKSI+CiAgICAgICAgICAgIDxnIGlkPSJHcm91cC0zIiB0cmFuc2Zvcm09InRyYW5zbGF0ZSgxMjM1LjAwMDAwMCwgNzkuMDAwMDAwKSI+CiAgICAgICAgICAgICAgICA8cG9seWdvbiBpZD0iUGF0aC0yMCIgZmlsbD0iIzAyNjVCNCIgcG9pbnRzPSIyLjM3NjIzNzYyIDgwIDM4LjA0NzY2NjcgODAgNTcuODIxNzgyMiA3My44MDU3NTkyIDU3LjgyMTc4MjIgMzIuNzU5MjczOSAzOS4xNDAyMjc4IDMxLjY4MzE2ODMiPjwvcG9seWdvbj4KICAgICAgICAgICAgICAgIDxwYXRoIGQ9Ik0zNS4wMDc3MTgsODAgQzQyLjkwNjIwMDcsNzYuNDU0OTM1OCA0Ny41NjQ5MTY3LDcxLjU0MjI2NzEgNDguOTgzODY2LDY1LjI2MTk5MzkgQzUxLjExMjI4OTksNTUuODQxNTg0MiA0MS42NzcxNzk1LDQ5LjIxMjIyODQgMjUuNjIzOTg0Niw0OS4yMTIyMjg0IEMyNS40ODQ5Mjg5LDQ5LjEyNjg0NDggMjkuODI2MTI5Niw0My4yODM4MjQ4IDM4LjY0NzU4NjksMzEuNjgzMTY4MyBMNzIuODcxMjg3MSwzMi41NTQ0MjUgTDY1LjI4MDk3Myw2Ny42NzYzNDIxIEw1MS4xMTIyODk5LDc3LjM3NjE0NCBMMzUuMDA3NzE4LDgwIFoiIGlkPSJQYXRoLTIyIiBmaWxsPSIjMDAyODY4Ij48L3BhdGg+CiAgICAgICAgICAgICAgICA8cGF0aCBkPSJNMCwzNy43MzA0NDA1IEwyNy4xMTQ1MzcsMC4yNTcxMTE0MzYgQzYyLjM3MTUxMjMsLTEuOTkwNzE3MDEgODAsMTAuNTAwMzkyNyA4MCwzNy43MzA0NDA1IEM4MCw2NC45NjA0ODgyIDY0Ljc3NjUwMzgsNzkuMDUwMzQxNCAzNC4zMjk1MTEzLDgwIEM0Ny4wNTUzNDg5LDc3LjU2NzA4MDggNTMuNDE4MjY3Nyw3MC4zMTM2MTAzIDUzLjQxODI2NzcsNTguMjM5NTg4NSBDNTMuNDE4MjY3Nyw0MC4xMjg1NTU3IDM2LjMwMzk1NDQsMzcuNzMwNDQwNSAyNS4yMjc0MTcsMzcuNzMwNDQwNSBDMTcuODQzMDU4NiwzNy43MzA0NDA1IDkuNDMzOTE5NjYsMzcuNzMwNDQwNSAwLDM3LjczMDQ0MDUgWiIgaWQ9IlBhdGgtMTkiIGZpbGw9IiMzNzkzRUYiPjwvcGF0aD4KICAgICAgICAgICAgPC9nPgogICAgICAgIDwvZz4KICAgIDwvZz4KPC9zdmc+' > </img>\n","Created in <span style='font-weight:600;margin-left:4px;'>Deepnote</span></a>\n"]}],"metadata":{"deepnote":{},"deepnote_execution_queue":[],"deepnote_notebook_id":"90ee66ed8ee74f0dad849c869f1da806","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.9.13"}},"nbformat":4,"nbformat_minor":0}