openai-cookbook/examples/Reproducible_outputs_with_the_seed_parameter.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"cell_id":"67bb097e130b41099c9d257dc06a4054","deepnote_cell_type":"markdown"},"source":["# How to make your completions outputs reproducible with the new seed parameter\n","\n","**TLDR**: Developers can now specify `seed` parameter in the Chat Completion request to receive (mostly) consistent outputs. To help you keep track of these changes, we expose the `system_fingerprint` field. If this value is different, you may see different outputs due to changes we've made on our systems. Please note that this feature is in beta and only currently supported for `gpt-4-1106-preview` and `gpt-3.5-turbo-1106`.\n","\n","### Context\n","\n","Reproducibility has always been a big request from user communities when using our APIs. For instance, when granted the capability of getting reproducible numerical result, users can unlock quite a bit of use cases that’s sensitive to numerical changes.\n","\n","#### Model level features for consistent outputs\n","\n","The Chat Completions and Completions APIs are non-deterministic by default (which means model outputs may differ from request to request), but now offer some control towards deterministic outputs using a few model level controls.\n","\n","This can unlock consistent completions which enables full control on the model behaviors for anything built on top of the APIs, and quite useful for reproducing results and testing so you know get peace of mind from knowing exactly what you’d get.\n","\n","#### Implementing consistent outputs\n","\n","To receive _mostly_ deterministic outputs across API calls:\n","\n","- Set the `seed` parameter to any integer of your choice, but use the same value across requests. For example, `12345`.\n","- Set all other parameters (prompt, temperature, top_p, etc.) to the same values across requests.\n","- In the response, check the `system_fingerprint` field. The system fingerprint is an identifier for the current combination of model weights, infrastructure, and other configuration options used by OpenAI servers to generate the completion. It changes whenever you change request parameters, or OpenAI updates numerical configuration of the infrastructure serving our models (which may happen a few times a year).\n","\n","If the `seed`, request parameters, and `system_fingerprint` all match across your requests, then model outputs will mostly be identical. There is a small chance that responses differ even when request parameters and `system_fingerprint` match, due to the inherent non-determinism of our models.\n"]},{"cell_type":"markdown","metadata":{"cell_id":"f49611fa59af4303883d76c491095fea","deepnote_cell_type":"markdown"},"source":["### Model level controls for consistent outputs - `seed` and `system_fingerprint`\n","\n","##### `seed`\n","\n","If specified, our system will make a best effort to sample deterministically, such that repeated requests with the same seed and parameters should return the same result. Determinism is not guaranteed, and you should refer to the `system_fingerprint` response parameter to monitor changes in the backend.\n","\n","##### `system_fingerprint`\n","\n","This fingerprint represents the backend configuration that the model runs with. It can be used in conjunction with the seed request parameter to understand when backend changes have been made that might impact determinism.This is the indicator on whether users should expect \"almost always the same result\".\n"]},{"cell_type":"markdown","metadata":{"cell_id":"cc6cd37b9a2243aaa4688ef8832512eb","deepnote_cell_type":"markdown"},"source":["## Example: Generating a short excerpt with a fixed seed\n","\n","In this example, we will demonstrate how to generate a short excerpt using a fixed seed. This can be particularly useful in scenarios where you need to generate consistent results for testing, debugging, or for applications that require consistent outputs."]},{"cell_type":"markdown","metadata":{},"source":["### Python SDK\n","\n","> **Note**\n","> Switch to latest version of the SDK (`1.3.3` at time of writing)."]},{"cell_type":"code","execution_count":null,"metadata":{},"outputs":[],"source":["!pip install --upgrade openai # Switch to the latest version of OpenAI (1.3.3 at time of writing)"]},{"cell_type":"code","execution_count":12,"metadata":{"cell_id":"48fd2d4c95ad465090ef97254a4a10d2","deepnote_cell_type":"code"},"outputs":[],"source":["import openai\n","import asyncio\n","from IPython.display import display, HTML\n","\n","from utils.embeddings_utils import (\n","    get_embedding,\n","    distances_from_embeddings\n",")\n","\n","GPT_MODEL = \"gpt-3.5-turbo-1106\""]},{"cell_type":"code","execution_count":13,"metadata":{"cell_id":"e54e0958be3746d39b6e4c16c59b395a","deepnote_cell_type":"code","deepnote_to_be_reexecuted":false,"execution_millis":5,"execution_start":1699034108287,"source_hash":null},"outputs":[],"source":["async def get_chat_response(\n","    system_message: str, user_request: str, seed: int = None, temperature: float = 0.7\n","):\n","    try:\n","        messages = [\n","            {\"role\": \"system\", \"content\": system_message},\n","            {\"role\": \"user\", \"content\": user_request},\n","        ]\n","\n","        response = openai.chat.completions.create(\n","            model=GPT_MODEL,\n","            messages=messages,\n","            seed=seed,\n","            max_tokens=200,\n","            temperature=temperature,\n","        )\n","\n","        response_content = response.choices[0].message.content\n","        system_fingerprint = response.system_fingerprint\n","        prompt_tokens = response.usage.prompt_tokens\n","        completion_tokens = response.usage.total_tokens - response.usage.prompt_tokens\n","\n","        table = f\"\"\"\n","        <table>\n","        <tr><th>Response</th><td>{response_content}</td></tr>\n","        <tr><th>System Fingerprint</th><td>{system_fingerprint}</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>{prompt_tokens}</td></tr>\n","        <tr><th>Number of completion tokens</th><td>{completion_tokens}</td></tr>\n","        </table>\n","        \"\"\"\n","        display(HTML(table))\n","\n","        return response_content\n","    except Exception as e:\n","        print(f\"An error occurred: {e}\")\n","        return None\n","\n","def calculate_average_distance(responses):\n","    \"\"\"\n","    This function calculates the average distance between the embeddings of the responses.\n","    The distance between embeddings is a measure of how similar the responses are.\n","    \"\"\"\n","    # Calculate embeddings for each response\n","    response_embeddings = [get_embedding(response) for response in responses]\n","\n","    # Compute distances between the first response and the rest\n","    distances = distances_from_embeddings(response_embeddings[0], response_embeddings[1:])\n","\n","    # Calculate the average distance\n","    average_distance = sum(distances) / len(distances)\n","\n","    # Return the average distance\n","    return average_distance"]},{"cell_type":"markdown","metadata":{"cell_id":"dfa39a438aa948cc910a46254df937af","deepnote_cell_type":"text-cell-p","formattedRanges":[]},"source":["First, let's try generating few different versions of a short excerpt about \"a journey to Mars\" without the `seed` parameter. This is the default behavior:"]},{"cell_type":"code","execution_count":14,"metadata":{"cell_id":"9d09f63309c449e4929364caccfd7065","deepnote_cell_type":"code","deepnote_to_be_reexecuted":false,"execution_millis":964,"execution_start":1699034108745,"source_hash":null},"outputs":[{"name":"stdout","output_type":"stream","text":["Output 1\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's Mars mission reaches critical stage as spacecraft successfully enters orbit around the red planet. The historic journey, which began over a year ago, has captured the world's attention as scientists and astronauts prepare to land on Mars for the first time. The mission is expected to provide valuable insights into the planet's geology, atmosphere, and potential for sustaining human life in the future.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>76</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Output 2\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's Perseverance rover successfully landed on Mars, marking a major milestone in the mission to explore the red planet. The rover is equipped with advanced scientific instruments to search for signs of ancient microbial life and collect samples of rock and soil for future return to Earth. This historic achievement paves the way for further exploration and potential human missions to Mars in the near future.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>76</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Output 3\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"SpaceX successfully launched the first manned mission to Mars yesterday, marking a historic milestone in space exploration. The crew of four astronauts will spend the next six months traveling to the red planet, where they will conduct groundbreaking research and experiments. This mission represents a significant step towards establishing a human presence on Mars and paves the way for future interplanetary travel.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>72</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Output 4\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's latest Mars mission exceeds expectations as the Perseverance rover uncovers tantalizing clues about the Red Planet's past. Scientists are thrilled by the discovery of ancient riverbeds and sedimentary rocks, raising hopes of finding signs of past life on Mars. With this exciting progress, the dream of sending humans to Mars feels closer than ever before.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>72</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Output 5\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's Perseverance Rover Successfully Lands on Mars, Begins Exploration Mission\n","\n","In a historic moment for space exploration, NASA's Perseverance rover has successfully landed on the surface of Mars. After a seven-month journey, the rover touched down in the Jezero Crater, a location scientists believe may have once held a lake and could potentially contain signs of ancient microbial life.\n","\n","The rover's primary mission is to search for evidence of past life on Mars and collect rock and soil samples for future return to Earth. Equipped with advanced scientific instruments, including cameras, spectrometers, and a drill, Perseverance will begin its exploration of the Martian surface, providing valuable data and insights into the planet's geology and potential habitability.\n","\n","This successful landing marks a significant milestone in humanity's quest to understand the red planet and paves the way for future manned missions to Mars. NASA's Perseverance rover is poised to unravel the mysteries of Mars and unlock new possibilities</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>200</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["The average similarity between responses is: 0.1136714512418833\n"]}],"source":["topic = \"a journey to Mars\"\n","system_message = \"You are a helpful assistant.\"\n","user_request = f\"Generate a short excerpt of news about {topic}.\"\n","\n","responses = []\n","\n","\n","async def get_response(i):\n","    print(f'Output {i + 1}\\n{\"-\" * 10}')\n","    response = await get_chat_response(\n","        system_message=system_message, user_request=user_request\n","    )\n","    return response\n","\n","\n","responses = await asyncio.gather(*[get_response(i) for i in range(5)])\n","average_distance = calculate_average_distance(responses)\n","print(f\"The average similarity between responses is: {average_distance}\")"]},{"cell_type":"markdown","metadata":{"cell_id":"e7eaf30e13ac4841b11dcffc505379c1","deepnote_cell_type":"markdown"},"source":["Now, let's try to tun the same code with a constant `seed` of 123 and `temperature` of 0 and compare the responses and `system_fingerprint`."]},{"cell_type":"code","execution_count":15,"metadata":{"cell_id":"a5754b8ef4074cf7adb479d44bebd97b","deepnote_cell_type":"code"},"outputs":[{"name":"stdout","output_type":"stream","text":["Output 1\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's Perseverance Rover Successfully Lands on Mars\n","\n","In a historic achievement, NASA's Perseverance rover has successfully landed on the surface of Mars, marking a major milestone in the exploration of the red planet. The rover, which traveled over 293 million miles from Earth, is equipped with state-of-the-art instruments designed to search for signs of ancient microbial life and collect rock and soil samples for future return to Earth. This mission represents a significant step forward in our understanding of Mars and the potential for human exploration of the planet in the future.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>113</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Output 2\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's Perseverance rover successfully lands on Mars, marking a historic milestone in space exploration. The rover is equipped with advanced scientific instruments to search for signs of ancient microbial life and collect samples for future return to Earth. This mission paves the way for future human exploration of the red planet, as scientists and engineers continue to push the boundaries of space travel and expand our understanding of the universe.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>81</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Output 3\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's Perseverance rover successfully lands on Mars, marking a historic milestone in space exploration. The rover is equipped with advanced scientific instruments to search for signs of ancient microbial life and collect samples for future return to Earth. This mission paves the way for future human exploration of the red planet, as NASA continues to push the boundaries of space exploration.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>72</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Output 4\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's Perseverance rover successfully lands on Mars, marking a historic milestone in space exploration. The rover is equipped with advanced scientific instruments to search for signs of ancient microbial life and collect samples for future return to Earth. This mission paves the way for future human exploration of the red planet, as scientists and engineers continue to push the boundaries of space travel and expand our understanding of the universe.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>81</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["Output 5\n","----------\n"]},{"data":{"text/html":["\n","        <table>\n","        <tr><th>Response</th><td>\"NASA's Perseverance rover successfully lands on Mars, marking a historic milestone in space exploration. The rover is equipped with advanced scientific instruments to search for signs of ancient microbial life and collect samples for future return to Earth. This mission paves the way for future human exploration of the red planet, as scientists and engineers continue to push the boundaries of space travel.\"</td></tr>\n","        <tr><th>System Fingerprint</th><td>fp_772e8125bb</td></tr>\n","        <tr><th>Number of prompt tokens</th><td>29</td></tr>\n","        <tr><th>Number of completion tokens</th><td>74</td></tr>\n","        </table>\n","        "],"text/plain":["<IPython.core.display.HTML object>"]},"metadata":{},"output_type":"display_data"},{"name":"stdout","output_type":"stream","text":["The average distance between responses is: 0.0449054397632461\n"]}],"source":["SEED = 123\n","responses = []\n","\n","\n","async def get_response(i):\n","    print(f'Output {i + 1}\\n{\"-\" * 10}')\n","    response = await get_chat_response(\n","        system_message=system_message,\n","        seed=SEED,\n","        temperature=0,\n","        user_request=user_request,\n","    )\n","    return response\n","\n","\n","responses = await asyncio.gather(*[get_response(i) for i in range(5)])\n","\n","average_distance = calculate_average_distance(responses)\n","print(f\"The average distance between responses is: {average_distance}\")"]},{"cell_type":"markdown","metadata":{},"source":["As we can observe, the `seed` parameter allows us to generate much more consistent results."]},{"cell_type":"markdown","metadata":{"cell_id":"f6c8ae9a6e29451baaeb52b7203fbea8","deepnote_cell_type":"markdown"},"source":["## Conclusion\n","\n","We demonstrated how to use a fixed integer `seed` to generate consistent outputs from our model. This is particularly useful in scenarios where reproducibility is important. However, it's important to note that while the `seed` ensures consistency, it does not guarantee the quality of the output. Note that when you want to use reproducible outputs, you need to set the `seed` to the same integer across Chat Completions calls. You should also match any other parameters like `temperature`, `max_tokens` etc. Further extension of reproducible outputs could be to use consistent `seed` when benchmarking/evaluating the performance of different prompts or models, to ensure that each version is evaluated under the same conditions, making the comparisons fair and the results reliable."]}],"metadata":{"deepnote":{},"deepnote_execution_queue":[],"deepnote_notebook_id":"90ee66ed8ee74f0dad849c869f1da806","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.11.5"}},"nbformat":4,"nbformat_minor":0}