"When you call the OpenAI API repeatedly, you may encounter error messages that say `429: 'Too Many Requests'` or `RateLimitError`. These error messages come from exceeding the API's rate limits.\n",
"This guide shares tips for avoiding and handling rate limit errors.\n",
"\n",
"To see an example script for throttling parallel requests to avoid rate limit errors, see [api_request_parallel_processor.py](api_request_parallel_processor.py).\n",
"Rate limits are a common practice for APIs, and they're put in place for a few different reasons.\n",
"\n",
"- First, they help protect against abuse or misuse of the API. For example, a malicious actor could flood the API with requests in an attempt to overload it or cause disruptions in service. By setting rate limits, OpenAI can prevent this kind of activity.\n",
"- Second, rate limits help ensure that everyone has fair access to the API. If one person or organization makes an excessive number of requests, it could bog down the API for everyone else. By throttling the number of requests that a single user can make, OpenAI ensures that everyone has an opportunity to use the API without experiencing slowdowns.\n",
"- Lastly, rate limits can help OpenAI manage the aggregate load on its infrastructure. If requests to the API increase dramatically, it could tax the servers and cause performance issues. By setting rate limits, OpenAI can help maintain a smooth and consistent experience for all users.\n",
"- [Help Center: Is API usage subject to any rate limits?](https://help.openai.com/en/articles/5955598-is-api-usage-subject-to-any-rate-limits)\n",
"- [Help Center: How can I solve 429: 'Too Many Requests' errors?](https://help.openai.com/en/articles/5955604-how-can-i-solve-429-too-many-requests-errors)\n",
"A rate limit error will occur when API requests are sent too quickly. If using the OpenAI Python library, they will look something like:\n",
"\n",
"```\n",
"RateLimitError: Rate limit reached for default-codex in organization org-{id} on requests per min. Limit: 20.000000 / min. Current: 24.000000 / min. Contact support@openai.com if you continue to have issues or if you’d like to request an increase.\n",
"```\n",
"\n",
"Below is example code for triggering a rate limit error."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import openai # for making OpenAI API requests\n",
"One easy way to avoid rate limit errors is to automatically retry requests with a random exponential backoff. Retrying with exponential backoff means performing a short sleep when a rate limit error is hit, then retrying the unsuccessful request. If the request is still unsuccessful, the sleep length is increased and the process is repeated. This continues until the request is successful or until a maximum number of retries is reached.\n",
"\n",
"This approach has many benefits:\n",
"\n",
"- Automatic retries means you can recover from rate limit errors without crashes or missing data\n",
"- Exponential backoff means that your first retries can be tried quickly, while still benefiting from longer delays if your first few retries fail\n",
"- Adding random jitter to the delay helps retries from all hitting at the same time\n",
"\n",
"Note that unsuccessful requests contribute to your per-minute limit, so continuously resending a request won’t work.\n",
"\n",
"Below are a few example solutions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Example #1: Using the Tenacity library\n",
"\n",
"[Tenacity](https://tenacity.readthedocs.io/en/latest/) is an Apache 2.0 licensed general-purpose retrying library, written in Python, to simplify the task of adding retry behavior to just about anything.\n",
"\n",
"To add exponential backoff to your requests, you can use the `tenacity.retry` [decorator](https://peps.python.org/pep-0318/). The following example uses the `tenacity.wait_random_exponential` function to add random exponential backoff to a request.\n",
"\n",
"Note that the Tenacity library is a third-party tool, and OpenAI makes no guarantees about its reliability or security."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<OpenAIObject text_completion id=cmpl-5oowO391reUW8RGVfFyzBM1uBs4A5 at 0x10d8cae00> JSON: {\n",
" \"choices\": [\n",
" {\n",
" \"finish_reason\": \"length\",\n",
" \"index\": 0,\n",
" \"logprobs\": null,\n",
" \"text\": \" a little girl dreamed of becoming a model.\\n\\nNowadays, that dream\"\n",
"## How to maximize throughput of batch processing given rate limits\n",
"\n",
"If you're processing real-time requests from users, backoff and retry is a great strategy to minimize latency while avoiding rate limit errors.\n",
"\n",
"However, if you're processing large volumes of batch data, where throughput matters more than latency, there are a few other things you can do in addition to backoff and retry.\n",
"\n",
"### Proactively adding delay between requests\n",
"\n",
"If you are constantly hitting the rate limit, then backing off, then hitting the rate limit again, then backing off again, it's possible that a good fraction of your request budget will be 'wasted' on requests that need to be retried. This limits your processing throughput, given a fixed rate limit.\n",
"Here, one potential solution is to calculate your rate limit and add a delay equal to its reciprocal (e.g., if your rate limit 20 requests per minute, add a delay of 3–6 seconds to each request). This can help you operate near the rate limit ceiling without hitting it and incurring wasted requests.\n",
" \"\"\"Delay a completion by a specified amount of time.\"\"\"\n",
"\n",
" # Sleep for the delay\n",
" time.sleep(delay_in_seconds)\n",
"\n",
" # Call the Completion API and return the result\n",
" return openai.Completion.create(**kwargs)\n",
"\n",
"\n",
"# Calculate the delay based on your rate limit\n",
"rate_limit_per_minute = 20\n",
"delay = 60.0 / rate_limit_per_minute\n",
"\n",
"delayed_completion(\n",
" delay_in_seconds=delay,\n",
" model=\"text-davinci-002\",\n",
" prompt=\"Once upon a time,\"\n",
")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"### Batching requests\n",
"\n",
"The OpenAI API has separate limits for requests per minute and tokens per minute.\n",
"\n",
"If you're hitting the limit on requests per minute, but have headroom on tokens per minute, you can increase your throughput by batching multiple tasks into each request. This will allow you to process more tokens per minute, especially with the smaller models.\n",
"\n",
"Sending in a batch of prompts works exactly the same as a normal API call, except that pass in a list of strings to `prompt` parameter instead of a single string.\n",
"\n",
"**Warning:** the response object may not return completions in the order of the prompts, so always remember to match responses back to prompts using the `index` field.\n",
"\n",
"#### Example without batching"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Once upon a time, before there were grandiloquent tales of the massacre at Fort Mims, there were stories of\n",
"Once upon a time, a full-sized search and rescue was created. However, CIDIs are the addition of requiring\n",
"Once upon a time, Schubert was hot with the films. “Schubert sings of honey, flowers,\n",
"Once upon a time, you could watch these films on your VCR, sometimes years after their initial theatrical release, and there\n",
"Once upon a time, there was a forest. In that forest, the forest animals ruled. The forest animals had their homes\n",
"Once upon a time, there were two programs that complained about false positive scans. Peacock and Midnight Manager alike, only\n",
"Once upon a time, a long, long time ago, tragedy struck. it was the darkest of nights, and there was\n",
"Once upon a time, when Adam was a perfect little gentleman, he was presented at Court as a guarantee of good character.\n",
"Once upon a time, Adam and Eve made a mistake. They ate the fruit from the tree of immortality and split the consequences\n",
"Once upon a time, there was a set of programming fundamental principles known as the \"X model.\" This is a set of\n"
]
}
],
"source": [
"import openai # for making OpenAI API requests\n",
"\n",
"\n",
"num_stories = 10\n",
"prompt = \"Once upon a time,\"\n",
"\n",
"# serial example, with one story completion per request\n",
"for _ in range(num_stories):\n",
" response = openai.Completion.create(\n",
" model=\"curie\",\n",
" prompt=prompt,\n",
" max_tokens=20,\n",
" )\n",
"\n",
" # print story\n",
" print(prompt + response.choices[0].text)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Example with batching"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Once upon a time, there were two sisters, Eliza Pickering and Ariana 'Ari' Lucas. When these lovely\n",
"Once upon a time, Keene was stung by a worm — actually, probably a python — snaking through his leg\n",
"Once upon a time, there was a professor of physics during the depression. It was difficult, during this time, to get\n",
"Once upon a time, before you got sick, you told stories to all and sundry, and your listeners believed in you\n",
"Once upon a time, there was one very old nice donkey. He was incredibly smart, in a very old, kind of\n",
"Once upon a time, the property of a common lodging house was a common cup for all the inhabitants. Betimes a constant\n",
"Once upon a time, in an unspecified country, there was a witch who had an illegal product. It was highly effective,\n",
"Once upon a time, a long time ago, I turned 13, my beautiful dog Duncan swept me up into his jaws like\n",
"Once upon a time, as a thoroughly reformed creature from an army of Nazis, he took On Judgement Day myself and his\n",
"Once upon a time, Capcom made a game for the Atari VCS called Missile Command. While it was innovative at the time\n"
]
}
],
"source": [
"import openai # for making OpenAI API requests\n",
"\n",
"\n",
"num_stories = 10\n",
"prompts = [\"Once upon a time,\"] * num_stories\n",
"\n",
"# batched example, with 10 stories completions per request\n",
"We've written an example script for parallel processing large quantities of API requests: [api_request_parallel_processor.py](https://github.com/openai/openai-cookbook/blob/main/examples/api_request_parallel_processor.py).\n",
