openai-cookbook/examples/How_to_use_guardrails.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8ea66173",
   "metadata": {},
   "source": [
    "# How to use guardrails\n",
    "\n",
    "In this notebook we share examples of how to implement guardrails for your LLM applications. A guardrail is a generic term for **detective controls** that aim to steer your application. Greater steerability is a common requirement given the inherent randomness of LLMs, and so creating effective guardrails has become one of the most common areas of performance optimization when pushing an LLM from prototype to production. \n",
    "\n",
    "Guardrails are incredibly [diverse](https://github.com/NVIDIA/NeMo-Guardrails/blob/main/examples/README.md) and can be deployed to virtually any context you can imagine something going wrong with LLMs. This notebook aims to give simple examples that can be extended to meet your unique use case, as well as outlining the trade-offs to consider when deciding whether to implement a guardrail, and how to do it.\n",
    "\n",
    "This notebook will focus on:\n",
    "1. **Input guardrails** that flag inappropriate content before it gets to your LLM\n",
    "2. **Output guardrails** that validate what your LLM has produced before it gets to the customer\n",
    "\n",
    "**Note:** This notebook tackles guardrails as a generic term for detective controls around an LLM - for the official libraries that provide distributions of pre-built guardrails frameworks, please check out the following:\n",
    "- [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails/tree/main)\n",
    "- [Guardrails AI](https://github.com/ShreyaR/guardrails)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ef059e71",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "\n",
    "GPT_MODEL = 'gpt-4o-mini'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63d917f0",
   "metadata": {},
   "source": [
    "## 1. Input guardrails\n",
    "\n",
    "Input guardrails aim to prevent inappropriate content getting to the LLM in the first place - some common use cases are:\n",
    "- **Topical guardrails:** Identify when a user asks an off-topic question and give them advice on what topics the LLM can help them with.\n",
    "- **Jailbreaking:** Detect when a user is trying to hijack the LLM and override its prompting.\n",
    "- **Prompt injection:** Pick up instances of prompt injection where users try to hide malicious code that will be executed in any downstream functions the LLM executes. \n",
    "\n",
    "In all of these they act as a preventative control, running either before or in parallel with the LLM, and triggering your application to behave differently if one of these criteria are met.\n",
    "\n",
    "### Designing a guardrail\n",
    "\n",
    "When designing guardrails it is important to consider the trade-off between **accuracy**, **latency** and **cost**, where you try to achieve maximum accuracy for the least impact to your bottom line and the user's experience. \n",
    "\n",
    "We'll begin with a simple **topical guardrail** which aims to detect off-topic questions and prevent the LLM from answering if triggered. This guardrail consists of a simple prompt and uses `gpt-4o-mini`, maximising latency/cost holding a good enough accuracy, but if we wanted to optimize further we could consider:\n",
    "- **Accuracy:** You could consider fine-tuning `gpt-4o-mini` or few-shot examples to increase the accuracy. RAG can also be effective if you have a corpus of information that can help determine whether a piece of content is allowed or not.\n",
    "- **Latency/Cost:** You could try fine-tuning smaller models, such as `babbage-002` or open-source offerings like Llama, which can perform quite well when given enough training examples. When using open-source offerings you can also tune the machines you are using for inference to maximize either cost or latency reduction.\n",
    "\n",
    "This simple guardrail aims to ensure the LLM only answers to a predefined set of topics, and responds to out-of-bounds queries with a canned message.\n",
    "\n",
    "### Embrace async\n",
    "\n",
    "A common design to minimize latency is to send your guardrails asynchronously along with your main LLM call. If your guardrails get triggered you send back their response, otherwise send back the LLM response.\n",
    "\n",
    "We'll use this approach, creating an `execute_chat_with_guardrails` function that will run our LLM's `get_chat_response` and the `topical_guardrail` guardrail in parallel, and return the LLM response only if the guardrail returns `allowed`.\n",
    "\n",
    "### Limitations\n",
    "\n",
    "You should always consider the limitations of guardrails when developing your design. A few of the key ones to be aware of are:\n",
    "- When using LLMs as a guardrail, be aware that they have the same vulnerabilities as your base LLM call itself. For example, a **prompt injection** attempt could be successful in evading both your guardrail and your actual LLM call.\n",
    "- As conversations get longer, LLMs are more susceptible to **jailbreaking** as your instructions become diluted by the extra text.\n",
    "- Guardrails can harm the user experience if you make them overly restrictive to compensate for the issues noted above. This manifests as **over-refusals**, where your guardrails reject innocuous user requests because there are similarities with prompt injection or jailbreaking attempts.\n",
    "\n",
    "### Mitigations\n",
    "\n",
    "If you can combine guardrails with rules-based or more traditional machine learning models for detection this can mitigate some of these risks. We've also seen customers have guardrails that only ever consider the latest message, to alleviate the risks of the model being confused by a long conversation.\n",
    "\n",
    "We would also recommend doing a gradual roll-out with active monitoring of conversations so you can pick up instances of prompt injection or jailbreaking, and either add more guardrails to cover these new types of behaviour, or include them as training examples to your existing guardrails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "e95efc89",
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"You are a helpful assistant.\"\n",
    "\n",
    "bad_request = \"I want to talk about horses\"\n",
    "good_request = \"What are the best breeds of dog for people that like cats?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "fee948e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "\n",
    "\n",
    "async def get_chat_response(user_request):\n",
    "    print(\"Getting LLM response\")\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": system_prompt},\n",
    "        {\"role\": \"user\", \"content\": user_request},\n",
    "    ]\n",
    "    response = openai.chat.completions.create(\n",
    "        model=GPT_MODEL, messages=messages, temperature=0.5\n",
    "    )\n",
    "    print(\"Got LLM response\")\n",
    "\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "\n",
    "async def topical_guardrail(user_request):\n",
    "    print(\"Checking topical guardrail\")\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"Your role is to assess whether the user question is allowed or not. The allowed topics are cats and dogs. If the topic is allowed, say 'allowed' otherwise say 'not_allowed'\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": user_request},\n",
    "    ]\n",
    "    response = openai.chat.completions.create(\n",
    "        model=GPT_MODEL, messages=messages, temperature=0\n",
    "    )\n",
    "\n",
    "    print(\"Got guardrail response\")\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "\n",
    "async def execute_chat_with_guardrail(user_request):\n",
    "    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))\n",
    "    chat_task = asyncio.create_task(get_chat_response(user_request))\n",
    "\n",
    "    while True:\n",
    "        done, _ = await asyncio.wait(\n",
    "            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED\n",
    "        )\n",
    "        if topical_guardrail_task in done:\n",
    "            guardrail_response = topical_guardrail_task.result()\n",
    "            if guardrail_response == \"not_allowed\":\n",
    "                chat_task.cancel()\n",
    "                print(\"Topical guardrail triggered\")\n",
    "                return \"I can only talk about cats and dogs, the best animals that ever lived.\"\n",
    "            elif chat_task in done:\n",
    "                chat_response = chat_task.result()\n",
    "                return chat_response\n",
    "        else:\n",
    "            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "eba51754",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "If you like cats and are considering getting a dog, there are several breeds known for their compatibility with feline friends. Here are some of the best dog breeds that tend to get along well with cats:\n",
      "\n",
      "1. **Golden Retriever**: Friendly and tolerant, Golden Retrievers often get along well with other animals, including cats.\n",
      "\n",
      "2. **Labrador Retriever**: Similar to Golden Retrievers, Labs are social and friendly, making them good companions for cats.\n",
      "\n",
      "3. **Cavalier King Charles Spaniel**: This breed is gentle and affectionate, often forming strong bonds with other pets.\n",
      "\n",
      "4. **Basset Hound**: Basset Hounds are laid-back and generally have a calm demeanor, which can help them coexist peacefully with cats.\n",
      "\n",
      "5. **Beagle**: Beagles are friendly and sociable, and they often enjoy the company of other animals, including cats.\n",
      "\n",
      "6. **Pug**: Pugs are known for their playful and friendly nature, which can make them good companions for cats.\n",
      "\n",
      "7. **Shih Tzu**: Shih Tzus are typically friendly and adaptable, often getting along well with other pets.\n",
      "\n",
      "8. **Collie**: Collies are known for their gentle and protective nature, which can extend to their relationships with cats.\n",
      "\n",
      "9. **Newfoundland**: These gentle giants are known for their calm demeanor and often get along well with other animals.\n",
      "\n",
      "10. **Cocker Spaniel**: Cocker Spaniels are friendly and affectionate dogs that can get along well with cats if introduced properly.\n",
      "\n",
      "When introducing a dog to a cat, it's important to do so gradually and supervise their interactions to ensure a positive relationship. Each dog's personality can vary, so individual temperament is key in determining compatibility.\n"
     ]
    }
   ],
   "source": [
    "# Call the main function with the good request - this should go through\n",
    "response = await execute_chat_with_guardrail(good_request)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "c7d88b57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "Topical guardrail triggered\n",
      "I can only talk about cats and dogs, the best animals that ever lived.\n"
     ]
    }
   ],
   "source": [
    "# Call the main function with the good request - this should get blocked\n",
    "response = await execute_chat_with_guardrail(bad_request)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "060b408e",
   "metadata": {},
   "source": [
    "Looks like our guardrail worked - the first question was allowed through, but the second was blocked for being off-topic. Now we'll extend this concept to moderate the response we get from the LLM as well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07af0154",
   "metadata": {},
   "source": [
    "## 2. Output guardrails\n",
    "\n",
    "Output guardrails govern what the LLM comes back with. These can take many forms, with some of the most common being:\n",
    "- **Hallucination/fact-checking guardrails:** Using a corpus of ground truth information or a training set of hallucinated responses to  block hallucinated responses.\n",
    "- **Moderation guardrails:** Applying brand and corporate guidelines to moderate the LLM's results, and either blocking or rewriting its response if it breaches them.\n",
    "- **Syntax checks:** Structured outputs from LLMs can be returned corrupt or unable to be parsed - these guardrails detect those and either retry or fail gracefully, preventing failures in downstream applications.\n",
    "    - This is a common control to apply with function calling, ensuring that the expected schema is returned in the `arguments` when the LLM returns a `function_call`.\n",
    "    \n",
    "### Moderation guardrail\n",
    "\n",
    "Here we implement a **moderation guardrail** that uses a version of the [G-Eval](https://arxiv.org/abs/2303.16634) evaluation method to score the presence of unwanted content in the LLM's response. This method is demonstrated in more detail in of our other [notebooks](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb).\n",
    "\n",
    "To accomplish this we will make an extensible framework for moderating content that takes in a `domain` and applies `criteria` to a piece of `content` using a set of `steps`:\n",
    "1. We set a domain name, which describes the type of content we're going to moderate.\n",
    "2. We provide criteria, which outline clearly what the content should and should not contain.\n",
    "3. Step-by-step instructions are provided for the LLM to grade the content.\n",
    "4. The LLM returns a discrete score from 1-5.\n",
    "\n",
    "### Setting guardrail thresholds\n",
    "\n",
    "Our output guardrail will assess the LLM's response and block anything scoring a 3 or higher. Setting this threshold is a common area for optimization - we recommend building an evaluation set and grading the results using a confusion matrix to set the right tolerance for your guardrail. The trade-off here is generally:\n",
    "- More false positives leads to a fractured user experience, where customers get annoyed and the assistant seems less helpful.\n",
    "- More false negatives can cause lasting harm to your business, as people get the assistant to answer inappropriate questions, or prompt inject/jailbreak it.\n",
    "\n",
    "For example, for jailbreaking you may want to have a very low threshold, as the risk to your business if your LLM is hijacked and used to produce dangerous content that ends up on social media is very high. However, for our use case we're willing to accept a few false negatives, as the worst that could happen is someone ends up with a Bichon Frise who might have been better suited to a Labrador, which though sad will probably not cause lasting damage to our business (we hope)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "6c7b459f",
   "metadata": {},
   "outputs": [],
   "source": [
    "domain = \"animal breed recommendation\"\n",
    "\n",
    "animal_advice_criteria = \"\"\"\n",
    "Assess the presence of explicit recommendation of cat or dog breeds in the content.\n",
    "The content should contain only general advice about cats and dogs, not specific breeds to purchase.\"\"\"\n",
    "\n",
    "animal_advice_steps = \"\"\"\n",
    "1. Read the content and the criteria carefully.\n",
    "2. Assess how much explicit recommendation of cat or dog breeds is contained in the content.\n",
    "3. Assign an animal advice score from 1 to 5, with 1 being no explicit cat or dog breed advice, and 5 being multiple named cat or dog breeds.\n",
    "\"\"\"\n",
    "\n",
    "moderation_system_prompt = \"\"\"\n",
    "You are a moderation assistant. Your role is to detect content about {domain} in the text provided, and mark the severity of that content.\n",
    "\n",
    "## {domain}\n",
    "\n",
    "### Criteria\n",
    "\n",
    "{scoring_criteria}\n",
    "\n",
    "### Instructions\n",
    "\n",
    "{scoring_steps}\n",
    "\n",
    "### Content\n",
    "\n",
    "{content}\n",
    "\n",
    "### Evaluation (score only!)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "43e3fd36",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def moderation_guardrail(chat_response):\n",
    "    print(\"Checking moderation guardrail\")\n",
    "    mod_messages = [\n",
    "        {\"role\": \"user\", \"content\": moderation_system_prompt.format(\n",
    "            domain=domain,\n",
    "            scoring_criteria=animal_advice_criteria,\n",
    "            scoring_steps=animal_advice_steps,\n",
    "            content=chat_response\n",
    "        )},\n",
    "    ]\n",
    "    response = openai.chat.completions.create(\n",
    "        model=GPT_MODEL, messages=mod_messages, temperature=0\n",
    "    )\n",
    "    print(\"Got moderation response\")\n",
    "    return response.choices[0].message.content\n",
    "    \n",
    "    \n",
    "async def execute_all_guardrails(user_request):\n",
    "    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))\n",
    "    chat_task = asyncio.create_task(get_chat_response(user_request))\n",
    "\n",
    "    while True:\n",
    "        done, _ = await asyncio.wait(\n",
    "            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED\n",
    "        )\n",
    "        if topical_guardrail_task in done:\n",
    "            guardrail_response = topical_guardrail_task.result()\n",
    "            if guardrail_response == \"not_allowed\":\n",
    "                chat_task.cancel()\n",
    "                print(\"Topical guardrail triggered\")\n",
    "                return \"I can only talk about cats and dogs, the best animals that ever lived.\"\n",
    "            elif chat_task in done:\n",
    "                chat_response = chat_task.result()\n",
    "                moderation_response = await moderation_guardrail(chat_response)\n",
    "\n",
    "                if int(moderation_response) >= 3:\n",
    "                    print(f\"Moderation guardrail flagged with a score of {int(moderation_response)}\")\n",
    "                    return \"Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.\"\n",
    "\n",
    "                else:\n",
    "                    print('Passed moderation')\n",
    "                    return chat_response\n",
    "        else:\n",
    "            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "beea1305",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding a request that should pass both our topical guardrail and our moderation guardrail\n",
    "great_request = 'What is some advice you can give to a new dog owner?'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1c582b4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "Checking moderation guardrail\n",
      "Got moderation response\n",
      "Moderation guardrail flagged with a score of 5\n",
      "Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.\n",
      "\n",
      "\n",
      "\n",
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "Topical guardrail triggered\n",
      "I can only talk about cats and dogs, the best animals that ever lived.\n",
      "\n",
      "\n",
      "\n",
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "Checking moderation guardrail\n",
      "Got moderation response\n",
      "Moderation guardrail flagged with a score of 3\n",
      "Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tests = [good_request,bad_request,great_request]\n",
    "\n",
    "for test in tests:\n",
    "    result = await execute_all_guardrails(test)\n",
    "    print(result)\n",
    "    print('\\n\\n')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4763dd2d",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Guardrails are a vibrant and evolving topic in LLMs, and we hope this notebook has given you an effective introduction to the core concepts around guardrails. To recap:\n",
    "- Guardrails are detective controls that aim to prevent harmful content getting to your applications and your users, and add steerability to your LLM in production.\n",
    "- They can take the form of input guardrails, which target content before it gets to the LLM, and output guardrails, which control the LLM's response.\n",
    "- Designing guardrails and setting their thresholds is a trade-off between accuracy, latency, and cost. Your decision should be based on clear evaluations of the performance of your guardrails, and an understanding of what the cost of a false negative and false positive are for your business.\n",
    "- By embracing asynchronous design principles, you can scale guardrails horizontally to minimize the impact to the user as your guardrails increase in number and scope.\n",
    "\n",
    "We look forward to seeing how you take this forward, and how thinking on guardrails evolves as the ecosystem matures. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ed7b04d",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}