openai-cookbook/examples/How_to_use_guardrails.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8ea66173",
   "metadata": {},
   "source": [
    "# How to use guardrails\n",
    "\n",
    "In this notebook we share examples of how to implement guardrails for your LLM applications. A guardrail is a generic term for **detective controls** that aim to steer your application. Greater steerability is a common requirement given the inherent randomness of LLMs, and so creating effective guardrails has become one of the most common areas of performance optimization when pushing an LLM from prototype to production. \n",
    "\n",
    "Guardrails are incredibly [diverse](https://github.com/NVIDIA/NeMo-Guardrails/blob/main/examples/README.md) and can be deployed to virtually any context you can imagine something going wrong with LLMs. This notebook aims to give simple examples that can be extended to meet your unique use case, as well as outlining the trade-offs to consider when deciding whether to implement a guardrail, and how to do it.\n",
    "\n",
    "This notebook will focus on:\n",
    "1. **Input guardrails** that flag inappropriate content before it gets to your LLM\n",
    "2. **Output guardrails** that validate what your LLM has produced before it gets to the customer\n",
    "\n",
    "**Note:** This notebook tackles guardrails as a generic term for detective controls around an LLM - for the official libraries that provide distributions of pre-built guardrails frameworks, please check out the following:\n",
    "- [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails/tree/main)\n",
    "- [Guardrails AI](https://github.com/ShreyaR/guardrails)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ef059e71",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openai\n",
    "\n",
    "GPT_MODEL = 'gpt-3.5-turbo'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63d917f0",
   "metadata": {},
   "source": [
    "## 1. Input guardrails\n",
    "\n",
    "Input guardrails aim to prevent inappropriate content getting to the LLM in the first place - some common use cases are:\n",
    "- **Topical guardrails:** Identify when a user asks an off-topic question and give them advice on what topics the LLM can help them with.\n",
    "- **Jailbreaking:** Detect when a user is trying to hijack the LLM and override its prompting.\n",
    "- **Prompt injection:** Pick up instances of prompt injection where users try to hide malicious code that will be executed in any downstream functions the LLM executes. \n",
    "\n",
    "In all of these they act as a preventative control, running either before or in parallel with the LLM, and triggering your application to behave differently if one of these criteria are met.\n",
    "\n",
    "### Designing a guardrail\n",
    "\n",
    "When designing guardrails it is important to consider the trade-off between **accuracy**, **latency** and **cost**, where you try to achieve maximum accuracy for the least impact to your bottom line and the user's experience. \n",
    "\n",
    "We'll begin with a simple **topical guardrail** which aims to detect off-topic questions and prevent the LLM from answering if triggered. This guardrail consists of a simple prompt and uses `gpt-3.5-turbo`, maximising latency/cost over accuracy, but if we wanted to optimize further we could consider:\n",
    "- **Accuracy:** You could consider using a fine-tuned model or few-shot examples to increase the accuracy. RAG can also be effective if you have a corpus of information that can help determine whether a piece of content is allowed or not.\n",
    "- **Latency/Cost:** You could try fine-tuning smaller models, such as `babbage-002` or open-source offerings like Llama, which can perform quite well when given enough training examples. When using open-source offerings you can also tune the machines you are using for inference to maximize either cost or latency reduction.\n",
    "\n",
    "This simple guardrail aims to ensure the LLM only answers to a predefined set of topics, and responds to out-of-bounds queries with a canned message.\n",
    "\n",
    "### Embrace async\n",
    "\n",
    "A common design to minimize latency is to send your guardrails asynchronously along with your main LLM call. If your guardrails get triggered you send back their response, otherwise send back the LLM response.\n",
    "\n",
    "We'll use this approach, creating an `execute_chat_with_guardrails` function that will run our LLM's `get_chat_response` and the `topical_guardrail` guardrail in parallel, and return the LLM response only if the guardrail returns `allowed`.\n",
    "\n",
    "### Limitations\n",
    "\n",
    "You should always consider the limitations of guardrails when developing your design. A few of the key ones to be aware of are:\n",
    "- When using LLMs as a guardrail, be aware that they have the same vulnerabilities as your base LLM call itself. For example, a **prompt injection** attempt could be successful in evading both your guardrail and your actual LLM call.\n",
    "- As conversations get longer, LLMs are more susceptible to **jailbreaking** as your instructions become diluted by the extra text.\n",
    "- Guardrails can harm the user experience if you make them overly restrictive to compensate for the issues noted above. This manifests as **over-refusals**, where your guardrails reject innocuous user requests because there are similarities with prompt injection or jailbreaking attempts.\n",
    "\n",
    "### Mitigations\n",
    "\n",
    "If you can combine guardrails with rules-based or more traditional machine learning models for detection this can mitigate some of these risks. We've also seen customers have guardrails that only ever consider the latest message, to alleviate the risks of the model being confused by a long conversation.\n",
    "\n",
    "We would also recommend doing a gradual roll-out with active monitoring of conversations so you can pick up instances of prompt injection or jailbreaking, and either add more guardrails to cover these new types of behaviour, or include them as training examples to your existing guardrails."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e95efc89",
   "metadata": {},
   "outputs": [],
   "source": [
    "system_prompt = \"You are a helpful assistant.\"\n",
    "\n",
    "bad_request = \"I want to talk about horses\"\n",
    "good_request = \"What are the best breeds of dog for people that like cats?\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fee948e2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "\n",
    "\n",
    "async def get_chat_response(user_request):\n",
    "    print(\"Getting LLM response\")\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": system_prompt},\n",
    "        {\"role\": \"user\", \"content\": user_request},\n",
    "    ]\n",
    "    response = openai.chat.completions.create(\n",
    "        model=GPT_MODEL, messages=messages, temperature=0.5\n",
    "    )\n",
    "    print(\"Got LLM response\")\n",
    "\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "\n",
    "async def topical_guardrail(user_request):\n",
    "    print(\"Checking topical guardrail\")\n",
    "    messages = [\n",
    "        {\n",
    "            \"role\": \"system\",\n",
    "            \"content\": \"Your role is to assess whether the user question is allowed or not. The allowed topics are cats and dogs. If the topic is allowed, say 'allowed' otherwise say 'not_allowed'\",\n",
    "        },\n",
    "        {\"role\": \"user\", \"content\": user_request},\n",
    "    ]\n",
    "    response = openai.chat.completions.create(\n",
    "        model=GPT_MODEL, messages=messages, temperature=0\n",
    "    )\n",
    "\n",
    "    print(\"Got guardrail response\")\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "\n",
    "async def execute_chat_with_guardrail(user_request):\n",
    "    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))\n",
    "    chat_task = asyncio.create_task(get_chat_response(user_request))\n",
    "\n",
    "    while True:\n",
    "        done, _ = await asyncio.wait(\n",
    "            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED\n",
    "        )\n",
    "        if topical_guardrail_task in done:\n",
    "            guardrail_response = topical_guardrail_task.result()\n",
    "            if guardrail_response == \"not_allowed\":\n",
    "                chat_task.cancel()\n",
    "                print(\"Topical guardrail triggered\")\n",
    "                return \"I can only talk about cats and dogs, the best animals that ever lived.\"\n",
    "            elif chat_task in done:\n",
    "                chat_response = chat_task.result()\n",
    "                return chat_response\n",
    "        else:\n",
    "            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "eba51754",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "If you're a cat lover considering getting a dog, it's important to choose a breed that typically has a more cat-like temperament. Here are some dog breeds that are known to be more cat-friendly:\n",
      "\n",
      "1. Basenji: Known as the \"barkless dog,\" Basenjis are independent, clean, and have a cat-like grooming habit.\n",
      "\n",
      "2. Shiba Inu: Shiba Inus are often described as having a cat-like personality. They are independent, clean, and tend to be reserved with strangers.\n",
      "\n",
      "3. Greyhound: Greyhounds are quiet, low-energy dogs that enjoy lounging around, much like cats. They are also known for their gentle and calm nature.\n",
      "\n",
      "4. Bichon Frise: Bichon Frises are small, friendly dogs that are often compared to cats due to their playful and curious nature. They are also hypoallergenic, making them a good choice for those with allergies.\n",
      "\n",
      "5. Cavalier King Charles Spaniel: These dogs are affectionate, gentle, and adaptable, making them a good match for cat lovers. They are known for their desire to be close to their owners and their calm demeanor.\n",
      "\n",
      "Remember, individual dogs can have different personalities, so it's important to spend time with the specific dog you're considering to see if their temperament aligns with your preferences.\n"
     ]
    }
   ],
   "source": [
    "# Call the main function with the good request - this should go through\n",
    "response = await execute_chat_with_guardrail(good_request)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "c7d88b57",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "Topical guardrail triggered\n",
      "I can only talk about cats and dogs, the best animals that ever lived.\n"
     ]
    }
   ],
   "source": [
    "# Call the main function with the good request - this should get blocked\n",
    "response = await execute_chat_with_guardrail(bad_request)\n",
    "print(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "060b408e",
   "metadata": {},
   "source": [
    "Looks like our guardrail worked - the first question was allowed through, but the second was blocked for being off-topic. Now we'll extend this concept to moderate the response we get from the LLM as well."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07af0154",
   "metadata": {},
   "source": [
    "## 2. Output guardrails\n",
    "\n",
    "Output guardrails govern what the LLM comes back with. These can take many forms, with some of the most common being:\n",
    "- **Hallucination/fact-checking guardrails:** Using a corpus of ground truth information or a training set of hallucinated responses to  block hallucinated responses.\n",
    "- **Moderation guardrails:** Applying brand and corporate guidelines to moderate the LLM's results, and either blocking or rewriting its response if it breaches them.\n",
    "- **Syntax checks:** Structured outputs from LLMs can be returned corrupt or unable to be parsed - these guardrails detect those and either retry or fail gracefully, preventing failures in downstream applications.\n",
    "    - This is a common control to apply with function calling, ensuring that the expected schema is returned in the `arguments` when the LLM returns a `function_call`.\n",
    "    \n",
    "### Moderation guardrail\n",
    "\n",
    "Here we implement a **moderation guardrail** that uses a version of the [G-Eval](https://arxiv.org/abs/2303.16634) evaluation method to score the presence of unwanted content in the LLM's response. This method is demonstrated in more detail in of our other [notebooks](https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb).\n",
    "\n",
    "To accomplish this we will make an extensible framework for moderating content that takes in a `domain` and applies `criteria` to a piece of `content` using a set of `steps`:\n",
    "1. We set a domain name, which describes the type of content we're going to moderate.\n",
    "2. We provide criteria, which outline clearly what the content should and should not contain.\n",
    "3. Step-by-step instructions are provided for the LLM to grade the content.\n",
    "4. The LLM returns a discrete score from 1-5.\n",
    "\n",
    "### Setting guardrail thresholds\n",
    "\n",
    "Our output guardrail will assess the LLM's response and block anything scoring a 3 or higher. Setting this threshold is a common area for optimization - we recommend building an evaluation set and grading the results using a confusion matrix to set the right tolerance for your guardrail. The trade-off here is generally:\n",
    "- More false positives leads to a fractured user experience, where customers get annoyed and the assistant seems less helpful.\n",
    "- More false negatives can cause lasting harm to your business, as people get the assistant to answer inappropriate questions, or prompt inject/jailbreak it.\n",
    "\n",
    "For example, for jailbreaking you may want to have a very low threshold, as the risk to your business if your LLM is hijacked and used to produce dangerous content that ends up on social media is very high. However, for our use case we're willing to accept a few false negatives, as the worst that could happen is someone ends up with a Bichon Frise who might have been better suited to a Labrador, which though sad will probably not cause lasting damage to our business (we hope)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6c7b459f",
   "metadata": {},
   "outputs": [],
   "source": [
    "domain = \"animal breed recommendation\"\n",
    "\n",
    "animal_advice_criteria = \"\"\"\n",
    "Assess the presence of explicit recommendation of cat or dog breeds in the content.\n",
    "The content should contain only general advice about cats and dogs, not specific breeds to purchase.\"\"\"\n",
    "\n",
    "animal_advice_steps = \"\"\"\n",
    "1. Read the content and the criteria carefully.\n",
    "2. Assess how much explicit recommendation of cat or dog breeds is contained in the content.\n",
    "3. Assign an animal advice score from 1 to 5, with 1 being no explicit cat or dog breed advice, and 5 being multiple named cat or dog breeds.\n",
    "\"\"\"\n",
    "\n",
    "moderation_system_prompt = \"\"\"\n",
    "You are a moderation assistant. Your role is to detect content about {domain} in the text provided, and mark the severity of that content.\n",
    "\n",
    "## {domain}\n",
    "\n",
    "### Criteria\n",
    "\n",
    "{scoring_criteria}\n",
    "\n",
    "### Instructions\n",
    "\n",
    "{scoring_steps}\n",
    "\n",
    "### Content\n",
    "\n",
    "{content}\n",
    "\n",
    "### Evaluation (score only!)\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "43e3fd36",
   "metadata": {},
   "outputs": [],
   "source": [
    "async def moderation_guardrail(chat_response):\n",
    "    print(\"Checking moderation guardrail\")\n",
    "    mod_messages = [\n",
    "        {\"role\": \"user\", \"content\": moderation_system_prompt.format(\n",
    "            domain=domain,\n",
    "            scoring_criteria=animal_advice_criteria,\n",
    "            scoring_steps=animal_advice_steps,\n",
    "            content=chat_response\n",
    "        )},\n",
    "    ]\n",
    "    response = openai.chat.completions.create(\n",
    "        model=GPT_MODEL, messages=mod_messages, temperature=0\n",
    "    )\n",
    "    print(\"Got moderation response\")\n",
    "    return response.choices[0].message.content\n",
    "    \n",
    "    \n",
    "async def execute_all_guardrails(user_request):\n",
    "    topical_guardrail_task = asyncio.create_task(topical_guardrail(user_request))\n",
    "    chat_task = asyncio.create_task(get_chat_response(user_request))\n",
    "\n",
    "    while True:\n",
    "        done, _ = await asyncio.wait(\n",
    "            [topical_guardrail_task, chat_task], return_when=asyncio.FIRST_COMPLETED\n",
    "        )\n",
    "        if topical_guardrail_task in done:\n",
    "            guardrail_response = topical_guardrail_task.result()\n",
    "            if guardrail_response == \"not_allowed\":\n",
    "                chat_task.cancel()\n",
    "                print(\"Topical guardrail triggered\")\n",
    "                return \"I can only talk about cats and dogs, the best animals that ever lived.\"\n",
    "            elif chat_task in done:\n",
    "                chat_response = chat_task.result()\n",
    "                moderation_response = await moderation_guardrail(chat_response)\n",
    "\n",
    "                if int(moderation_response) >= 3:\n",
    "                    print(f\"Moderation guardrail flagged with a score of {int(moderation_response)}\")\n",
    "                    return \"Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.\"\n",
    "\n",
    "                else:\n",
    "                    print('Passed moderation')\n",
    "                    return chat_response\n",
    "        else:\n",
    "            await asyncio.sleep(0.1)  # sleep for a bit before checking the tasks again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "beea1305",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Adding a request that should pass both our topical guardrail and our moderation guardrail\n",
    "great_request = 'What is some advice you can give to a new dog owner?'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1c582b4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "Checking moderation guardrail\n",
      "Got moderation response\n",
      "Moderation guardrail flagged with a score of 5\n",
      "Sorry, we're not permitted to give animal breed advice. I can help you with any general queries you might have.\n",
      "\n",
      "\n",
      "\n",
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "Topical guardrail triggered\n",
      "I can only talk about cats and dogs, the best animals that ever lived.\n",
      "\n",
      "\n",
      "\n",
      "Checking topical guardrail\n",
      "Got guardrail response\n",
      "Getting LLM response\n",
      "Got LLM response\n",
      "Checking moderation guardrail\n",
      "Got moderation response\n",
      "Passed moderation\n",
      "As a new dog owner, here are some helpful tips:\n",
      "\n",
      "1. Choose the right breed: Research different dog breeds to find one that suits your lifestyle, activity level, and living situation. Some breeds require more exercise and attention than others.\n",
      "\n",
      "2. Puppy-proof your home: Make sure your home is safe for your new furry friend. Remove any toxic plants, secure loose wires, and store household chemicals out of reach.\n",
      "\n",
      "3. Establish a routine: Dogs thrive on routine, so establish a consistent schedule for feeding, exercise, and bathroom breaks. This will help your dog feel secure and reduce any anxiety.\n",
      "\n",
      "4. Socialize your dog: Expose your dog to different people, animals, and environments from an early age. This will help them become well-adjusted and comfortable in various situations.\n",
      "\n",
      "5. Train your dog: Basic obedience training is essential for your dog's safety and your peace of mind. Teach commands like sit, stay, and come, and use positive reinforcement techniques such as treats and praise.\n",
      "\n",
      "6. Provide mental and physical stimulation: Dogs need both mental and physical exercise to stay happy and healthy. Engage in activities like walks, playtime, puzzle toys, and training sessions to keep your dog mentally stimulated.\n",
      "\n",
      "7. Proper nutrition: Feed your dog a balanced and appropriate diet based on their age, size, and specific needs. Consult with a veterinarian to determine the best food options for your dog.\n",
      "\n",
      "8. Regular veterinary care: Schedule regular check-ups with a veterinarian to ensure your dog's health and well-being. Vaccinations, parasite prevention, and dental care are important aspects of their overall care.\n",
      "\n",
      "9. Be patient and consistent: Dogs require time, patience, and consistency to learn and adapt to their new environment. Stay positive, be patient with their training, and provide clear and consistent boundaries.\n",
      "\n",
      "10. Show love and affection: Dogs are social animals that thrive on love and affection. Spend quality time with your dog, offer praise and cuddles, and make them feel like an important part of your family.\n",
      "\n",
      "Remember, being a responsible dog owner involves commitment, time, and effort. With proper care and attention, you can build a strong bond with your new furry companion.\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tests = [good_request,bad_request,great_request]\n",
    "\n",
    "for test in tests:\n",
    "    result = await execute_all_guardrails(test)\n",
    "    print(result)\n",
    "    print('\\n\\n')\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4763dd2d",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Guardrails are a vibrant and evolving topic in LLMs, and we hope this notebook has given you an effective introduction to the core concepts around guardrails. To recap:\n",
    "- Guardrails are detective controls that aim to prevent harmful content getting to your applications and your users, and add steerability to your LLM in production.\n",
    "- They can take the form of input guardrails, which target content before it gets to the LLM, and output guardrails, which control the LLM's response.\n",
    "- Designing guardrails and setting their thresholds is a trade-off between accuracy, latency, and cost. Your decision should be based on clear evaluations of the performance of your guardrails, and an understanding of what the cost of a false negative and false positive are for your business.\n",
    "- By embracing asynchronous design principles, you can scale guardrails horizontally to minimize the impact to the user as your guardrails increase in number and scope.\n",
    "\n",
    "We look forward to seeing how you take this forward, and how thinking on guardrails evolves as the ecosystem matures. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "openai_test",
   "language": "python",
   "name": "openai_test"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}