langchain/docs/docs/guides/productionization/safety/moderation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4927a727-b4c8-453c-8c83-bd87b4fcac14",
   "metadata": {},
   "source": [
    "# Moderation chain\n",
    "\n",
    "This notebook walks through examples of how to use a moderation chain, and several common ways for doing so. \n",
    "Moderation chains are useful for detecting text that could be hateful, violent, etc. This can be useful to apply on both user input, but also on the output of a Language Model. \n",
    "Some API providers specifically prohibit you, or your end users, from generating some \n",
    "types of harmful content. To comply with this (and to just generally prevent your application from being harmful) \n",
    "you may want to add a moderation chain to your sequences in order to make sure any output \n",
    "the LLM generates is not harmful.\n",
    "\n",
    "If the content passed into the moderation chain is harmful, there is not one best way to handle it.\n",
    "It probably depends on your application. Sometimes you may want to throw an error \n",
    "(and have your application handle that). Other times, you may want to return something to \n",
    "the user explaining that the text was harmful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6acf3505",
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install --upgrade --quiet  langchain langchain-openai"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4f5f6449-940a-4f5c-97c0-39b71c3e2a68",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import OpenAIModerationChain\n",
    "from langchain_core.prompts import ChatPromptTemplate\n",
    "from langchain_openai import OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "fcb8312b-7e7a-424f-a3ec-76738c9a9d21",
   "metadata": {},
   "outputs": [],
   "source": [
    "moderate = OpenAIModerationChain()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b24b9148-f6b0-4091-8ea8-d3fb281bd950",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = OpenAI()\n",
    "prompt = ChatPromptTemplate.from_messages([(\"system\", \"repeat after me: {input}\")])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1c8ed87c-9ca6-4559-bf60-d40e94a0af08",
   "metadata": {},
   "outputs": [],
   "source": [
    "chain = prompt | model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "5256b9bd-381a-42b0-bfa8-7e6d18f853cb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n\\nYou are stupid.'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chain.invoke({\"input\": \"you are stupid\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "fe6e3b33-dc9a-49d5-b194-ba750c58a628",
   "metadata": {},
   "outputs": [],
   "source": [
    "moderated_chain = chain | moderate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "d8ba0cbd-c739-4d23-be9f-6ae092bd5ffb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input': '\\n\\nYou are stupid',\n",
       " 'output': \"Text was found that violates OpenAI's content policy.\"}"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "moderated_chain.invoke({\"input\": \"you are stupid\"})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}