langchain/docs/extras/expression_language/how_to/fallbacks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "19c9cbd6",
   "metadata": {},
   "source": [
    "# Add fallbacks\n",
    "\n",
    "There are many possible points of failure in an LLM application, whether that be issues with LLM API's, poor model outputs, issues with other integrations, etc. Fallbacks help you gracefully handle and isolate these issues.\n",
    "\n",
    "Crucially, fallbacks can be applied not only on the LLM level but on the whole runnable level."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6bb9ba9",
   "metadata": {},
   "source": [
    "## Handling LLM API Errors\n",
    "\n",
    "This is maybe the most common use case for fallbacks. A request to an LLM API can fail for a variety of reasons - the API could be down, you could have hit rate limits, any number of things. Therefore, using fallbacks can help protect against these types of things.\n",
    "\n",
    "IMPORTANT: By default, a lot of the LLM wrappers catch errors and retry. You will most likely want to turn those off when working with fallbacks. Otherwise the first wrapper will keep on retrying and not failing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d3e893bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI, ChatAnthropic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4847c82d",
   "metadata": {},
   "source": [
    "First, let's mock out what happens if we hit a RateLimitError from OpenAI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "dfdd8bf5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from unittest.mock import patch\n",
    "from openai.error import RateLimitError"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "e6fdffc1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Note that we set max_retries = 0 to avoid retrying on RateLimits, etc\n",
    "openai_llm = ChatOpenAI(max_retries=0)\n",
    "anthropic_llm = ChatAnthropic()\n",
    "llm = openai_llm.with_fallbacks([anthropic_llm])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "584461ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hit error\n"
     ]
    }
   ],
   "source": [
    "# Let's use just the OpenAI LLm first, to show that we run into an error\n",
    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
    "    try:\n",
    "         print(openai_llm.invoke(\"Why did the chicken cross the road?\"))\n",
    "    except:\n",
    "        print(\"Hit error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "4fc1e673",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "content=' I don\\'t actually know why the chicken crossed the road, but here are some possible humorous answers:\\n\\n- To get to the other side!\\n\\n- It was too chicken to just stand there. \\n\\n- It wanted a change of scenery.\\n\\n- It wanted to show the possum it could be done.\\n\\n- It was on its way to a poultry farmers\\' convention.\\n\\nThe joke plays on the double meaning of \"the other side\" - literally crossing the road to the other side, or the \"other side\" meaning the afterlife. So it\\'s an anti-joke, with a silly or unexpected pun as the answer.' additional_kwargs={} example=False\n"
     ]
    }
   ],
   "source": [
    "# Now let's try with fallbacks to Anthropic\n",
    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
    "    try:\n",
    "         print(llm.invoke(\"Why did the the chicken cross the road?\"))\n",
    "    except:\n",
    "        print(\"Hit error\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f00bea25",
   "metadata": {},
   "source": [
    "We can use our \"LLM with Fallbacks\" as we would a normal LLM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "4f8eaaa0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "content=\" I don't actually know why the kangaroo crossed the road, but I'm happy to take a guess! Maybe the kangaroo was trying to get to the other side to find some tasty grass to eat. Or maybe it was trying to get away from a predator or other danger. Kangaroos do need to cross roads and other open areas sometimes as part of their normal activities. Whatever the reason, I'm sure the kangaroo looked both ways before hopping across!\" additional_kwargs={} example=False\n"
     ]
    }
   ],
   "source": [
    "from langchain.prompts import ChatPromptTemplate\n",
    "\n",
    "prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", \"You're a nice assistant who always includes a compliment in your response\"),\n",
    "        (\"human\", \"Why did the {animal} cross the road\"),\n",
    "    ]\n",
    ")\n",
    "chain = prompt | llm\n",
    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
    "    try:\n",
    "         print(chain.invoke({\"animal\": \"kangaroo\"}))\n",
    "    except:\n",
    "        print(\"Hit error\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef9f0f39-0b9f-4723-a394-f61c98c75d41",
   "metadata": {},
   "source": [
    "### Specifying errors to handle\n",
    "\n",
    "We can also specify the errors to handle if we want to be more specific about when the fallback is invoked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "e4069ca4-1c16-4915-9a8c-b2732869ae27",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hit error\n"
     ]
    }
   ],
   "source": [
    "llm = openai_llm.with_fallbacks([anthropic_llm], exceptions_to_handle=(KeyboardInterrupt,))\n",
    "\n",
    "chain = prompt | llm\n",
    "with patch('openai.ChatCompletion.create', side_effect=RateLimitError()):\n",
    "    try:\n",
    "         print(chain.invoke({\"animal\": \"kangaroo\"}))\n",
    "    except:\n",
    "        print(\"Hit error\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d62241b",
   "metadata": {},
   "source": [
    "## Fallbacks for Sequences\n",
    "\n",
    "We can also create fallbacks for sequences, that are sequences themselves. Here we do that with two different models: ChatOpenAI and then normal OpenAI (which does not use a chat model). Because OpenAI is NOT a chat model, you likely want a different prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "6d0b8056",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First let's create a chain with a ChatModel\n",
    "# We add in a string output parser here so the outputs between the two are the same type\n",
    "from langchain.schema.output_parser import StrOutputParser\n",
    "\n",
    "chat_prompt = ChatPromptTemplate.from_messages(\n",
    "    [\n",
    "        (\"system\", \"You're a nice assistant who always includes a compliment in your response\"),\n",
    "        (\"human\", \"Why did the {animal} cross the road\"),\n",
    "    ]\n",
    ")\n",
    "# Here we're going to use a bad model name to easily create a chain that will error\n",
    "chat_model = ChatOpenAI(model_name=\"gpt-fake\")\n",
    "bad_chain = chat_prompt | chat_model | StrOutputParser()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "8d1fc2a5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now lets create a chain with the normal OpenAI model\n",
    "from langchain.llms import OpenAI\n",
    "from langchain.prompts import PromptTemplate\n",
    "\n",
    "prompt_template = \"\"\"Instructions: You should always include a compliment in your response.\n",
    "\n",
    "Question: Why did the {animal} cross the road?\"\"\"\n",
    "prompt = PromptTemplate.from_template(prompt_template)\n",
    "llm = OpenAI()\n",
    "good_chain = prompt | llm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "283bfa44",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n\\nAnswer: The turtle crossed the road to get to the other side, and I have to say he had some impressive determination.'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can now create a final chain which combines the two\n",
    "chain = bad_chain.with_fallbacks([good_chain])\n",
    "chain.invoke({\"animal\": \"turtle\"})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}