Prompt-Engineering-Guide/notebooks/pe-litellm-intro.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "MZ01up0p7wOJ"
      },
      "source": [
        "## 🚅 liteLLM Demo\n",
        "### TLDR: Call 50+ LLM APIs using chatGPT Input/Output format\n",
        "https://github.com/BerriAI/litellm\n",
        "\n",
        "liteLLM is package to simplify calling **OpenAI, Azure, Llama2, Cohere, Anthropic, Huggingface API Endpoints**. LiteLLM manages\n",
        "\n",
        "* Translating inputs to the provider's `completion()` and `embedding()` endpoints\n",
        "* Guarantees consistent output, text responses will always be available at `['choices'][0]['message']['content']`\n",
        "* Exception mapping - common exceptions across providers are mapped to the OpenAI exception types\n",
        "\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "RZtzCnQS7rW-"
      },
      "source": [
        "## Installation and setting Params"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rsrN5W-N7L8d"
      },
      "outputs": [],
      "source": [
        "!pip install litellm"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "ArrWyG5b7QAG"
      },
      "outputs": [],
      "source": [
        "from litellm import completion\n",
        "import os"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "bbhJRt34_NJ1"
      },
      "source": [
        "## Set your API keys\n",
        "- liteLLM reads your .env, env variables or key manager for Auth\n",
        "\n",
        "Set keys for the models you want to use below"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "-h8Ga5cR7SvV"
      },
      "outputs": [],
      "source": [
        "# Only set keys for the LLMs you want to use\n",
        "os.environ['OPENAI_API_KEY'] = \"\" #@param\n",
        "os.environ[\"ANTHROPIC_API_KEY\"] = \"\" #@param\n",
        "os.environ[\"AZURE_API_BASE\"] = \"\" #@param\n",
        "os.environ[\"AZURE_API_VERSION\"] = \"\" #@param\n",
        "os.environ[\"AZURE_API_KEY\"] = \"\" #@param\n",
        "os.environ[\"REPLICATE_API_TOKEN\"] = \"\" #@param\n",
        "os.environ[\"COHERE_API_KEY\"] = \"\" #@param\n",
        "os.environ[\"HF_TOKEN\"] = \"\" #@param"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "MBujGiby8YBu"
      },
      "outputs": [],
      "source": [
        "messages = [{ \"content\": \"what's the weather in SF\",\"role\": \"user\"}]"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "fhqpKv6L8fBj"
      },
      "source": [
        "## Call chatGPT"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "speIkoX_8db4",
        "outputId": "331a6c65-f121-4e65-e121-bf8aaad05d9d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<OpenAIObject chat.completion id=chatcmpl-7vYWJYYUeFuhjCiOjI9JXK6gNmWk3 at 0x1067d42c0> JSON: {\n",
              "  \"id\": \"chatcmpl-7vYWJYYUeFuhjCiOjI9JXK6gNmWk3\",\n",
              "  \"object\": \"chat.completion\",\n",
              "  \"created\": 1693951747,\n",
              "  \"model\": \"gpt-3.5-turbo-0613\",\n",
              "  \"choices\": [\n",
              "    {\n",
              "      \"index\": 0,\n",
              "      \"message\": {\n",
              "        \"role\": \"assistant\",\n",
              "        \"content\": \"I apologize, but as an AI language model, I do not have real-time data. However, you can easily find the current weather conditions in San Francisco, California by checking a trusted weather website or using a weather app on your smartphone.\"\n",
              "      },\n",
              "      \"finish_reason\": \"stop\"\n",
              "    }\n",
              "  ],\n",
              "  \"usage\": {\n",
              "    \"prompt_tokens\": 13,\n",
              "    \"completion_tokens\": 48,\n",
              "    \"total_tokens\": 61\n",
              "  }\n",
              "}"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "completion(model=\"gpt-3.5-turbo\", messages=messages)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Q3jV1Uxv8zNo"
      },
      "source": [
        "## Call Claude-2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "V8yTWYzY8m9S",
        "outputId": "8b6dd32d-f9bf-4e89-886d-47cb8020f025"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'choices': [{'finish_reason': 'stop',\n",
              "   'index': 0,\n",
              "   'message': {'role': 'assistant',\n",
              "    'content': \" Unfortunately I do not have enough context to provide the current weather in San Francisco. To get the most accurate weather report, it's helpful if I know details like:\\n\\n- Exact location (city name, zip code, etc)\\n- Time frame (current conditions, forecast for a certain day/week, etc)\\n\\nIf you can provide some more specifics about what weather information you need for San Francisco, I'd be happy to look that up for you!\"}}],\n",
              " 'created': 1691880836.974166,\n",
              " 'model': 'claude-2',\n",
              " 'usage': {'prompt_tokens': 18, 'completion_tokens': 95, 'total_tokens': 113}}"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "completion(model=\"claude-2\", messages=messages)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "yu0LPDmW9PJa"
      },
      "source": [
        "## Call llama2 on replicate"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0GWV5mtO9Jbu",
        "outputId": "38538825-b271-406d-a437-f5cf0eb7e548"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'choices': [{'finish_reason': 'stop',\n",
              "   'index': 0,\n",
              "   'message': {'role': 'assistant',\n",
              "    'content': ' I\\'m happy to help! However, I must point out that the question \"what\\'s the weather in SF\" doesn\\'t make sense as \"SF\" could refer to multiple locations (San Francisco, South Florida, San Fernando, etc.). Could you please provide more context or specify which location you\\'re referring to? That way, I can give you an accurate answer.'}}],\n",
              " 'created': 1691880930.9003325,\n",
              " 'model': 'replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1',\n",
              " 'usage': {'prompt_tokens': 6, 'completion_tokens': 74, 'total_tokens': 80}}"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model = \"replicate/llama-2-70b-chat:2c1608e18606fad2812020dc541930f2d0495ce32eee50074220b87300bc16e1\"\n",
        "completion(model=model, messages=messages)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "HXdj5SEe9iLK"
      },
      "source": [
        "## Call Command-Nightly"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EaUq2xIx9fhr",
        "outputId": "55fe6f52-b58b-4729-948a-74dac4b431b2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<ModelResponse at 0x11cb0c3b0> JSON: {\n",
              "  \"choices\": [\n",
              "    {\n",
              "      \"finish_reason\": \"stop\",\n",
              "      \"index\": 0,\n",
              "      \"message\": {\n",
              "        \"content\": \" The weather in San Francisco can be quite unpredictable and varies throughout the year. In general, the city\",\n",
              "        \"role\": \"assistant\",\n",
              "        \"logprobs\": null\n",
              "      }\n",
              "    }\n",
              "  ],\n",
              "  \"created\": 1693951797.3149078,\n",
              "  \"model\": \"command-nightly\",\n",
              "  \"usage\": {\n",
              "    \"prompt_tokens\": 6,\n",
              "    \"completion_tokens\": 20,\n",
              "    \"total_tokens\": 26\n",
              "  }\n",
              "}"
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "completion(model=\"command-nightly\", messages=messages)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "1g9hSgsL9soJ"
      },
      "source": [
        "## Call Azure OpenAI"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "For azure openai calls ensure to add the `azure/` prefix to `model`. If your deployment-id is `chatgpt-test` set `model` = `azure/chatgpt-test`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AvLjR-PF-lt0",
        "outputId": "deff2db3-b003-48cd-ea62-c03a68a4464a"
      },
      "outputs": [],
      "source": [
        "completion(model=\"azure/chatgpt-test\", messages=messages)"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.6"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}