openai-cookbook/examples/o1/Using_reasoning_for_routine_generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using reasoning for routine generation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When developing customer service solutions, one of the initial steps involves transforming knowledge base articles into a set of routines that an LLM can comprehend and follow. A routine, in this context, refers to a set of step-by-step instructions designed specifically for the LLM to execute efficiently. Each routine is carefully structured so that a step corresponds to a clear action. Actions can include responding to a user, triggering a function call, or retrieving additional relevant knowledge.\n",
    "\n",
    "Most internal knowledge base articles are complex and structured for human interpretation. They often include intricate diagrams, multi-step processes, and decision trees that pose challenges for LLM-based solutions to reason through in a meaningful way. By breaking down these documents into routines, each instruction can be simplified and formatted in a way that guides the LLM through a series of small, manageable tasks. This granular approach reduces ambiguity, allowing the LLM to process the information methodically and reducing the risk of hallucination or deviation from the expected path.\n",
    "\n",
    "Converting these knowledge base articles into routines can be time-consuming and challenging, especially for companies attempting to build an automated pipeline. Each routine must account for various user scenarios, where actions need to be clearly defined. For instance, when a function call is necessary, the routine must specify the exact information to retrieve or the action to execute—whether it’s triggering an API, retrieving external data, or pulling in additional context. While automating this process with traditional GPT-class models can significantly reduce the manual effort involved, it often introduces new challenges. Some challenges include designing robust instructions that are specific enough for the LLM to follow consistently, capturing unique edge cases that may arise during customer interactions, providing high-quality few-shot examples to guide the model’s behavior, and in some cases, fine-tuning the model to achieve more reliable or specialized outcomes.\n",
    "\n",
    "o1 has demonstrated the capability to efficiently deconstruct these articles and convert them into sets of routines zero-shot, meaning that the LLM can understand and follow the instructions without extensive examples or prior training on similar tasks. This minimizes the prompting effort required, as the routine structure itself provides the necessary guidance for the LLM to complete each step. By breaking down tasks into specific actions and integrating function calls where needed, o1’s approach ensures that even complex workflows can be handled seamlessly by the LLM, leading to more effective and scalable customer service solutions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Selecting Knowledge Base Articles\n",
    "\n",
    "In this example, we will use a set of publicly available Help Center articles from the OpenAI website and convert them into internal routines that an LLM can execute. Besides transforming the policies into routines, we will also have the model generate functions that allow the LLM to perform actions on behalf of the user. This is necessary to allow the LLM to execute the same actions that human agents have, and access additional information that may not be immediately available just from the policy documentation.\n",
    "\n",
    "We will begin by using the following Help Center articles for conversion into routines:\n",
    "- [How do I delete my payment method](https://help.openai.com/en/articles/8156070-how-do-i-delete-my-payment-method)\n",
    "- [How can I get a Business Associate Agreement (BAA) with OpenAI?](https://help.openai.com/en/articles/8660679-how-can-i-get-a-business-associate-agreement-baa-with-openai)\n",
    "- [How can I set up prepaid billing?](https://help.openai.com/en/articles/8264644-how-can-i-set-up-prepaid-billing)\n",
    "- [How do I submit a VAT exemption request](https://help.openai.com/en/articles/7232908-how-do-i-submit-a-vat-exemption-request)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openai import OpenAI\n",
    "from IPython.display import display, HTML\n",
    "import pandas as pd\n",
    "from concurrent.futures import ThreadPoolExecutor\n",
    "import csv\n",
    "\n",
    "client = OpenAI()\n",
    "MODEL = 'o1-preview'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have our articles stored in an accessible csv. We will take the articles and pass them to o1-preview in parallel and generate the initial routines.\n",
    "\n",
    "Our instructions for converting to policy to a routine include:\n",
    "- Converting the policy from an external facing document to an internal SOP routine\n",
    "- Breaking down the policy in specific actions and sub-actions\n",
    "- Outlining specific conditions for moving between steps\n",
    "- Determing where external knowledge/actions may be required, and defining functions that we could use to get that information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "articles = []\n",
    "\n",
    "with open('../data/helpcenter_articles.csv', mode='r', encoding='utf-8') as file:\n",
    "    reader = csv.DictReader(file)\n",
    "    for row in reader:\n",
    "        articles.append({\n",
    "            \"policy\": row[\"policy\"],\n",
    "            \"content\": row[\"content\"]\n",
    "        })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "CONVERSION_PROMPT = \"\"\"\n",
    "You are a helpful assistant tasked with taking an external facing help center article and converting it into a internal-facing programmatically executable routine optimized for an LLM. \n",
    "The LLM using this routine will be tasked with reading the policy, answering incoming questions from customers, and helping drive the case toward resolution.\n",
    "\n",
    "Please follow these instructions:\n",
    "1. **Review the customer service policy carefully** to ensure every step is accounted for. It is crucial not to skip any steps or policies.\n",
    "2. **Organize the instructions into a logical, step-by-step order**, using the specified format.\n",
    "3. **Use the following format**:\n",
    "   - **Main actions are numbered** (e.g., 1, 2, 3).\n",
    "   - **Sub-actions are lettered** under their relevant main actions (e.g., 1a, 1b).\n",
    "      **Sub-actions should start on new lines**\n",
    "   - **Specify conditions using clear 'if...then...else' statements** (e.g., 'If the product was purchased within 30 days, then...').\n",
    "   - **For instructions that require more information from the customer**, provide polite and professional prompts to ask for additional information.\n",
    "   - **For actions that require data from external systems**, write a step to call a function using backticks for the function name (e.g., `call the check_delivery_date function`).\n",
    "      - **If a step requires the customer service agent to take an action** (e.g., process a refund), generate a function call for this action (e.g., `call the process_refund function`).\n",
    "      - **Define any new functions** by providing a brief description of their purpose and required parameters.\n",
    "   - **If there is an action an assistant can performon behalf of the user**, include a function call for this action (e.g., `call the change_email_address function`), and ensure the function is defined with its purpose and required parameters.\n",
    "      - This action may not be explicitly defined in the help center article, but can be done to help the user resolve their inquiry faster\n",
    "   - **The step prior to case resolution should always be to ask if there is anything more you can assist with**.\n",
    "   - **End with a final action for case resolution**: calling the `case_resolution` function should always be the final step.\n",
    "4. **Ensure compliance** by making sure all steps adhere to company policies, privacy regulations, and legal requirements.\n",
    "5. **Handle exceptions or escalations** by specifying steps for scenarios that fall outside the standard policy.\n",
    "\n",
    "**Important**: If at any point you are uncertain, respond with \"I don't know.\"\n",
    "\n",
    "Please convert the customer service policy into the formatted routine, ensuring it is easy to follow and execute programmatically.\n",
    "\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_routine(policy):\n",
    "    try:\n",
    "        messages = [\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": f\"\"\"\n",
    "                    {CONVERSION_PROMPT}\n",
    "\n",
    "                    POLICY:\n",
    "                    {policy}\n",
    "                \"\"\"\n",
    "            }\n",
    "        ]\n",
    "\n",
    "        response = client.chat.completions.create(\n",
    "            model=MODEL,\n",
    "            messages=messages\n",
    "        )\n",
    "        \n",
    "\n",
    "        return response.choices[0].message.content \n",
    "    except Exception as e:\n",
    "        print(f\"An error occurred: {e}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_article(article):\n",
    "    routine = generate_routine(article['content'])\n",
    "    return {\"policy\": article['policy'], \"content\": article['content'], \"routine\": routine}\n",
    "\n",
    "\n",
    "with ThreadPoolExecutor() as executor:\n",
    "    results = list(executor.map(process_article, articles))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll store the results of our routines in a dataframe and print them out so we can get an initial look."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th></th>\n",
       "      <th>policy</th>\n",
       "      <th>content</th>\n",
       "      <th>routine</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Delete Payment Method</td>\n",
       "      <td>How do I delete my payment method?<br>Updated over a week ago<br>We keep your payment method on file to cover any outstanding charges on your account. To stop charges to your payment method, please follow the steps below.<br><br>## ChatGPT <br>You can cancel your ChatGPT Plus subscription to stop further charges at any time:  <br>Click on 'My Plan' in the ChatGPT sidebar. <br>Click on 'Manage my subscription' in the pop-up window.<br>Select 'Cancel Plan'. <br>Please note that your cancellation will take effect the day after the next billing date, and you can continue using our services until then. To avoid being charged for your next billing period, please cancel your subscription at least 24 hours before your next billing date. <br><br>## API<br>We'll need to keep a payment method on file to account for any outstanding usage costs. You're welcome to cancel your pay-as-you-go service, by clicking 'Cancel paid account' in your billing overview. After the current month's invoice has been issued, the current card will no longer be charged. <br>If you'd like to continue using the service, add a new payment method in the billing overview page and select 'Set as default'. You'll then be able to delete the old payment method.<br></td>\n",
       "      <td>1. Verify the customer's account.<br>   a. Politely ask the customer for their email address or account ID to locate their account.<br>   b. `call the verify_customer_account(email_or_account_id)`.<br><br>2. Verify the customer's identity.<br>   a. Politely ask the customer to provide security information to confirm their identity (e.g., the last four digits of the payment method on file).<br>   b. `call the verify_customer_identity(account_id, security_information)`.<br>   c. If the customer's identity cannot be verified, then:<br>      - Inform the customer that we are unable to proceed without identity verification for security reasons.<br>      - Provide guidance on how they can verify their identity.<br>      - Proceed to step 6.<br><br>3. Determine the customer's account type.<br>   a. `call the check_account_type(account_id)`.<br><br>4. If the customer is a ChatGPT Plus subscriber, then:<br>   a. Ask the customer if they would like assistance with canceling their ChatGPT Plus subscription.<br>   b. If the customer agrees, then:<br>      - `call the cancel_subscription(account_id)`.<br>      - Inform the customer that their subscription has been canceled and the cancellation will take effect the day after the next billing date.<br>      - Remind the customer that they can continue using our services until then.<br>   c. Else:<br>      - Provide the following steps for the customer to cancel their subscription:<br>         - Click on **'My Plan'** in the ChatGPT sidebar.<br>         - Click on **'Manage my subscription'** in the pop-up window.<br>         - Select **'Cancel Plan'**.<br>      - Inform the customer about the cancellation effective date and continued access until then.<br>      - Advise the customer to cancel at least 24 hours before their next billing date to avoid being charged for the next billing period.<br><br>5. Else if the customer is an API user, then:<br>   a. Inform the customer that we need to keep a payment method on file to account for any outstanding usage costs.<br>   b. Ask the customer if they would like assistance with canceling their pay-as-you-go service.<br>   c. If the customer agrees, then:<br>      - `call the cancel_paid_account(account_id)`.<br>      - Inform the customer that after the current month's invoice has been issued, the current card will no longer be charged.<br>   d. Else:<br>      - Provide the following steps for the customer to cancel their pay-as-you-go service:<br>         - Go to the **billing overview** page.<br>         - Click on **'Cancel paid account'**.<br>      - Inform the customer that after the current month's invoice has been issued, the current card will no longer be charged.<br>   e. If the customer wants to continue using the service but change the payment method:<br>      - Ask the customer if they would like assistance with adding a new payment method and setting it as default.<br>      - If the customer agrees:<br>         - Politely request the new payment method details.<br>         - `call the add_payment_method(account_id, payment_details)`.<br>         - `call the set_default_payment_method(account_id, new_payment_method_id)`.<br>         - `call the delete_payment_method(account_id, old_payment_method_id)`.<br>         - Inform the customer that the old payment method has been deleted and the new one is set as default.<br>      - Else:<br>         - Instruct the customer to add a new payment method in the billing overview page.<br>         - Ask them to select **'Set as default'** for the new payment method.<br>         - Inform them that they can then delete the old payment method.<br><br>6. Ask the customer if there is anything else you can assist them with.<br><br>7. `call the case_resolution()`.<br><br>---<br><br>**Function Definitions:**<br><br>- `verify_customer_account(email_or_account_id)`: Verifies the customer's account using their email address or account ID.  <br>  **Parameters:** `email_or_account_id`<br><br>- `verify_customer_identity(account_id, security_information)`: Confirms the customer's identity using security information.  <br>  **Parameters:** `account_id`, `security_information`<br><br>- `check_account_type(account_id)`: Retrieves the customer's account type (ChatGPT Plus subscriber or API user).  <br>  **Parameters:** `account_id`<br><br>- `cancel_subscription(account_id)`: Cancels the ChatGPT Plus subscription for the customer.  <br>  **Parameters:** `account_id`<br><br>- `cancel_paid_account(account_id)`: Cancels the API pay-as-you-go service for the customer.  <br>  **Parameters:** `account_id`<br><br>- `add_payment_method(account_id, payment_details)`: Adds a new payment method to the customer's account.  <br>  **Parameters:** `account_id`, `payment_details`<br><br>- `set_default_payment_method(account_id, payment_method_id)`: Sets a payment method as the default for the customer.  <br>  **Parameters:** `account_id`, `payment_method_id`<br><br>- `delete_payment_method(account_id, payment_method_id)`: Deletes a payment method from the customer's account.  <br>  **Parameters:** `account_id`, `payment_method_id`<br><br>- `case_resolution()`: Resolves the case and marks it as completed.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Business Associate Agreement</td>\n",
       "      <td>How can I get a Business Associate Agreement (BAA) with OpenAI?<br>Information about HIPAA compliance for healthcare companies<br><br>The Health Insurance Portability and Accountability Act (HIPAA) is a U.S. federal law that requires privacy and security protections for protected health information (PHI). Our API platform can be a great fit for any covered entity or business associate looking to process protected health information, and we’d be happy to assist you in fulfilling your HIPAA compliance. To use our API platform, you’ll first need a BAA with OpenAI.<br><br><br>How do I get started?<br>If you require a BAA before you can use our API, email us at baa@openai.com with details about your company and use case.<br><br>Our team will respond within 1-2 business days. We review each BAA request on a case-by-case basis and may need additional information. The process is usually completed within a few business days.<br><br><br>Can I get a BAA for ChatGPT?<br>If you're interested in exploring a BAA for ChatGPT Enterprise, please contact sales.<br> <br><br>What happens if I’m not approved?<br>We are able to approve most customers that request BAAs, but occasionally a use case doesn’t pass our team's evaluation. In that case, we’ll give feedback and context as to why that is and give you the opportunity to update your intended use of our API and re-apply.<br><br> <br>Are all API services covered by the BAA?<br>No, only endpoints that are eligible for zero retention are covered by the BAA. You can see a list of those endpoints. <br><br> <br>Is an enterprise agreement requirement to sign a BAA?<br>No, an enterprise agreement is not required to sign a BAA.<br></td>\n",
       "      <td>1. Thank the customer and ask for clarification:<br>   a. \"Thank you for reaching out! Could you please specify whether you require a Business Associate Agreement (BAA) for using our API or for ChatGPT Enterprise?\"<br><br>2. If the customer requires a BAA for the API, then:<br>   a. Inform the customer: \"To obtain a BAA for our API, please email baa@openai.com with details about your company and use case.\"<br>   b. Inform the customer: \"Our team will respond within 1-2 business days.\"<br>   c. Inform the customer: \"We review each BAA request on a case-by-case basis and may need additional information.\"<br>   d. Inform the customer: \"The process is usually completed within a few business days.\"<br>   e. Inform the customer: \"Please note that only endpoints eligible for zero data retention are covered by the BAA.\"<br>      i. Call the `provide_list_of_zero_retention_endpoints` function.<br>   f. Inform the customer: \"An enterprise agreement is not required to sign a BAA.\"<br><br>3. If the customer requires a BAA for ChatGPT Enterprise, then:<br>   a. Inform the customer: \"To explore a BAA for ChatGPT Enterprise, please contact our sales team.\"<br>      i. Call the `provide_sales_contact_information` function.<br><br>4. If the customer is not approved, then:<br>   a. Inform the customer: \"We are able to approve most customers that request BAAs, but occasionally a use case doesn't pass our team's evaluation.\"<br>   b. Inform the customer: \"In that case, we'll provide feedback and context as to why and give you the opportunity to update your intended use of our API and re-apply.\"<br><br>5. Ask the customer if there is anything else you can assist with:<br>   a. \"Is there anything else I can assist you with today?\"<br><br>6. Call the `case_resolution` function.<br><br>---<br><br>**Function Definitions:**<br><br>- `provide_list_of_zero_retention_endpoints`:<br>  - **Purpose**: Provides the customer with a list of API endpoints that are eligible for zero data retention under the BAA.<br>  - **Parameters**: None.<br><br>- `provide_sales_contact_information`:<br>  - **Purpose**: Provides the customer with contact information to reach our sales team for ChatGPT Enterprise inquiries.<br>  - **Parameters**: None.<br><br>- `case_resolution`:<br>  - **Purpose**: Finalizes the case and marks it as resolved.<br>  - **Parameters**: None.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Set up prepaid billing</td>\n",
       "      <td>How can I set up prepaid billing?<br><br>How it works<br>Prepaid billing allows API users to pre-purchase usage. The credits you've bought will be applied to your monthly invoice. This means that any API usage you incur will first be deducted from the prepaid credits. If your usage exceeds the credits you've purchased, you'll then be billed for the additional amount.<br>Prepaid billing helps developers know what they are committing to upfront which can provide more predictability for budgeting and spend management. <br><br><br>Setting up prepaid billing<br>If you're on a Monthly Billing plan, you may also choose to switch to prepaid billing and purchase credits upfront for API usage. <br>- Go to your billing overview in your account settings<br>- Click \"Start payment plan\" (you may see variations like \"Buy credits\")<br>    Note: If you previously had an arrears billing plan, you'll need to cancel this existing payment plan first.<br>- Choose the initial amount of credits you want to purchase. The minimum purchase is $5. The maximum purchase will be based on your trust tier.<br>- Confirm and purchase your initial amount of credits.<br>- Use auto-recharge to set an automatic recharge amount, which is the amount of credits that will be added to your account when your balance falls below a set threshold.<br><br>Please note that any purchased credits will expire after 1 year and they are non-refundable. <br>After you’ve purchased credits, you should be able to start using the API. Note that there may be a couple minutes of delay while our systems update to reflect your credit balance.<br><br><br>Purchasing additional credits<br>Once you’ve consumed all your credits, your API requests will start returning an error letting you know you’ve hit your billing quota. If you’d like to continue your API usage, you can return to the billing portal and use the “Add to balance” button to purchase additional credits.<br><br> <br>Delayed billing<br>Due to the complexity of our billing and processing systems, there may be delays in our ability to cut off access after you consume all of your credits. This excess usage may appear as a negative credit balance in your billing dashboard, and will be deducted from your next credit purchase.<br></td>\n",
       "      <td>1. `call check_billing_plan(user_id)`<br>   - **Function:** `check_billing_plan(user_id)`<br>     - **Purpose:** Retrieves the user's current billing plan (e.g., Monthly Billing, Prepaid Billing, or Arrears Billing).<br>     - **Parameters:**<br>       - `user_id`: The unique identifier of the user.<br><br>2. If the user has an arrears billing plan:<br>   2a. Inform the user: \"Please note that since you have an arrears billing plan, you'll need to cancel your existing payment plan before switching to prepaid billing. Would you like assistance with cancelling your current plan?\"<br>   2b. If the user agrees, `call cancel_payment_plan(user_id)`<br>       - **Function:** `cancel_payment_plan(user_id)`<br>         - **Purpose:** Cancels the user's current arrears billing plan.<br>         - **Parameters:**<br>           - `user_id`: The unique identifier of the user.<br><br>3. Guide the user to set up prepaid billing:<br>   3a. Instruct the user: \"Please go to your billing overview in your account settings.\"<br>   3b. Instruct the user: \"Click on 'Start payment plan' (you may see variations like 'Buy credits').\"<br>   3c. Inform the user: \"Choose the initial amount of credits you want to purchase. The minimum purchase is $5, and the maximum purchase will be based on your trust tier.\"<br>   3d. Instruct the user: \"Confirm and purchase your initial amount of credits.\"<br>   3e. Suggest to the user: \"You can set up auto-recharge to automatically add credits to your account when your balance falls below a set threshold.\"<br><br>4. Inform the user about credit expiration and refund policy:<br>   4a. Inform the user: \"Please note that any purchased credits will expire after 1 year and they are non-refundable.\"<br><br>5. Inform the user about activation time:<br>   5a. Inform the user: \"After you’ve purchased credits, you should be able to start using the API. Note that there may be a couple of minutes delay while our systems update to reflect your credit balance.\"<br><br>6. Ask the user: \"Is there anything else I can assist you with today?\"<br><br>7. If the user has no further questions, `call case_resolution()`<br>   - **Function:** `case_resolution()`<br>     - **Purpose:** Marks the case as resolved and ends the interaction.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>VAT Exemption request</td>\n",
       "      <td>How do I submit a VAT exemption request?<br>Updated over a week ago<br>If you are eligible for a tax exemption and would like to apply it to your account, please follow these steps: <br><br>Depending on the state and if required: <br>1. Obtain a current tax exemption certificate from your state or local government and/or your fully completed non-profit sales tax exemption forms. The certificate/forms should include: <br>    Your name and address<br>    Tax exemption number<br>    Signatures and dates, etc. <br>2. Send us a copy of your certificate using the chat widget in the bottom-right corner. Please include your organization id, invoice number or email address associated with the account, so we can easily find you. Instructions on how to find these items are below.<br><br>3. Once we receive your certificate/form, we will review it and apply the tax exemption to your account. You will receive a confirmation email once the exemption has been applied. <br><br>Please note that the tax exemption will only apply to future purchases. We cannot apply VAT exemptions retroactively.<br><br> <br><br>Where to find your account data<br>In order to submit a Value Added Tax ('VAT') exemption request you will need the following from your organization Billing preferences:<br>    1. Company name<br>    2. Billing email<br>    3. Primary business address<br>    4. Business tax ID</td>\n",
       "      <td>1. Greet the customer and acknowledge their request.<br>   1a. Say: \"Certainly, I'd be happy to assist you with submitting a VAT exemption request.\"<br><br>2. Request necessary information from the customer.<br>   2a. Politely ask for the following:<br>       - \"Could you please provide your **company name**?\"<br>       - \"May I have the **billing email** associated with your account?\"<br>       - \"Could you provide your **primary business address**?\"<br>       - \"Please provide your **business tax ID**.\"<br>   2b. If the customer needs assistance finding this information, provide guidance.<br>       - Say: \"You can find this information in your organization's billing preferences.\"<br><br>3. Request a copy of their current tax exemption certificate.<br>   3a. Say: \"Could you please send us a copy of your current **tax exemption certificate**? It should include your name and address, tax exemption number, signatures, and dates.\"<br><br>4. Instruct the customer on how to send the certificate.<br>   4a. Say: \"You can send us a copy of your certificate using the **chat widget in the bottom-right corner**.\"<br>   4b. Say: \"Please include your **organization ID**, **invoice number**, or the **email address associated with your account** so we can easily find you.\"<br><br>5. Once the customer has provided the required information and certificate:<br>   5a. `call the process_vat_exemption_request function` with parameters: company_name, billing_email, business_address, business_tax_id, tax_exemption_certificate, account_identifier.<br>   5b. **Define `process_vat_exemption_request function`**:<br>       - **Purpose**: To review and apply the VAT exemption to the customer's account based on the provided information and certificate.<br>       - **Parameters**:<br>         - company_name<br>         - billing_email<br>         - business_address<br>         - business_tax_id<br>         - tax_exemption_certificate<br>         - account_identifier (organization ID/invoice number/email address)<br><br>6. Inform the customer:<br>   6a. Say: \"Thank you. Once we have reviewed your certificate, we will apply the VAT exemption to your account.\"<br>   6b. Say: \"You will receive a confirmation email once the exemption has been applied.\"<br>   6c. Say: \"Please note that the tax exemption will only apply to future purchases. We cannot apply VAT exemptions retroactively.\"<br>   6d. If the customer requests to apply the VAT exemption to past purchases:<br>       - Say: \"I apologize, but we're unable to apply VAT exemptions to past purchases. The tax exemption will only apply to future purchases once it's applied to your account.\"<br><br>7. Ask if there is anything more you can assist with.<br>   7a. Say: \"Is there anything else I can assist you with today?\"<br><br>8. `call the case_resolution function`</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = pd.DataFrame(results)\n",
    "\n",
    "# Set display options to show all text in the dataframe cells\n",
    "pd.set_option('display.max_colwidth', None)\n",
    "\n",
    "# Function to display formatted text in HTML\n",
    "def display_formatted_dataframe(df):\n",
    "    def format_text(text):\n",
    "        return text.replace('\\n', '<br>')\n",
    "\n",
    "    df_formatted = df.copy()\n",
    "    df_formatted['content'] = df_formatted['content'].apply(format_text)\n",
    "    df_formatted['routine'] = df_formatted['routine'].apply(format_text)\n",
    "    \n",
    "    display(HTML(df_formatted.to_html(escape=False, justify='left')))\n",
    "\n",
    "display_formatted_dataframe(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Results\n",
    "\n",
    "Upon reviewing the generated routines, we can derive several insights:\n",
    "- Sample Responses: The model effectively generates sample responses that the LLM can utilize when executing the policy (e.g., “Instruct the user: ‘Confirm and purchase your initial amount of credits.’”).\n",
    "- Discrete Steps: The model excels at decomposing the problem into discrete actions that the LLM needs to execute. Each instruction is clearly defined and easy to interpret.\n",
    "- Function Definitions: The routines’ outputs include clearly defined functions to retrieve external information or trigger actions (e.g., `review_and_apply_tax_exemption`, `get_billing_plan`, `update_payment_method`).\n",
    "    - This is crucial for any successful routine because LLMs often need to interact with external systems. Leveraging function calls is an effective way to interact with those systems and execute actions.\n",
    "- IFTTT Logic: The model effectively employs IFTTT (If This, Then That) logic, which is ideal for an LLM (e.g., “If the customer requests assistance, proceed to step 3f.”).\n",
    "    - This type of translation becomes extremely valuable when the original knowledge base articles contain complex workflows and diagrams. Such complexity may not be easily understood by humans, and even less so by an LLM. IFTTT logic is easily comprehensible and works well for customer service solution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where do we go from here?\n",
    "\n",
    "These routines can now be integrated into agentic systems to address specific customer issues. When a customer requests assistance with tasks such as setting up prepaid billing, we can employ a classifier to determine the appropriate routine to retrieve and provide that to the LLM to interact directly with the customer. Beyond providing instructions to the user on *how* to set up billing, the system can also perform the action on their behalf.\n",
    "\n",
    "Before deploying these routines into production, we should develop comprehensive evaluations to test and validate the quality of the model’s responses. This process may necessitate adjusting the routines to ensure compliance and effectiveness."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}