openai-cookbook/examples/Whisper_correct_misspelling.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Addressing transcription misspellings: prompt vs post-processing\n",
    "\n",
    "We are addressing the problem of enhancing the precision of transcriptions, particularly when it comes to company names and product references. Our solution involves a dual strategy that utilizes both the Whisper prompt parameter and GPT-4's post-processing capabilities. \n",
    "\n",
    "Two approaches to correct inaccuracies are:\n",
    "\n",
    "- We input a list of correct spellings directly into Whisper's prompt parameter to guide the initial transcription.\n",
    "\n",
    "- We utilized GPT-4 to fix misspellings post transcription, again using the same list of correct spellings in the prompt.\n",
    "\n",
    "These strategies aimed at ensuring precise transcription of unfamilar proper nouns.\n",
    "\n",
    "## Setup\n",
    "\n",
    "To get started, let's:\n",
    "\n",
    "- Import the OpenAI Python library (if you don't have it, you'll need to install it with ```pip install openai```)\n",
    "- Download the audio file example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# imports\n",
    "from openai import OpenAI  # for making OpenAI API calls\n",
    "import urllib  # for downloading example audio files\n",
    "import os  # for accessing environment variables\n",
    "\n",
    "client = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('data/ZyntriQix.wav', <http.client.HTTPMessage at 0x10559a910>)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# set download paths\n",
    "ZyntriQix_remote_filepath = \"https://cdn.openai.com/API/examples/data/ZyntriQix.wav\"\n",
    "\n",
    "\n",
    "# set local save locations\n",
    "ZyntriQix_filepath = \"data/ZyntriQix.wav\"\n",
    "\n",
    "# download example audio files and save locally\n",
    "urllib.request.urlretrieve(ZyntriQix_remote_filepath, ZyntriQix_filepath)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting our baseline with a fictitious audio recording\n",
    "\n",
    "Our reference point is a monologue, which was generated by ChatGPT from prompts given by the author. The author then voiced this content. So, the author both guided the ChatGPT's output with prompts and brought it to life by speaking it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our fictitious company, ZyntriQix, offers a range of tech products. These include Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, and DigiFractal Matrix. We also spearhead several initiatives such as PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., and F.L.I.N.T."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a wrapper function for seeing how prompts affect transcriptions\n",
    "def transcribe(prompt: str, audio_filepath) -> str:\n",
    "    \"\"\"Given a prompt, transcribe the audio file.\"\"\"\n",
    "    transcript = client.audio.transcriptions.create(\n",
    "        file=open(audio_filepath, \"rb\"),\n",
    "        model=\"whisper-1\",\n",
    "        prompt=prompt,\n",
    "    )\n",
    "    return transcript.text\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Have you heard of ZentricX? This tech giant boasts products like Digi-Q+, Synapse 5, VortiCore V8, Echo Nix Array, and not to forget the latest Orbital Link 7 and Digifractal Matrix. Their innovation arsenal also includes the Pulse framework, Wrapped system, they've developed a brick infrastructure court system, and launched the Flint initiative, all highlighting their commitment to relentless innovation. ZentricX, in just 30 years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# baseline transcription with no prompt\n",
    "transcribe(prompt=\"\", audio_filepath=ZyntriQix_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whisper transcribed our company name, product names, and miscapitalized our acronyms incorrectly. Let's pass the correct names as a list in the prompt. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system. They've developed a B.R.I.C.K. infrastructure, Q.U.A.R.T. system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix in just 30 years has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?\""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add the correct spelling names to the prompt\n",
    "transcribe(\n",
    "    prompt=\"ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.\",\n",
    "    audio_filepath=ZyntriQix_filepath,\n",
    ")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When passing the list of product names, some of the product names are transcribed correctly while others are still misspelled. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Have you heard of ZentricX? This tech giant boasts products like DigiCube Plus, Synapse 5, VortiCore V8, EchoNix Array, and not to forget the latest Orbital Link 7 and Digifractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system. They've developed a brick infrastructure court system and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZentricX in just 30 years has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?\""
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# add a full product list to the prompt\n",
    "transcribe(\n",
    "    prompt=\"ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, AstroPixel Array, QuantumFlare Five, CyberPulse Six, VortexDrive Matrix, PhotonLink Ten, TriCircuit Array, PentaSync Seven, UltraWave Eight, QuantumVertex Nine, HyperHelix X, DigiSpiral Z, PentaQuark Eleven, TetraCube Twelve, GigaPhase Thirteen, EchoNeuron Fourteen, FusionPulse V15, MetaQuark Sixteen, InfiniCircuit Seventeen, TeraPulse Eighteen, ExoMatrix Nineteen, OrbiSync Twenty, QuantumHelix TwentyOne, NanoPhase TwentyTwo, TeraFractal TwentyThree, PentaHelix TwentyFour, ExoCircuit TwentyFive, HyperQuark TwentySix, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.\",\n",
    "    audio_filepath=ZyntriQix_filepath,\n",
    ")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You can use GPT-4 to fix spelling mistakes\n",
    "\n",
    "Leveraging GPT-4 proves especially useful when the speech content is unknown beforehand and we have a list of product names readily available.\n",
    "\n",
    "The post-processing technique using GPT-4 is notably more scalable than depending solely on Whisper's prompt parameter, which has a token limit of 244. GPT-4 allows us to process larger lists of correct spellings, making it a more robust method for handling extensive product lists.\n",
    "\n",
    "However, this post-processing technique isn't without limitations. It's constrained by the context window of the chosen model, which may pose challenges when dealing with vast numbers of unique terms. For instance, companies with thousands of SKUs may find that the context window of GPT-4 is insufficient to handle their requirements, and they might need to explore alternative solutions.\n",
    "\n",
    "Interestingly, the GPT-4 post-processing technique seems more reliable than using Whisper alone. This method, which leverages a product list, enhances the reliability of our results. However, this increased reliability comes at a price, as using this approach can increase costs and can result in higher latency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define a wrapper function for seeing how prompts affect transcriptions\n",
    "def transcribe_with_spellcheck(system_message, audio_filepath):\n",
    "    completion = client.chat.completions.create(\n",
    "        model=\"gpt-4\",\n",
    "        temperature=0,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_message},\n",
    "            {\n",
    "                \"role\": \"user\",\n",
    "                \"content\": transcribe(prompt=\"\", audio_filepath=audio_filepath),\n",
    "            },\n",
    "        ],\n",
    "    )\n",
    "    return completion.choices[0].message.content\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's input the original product list into GPT-4 and evaluate its performance. By doing so, we aim to assess the AI model's ability to correctly spell the proprietary product names, even with no prior knowledge of the exact terms to appear in the transcription. In our experiment, GPT-4 was successful in correctly spelling our product names, confirming its potential as a reliable tool for ensuring transcription accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system, they've developed a B.R.I.C.K. infrastructure court system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix, in just 30 years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?\n"
     ]
    }
   ],
   "source": [
    "system_prompt = \"You are a helpful assistant for the company ZyntriQix. Your task is to correct any spelling discrepancies in the transcribed text. Make sure that the names of the following products are spelled correctly: ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.\"\n",
    "new_text = transcribe_with_spellcheck(system_prompt, audio_filepath=ZyntriQix_filepath)\n",
    "print(new_text)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, we supplied a comprehensive product list that included all the previously used spellings, along with additional new names. This scenario simulates a real-life situation where we have a substantial SKU list and uncertain about the exact terms to appear in the transcription. Feeding this extensive list of product names into the system resulted in a correctly transcribed output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system, they've developed a B.R.I.C.K. infrastructure court system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix, in just 30 years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?\n"
     ]
    }
   ],
   "source": [
    "system_prompt = \"You are a helpful assistant for the company ZyntriQix. Your task is to correct any spelling discrepancies in the transcribed text. Make sure that the names of the following products are spelled correctly: ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array,  OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, AstroPixel Array, QuantumFlare Five, CyberPulse Six, VortexDrive Matrix, PhotonLink Ten, TriCircuit Array, PentaSync Seven, UltraWave Eight, QuantumVertex Nine, HyperHelix X, DigiSpiral Z, PentaQuark Eleven, TetraCube Twelve, GigaPhase Thirteen, EchoNeuron Fourteen, FusionPulse V15, MetaQuark Sixteen, InfiniCircuit Seventeen, TeraPulse Eighteen, ExoMatrix Nineteen, OrbiSync Twenty, QuantumHelix TwentyOne, NanoPhase TwentyTwo, TeraFractal TwentyThree, PentaHelix TwentyFour, ExoCircuit TwentyFive, HyperQuark TwentySix, GigaLink TwentySeven, FusionMatrix TwentyEight, InfiniFractal TwentyNine, MetaSync Thirty, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T. Only add necessary punctuation such as periods, commas, and capitalization, and use only the context provided.\"\n",
    "new_text = transcribe_with_spellcheck(system_prompt, audio_filepath=ZyntriQix_filepath)\n",
    "print(new_text)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are employing GPT-4 as a spell checker, using the same list of correct spellings that was previously used in the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The misspelled words are: ZentricX, Digi-Q+, Synapse 5, VortiCore V8, Echo Nix Array, Orbital Link 7, Digifractal Matrix, Pulse, Wrapped, brick, Flint, and 30. The total number of misspelled words is 12.\n",
      "\n",
      "The corrected paragraph is:\n",
      "\n",
      "Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system, they've developed a B.R.I.C.K. infrastructure court system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix, in just MetaSync Thirty years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?\n"
     ]
    }
   ],
   "source": [
    "system_prompt = \"You are a helpful assistant for the company ZyntriQix. Your first task is to list the words that are not spelled correctly according to the list provided to you and to tell me the number of misspelled words. Your next task is to insert those correct words in place of the misspelled ones. List: ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array,  OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, AstroPixel Array, QuantumFlare Five, CyberPulse Six, VortexDrive Matrix, PhotonLink Ten, TriCircuit Array, PentaSync Seven, UltraWave Eight, QuantumVertex Nine, HyperHelix X, DigiSpiral Z, PentaQuark Eleven, TetraCube Twelve, GigaPhase Thirteen, EchoNeuron Fourteen, FusionPulse V15, MetaQuark Sixteen, InfiniCircuit Seventeen, TeraPulse Eighteen, ExoMatrix Nineteen, OrbiSync Twenty, QuantumHelix TwentyOne, NanoPhase TwentyTwo, TeraFractal TwentyThree, PentaHelix TwentyFour, ExoCircuit TwentyFive, HyperQuark TwentySix, GigaLink TwentySeven, FusionMatrix TwentyEight, InfiniFractal TwentyNine, MetaSync Thirty, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.\"\n",
    "new_text = transcribe_with_spellcheck(system_prompt, audio_filepath=ZyntriQix_filepath)\n",
    "print(new_text)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}