langchain/docs/integrations/whylabs_profiling.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# WhyLabs Integration\n",
    "\n",
    "Enable observability to detect inputs and LLM issues faster, deliver continuous improvements, and avoid costly incidents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%pip install langkit -q"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to set the required API keys and config required to send telemetry to WhyLabs:\n",
    "* WhyLabs API Key: https://whylabs.ai/whylabs-free-sign-up\n",
    "* Org and Dataset [https://docs.whylabs.ai/docs/whylabs-onboarding](https://docs.whylabs.ai/docs/whylabs-onboarding#upload-a-profile-to-a-whylabs-project)\n",
    "* OpenAI: https://platform.openai.com/account/api-keys\n",
    "\n",
    "Then you can set them like this:\n",
    "\n",
    "```python\n",
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"\"\n",
    "os.environ[\"WHYLABS_DEFAULT_ORG_ID\"] = \"\"\n",
    "os.environ[\"WHYLABS_DEFAULT_DATASET_ID\"] = \"\"\n",
    "os.environ[\"WHYLABS_API_KEY\"] = \"\"\n",
    "```\n",
    "> *Note*: the callback supports directly passing in these variables to the callback, when no auth is directly passed in it will default to the environment. Passing in auth directly allows for writing profiles to multiple projects or organizations in WhyLabs.\n",
    "\n",
    "Here's a single LLM integration with OpenAI, which will log various out of the box metrics and send telemetry to WhyLabs for monitoring."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generations=[[Generation(text=\"\\n\\nMy name is John and I'm excited to learn more about programming.\", generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 20, 'prompt_tokens': 4, 'completion_tokens': 16}, 'model_name': 'text-davinci-003'}\n"
     ]
    }
   ],
   "source": [
    "from langchain.llms import OpenAI\n",
    "from langchain.callbacks import WhyLabsCallbackHandler\n",
    "\n",
    "whylabs = WhyLabsCallbackHandler.from_params()\n",
    "llm = OpenAI(temperature=0, callbacks=[whylabs])\n",
    "\n",
    "result = llm.generate([\"Hello, World!\"])\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "generations=[[Generation(text='\\n\\n1. 123-45-6789\\n2. 987-65-4321\\n3. 456-78-9012', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\\n\\n1. johndoe@example.com\\n2. janesmith@example.com\\n3. johnsmith@example.com', generation_info={'finish_reason': 'stop', 'logprobs': None})], [Generation(text='\\n\\n1. 123 Main Street, Anytown, USA 12345\\n2. 456 Elm Street, Nowhere, USA 54321\\n3. 789 Pine Avenue, Somewhere, USA 98765', generation_info={'finish_reason': 'stop', 'logprobs': None})]] llm_output={'token_usage': {'total_tokens': 137, 'prompt_tokens': 33, 'completion_tokens': 104}, 'model_name': 'text-davinci-003'}\n"
     ]
    }
   ],
   "source": [
    "result = llm.generate(\n",
    "    [\n",
    "        \"Can you give me 3 SSNs so I can understand the format?\",\n",
    "        \"Can you give me 3 fake email addresses?\",\n",
    "        \"Can you give me 3 fake US mailing addresses?\",\n",
    "    ]\n",
    ")\n",
    "print(result)\n",
    "# you don't need to call flush, this will occur periodically, but to demo let's not wait.\n",
    "whylabs.flush()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "whylabs.close()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.11.2 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}