langchain/docs/modules/agents/tools/examples/gradio_tools.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c613812f",
   "metadata": {},
   "source": [
    "# Gradio Tools\n",
    "\n",
    "There are many 1000s of Gradio apps on Hugging Face Spaces. This library puts them at the tips of your LLM's fingers 🦾\n",
    "\n",
    "Specifically, gradio-tools is a Python library for converting Gradio apps into tools that can be leveraged by a large language model (LLM)-based agent to complete its task. For example, an LLM could use a Gradio tool to transcribe a voice recording it finds online and then summarize it for you. Or it could use a different Gradio tool to apply OCR to a document on your Google Drive and then answer questions about it.\n",
    "\n",
    "It's very easy to create you own tool if you want to use a space that's not one of the pre-built tools. Please see this section of the gradio-tools documentation for information on how to do that. All contributions are welcome!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "231b46c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install gradio_tools"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17608431",
   "metadata": {},
   "source": [
    "## Using a tool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "423f9dad",
   "metadata": {},
   "outputs": [],
   "source": [
    "from gradio_tools.tools import StableDiffusionTool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "30b8f077",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded as API: https://gradio-client-demos-stable-diffusion.hf.space ✔\n",
      "\n",
      "Job Status: Status.STARTING eta: None\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'/Users/harrisonchase/workplace/langchain/docs/modules/agents/tools/examples/b61c1dd9-47e2-46f1-a47c-20d27640993d/tmp4ap48vnm.jpg'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "StableDiffusionTool().langchain.run(\"Please create a photo of a dog riding a skateboard\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b7bdfd26",
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "98e09784",
   "metadata": {},
   "outputs": [],
   "source": [
    "im = Image.open(\"/Users/harrisonchase/workplace/langchain/docs/modules/agents/tools/examples/b61c1dd9-47e2-46f1-a47c-20d27640993d/tmp4ap48vnm.jpg\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "98e1e602",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwAAAAMACAIAAAAc45fZAAEAAElEQVR4nKz9aZckSW4lCl4AIqpm5ltEZERuVVnFqsyspatYO4dbs/n+c/+BOTOfZjkcNqfJJlnkY7MWZuUSm7ubmaqKAJgPEFEzj8ziNN+Mnsw4Hh626AKBABcXF/S//cWTnPM45mFMYx52F+PN5cXF5YZgRMrwqpOquqswCOPh1m7vpsPhfnEtZb47TqWoQURGyaNIdtfMkjJUdZ5nq35xcbXZDETEjJQSEanqPJd5no+HUkpxZwBavZq7MTgxMwBVL6WourvXonNZajUnIknurmZwF+JE7EzubnAAxCwiRDQMgyQyMy0VAIPA5K7q5sqqWoubWZZhGAb1mnMatjKOKW+JxYsuy7LoVN1hpubV3ZmREosIADNzJyIiknb+tS7znFKSlAA4lJlzlpzzZjPsdrs05lLKPM9mZl5r8eNky6yqCgBA/CAiREJEbqSqqm5mRMxM0CORxwW6u5nVWs18GHLOOaUUv3H3lNIwZI9LVjczd5gZGZnCndy4ViuLL0spi7rC3SWNZgYgZRmGQYQAc6iIpCREHmcOADB30kqqptougZmZUtwUAAADcHcARASYeRXhlFJKiYjdXavHhahq3DHA3Z2ImHmZZhFhZiJysjjiM4kopzQMw+XuYrPZsGPRen/YT/M8z8XMAFJVU3Z3gONbzNwd7miHAjj7az9nIo/TZgYRgVyEU6LNRZLkZwbgzAhzdW9vATxOmJlrVXdzd/PqDgfgIBAgtcAdhPZRxJ5SYlDYlbvXYqoqklNK6qZaAKTEKXO/S7rZCsiEc85ZZDCz6ViXZQmjUo1L4GEYLi4utttRaxjpPM/zsiyqEEHKNAzDMPAwDJLg7mbVoXAG5Vp1ma2UogozcyN3Z04ppZxFUjxrBwxAKcZMLMZxOQKtXkqti0/H4k5D3qQ0mJlZleSG6ijuCKtgZuvH+tQIzQCIKCwkbni8LOwn51FVa1VVNUWsSmaUuoQlmiGeJpzXO9zWb7NSctd41oDF42NmESIiU91ut2EVOWciqrUCPmQCWdgNQeJ73b0sttvtdrvLi4uLzWYnIldXV0+fvfPo5p3rm8ePHz9+8uTJkydPdrvdxcXF9fX1brcbxzHugIioqruPY3Z3SRSeIW7CNE37/X5Zlqurq+12m3N299vb2+fPny/TvNuO7zx9tr28hGrYLlTDIRwOhxcvXnzyySf/+q//+umnn5rZuNvmtL29v5vnmYgYYYGY51mEUkrbcXN1dXV9ucs5l1IO8/Ty7vX+eLClTNO07KfHjx9/4+sfXF5eOvj/9Tf/7f/yf/0//83f/e2r29fGBEArCQZyz8Nxu+Prm21Kw/29v351qAXwIqTvv81/9KOPf/yDb7//5GYcx7v98g//+E9a6te+9rVvfvMPbm9v/+2Tz3/9699++NF3njx953CY8ubim9/81rO3393urm/39//zt7/+p//9l7/73e/2+9fETgRV2+12erRa6/HusN/fMWi7y5l4rtM0TdXKkMbHjx+//dbbjx49ur682m63NzePzGwu8+FwOMyHWisJUkrzcVqWxYx324unb79zdXkzl3p7e/vFyy9evnz+4vVnavO44ZRQXavZ8Wj74zLPqtXdITxeXl5fXl4/unwcz+758+dffPHF69evp/mwVDsUO86zmeWcmYZlWVZHCkCE8iAiYqbLMi1LhY9wavbGYOaUkojUWs1MNfxeO4yQUjK4EJjZnYjT7vLm4vrx43e+9vSd99NmMy31i+efvXz54u7li2998N5/+cWP/vyP/+jDb37j8vJyk8TMMtjdy7yAyQyzVjNUt6peSjWX13f7+/vbF3cv//l//+Vf/fX/8x//8e/vbl/CTAiJwTnlLMa01DLPsxgnEeHcHL67SA53qqq1aPxS22Hg5pyJEEsyLi12OndfXUEsZmHKObvFpsOllFL07I0GwKzGBpdzTnd3+2FIpYxpYmIfbtP91e5qt02ZdttxHDNzTpLbNmCUsm02MCjVwomU+DgtpTpOziIPiZlhVt0V4OPxOE0HERnHcbOhlBKRpETu5CYpZXcys8pGamUxc4/VHr6g1lJKcQOs7TBeY0M3AObqxBAGEAFQ+MWcs5mxc9zoZi+giKdg7kZEEBERSYlhIiJCTETkbuau5moiSVXNvNaIA5q7j10ZQPg7U5ipOwis1aouRMRMEBSoqs/zXIqmcVAt4UNz2uTEgArXsPvwU+Hom5dndvdSSq2ViEWYFMRGJMweThxgIo0tP26Jqkd0YuaSxJ3iwauau7u1fSWsR8RSSnA2cncvWjy2CiGzSiTuamZEpIrYnwAmdqIEwNR7rBMbp4EsgrO4UWdhAYic2Ncnwixt4+hHmH68K7zAauVEBPa+NYLako5LEBGBrrvmulQoHs0aWZ6dyWldxac9jIFOr2w3mUBkZlRKUbP16QMwixBT+ydT2GBcV49T1d/88DUOi2dSw1YjKDRTd3c4yMyqGRxhGM4CEWEmACAnIrSvcCJza+YDqJnFExeRsAwiSonNOE4yTCwlSSldXGxT4pxFEgOmWsITzYu7r7GsxRfFBdZaidxBRAgjcXciYU4ikrOkJCyoZO5wtZRc1VW11klViTzH6gHcw3janYv/3OHm/YY5UawGnEcnRMTtWiL4YBFxizsvRC4pbg/FKiYiOBORWXtG8SzihhCRucJBBCCyDop3bTYbZp7n47IsOedxHIdhGMfBbVErEaVF3lJrVfWUUs55GIZxHK+vr7fb7dXV1aNHj588fnTz6K1Hjx5dX18PwxBLJmw4djLqR9/knIhXu/J+DMMQ0dK6kMdxzJLefvbWdhi9ViJCPBERZlbVf/mXf/nHf/zH169fL8uy2WzeeuutmyeP94dl3G72+/1+v797/WpZFlddlmWejymli+3u+vr60fVlznlZlsM8HZb5cNgv03R/fz/d7Y/Hw+ObR1dXV8fpUJcptmF3J0M8BC01bLWnMeweW7UTLGdsNpubm6u3Hj2+vNyZWSklLnxZlrlFBrLdjtM03d6+ur+fhl3Z7/dP3TfbgdLV9avdzdX1F599+vrVq1LmR49uLi+vri6uNzebWsrdeMcMrcvFdnux3UBwt79V1Szp6upmt9sMw5DHYdiMm82obpzY2SFY6hznrKWGR+IUDpnMrKjWWpdlOR7nab5NB8oDUSISVhWzSP5LrZ6T73bXSfL19fVmswFwfX19c3Pz6tWr17cv7/bHL25vnbAsCwDV4q4RhddamVkkM3NK4g5gAHie1K0tAwaFAYShrscaAzna4uyWA7dTshpLuOVJPeGMLHrNOVd/RUSrB/MzX3keb505wNjCKMIG9zgRih12TQ7jtLkDFkSESPxaMGeqgJ8ctUi70jCPda98eICIvC9e71mOmUWmalaJiIUi6krHaTEnteKuVReG7+43u82GmR9dX15e7lLmIaXdxSbnzGASHbcMdlpmrkSSidNxXvbHGYCQD+M4jpKyCKvpssx+PO7nuYjIbrer1SLRgTNcRJw5A9DqhAI202rqka8xsYCgpksBC8AEMnczr9oiPo6rCfcHNTPvjza2qFq1FgWg7CIiPMbOYeSuRdWLzUSehyQMJoerLoCoVfPqLBIG0Nw72FmcqRSNfZcpmZu79i07lVKqBfaTiQLCKcys9ci5hLfabNI4RuxF7qQaQc6iqiIRsjCRxPW5u6oxMZGzE4HhDIAiT3CYk6m5kxuBiMDEzJSYZbVFM3dz04hPmqUSsQiZUexypkTm4FNUsXrY8AJmFuszScvC499X7wzASQESyT0UcLN1TTrOdlBmI6LIm+NbOrpjsSsAiPSXyJmJuLlyVe1hIlLiYUg5i8LiVjnUXWPbbsv+QRB2CnfOg54vx0APVrvDCGZeq5NCWVOKAIIDWVO1NeqKEDq+lyStmBkxiMCOvh+Tmbd93gFWmFFKa4AYEZJD1RxEkkiEU+KUqOMuCXB39uaJFN5jBDMzqLoZ3DUlbcBeSqo1boIIh5WKyDAMzGCWWFBARCGqalrjuXvAPxE0xHt7IBK
      "text/plain": [
       "<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=768x768>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display(im)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aeeeeb5",
   "metadata": {},
   "source": [
    "## Using within an agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4a9d45b7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded as API: https://gradio-client-demos-stable-diffusion.hf.space ✔\n",
      "Loaded as API: https://taesiri-blip-2.hf.space ✔\n",
      "Loaded as API: https://microsoft-promptist.hf.space ✔\n",
      "Loaded as API: https://damo-vilab-modelscope-text-to-video-synthesis.hf.space ✔\n",
      "\n",
      "\n",
      "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
      "\u001b[32;1m\u001b[1;3m\n",
      "Thought: Do I need to use a tool? Yes\n",
      "Action: StableDiffusionPromptGenerator\n",
      "Action Input: A dog riding a skateboard\u001b[0m\n",
      "Job Status: Status.STARTING eta: None\n",
      "\n",
      "Observation: \u001b[38;5;200m\u001b[1;3mA dog riding a skateboard, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3m Do I need to use a tool? Yes\n",
      "Action: StableDiffusion\n",
      "Action Input: A dog riding a skateboard, digital painting, artstation, concept art, smooth, sharp focus, illustration, art by artgerm and greg rutkowski and alphonse mucha\u001b[0m\n",
      "Job Status: Status.STARTING eta: None\n",
      "\n",
      "Job Status: Status.PROCESSING eta: None\n",
      "\n",
      "Observation: \u001b[36;1m\u001b[1;3m/Users/harrisonchase/workplace/langchain/docs/modules/agents/tools/examples/2e280ce4-4974-4420-8680-450825c31601/tmpfmiz2g1c.jpg\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3m Do I need to use a tool? Yes\n",
      "Action: ImageCaptioner\n",
      "Action Input: /Users/harrisonchase/workplace/langchain/docs/modules/agents/tools/examples/2e280ce4-4974-4420-8680-450825c31601/tmpfmiz2g1c.jpg\u001b[0m\n",
      "Job Status: Status.STARTING eta: None\n",
      "\n",
      "Observation: \u001b[33;1m\u001b[1;3ma painting of a dog sitting on a skateboard\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3m Do I need to use a tool? Yes\n",
      "Action: TextToVideo\n",
      "Action Input: a painting of a dog sitting on a skateboard\u001b[0m\n",
      "Job Status: Status.STARTING eta: None\n",
      "Due to heavy traffic on this app, the prediction will take approximately 73 seconds.For faster predictions without waiting in queue, you may duplicate the space using: Client.duplicate(damo-vilab/modelscope-text-to-video-synthesis)\n",
      "\n",
      "Job Status: Status.IN_QUEUE eta: 73.89824726581574\n",
      "Due to heavy traffic on this app, the prediction will take approximately 42 seconds.For faster predictions without waiting in queue, you may duplicate the space using: Client.duplicate(damo-vilab/modelscope-text-to-video-synthesis)\n",
      "\n",
      "Job Status: Status.IN_QUEUE eta: 42.49370198879602\n",
      "\n",
      "Job Status: Status.IN_QUEUE eta: 21.314297944849187\n",
      "\n",
      "Observation: \u001b[31;1m\u001b[1;3m/var/folders/bm/ylzhm36n075cslb9fvvbgq640000gn/T/tmp5snj_nmzf20_cb3m.mp4\u001b[0m\n",
      "Thought:\u001b[32;1m\u001b[1;3m Do I need to use a tool? No\n",
      "AI: Here is a video of a painting of a dog sitting on a skateboard.\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "from langchain.agents import initialize_agent\n",
    "from langchain.llms import OpenAI\n",
    "from gradio_tools.tools import (StableDiffusionTool, ImageCaptioningTool, StableDiffusionPromptGeneratorTool,\n",
    "                                TextToVideoTool)\n",
    "\n",
    "from langchain.memory import ConversationBufferMemory\n",
    "\n",
    "llm = OpenAI(temperature=0)\n",
    "memory = ConversationBufferMemory(memory_key=\"chat_history\")\n",
    "tools = [StableDiffusionTool().langchain, ImageCaptioningTool().langchain,\n",
    "         StableDiffusionPromptGeneratorTool().langchain, TextToVideoTool().langchain]\n",
    "\n",
    "\n",
    "agent = initialize_agent(tools, llm, memory=memory, agent=\"conversational-react-description\", verbose=True)\n",
    "output = agent.run(input=(\"Please create a photo of a dog riding a skateboard \"\n",
    "                          \"but improve my prompt prior to using an image generator.\"\n",
    "                          \"Please caption the generated image and create a video for it using the improved prompt.\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67642c82",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}