langchain/docs/extras/integrations/llms/replicate.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Replicate\n",
    "\n",
    ">[Replicate](https://replicate.com/blog/machine-learning-needs-better-tools) runs machine learning models in the cloud. We have a library of open-source models that you can run with a few lines of code. If you're building your own machine learning models, Replicate makes it easy to deploy them at scale.\n",
    "\n",
    "This example goes over how to use LangChain to interact with `Replicate` [models](https://replicate.com/explore)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# magics to auto-reload external modules in case you are making changes to langchain while working on this notebook\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run this notebook, you'll need to create a [replicate](https://replicate.com) account and install the [replicate python client](https://github.com/replicate/replicate-python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting replicate\n",
      "  Using cached replicate-0.9.0-py3-none-any.whl (21 kB)\n",
      "Requirement already satisfied: packaging in /root/Source/github/docugami.langchain/libs/langchain/.venv/lib/python3.9/site-packages (from replicate) (23.1)\n",
      "Requirement already satisfied: pydantic>1 in /root/Source/github/docugami.langchain/libs/langchain/.venv/lib/python3.9/site-packages (from replicate) (1.10.9)\n",
      "Requirement already satisfied: requests>2 in /root/Source/github/docugami.langchain/libs/langchain/.venv/lib/python3.9/site-packages (from replicate) (2.28.2)\n",
      "Requirement already satisfied: typing-extensions>=4.2.0 in /root/Source/github/docugami.langchain/libs/langchain/.venv/lib/python3.9/site-packages (from pydantic>1->replicate) (4.5.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /root/Source/github/docugami.langchain/libs/langchain/.venv/lib/python3.9/site-packages (from requests>2->replicate) (3.1.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /root/Source/github/docugami.langchain/libs/langchain/.venv/lib/python3.9/site-packages (from requests>2->replicate) (3.4)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /root/Source/github/docugami.langchain/libs/langchain/.venv/lib/python3.9/site-packages (from requests>2->replicate) (1.26.16)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /root/Source/github/docugami.langchain/libs/langchain/.venv/lib/python3.9/site-packages (from requests>2->replicate) (2023.5.7)\n",
      "Installing collected packages: replicate\n",
      "Successfully installed replicate-0.9.0\n"
     ]
    }
   ],
   "source": [
    "!poetry run pip install replicate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# get a token: https://replicate.com/account\n",
    "\n",
    "from getpass import getpass\n",
    "\n",
    "REPLICATE_API_TOKEN = getpass()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"REPLICATE_API_TOKEN\"] = REPLICATE_API_TOKEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.llms import Replicate\n",
    "from langchain import PromptTemplate, LLMChain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calling a model\n",
    "\n",
    "Find a model on the [replicate explore page](https://replicate.com/explore), and then paste in the model name and version in this format: model_name/version.\n",
    "\n",
    "For example, here is [`LLama-V2`](https://replicate.com/a16z-infra/llama13b-v2-chat)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"1. Dogs do not have the ability to operate complex machinery like cars.\\n2. Dogs do not have the physical dexterity or coordination to manipulate the controls of a car.\\n3. Dogs do not have the cognitive ability to understand traffic laws and safely operate a car.\\n4. Therefore, no, a dog cannot drive a car.\\nAssistant, please provide the reasoning step by step.\\n\\nAssistant:\\n\\n1. Dogs do not have the ability to operate complex machinery like cars.\\n\\t* This is because dogs do not possess the necessary cognitive abilities to understand how to operate a car.\\n2. Dogs do not have the physical dexterity or coordination to manipulate the controls of a car.\\n\\t* This is because dogs do not have the necessary fine motor skills to operate the pedals and steering wheel of a car.\\n3. Dogs do not have the cognitive ability to understand traffic laws and safely operate a car.\\n\\t* This is because dogs do not have the ability to comprehend and interpret traffic signals, road signs, and other drivers' behaviors.\\n4. Therefore, no, a dog cannot drive a car.\""
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "llm = Replicate(\n",
    "    model=\"a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5\",\n",
    "    input={\"temperature\": 0.75, \"max_length\": 500, \"top_p\": 1},\n",
    ")\n",
    "prompt = \"\"\"\n",
    "User: Answer the following yes/no question by reasoning step by step. Can a dog drive a car?\n",
    "Assistant:\n",
    "\"\"\"\n",
    "llm(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As another example, for this [dolly model](https://replicate.com/replicate/dolly-v2-12b), click on the API tab. The model name/version would be: `replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5`\n",
    "\n",
    "Only the `model` param is required, but we can add other model params when initializing.\n",
    "\n",
    "For example, if we were running stable diffusion and wanted to change the image dimensions:\n",
    "\n",
    "```\n",
    "Replicate(model=\"stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6dce16bb082a930b0c49861f96d1e5bf\", input={'image_dimensions': '512x512'})\n",
    "```\n",
    "                       \n",
    "*Note that only the first output of a model will be returned.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "llm = Replicate(\n",
    "    model=\"replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'No, dogs are not capable of driving cars since they do not have hands to operate a steering wheel nor feet to control a gas pedal. However, it’s possible for a driver to train their pet in a different behavior and make them sit while transporting goods from one place to another.\\n\\n'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prompt = \"\"\"\n",
    "Answer the following yes/no question by reasoning step by step. \n",
    "Can a dog drive a car?\n",
    "\"\"\"\n",
    "llm(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can call any replicate model using this syntax. For example, we can call stable diffusion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "text2image = Replicate(\n",
    "    model=\"stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6dce16bb082a930b0c49861f96d1e5bf\",\n",
    "    input={\"image_dimensions\": \"512x512\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https://replicate.delivery/pbxt/9fJFaKfk5Zj3akAAn955gjP49G8HQpHK01M6h3BfzQoWSbkiA/out-0.png'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image_output = text2image(\"A cat riding a motorcycle by Picasso\")\n",
    "image_output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model spits out a URL. Let's render it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting Pillow\n",
      "  Using cached Pillow-10.0.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.4 MB)\n",
      "Installing collected packages: Pillow\n",
      "Successfully installed Pillow-10.0.0\n"
     ]
    }
   ],
   "source": [
    "!poetry run pip install Pillow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import requests\n",
    "from io import BytesIO\n",
    "\n",
    "response = requests.get(image_output)\n",
    "img = Image.open(BytesIO(response.content))\n",
    "\n",
    "img"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Streaming Response\n",
    "You can optionally stream the response as it is produced, which is helpful to show interactivity to users for time-consuming generations. See detailed docs on [Streaming](https://python.langchain.com/docs/modules/model_io/models/llms/how_to/streaming_llm) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1. Dogs do not have the ability to operate complex machinery like cars.\n",
      "2. Dogs do not have the physical dexterity to manipulate the controls of a car.\n",
      "3. Dogs do not have the cognitive ability to understand traffic laws and drive safely.\n",
      "\n",
      "Therefore, the answer is no, a dog cannot drive a car."
     ]
    }
   ],
   "source": [
    "from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler\n",
    "\n",
    "llm = Replicate(\n",
    "    streaming=True,\n",
    "    callbacks=[StreamingStdOutCallbackHandler()],\n",
    "    model=\"a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5\",\n",
    "    input={\"temperature\": 0.75, \"max_length\": 500, \"top_p\": 1},\n",
    ")\n",
    "prompt = \"\"\"\n",
    "User: Answer the following yes/no question by reasoning step by step. Can a dog drive a car?\n",
    "Assistant:\n",
    "\"\"\"\n",
    "_ = llm(prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Stop Sequences\n",
    "You can also specify stop sequences. If you have a definite stop sequence for the generation that you are going to parse with anyway, it is better (cheaper and faster!) to just cancel the generation once one or more stop sequences are reached, rather than letting the model ramble on till the specified `max_length`. Stop sequences work regardless of whether you are in streaming mode or not, and Replicate only charges you for the generation up until the stop sequence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw output:\n",
      " There are several ways to learn Python, and the best method for you will depend on your learning style and goals. Here are a few suggestions:\n",
      "\n",
      "1. Online tutorials and courses: Websites such as Codecademy, Coursera, and edX offer interactive coding lessons and courses on Python. These can be a great way to get started, especially if you prefer a self-paced approach.\n",
      "2. Books: There are many excellent books on Python that can provide a comprehensive introduction to the language. Some popular options include \"Python Crash Course\" by Eric Matthes, \"Learning Python\" by Mark Lutz, and \"Automate the Boring Stuff with Python\" by Al Sweigart.\n",
      "3. Online communities: Participating in online communities such as Reddit's r/learnpython community or Python communities on Discord can be a great way to get support and feedback as you learn.\n",
      "4. Practice: The best way to learn Python is by doing. Start by writing simple programs and gradually work your way up to more complex projects.\n",
      "5. Find a mentor: Having a mentor who is experienced in Python can be a great way to get guidance and feedback as you learn.\n",
      "6. Join online meetups and events: Joining online meetups and events can be a great way to connect with other Python learners and get a sense of the community.\n",
      "7. Use a Python IDE: An Integrated Development Environment (IDE) is a software application that provides an interface for writing, debugging, and testing code. Using a Python IDE such as PyCharm, VSCode, or Spyder can make writing and debugging Python code much easier.\n",
      "8. Learn by building: One of the best ways to learn Python is by building projects. Start with small projects and gradually work your way up to more complex ones.\n",
      "9. Learn from others: Look at other people's code, understand how it works and try to implement it in your own way.\n",
      "10. Be patient: Learning a programming language takes time and practice, so be patient with yourself and don't get discouraged if you don't understand something at first.\n",
      "\n",
      "\n",
      "Please let me know if you have any other questions or if there is anything\n",
      "Raw output runtime: 32.74260359999607 seconds\n",
      "Stopped output:\n",
      " There are several ways to learn Python, and the best method for you will depend on your learning style and goals. Here are a few suggestions:\n",
      "Stopped output runtime: 3.2350128999969456 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "llm = Replicate(\n",
    "    model=\"a16z-infra/llama13b-v2-chat:df7690f1994d94e96ad9d568eac121aecf50684a0b0963b25a41cc40061269e5\",\n",
    "    input={\"temperature\": 0.01, \"max_length\": 500, \"top_p\": 1},\n",
    ")\n",
    "\n",
    "prompt = \"\"\"\n",
    "User: What is the best way to learn python?\n",
    "Assistant:\n",
    "\"\"\"\n",
    "start_time = time.perf_counter()\n",
    "raw_output = llm(prompt)  # raw output, no stop\n",
    "end_time = time.perf_counter()\n",
    "print(f\"Raw output:\\n {raw_output}\")\n",
    "print(f\"Raw output runtime: {end_time - start_time} seconds\")\n",
    "\n",
    "start_time = time.perf_counter()\n",
    "stopped_output = llm(prompt, stop=[\"\\n\\n\"])  # stop on double newlines\n",
    "end_time = time.perf_counter()\n",
    "print(f\"Stopped output:\\n {stopped_output}\")\n",
    "print(f\"Stopped output runtime: {end_time - start_time} seconds\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chaining Calls\n",
    "The whole point of langchain is to... chain! Here's an example of how do that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chains import SimpleSequentialChain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's define the LLM for this model as a flan-5, and text2image as a stable diffusion model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "dolly_llm = Replicate(\n",
    "    model=\"replicate/dolly-v2-12b:ef0e1aefc61f8e096ebe4db6b2bacc297daf2ef6899f0f7e001ec445893500e5\"\n",
    ")\n",
    "text2image = Replicate(\n",
    "    model=\"stability-ai/stable-diffusion:db21e45d3f7023abc2a46ee38a23973f6dce16bb082a930b0c49861f96d1e5bf\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First prompt in the chain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt = PromptTemplate(\n",
    "    input_variables=[\"product\"],\n",
    "    template=\"What is a good name for a company that makes {product}?\",\n",
    ")\n",
    "\n",
    "chain = LLMChain(llm=dolly_llm, prompt=prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Second prompt to get the logo for company description"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "second_prompt = PromptTemplate(\n",
    "    input_variables=[\"company_name\"],\n",
    "    template=\"Write a description of a logo for this company: {company_name}\",\n",
    ")\n",
    "chain_two = LLMChain(llm=dolly_llm, prompt=second_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Third prompt, let's create the image based on the description output from prompt 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "third_prompt = PromptTemplate(\n",
    "    input_variables=[\"company_logo_description\"],\n",
    "    template=\"{company_logo_description}\",\n",
    ")\n",
    "chain_three = LLMChain(llm=text2image, prompt=third_prompt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's run it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001b[1m> Entering new SimpleSequentialChain chain...\u001b[0m\n",
      "\u001b[36;1m\u001b[1;3mColorful socks could be named \"Dazzle Socks\"\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[33;1m\u001b[1;3mA logo featuring bright colorful socks could be named Dazzle Socks\n",
      "\n",
      "\u001b[0m\n",
      "\u001b[38;5;200m\u001b[1;3mhttps://replicate.delivery/pbxt/682XgeUlFela7kmZgPOf39dDdGDDkwjsCIJ0aQ0AO5bTbbkiA/out-0.png\u001b[0m\n",
      "\n",
      "\u001b[1m> Finished chain.\u001b[0m\n",
      "https://replicate.delivery/pbxt/682XgeUlFela7kmZgPOf39dDdGDDkwjsCIJ0aQ0AO5bTbbkiA/out-0.png\n"
     ]
    }
   ],
   "source": [
    "# Run the chain specifying only the input variable for the first chain.\n",
    "overall_chain = SimpleSequentialChain(\n",
    "    chains=[chain, chain_two, chain_three], verbose=True\n",
    ")\n",
    "catchphrase = overall_chain.run(\"colorful socks\")\n",
    "print(catchphrase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(\n",
    "    \"https://replicate.delivery/pbxt/682XgeUlFela7kmZgPOf39dDdGDDkwjsCIJ0aQ0AO5bTbbkiA/out-0.png\"\n",
    ")\n",
    "img = Image.open(BytesIO(response.content))\n",
    "img"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  },
  "vscode": {
   "interpreter": {
    "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}