langchain/docs/extras/integrations/llms/anyscale.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9597802c",
   "metadata": {},
   "source": [
    "# Anyscale\n",
    "\n",
    "[Anyscale](https://www.anyscale.com/) is a fully-managed [Ray](https://www.ray.io/) platform, on which you can build, deploy, and manage scalable AI and Python applications\n",
    "\n",
    "This example goes over how to use LangChain to interact with `Anyscale` [service](https://docs.anyscale.com/productionize/services-v2/get-started). \n",
    "\n",
    "It will send the requests to Anyscale Service endpoint, which is concatenate `ANYSCALE_SERVICE_URL` and `ANYSCALE_SERVICE_ROUTE`, with a token defined in `ANYSCALE_SERVICE_TOKEN`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5472a7cd-af26-48ca-ae9b-5f6ae73c74d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"ANYSCALE_SERVICE_URL\"] = ANYSCALE_SERVICE_URL\n",
    "os.environ[\"ANYSCALE_SERVICE_ROUTE\"] = ANYSCALE_SERVICE_ROUTE\n",
    "os.environ[\"ANYSCALE_SERVICE_TOKEN\"] = ANYSCALE_SERVICE_TOKEN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fb585dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.llms import Anyscale\n",
    "from langchain import PromptTemplate, LLMChain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "035dea0f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "template = \"\"\"Question: {question}\n",
    "\n",
    "Answer: Let's think step by step.\"\"\"\n",
    "\n",
    "prompt = PromptTemplate(template=template, input_variables=[\"question\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f3458d9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "llm = Anyscale()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a641dbd9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "llm_chain = LLMChain(prompt=prompt, llm=llm)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f844993",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"When was George Washington president?\"\n",
    "\n",
    "llm_chain.run(question)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "42f05b34-1a44-4cbd-8342-35c1572b6765",
   "metadata": {},
   "source": [
    "With Ray, we can distribute the queries without asyncrhonized implementation. This not only applies to Anyscale LLM model, but to any other Langchain LLM models which do not have `_acall` or `_agenerate` implemented"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08b23adc-2b29-4c38-b538-47b3c3d840a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_list = [\n",
    "    \"When was George Washington president?\",\n",
    "    \"Explain to me the difference between nuclear fission and fusion.\",\n",
    "    \"Give me a list of 5 science fiction books I should read next.\",\n",
    "    \"Explain the difference between Spark and Ray.\",\n",
    "    \"Suggest some fun holiday ideas.\",\n",
    "    \"Tell a joke.\",\n",
    "    \"What is 2+2?\",\n",
    "    \"Explain what is machine learning like I am five years old.\",\n",
    "    \"Explain what is artifical intelligence.\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b45abb9-b764-497d-af99-0df1d4e335e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ray\n",
    "\n",
    "\n",
    "@ray.remote\n",
    "def send_query(llm, prompt):\n",
    "    resp = llm(prompt)\n",
    "    return resp\n",
    "\n",
    "\n",
    "futures = [send_query.remote(llm, prompt) for prompt in prompt_list]\n",
    "results = ray.get(futures)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}