langchain/docs/docs/integrations/llms/anyscale.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9597802c",
   "metadata": {},
   "source": [
    "# Anyscale\n",
    "\n",
    "[Anyscale](https://www.anyscale.com/) is a fully-managed [Ray](https://www.ray.io/) platform, on which you can build, deploy, and manage scalable AI and Python applications\n",
    "\n",
    "This example goes over how to use LangChain to interact with [Anyscale Endpoint](https://app.endpoints.anyscale.com/). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "515070aa-e241-480e-8d9a-afdf52f35322",
   "metadata": {},
   "outputs": [],
   "source": [
    "ANYSCALE_API_BASE = \"...\"\n",
    "ANYSCALE_API_KEY = \"...\"\n",
    "ANYSCALE_MODEL_NAME = \"...\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5472a7cd-af26-48ca-ae9b-5f6ae73c74d2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ[\"ANYSCALE_API_BASE\"] = ANYSCALE_API_BASE\n",
    "os.environ[\"ANYSCALE_API_KEY\"] = ANYSCALE_API_KEY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fb585dd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.chains import LLMChain\n",
    "from langchain_community.llms import Anyscale\n",
    "from langchain_core.prompts import PromptTemplate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "035dea0f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "template = \"\"\"Question: {question}\n",
    "\n",
    "Answer: Let's think step by step.\"\"\"\n",
    "\n",
    "prompt = PromptTemplate.from_template(template)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f3458d9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "llm = Anyscale(model_name=ANYSCALE_MODEL_NAME)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a641dbd9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "llm_chain = prompt | llm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f844993",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "question = \"When was George Washington president?\"\n",
    "\n",
    "llm_chain.invoke({\"question\": question})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42f05b34-1a44-4cbd-8342-35c1572b6765",
   "metadata": {},
   "source": [
    "With Ray, we can distribute the queries without asynchronized implementation. This not only applies to Anyscale LLM model, but to any other Langchain LLM models which do not have `_acall` or `_agenerate` implemented"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08b23adc-2b29-4c38-b538-47b3c3d840a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_list = [\n",
    "    \"When was George Washington president?\",\n",
    "    \"Explain to me the difference between nuclear fission and fusion.\",\n",
    "    \"Give me a list of 5 science fiction books I should read next.\",\n",
    "    \"Explain the difference between Spark and Ray.\",\n",
    "    \"Suggest some fun holiday ideas.\",\n",
    "    \"Tell a joke.\",\n",
    "    \"What is 2+2?\",\n",
    "    \"Explain what is machine learning like I am five years old.\",\n",
    "    \"Explain what is artifical intelligence.\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b45abb9-b764-497d-af99-0df1d4e335e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ray\n",
    "\n",
    "\n",
    "@ray.remote(num_cpus=0.1)\n",
    "def send_query(llm, prompt):\n",
    "    resp = llm.invoke(prompt)\n",
    "    return resp\n",
    "\n",
    "\n",
    "futures = [send_query.remote(llm, prompt) for prompt in prompt_list]\n",
    "results = ray.get(futures)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "vscode": {
   "interpreter": {
    "hash": "a0a0263b650d907a3bfe41c0f8d6a63a071b884df3cfdc1579f00cdc1aed6b03"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}