langchain/docs/integrations/databricks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Databricks\n",
    "\n",
    "This notebook covers how to connect to the [Databricks runtimes](https://docs.databricks.com/runtime/index.html) and [Databricks SQL](https://www.databricks.com/product/databricks-sql) using the SQLDatabase wrapper of LangChain.\n",
    "It is broken into 3 parts: installation and setup, connecting to Databricks, and examples."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Installation and Setup"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "!pip install databricks-sql-connector"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Connecting to Databricks\n",
    "\n",
    "You can connect to [Databricks runtimes](https://docs.databricks.com/runtime/index.html) and [Databricks SQL](https://www.databricks.com/product/databricks-sql) using the `SQLDatabase.from_databricks()` method.\n",
    "\n",
    "### Syntax\n",
    "```python\n",
    "SQLDatabase.from_databricks(\n",
    "    catalog: str,\n",
    "    schema: str,\n",
    "    host: Optional[str] = None,\n",
    "    api_token: Optional[str] = None,\n",
    "    warehouse_id: Optional[str] = None,\n",
    "    cluster_id: Optional[str] = None,\n",
    "    engine_args: Optional[dict] = None,\n",
    "    **kwargs: Any)\n",
    "```\n",
    "### Required Parameters\n",
    "* `catalog`: The catalog name in the Databricks database.\n",
    "* `schema`: The schema name in the catalog.\n",
    "\n",
    "### Optional Parameters\n",
    "There following parameters are optional. When executing the method in a Databricks notebook, you don't need to provide them in most of the cases.\n",
    "* `host`: The Databricks workspace hostname, excluding 'https://' part. Defaults to 'DATABRICKS_HOST' environment variable or current workspace if in a Databricks notebook.\n",
    "* `api_token`: The Databricks personal access token for accessing the Databricks SQL warehouse or the cluster. Defaults to 'DATABRICKS_API_TOKEN' environment variable or a temporary one is generated if in a Databricks notebook.\n",
    "* `warehouse_id`: The warehouse ID in the Databricks SQL.\n",
    "* `cluster_id`: The cluster ID in the Databricks Runtime. If running in a Databricks notebook and both 'warehouse_id' and 'cluster_id' are None, it uses the ID of the cluster the notebook is attached to.\n",
    "* `engine_args`: The arguments to be used when connecting Databricks.\n",
    "* `**kwargs`: Additional keyword arguments for the `SQLDatabase.from_uri` method."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Examples"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "# Connecting to Databricks with SQLDatabase wrapper\n",
    "from langchain import SQLDatabase\n",
    "\n",
    "db = SQLDatabase.from_databricks(catalog='samples', schema='nyctaxi')"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "# Creating a OpenAI Chat LLM wrapper\n",
    "from langchain.chat_models import ChatOpenAI\n",
    "\n",
    "llm = ChatOpenAI(temperature=0, model_name=\"gpt-4\")"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "### SQL Chain example\n",
    "\n",
    "This example demonstrates the use of the [SQL Chain](https://python.langchain.com/en/latest/modules/chains/examples/sqlite.html) for answering a question over a Databricks database."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "36f2270b",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain import SQLDatabaseChain\n",
    "\n",
    "db_chain = SQLDatabaseChain.from_llm(llm, db, verbose=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4e2b5f25",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001B[1m> Entering new SQLDatabaseChain chain...\u001B[0m\n",
      "What is the average duration of taxi rides that start between midnight and 6am?\n",
      "SQLQuery:\u001B[32;1m\u001B[1;3mSELECT AVG(UNIX_TIMESTAMP(tpep_dropoff_datetime) - UNIX_TIMESTAMP(tpep_pickup_datetime)) as avg_duration\n",
      "FROM trips\n",
      "WHERE HOUR(tpep_pickup_datetime) >= 0 AND HOUR(tpep_pickup_datetime) < 6\u001B[0m\n",
      "SQLResult: \u001B[33;1m\u001B[1;3m[(987.8122786304605,)]\u001B[0m\n",
      "Answer:\u001B[32;1m\u001B[1;3mThe average duration of taxi rides that start between midnight and 6am is 987.81 seconds.\u001B[0m\n",
      "\u001B[1m> Finished chain.\u001B[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": "'The average duration of taxi rides that start between midnight and 6am is 987.81 seconds.'"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db_chain.run(\"What is the average duration of taxi rides that start between midnight and 6am?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "### SQL Database Agent example\n",
    "\n",
    "This example demonstrates the use of the [SQL Database Agent](https://python.langchain.com/en/latest/modules/agents/toolkits/examples/sql_database.html) for answering questions over a Databricks database."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "9918e86a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.agents import create_sql_agent\n",
    "from langchain.agents.agent_toolkits import SQLDatabaseToolkit\n",
    "\n",
    "toolkit = SQLDatabaseToolkit(db=db, llm=llm)\n",
    "agent = create_sql_agent(\n",
    "    llm=llm,\n",
    "    toolkit=toolkit,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "c484a76e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\u001B[1m> Entering new AgentExecutor chain...\u001B[0m\n",
      "\u001B[32;1m\u001B[1;3mAction: list_tables_sql_db\n",
      "Action Input: \u001B[0m\n",
      "Observation: \u001B[38;5;200m\u001B[1;3mtrips\u001B[0m\n",
      "Thought:\u001B[32;1m\u001B[1;3mI should check the schema of the trips table to see if it has the necessary columns for trip distance and duration.\n",
      "Action: schema_sql_db\n",
      "Action Input: trips\u001B[0m\n",
      "Observation: \u001B[33;1m\u001B[1;3m\n",
      "CREATE TABLE trips (\n",
      "\ttpep_pickup_datetime TIMESTAMP, \n",
      "\ttpep_dropoff_datetime TIMESTAMP, \n",
      "\ttrip_distance FLOAT, \n",
      "\tfare_amount FLOAT, \n",
      "\tpickup_zip INT, \n",
      "\tdropoff_zip INT\n",
      ") USING DELTA\n",
      "\n",
      "/*\n",
      "3 rows from trips table:\n",
      "tpep_pickup_datetime\ttpep_dropoff_datetime\ttrip_distance\tfare_amount\tpickup_zip\tdropoff_zip\n",
      "2016-02-14 16:52:13+00:00\t2016-02-14 17:16:04+00:00\t4.94\t19.0\t10282\t10171\n",
      "2016-02-04 18:44:19+00:00\t2016-02-04 18:46:00+00:00\t0.28\t3.5\t10110\t10110\n",
      "2016-02-17 17:13:57+00:00\t2016-02-17 17:17:55+00:00\t0.7\t5.0\t10103\t10023\n",
      "*/\u001B[0m\n",
      "Thought:\u001B[32;1m\u001B[1;3mThe trips table has the necessary columns for trip distance and duration. I will write a query to find the longest trip distance and its duration.\n",
      "Action: query_checker_sql_db\n",
      "Action Input: SELECT trip_distance, tpep_dropoff_datetime - tpep_pickup_datetime as duration FROM trips ORDER BY trip_distance DESC LIMIT 1\u001B[0m\n",
      "Observation: \u001B[31;1m\u001B[1;3mSELECT trip_distance, tpep_dropoff_datetime - tpep_pickup_datetime as duration FROM trips ORDER BY trip_distance DESC LIMIT 1\u001B[0m\n",
      "Thought:\u001B[32;1m\u001B[1;3mThe query is correct. I will now execute it to find the longest trip distance and its duration.\n",
      "Action: query_sql_db\n",
      "Action Input: SELECT trip_distance, tpep_dropoff_datetime - tpep_pickup_datetime as duration FROM trips ORDER BY trip_distance DESC LIMIT 1\u001B[0m\n",
      "Observation: \u001B[36;1m\u001B[1;3m[(30.6, '0 00:43:31.000000000')]\u001B[0m\n",
      "Thought:\u001B[32;1m\u001B[1;3mI now know the final answer.\n",
      "Final Answer: The longest trip distance is 30.6 miles and it took 43 minutes and 31 seconds.\u001B[0m\n",
      "\n",
      "\u001B[1m> Finished chain.\u001B[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": "'The longest trip distance is 30.6 miles and it took 43 minutes and 31 seconds.'"
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent.run(\"What is the longest trip distance and how long did it take?\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}