langchain/docs/extras/integrations/vectorstores/starrocks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "59723cea",
   "metadata": {},
   "source": [
    "# StarRocks\n",
    "\n",
    ">[StarRocks](https://www.starrocks.io/) is a High-Performance Analytical Database.\n",
    "`StarRocks` is a next-gen sub-second MPP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics and ad-hoc query.\n",
    "\n",
    ">Usually `StarRocks` is categorized into OLAP, and it has showed excellent performance in [ClickBench — a Benchmark For Analytical DBMS](https://benchmark.clickhouse.com/). Since it has a super-fast vectorized execution engine, it could also be used as a fast vectordb.\n",
    "\n",
    "Here we'll show how to use the StarRocks Vector Store."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1685854f",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "311d44bb-4aca-4f3b-8f97-5e1f29238e40",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install pymysql"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c891bba",
   "metadata": {},
   "source": [
    "Set `update_vectordb = False` at the beginning. If there is no docs updated, then we don't need to rebuild the embeddings of docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "3c85fb93",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/dirlt/utils/py3env/lib/python3.9/site-packages/requests/__init__.py:102: RequestsDependencyWarning: urllib3 (1.26.7) or chardet (5.1.0)/charset_normalizer (2.0.9) doesn't match a supported version!\n",
      "  warnings.warn(\"urllib3 ({}) or chardet ({})/charset_normalizer ({}) doesn't match a supported \"\n"
     ]
    }
   ],
   "source": [
    "from langchain.embeddings.openai import OpenAIEmbeddings\n",
    "from langchain.vectorstores import StarRocks\n",
    "from langchain.vectorstores.starrocks import StarRocksSettings\n",
    "from langchain.vectorstores import Chroma\n",
    "from langchain.text_splitter import CharacterTextSplitter, TokenTextSplitter\n",
    "from langchain import OpenAI, VectorDBQA\n",
    "from langchain.document_loaders import DirectoryLoader\n",
    "from langchain.chains import RetrievalQA\n",
    "from langchain.document_loaders import TextLoader, UnstructuredMarkdownLoader\n",
    "\n",
    "update_vectordb = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee821c00",
   "metadata": {},
   "source": [
    "## Load docs and split them into tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ba0cfd",
   "metadata": {},
   "source": [
    "Load all markdown files under the `docs` directory\n",
    "\n",
    "for starrocks documents, you can clone repo from https://github.com/StarRocks/starrocks, and there is `docs` directory in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "85912696",
   "metadata": {},
   "outputs": [],
   "source": [
    "loader = DirectoryLoader(\n",
    "    \"./docs\", glob=\"**/*.md\", loader_cls=UnstructuredMarkdownLoader\n",
    ")\n",
    "documents = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b415fe2a",
   "metadata": {},
   "source": [
    "Split docs into tokens, and set `update_vectordb = True` because there are new docs/tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "07e8acff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load text splitter and split docs into snippets of text\n",
    "text_splitter = TokenTextSplitter(chunk_size=400, chunk_overlap=50)\n",
    "split_docs = text_splitter.split_documents(documents)\n",
    "\n",
    "# tell vectordb to update text embeddings\n",
    "update_vectordb = True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1f365370",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Document(page_content='Compile StarRocks with Docker\\n\\nThis topic describes how to compile StarRocks using Docker.\\n\\nOverview\\n\\nStarRocks provides development environment images for both Ubuntu 22.04 and CentOS 7.9. With the image, you can launch a Docker container and compile StarRocks in the container.\\n\\nStarRocks version and DEV ENV image\\n\\nDifferent branches of StarRocks correspond to different development environment images provided on StarRocks Docker Hub.\\n\\nFor Ubuntu 22.04:\\n\\n| Branch name | Image name              |\\n  | --------------- | ----------------------------------- |\\n  | main            | starrocks/dev-env-ubuntu:latest     |\\n  | branch-3.0      | starrocks/dev-env-ubuntu:3.0-latest |\\n  | branch-2.5      | starrocks/dev-env-ubuntu:2.5-latest |\\n\\nFor CentOS 7.9:\\n\\n| Branch name | Image name                       |\\n  | --------------- | ------------------------------------ |\\n  | main            | starrocks/dev-env-centos7:latest     |\\n  | branch-3.0      | starrocks/dev-env-centos7:3.0-latest |\\n  | branch-2.5      | starrocks/dev-env-centos7:2.5-latest |\\n\\nPrerequisites\\n\\nBefore compiling StarRocks, make sure the following requirements are satisfied:\\n\\nHardware\\n\\n', metadata={'source': 'docs/developers/build-starrocks/Build_in_docker.md'})"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "split_docs[-20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "50012b29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# docs  = 657, # splits = 2802\n"
     ]
    }
   ],
   "source": [
    "print(\"# docs  = %d, # splits = %d\" % (len(documents), len(split_docs)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5371f152",
   "metadata": {},
   "source": [
    "## Create vectordb instance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15702d9c",
   "metadata": {},
   "source": [
    "### Use StarRocks as vectordb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ced7dbe1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def gen_starrocks(update_vectordb, embeddings, settings):\n",
    "    if update_vectordb:\n",
    "        docsearch = StarRocks.from_documents(split_docs, embeddings, config=settings)\n",
    "    else:\n",
    "        docsearch = StarRocks(embeddings, settings)\n",
    "    return docsearch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15d86fda",
   "metadata": {},
   "source": [
    "## Convert tokens into embeddings and put them into vectordb"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff1322ea",
   "metadata": {},
   "source": [
    "Here we use StarRocks as vectordb, you can configure StarRocks instance via `StarRocksSettings`.\n",
    "\n",
    "Configuring StarRocks instance is pretty much like configuring mysql instance. You need to specify:\n",
    "1. host/port\n",
    "2. username(default: 'root')\n",
    "3. password(default: '')\n",
    "4. database(default: 'default')\n",
    "5. table(default: 'langchain')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "26410d9b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Inserting data...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2802/2802 [02:26<00:00, 19.11it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[92m\u001b[1mzya.langchain @ 127.0.0.1:41003\u001b[0m\n",
      "\n",
      "\u001b[1musername: root\u001b[0m\n",
      "\n",
      "Table Schema:\n",
      "----------------------------------------------------------------------------\n",
      "|\u001b[94mname                    \u001b[0m|\u001b[96mtype                    \u001b[0m|\u001b[96mkey                     \u001b[0m|\n",
      "----------------------------------------------------------------------------\n",
      "|\u001b[94mid                      \u001b[0m|\u001b[96mvarchar(65533)          \u001b[0m|\u001b[96mtrue                    \u001b[0m|\n",
      "|\u001b[94mdocument                \u001b[0m|\u001b[96mvarchar(65533)          \u001b[0m|\u001b[96mfalse                   \u001b[0m|\n",
      "|\u001b[94membedding               \u001b[0m|\u001b[96marray<float>            \u001b[0m|\u001b[96mfalse                   \u001b[0m|\n",
      "|\u001b[94mmetadata                \u001b[0m|\u001b[96mvarchar(65533)          \u001b[0m|\u001b[96mfalse                   \u001b[0m|\n",
      "----------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "embeddings = OpenAIEmbeddings()\n",
    "\n",
    "# configure starrocks settings(host/port/user/pw/db)\n",
    "settings = StarRocksSettings()\n",
    "settings.port = 41003\n",
    "settings.host = \"127.0.0.1\"\n",
    "settings.username = \"root\"\n",
    "settings.password = \"\"\n",
    "settings.database = \"zya\"\n",
    "docsearch = gen_starrocks(update_vectordb, embeddings, settings)\n",
    "\n",
    "print(docsearch)\n",
    "\n",
    "update_vectordb = False"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bde66626",
   "metadata": {},
   "source": [
    "## Build QA and ask question to it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "84921814",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " No, profile is not enabled by default. To enable profile, set the variable `enable_profile` to `true` using the command `set enable_profile = true;`\n"
     ]
    }
   ],
   "source": [
    "llm = OpenAI()\n",
    "qa = RetrievalQA.from_chain_type(\n",
    "    llm=llm, chain_type=\"stuff\", retriever=docsearch.as_retriever()\n",
    ")\n",
    "query = \"is profile enabled by default? if not, how to enable profile?\"\n",
    "resp = qa.run(query)\n",
    "print(resp)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}