2023-12-13 21:26:53 +00:00
{
"cells": [
{
"cell_type": "markdown",
"id": "SzvBjdID1V3m",
"metadata": {
"id": "SzvBjdID1V3m"
},
"source": [
"# Multi-modal RAG with Google Cloud"
]
},
{
"cell_type": "markdown",
"id": "4tfidrmE1Zlo",
"metadata": {
"id": "4tfidrmE1Zlo"
},
"source": [
"This tutorial demonstrates how to implement the Option 2 described [here](https://github.com/langchain-ai/langchain/blob/master/cookbook/Multi_modal_RAG.ipynb) with Generative API on Google Cloud."
]
},
{
"cell_type": "markdown",
"id": "84fcd59f-2eaf-4a76-ad1a-96d6db70bf42",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"Install the required dependencies, and create an API key for your Google service."
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "6b1e10dd-25de-4c0a-9577-f36e72518f89",
"metadata": {},
"outputs": [],
"source": [
"%pip install -U --quiet langchain langchain_community openai chromadb langchain-experimental\n",
"%pip install --quiet \"unstructured[all-docs]\" pypdf pillow pydantic lxml pillow matplotlib chromadb tiktoken"
]
},
{
"cell_type": "markdown",
"id": "pSInKtCZ32mt",
"metadata": {
"id": "pSInKtCZ32mt"
},
"source": [
"## Data loading"
]
},
{
"cell_type": "markdown",
"id": "Iv2R8-lJ37dG",
"metadata": {
"id": "Iv2R8-lJ37dG"
},
"source": [
"We use a zip file with a sub-set of the extracted images and pdf from [this](https://cloudedjudgement.substack.com/p/clouded-judgement-111023) blog post. If you want to follow the full flow, please, use the original [example](https://github.com/langchain-ai/langchain/blob/master/cookbook/Multi_modal_RAG.ipynb)."
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "d999f3fe-c165-4772-b63e-ffe4dd5b03cf",
"metadata": {},
"outputs": [],
"source": [
"# First download\n",
"import logging\n",
"import zipfile\n",
"\n",
"import requests\n",
"\n",
"logging.basicConfig(level=logging.INFO)\n",
"\n",
"data_url = \"https://storage.googleapis.com/benchmarks-artifacts/langchain-docs-benchmarking/cj.zip\"\n",
"result = requests.get(data_url)\n",
"filename = \"cj.zip\"\n",
"with open(filename, \"wb\") as file:\n",
" file.write(result.content)\n",
"\n",
"with zipfile.ZipFile(filename, \"r\") as zip_ref:\n",
" zip_ref.extractall()"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "eGUfuevMUA6R",
"metadata": {},
"outputs": [],
"source": [
2024-01-02 21:47:11 +00:00
"from langchain_community.document_loaders import PyPDFLoader\n",
2023-12-13 21:26:53 +00:00
"\n",
"loader = PyPDFLoader(\"./cj/cj.pdf\")\n",
"docs = loader.load()\n",
"tables = []\n",
"texts = [d.page_content for d in docs]"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "Fst17fNHWYcq",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"21"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(texts)"
]
},
{
"cell_type": "markdown",
"id": "vjfcg_Vn3_1C",
"metadata": {
"id": "vjfcg_Vn3_1C"
},
"source": [
"## Multi-vector retriever"
]
},
{
"cell_type": "markdown",
"id": "1ynRqJn04BFG",
"metadata": {
"id": "1ynRqJn04BFG"
},
"source": [
"Let's generate text and image summaries and save them to a ChromaDB vectorstore."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "kWDWfSDBMPl8",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:numexpr.utils:Note: NumExpr detected 12 cores but \"NUMEXPR_MAX_THREADS\" not set, so enforcing safe limit of 8.\n",
"INFO:numexpr.utils:NumExpr defaulting to 8 threads.\n"
]
}
],
"source": [
"from langchain.prompts import PromptTemplate\n",
2024-01-02 20:32:16 +00:00
"from langchain_community.chat_models import ChatVertexAI\n",
"from langchain_community.llms import VertexAI\n",
2023-12-13 21:26:53 +00:00
"from langchain_core.messages import AIMessage\n",
2024-01-02 20:09:45 +00:00
"from langchain_core.output_parsers import StrOutputParser\n",
2023-12-13 21:26:53 +00:00
"from langchain_core.runnables import RunnableLambda\n",
"\n",
"\n",
"# Generate summaries of text elements\n",
"def generate_text_summaries(texts, tables, summarize_texts=False):\n",
" \"\"\"\n",
" Summarize text elements\n",
" texts: List of str\n",
" tables: List of str\n",
" summarize_texts: Bool to summarize texts\n",
" \"\"\"\n",
"\n",
" # Prompt\n",
" prompt_text = \"\"\"You are an assistant tasked with summarizing tables and text for retrieval. \\\n",
" These summaries will be embedded and used to retrieve the raw text or table elements. \\\n",
" Give a concise summary of the table or text that is well optimized for retrieval. Table or text: {element} \"\"\"\n",
" prompt = PromptTemplate.from_template(prompt_text)\n",
" empty_response = RunnableLambda(\n",
" lambda x: AIMessage(content=\"Error processing document\")\n",
" )\n",
" # Text summary chain\n",
" model = VertexAI(\n",
" temperature=0, model_name=\"gemini-pro\", max_output_tokens=1024\n",
" ).with_fallbacks([empty_response])\n",
" summarize_chain = {\"element\": lambda x: x} | prompt | model | StrOutputParser()\n",
"\n",
" # Initialize empty summaries\n",
" text_summaries = []\n",
" table_summaries = []\n",
"\n",
" # Apply to text if texts are provided and summarization is requested\n",
" if texts and summarize_texts:\n",
" text_summaries = summarize_chain.batch(texts, {\"max_concurrency\": 1})\n",
" elif texts:\n",
" text_summaries = texts\n",
"\n",
" # Apply to tables if tables are provided\n",
" if tables:\n",
" table_summaries = summarize_chain.batch(tables, {\"max_concurrency\": 1})\n",
"\n",
" return text_summaries, table_summaries\n",
"\n",
"\n",
"# Get text, table summaries\n",
"text_summaries, table_summaries = generate_text_summaries(\n",
" texts, tables, summarize_texts=True\n",
")"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "F0NnyUl48yYb",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"21"
]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(text_summaries)"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "PeK9bzXv3olF",
"metadata": {},
"outputs": [],
"source": [
"import base64\n",
"import os\n",
"\n",
2024-01-02 20:09:45 +00:00
"from langchain_core.messages import HumanMessage\n",
2023-12-13 21:26:53 +00:00
"\n",
"\n",
"def encode_image(image_path):\n",
" \"\"\"Getting the base64 string\"\"\"\n",
" with open(image_path, \"rb\") as image_file:\n",
" return base64.b64encode(image_file.read()).decode(\"utf-8\")\n",
"\n",
"\n",
"def image_summarize(img_base64, prompt):\n",
" \"\"\"Make image summary\"\"\"\n",
" model = ChatVertexAI(model_name=\"gemini-pro-vision\", max_output_tokens=1024)\n",
"\n",
" msg = model(\n",
" [\n",
" HumanMessage(\n",
" content=[\n",
" {\"type\": \"text\", \"text\": prompt},\n",
" {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\"url\": f\"data:image/jpeg;base64,{img_base64}\"},\n",
" },\n",
" ]\n",
" )\n",
" ]\n",
" )\n",
" return msg.content\n",
"\n",
"\n",
"def generate_img_summaries(path):\n",
" \"\"\"\n",
" Generate summaries and base64 encoded strings for images\n",
" path: Path to list of .jpg files extracted by Unstructured\n",
" \"\"\"\n",
"\n",
" # Store base64 encoded images\n",
" img_base64_list = []\n",
"\n",
" # Store image summaries\n",
" image_summaries = []\n",
"\n",
" # Prompt\n",
" prompt = \"\"\"You are an assistant tasked with summarizing images for retrieval. \\\n",
" These summaries will be embedded and used to retrieve the raw image. \\\n",
" Give a concise summary of the image that is well optimized for retrieval.\"\"\"\n",
"\n",
" # Apply to images\n",
" for img_file in sorted(os.listdir(path)):\n",
" if img_file.endswith(\".jpg\"):\n",
" img_path = os.path.join(path, img_file)\n",
" base64_image = encode_image(img_path)\n",
" img_base64_list.append(base64_image)\n",
" image_summaries.append(image_summarize(base64_image, prompt))\n",
"\n",
" return img_base64_list, image_summaries\n",
"\n",
"\n",
"# Image summaries\n",
"img_base64_list, image_summaries = generate_img_summaries(\"./cj\")"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "6WDYpDFzjocl",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"5"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(image_summaries)"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "cWyWfZ-XB6cS",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"INFO:chromadb.telemetry.product.posthog:Anonymized telemetry enabled. See https://docs.trychroma.com/telemetry for more information.\n"
]
}
],
"source": [
"import uuid\n",
"\n",
"from langchain.retrievers.multi_vector import MultiVectorRetriever\n",
"from langchain.storage import InMemoryStore\n",
2024-01-02 20:32:16 +00:00
"from langchain_community.embeddings import VertexAIEmbeddings\n",
2024-01-02 21:47:11 +00:00
"from langchain_community.vectorstores import Chroma\n",
2024-01-02 20:09:45 +00:00
"from langchain_core.documents import Document\n",
2023-12-13 21:26:53 +00:00
"\n",
"\n",
"def create_multi_vector_retriever(\n",
" vectorstore, text_summaries, texts, table_summaries, tables, image_summaries, images\n",
"):\n",
" \"\"\"\n",
" Create retriever that indexes summaries, but returns raw images or texts\n",
" \"\"\"\n",
"\n",
" # Initialize the storage layer\n",
" store = InMemoryStore()\n",
" id_key = \"doc_id\"\n",
"\n",
" # Create the multi-vector retriever\n",
" retriever = MultiVectorRetriever(\n",
" vectorstore=vectorstore,\n",
" docstore=store,\n",
" id_key=id_key,\n",
" )\n",
"\n",
" # Helper function to add documents to the vectorstore and docstore\n",
" def add_documents(retriever, doc_summaries, doc_contents):\n",
" doc_ids = [str(uuid.uuid4()) for _ in doc_contents]\n",
" summary_docs = [\n",
" Document(page_content=s, metadata={id_key: doc_ids[i]})\n",
" for i, s in enumerate(doc_summaries)\n",
" ]\n",
" retriever.vectorstore.add_documents(summary_docs)\n",
" retriever.docstore.mset(list(zip(doc_ids, doc_contents)))\n",
"\n",
" # Add texts, tables, and images\n",
" # Check that text_summaries is not empty before adding\n",
" if text_summaries:\n",
" add_documents(retriever, text_summaries, texts)\n",
" # Check that table_summaries is not empty before adding\n",
" if table_summaries:\n",
" add_documents(retriever, table_summaries, tables)\n",
" # Check that image_summaries is not empty before adding\n",
" if image_summaries:\n",
" add_documents(retriever, image_summaries, images)\n",
"\n",
" return retriever\n",
"\n",
"\n",
"# The vectorstore to use to index the summaries\n",
"vectorstore = Chroma(\n",
" collection_name=\"mm_rag_cj_blog\",\n",
" embedding_function=VertexAIEmbeddings(model_name=\"textembedding-gecko@latest\"),\n",
")\n",
"\n",
"# Create retriever\n",
"retriever_multi_vector_img = create_multi_vector_retriever(\n",
" vectorstore,\n",
" text_summaries,\n",
" texts,\n",
" table_summaries,\n",
" tables,\n",
" image_summaries,\n",
" img_base64_list,\n",
")"
]
},
{
"cell_type": "markdown",
"id": "NGDkkMFfCg4j",
"metadata": {
"id": "NGDkkMFfCg4j"
},
"source": [
"## Building a RAG"
]
},
{
"cell_type": "markdown",
"id": "8TzOcHVsCmBc",
"metadata": {
"id": "8TzOcHVsCmBc"
},
"source": [
"Let's build a retriever:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "GlwCErBaCKQW",
"metadata": {},
"outputs": [],
"source": [
"import io\n",
"import re\n",
"\n",
"from IPython.display import HTML, display\n",
2024-01-02 20:09:45 +00:00
"from langchain_core.runnables import RunnableLambda, RunnablePassthrough\n",
2023-12-13 21:26:53 +00:00
"from PIL import Image\n",
"\n",
"\n",
"def plt_img_base64(img_base64):\n",
" \"\"\"Disply base64 encoded string as image\"\"\"\n",
" # Create an HTML img tag with the base64 string as the source\n",
" image_html = f'<img src=\"data:image/jpeg;base64,{img_base64}\" />'\n",
" # Display the image by rendering the HTML\n",
" display(HTML(image_html))\n",
"\n",
"\n",
"def looks_like_base64(sb):\n",
" \"\"\"Check if the string looks like base64\"\"\"\n",
" return re.match(\"^[A-Za-z0-9+/]+[=]{0,2}$\", sb) is not None\n",
"\n",
"\n",
"def is_image_data(b64data):\n",
" \"\"\"\n",
" Check if the base64 data is an image by looking at the start of the data\n",
" \"\"\"\n",
" image_signatures = {\n",
" b\"\\xFF\\xD8\\xFF\": \"jpg\",\n",
" b\"\\x89\\x50\\x4E\\x47\\x0D\\x0A\\x1A\\x0A\": \"png\",\n",
" b\"\\x47\\x49\\x46\\x38\": \"gif\",\n",
" b\"\\x52\\x49\\x46\\x46\": \"webp\",\n",
" }\n",
" try:\n",
" header = base64.b64decode(b64data)[:8] # Decode and get the first 8 bytes\n",
" for sig, format in image_signatures.items():\n",
" if header.startswith(sig):\n",
" return True\n",
" return False\n",
" except Exception:\n",
" return False\n",
"\n",
"\n",
"def resize_base64_image(base64_string, size=(128, 128)):\n",
" \"\"\"\n",
" Resize an image encoded as a Base64 string\n",
" \"\"\"\n",
" # Decode the Base64 string\n",
" img_data = base64.b64decode(base64_string)\n",
" img = Image.open(io.BytesIO(img_data))\n",
"\n",
" # Resize the image\n",
" resized_img = img.resize(size, Image.LANCZOS)\n",
"\n",
" # Save the resized image to a bytes buffer\n",
" buffered = io.BytesIO()\n",
" resized_img.save(buffered, format=img.format)\n",
"\n",
" # Encode the resized image to Base64\n",
" return base64.b64encode(buffered.getvalue()).decode(\"utf-8\")\n",
"\n",
"\n",
"def split_image_text_types(docs):\n",
" \"\"\"\n",
" Split base64-encoded images and texts\n",
" \"\"\"\n",
" b64_images = []\n",
" texts = []\n",
" for doc in docs:\n",
" # Check if the document is of type Document and extract page_content if so\n",
" if isinstance(doc, Document):\n",
" doc = doc.page_content\n",
" if looks_like_base64(doc) and is_image_data(doc):\n",
" doc = resize_base64_image(doc, size=(1300, 600))\n",
" b64_images.append(doc)\n",
" else:\n",
" texts.append(doc)\n",
" if len(b64_images) > 0:\n",
" return {\"images\": b64_images[:1], \"texts\": []}\n",
" return {\"images\": b64_images, \"texts\": texts}\n",
"\n",
"\n",
"def img_prompt_func(data_dict):\n",
" \"\"\"\n",
" Join the context into a single string\n",
" \"\"\"\n",
" formatted_texts = \"\\n\".join(data_dict[\"context\"][\"texts\"])\n",
" messages = []\n",
"\n",
" # Adding the text for analysis\n",
" text_message = {\n",
" \"type\": \"text\",\n",
" \"text\": (\n",
" \"You are financial analyst tasking with providing investment advice.\\n\"\n",
" \"You will be given a mixed of text, tables, and image(s) usually of charts or graphs.\\n\"\n",
" \"Use this information to provide investment advice related to the user question. \\n\"\n",
" f\"User-provided question: {data_dict['question']}\\n\\n\"\n",
" \"Text and / or tables:\\n\"\n",
" f\"{formatted_texts}\"\n",
" ),\n",
" }\n",
" messages.append(text_message)\n",
" # Adding image(s) to the messages if present\n",
" if data_dict[\"context\"][\"images\"]:\n",
" for image in data_dict[\"context\"][\"images\"]:\n",
" image_message = {\n",
" \"type\": \"image_url\",\n",
" \"image_url\": {\"url\": f\"data:image/jpeg;base64,{image}\"},\n",
" }\n",
" messages.append(image_message)\n",
" return [HumanMessage(content=messages)]\n",
"\n",
"\n",
"def multi_modal_rag_chain(retriever):\n",
" \"\"\"\n",
" Multi-modal RAG chain\n",
" \"\"\"\n",
"\n",
" # Multi-modal LLM\n",
" model = ChatVertexAI(\n",
" temperature=0, model_name=\"gemini-pro-vision\", max_output_tokens=1024\n",
" )\n",
"\n",
" # RAG pipeline\n",
" chain = (\n",
" {\n",
" \"context\": retriever | RunnableLambda(split_image_text_types),\n",
" \"question\": RunnablePassthrough(),\n",
" }\n",
" | RunnableLambda(img_prompt_func)\n",
" | model\n",
" | StrOutputParser()\n",
" )\n",
"\n",
" return chain\n",
"\n",
"\n",
"# Create RAG chain\n",
"chain_multimodal_rag = multi_modal_rag_chain(retriever_multi_vector_img)"
]
},
{
"cell_type": "markdown",
"id": "BS4hNKqCCp8u",
"metadata": {
"id": "BS4hNKqCCp8u"
},
"source": [
"Let's check that we get images as documents:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "Q7GrwFC_FGwr",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"4"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"query = \"What are the EV / NTM and NTM rev growth for MongoDB, Cloudflare, and Datadog?\"\n",
"docs = retriever_multi_vector_img.get_relevant_documents(query, limit=1)\n",
"\n",
"# We get 2 docs\n",
"len(docs)"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "unnxB5M_FLCD",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"<img src=\"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wAARCAKyBe0DASIAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD3+iiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiikJoAWim5paBXFopKM0DFopM0UALRSUZoAWim55pc0CuLQaTNFAwoNFHWgQoopBS0DCiiigAooooAKKTNGaAFpKM0UALSUUUAFLSUZoAWkoozQIWkozRQhhRRRmgApaSloAKKKKACiikoAWikzRmgBaQ0ZooAKKXtSUgsKKKTNGaYhaKSjNAxaKTNGaAClpKKBBRRRQMWiikoAKXtSUZoAKKKKBAaKM0UDFopKM0CuLSGijNAwoozRQIWikooGLRRRQAUUUUAFFFFABRRRQAUUUmaAFopM0maQri0tJRTGKKSiigAFFAzRQDCg0UUCClopKBi0lLRQAUUUUAFFFJQAtBpM0UAFFFFAhTSUUUDCiiigBe1JRmigApaSjtQAtFIKWgAooooAKKKKACiiigAooooAKKTNFAri0UmaM0DFopM0ZoAWiiigAooooAKKKKACiikoAWikzRQAtFFJQAtFJzmloAKKSjNAC0UmaKAFopKKAFopM0maAHUlGaKAFopuaWkAtFJS0wEpaKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKACiiigAooooAKKKKAEry7xNbT3/jJ7RLwW6uPvO5CivUq8r8UWtveeM3hurtbWFh80rdqun8RnU+EfF4NuvNXGv2rHPQSnn9au+K1m0jXNNv0kYRnajYJwT3/AEqjaeH9AtruKf8A4Se3JjYNjAGf1rq/F1it/wCGHaP940QDoR39/wAquUrNNmcYXujQ1bU0tfD018rj/VbkPqSOKxfAVpKunS307OXuGJG45+WuRk1qTWNF03Ro8+aXCsM+nAr1TT7ZLSxhgQYVFAxSklFepUXzNLsJqd0LPTLi4Y4EaFs1keDoZI9EWWRmZp2L8nOKb41mZNAeBPvzsIx+NbOmwC1063hHG1BULSLNHrJHIfEh5EtLUxuyncfunGaseGNcbUPD1zbTMRdW8TAg9SMdag+IvMFl/v1meJbSfQLmPVbIYiuI9kqjpkjFXBJxszOV+a6NHwSkt5oN9D5rb3YqGYk461zviDwzqOiQJPLqXmLI+0Kpbiun+HB3aVOf+mn+NP8AiIP+JXa/9dqbdp2EtYNmLb+E9QsdPbU5NQDx+Rv2AtnkZql4O1Ca01yEzyP5M+VyzZGea767/wCRMb/r1H8q88jiMXhi31BF+a3usk+3FVF8zaYnGyTRf8d3kt1rbQwSuqW0W5sEgdq7Twg5fwzZFiSTH1J9zXBQh9R0HWdWmyXkO1fpzxXeeD/+RXsf+uf9amStGwJ807m6KWkFLWJugooooGFeefEDzJNSsYY5zF5mFzuIAzXoRrzr4hRxzalYpLJ5aHgueij1qofEianwspDwXc4GPENr/wB/T/jV/wAU6dc6foGnTxzs8lqQHdGOG54rKXwz4eyCfE8Hr0H+Nd7PZ22oeGXtoZVniMR2OvRiOn61pKVramMI3ZNpuox3OhRXhcYEQLN7gc1zHgyObUtYvdWldym4qg3cZ71zUOuSWfhi60YkrN5uwc84Oc/rXo3hfTf7N0O3iK4kZdz+5oaUbscZN2ibLHahPoK5bwnvurzUdQLMUllKrk8cVsa9dix0S6nJwVQ4+tV/Cdt9n8PW2R8zrvb6mslszSS1QviVdTk0lo9LUmdjgkEAgfjXF6xolx4d06DU01C4+2bhuRmBBPtXoWp6jBpdlJdXDYjQZ+v0rhbW3v8Axpqi3l4DHp0TZjQd6umTUaLPiPWby5sdLsYnMUl4gMjDr0qs1rP4O1yxaO8lmt7khHWQ55P/AOurHi+JbXW9HkxiBPlyBwKXxiwutV0eCEhpBKrkDsMjmrjboKWt/Qua1pWq61riQGSSHTAoO+NgCTWdo7T6H4y/smG7kuLZlJIc5IODWx4n8S/2NHHaWgD30qgKOuKg8M+H57BJdUvi0moTKWwecHFJOy8hT38x3jPW5Le1/s2yJN3OOdv8K0nw9eR9Ck8xyzCZsknPYVzwbWra41C8utEnnllyqysCAi/l9K0fh1d3QgktfsbeRvZjPngNjpQo+42Nu8kd/TqaKdWJsFFFFIAryvXrO41Lxrc2iXwtlIzukchRjNeqV5P4gs7W+8bXEF3eLaQkZMrdjzWlP4jOp8JatvBt0tzGRr1q+GB2iQkn9ateJTNo3iyyvxI4hlwrDJx15/nVKx0HQbK+huE8S27tG2QuAM/rXT+NrEX/AIcMqfM0R3qR6d60bs0ZxV4s0de1JbLw7Pdq4DGP937k1meA7KSHRvtczOZbg7vmOeO1cfNrL6/ZaVpKZ8wNhxn04H6V6paQJa2scMYwiLgCoa5UWndog1i6FnpF1OTjYhrO8IQvD4egMhZnk/eEk561X8bTMNKjtk+/cShP8/lXQWcAtrSKFfuooAqb2iVe8tDh/iRJLGLMxOyn0U4zV3Q9bOpeFLuKViLuCBg479ODVb4gAG500H+//Ws3xRa3Hh7UH1CzH+j3iFJFHTJHP86uKTjZkSvzXRq+Dopb7wvdweaQ8jEBmJOK5jX/AA1qOhxRPNqPmCVtoClhj/Oa674dnOiOf9s1F8Rf+PSw/wCu3+FPVTsS/eg2ZK+FNQ0zTZtQl1ASJ5O7YC2ecVV8D381trUSzyOY7hSBubIJHT+dd1rA/wCKOl/691/kK8+VTaeG9M1ONMtBcEsfb5cU4yve4pRslYseNbya81udYZXWO1T5sEgdv8a73wsxfw1YFiSTCuSa88SN7rw1qeqTZMk0mAfavQvCn/Is2H/XFaUlaNi4u8rmzRS4orE1CkpaKBmZr5I0K8Kkg+XwRXn/AIF12Szvvsl07eTOcIzEn5q9A8Qf8gG8/wCudef2OjNqHgdLq3GLq2lZ0I64GCf5VpTtZ3Mqid1Y1llc/EqVd7bMDjPHQUniXwnqF1dXWpRaiscQBfy8tnisbwtfS6j4uSecYkK7W/Dj+lek6v8A8ge7/wCuTfyon7iVhU3zN3PL9A8MahrcD3EWoCMRyYIZm5qtqj3dl4gmCyyMbZgThjjg12vw7/5BFz/12NYclqLzxdrEBG4tHJge+Diteb3rMz5bw5jY8V6vu8LWogdt91hQV+n+Iql8OzKl7qEMsjOybQckn
],
"text/plain": [
"<IPython.core.display.HTML object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plt_img_base64(docs[0])"
]
},
{
"cell_type": "markdown",
"id": "YUkGZXqsCtF6",
"metadata": {
"id": "YUkGZXqsCtF6"
},
"source": [
"And let's run our RAG on the same query:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "LsPTehdK-T-_",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"' | Company | EV / NTM Rev | NTM Rev Growth |\\n|---|---|---|\\n| MongoDB | 14.6x | 17% |\\n| Cloudflare | 13.4x | 28% |\\n| Datadog | 13.1x | 19% |'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"chain_multimodal_rag.invoke(query)"
]
},
{
"cell_type": "markdown",
"id": "XpLQB6dEfQX-",
"metadata": {
"id": "XpLQB6dEfQX-"
},
"source": [
"As we can see, the model was able to figure out the the right values that are relevant to answer the question."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.2"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
