langchain/docs/extras/integrations/document_loaders/image_captions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ddb208a0-617e-433e-b9de-69099bc456f8",
   "metadata": {},
   "source": [
    "# Image captions\n",
    "\n",
    "By default, the loader utilizes the pre-trained [Salesforce BLIP image captioning model](https://huggingface.co/Salesforce/blip-image-captioning-base).\n",
    "\n",
    "\n",
    "This notebook shows how to use the `ImageCaptionLoader` to generate a query-able index of image captions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f78585a-a2fa-4ece-834f-66692b959efb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac0a2a76-c36a-4952-b511-7906ca840e08",
   "metadata": {
    "pycharm": {
     "is_executing": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.document_loaders import ImageCaptionLoader"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faefe80f-08f2-4683-a325-4efd61fae0bf",
   "metadata": {},
   "source": [
    "### Prepare a list of image urls from Wikimedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a400568-5fea-47e6-8703-d9c1a1cc00ea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "list_image_urls = [\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/thumb/5/5a/Hyla_japonica_sep01.jpg/260px-Hyla_japonica_sep01.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/thumb/7/71/Tibur%C3%B3n_azul_%28Prionace_glauca%29%2C_canal_Fayal-Pico%2C_islas_Azores%2C_Portugal%2C_2020-07-27%2C_DD_14.jpg/270px-Tibur%C3%B3n_azul_%28Prionace_glauca%29%2C_canal_Fayal-Pico%2C_islas_Azores%2C_Portugal%2C_2020-07-27%2C_DD_14.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/thumb/2/21/Thure_de_Thulstrup_-_Battle_of_Shiloh.jpg/251px-Thure_de_Thulstrup_-_Battle_of_Shiloh.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/thumb/2/21/Passion_fruits_-_whole_and_halved.jpg/270px-Passion_fruits_-_whole_and_halved.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/thumb/5/5e/Messier83_-_Heic1403a.jpg/277px-Messier83_-_Heic1403a.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/thumb/b/b6/2022-01-22_Men%27s_World_Cup_at_2021-22_St._Moritz%E2%80%93Celerina_Luge_World_Cup_and_European_Championships_by_Sandro_Halank%E2%80%93257.jpg/288px-2022-01-22_Men%27s_World_Cup_at_2021-22_St._Moritz%E2%80%93Celerina_Luge_World_Cup_and_European_Championships_by_Sandro_Halank%E2%80%93257.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/thumb/9/99/Wiesen_Pippau_%28Crepis_biennis%29-20220624-RM-123950.jpg/224px-Wiesen_Pippau_%28Crepis_biennis%29-20220624-RM-123950.jpg\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be585acd-6e28-4400-9e8f-17fdde11e02c",
   "metadata": {},
   "source": [
    "### Create the loader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb392517-72d8-416e-852c-da90b77267ed",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "loader = ImageCaptionLoader(path_images=list_image_urls)\n",
    "list_docs = loader.load()\n",
    "list_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f56db67-99bb-4543-ba40-1871a58b2da5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from PIL import Image\n",
    "import requests\n",
    "\n",
    "Image.open(requests.get(list_image_urls[0], stream=True).raw).convert(\"RGB\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "52193308-e2c5-4757-8f86-a73c07510f73",
   "metadata": {},
   "source": [
    "### Create the index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b7a15ac-d2c7-4359-9c5c-a543c8eebf80",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from langchain.indexes import VectorstoreIndexCreator\n",
    "\n",
    "index = VectorstoreIndexCreator().from_loaders([loader])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "677398d8-6ab7-4224-8e4a-4b94a7fb2a94",
   "metadata": {},
   "source": [
    "### Query"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e03e31c6-3018-434d-bcad-5c25144509e1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "query = \"What's the painting about?\"\n",
    "index.query(query)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3ec2b5a-9c03-4e32-b571-be5af9a22223",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "query = \"What kind of images are there?\"\n",
    "index.query(query)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}