{ "cells": [ { "cell_type": "markdown", "id": "fd7e6164", "metadata": {}, "source": [ "# Using GPT-4V to tag & caption images\n", "\n", "This notebook explores how to leverage GPT-4V to tag & caption images. \n", "\n", "We can leverage the multimodal capabilities of GPT-4V to provide input images along with additional context on what they represent, and prompt the model to output tags or image descriptions. The image descriptions can then be further refined with a language model (in this notebook, we'll use GPT-4-turbo) to generate captions. \n", "\n", "Generating text content from images can be useful for multiple use cases, especially use cases involving search. \n", "We will illustrate a search use case in this notebook by using generated keywords and product captions to search for products - both from a text input and an image input.\n", "\n", "As an example, we will use a dataset of Amazon furniture items, tag them with relevant keywords and generate short, descriptive captions." ] }, { "cell_type": "markdown", "id": "a00b5267", "metadata": {}, "source": [ "## Setup" ] }, { "cell_type": "code", "execution_count": null, "id": "176a7836", "metadata": {}, "outputs": [], "source": [ "# Install dependencies if needed\n", "%pip install openai\n", "%pip install scikit-learn" ] }, { "cell_type": "code", "execution_count": 2, "id": "5b4c63de", "metadata": {}, "outputs": [], "source": [ "from IPython.display import Image, display\n", "import pandas as pd\n", "from sklearn.metrics.pairwise import cosine_similarity\n", "import numpy as np\n", "from openai import OpenAI\n", "\n", "# Initializing OpenAI client - see https://platform.openai.com/docs/quickstart?context=python\n", "client = OpenAI()" ] }, { "cell_type": "code", "execution_count": 14, "id": "1b0fa62d", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", " | asin | \n", "url | \n", "title | \n", "brand | \n", "price | \n", "availability | \n", "categories | \n", "primary_image | \n", "images | \n", "upc | \n", "... | \n", "color | \n", "material | \n", "style | \n", "important_information | \n", "product_overview | \n", "about_item | \n", "description | \n", "specifications | \n", "uniq_id | \n", "scraped_at | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "B0CJHKVG6P | \n", "https://www.amazon.com/dp/B0CJHKVG6P | \n", "GOYMFK 1pc Free Standing Shoe Rack, Multi-laye... | \n", "GOYMFK | \n", "$24.99 | \n", "Only 13 left in stock - order soon. | \n", "['Home & Kitchen', 'Storage & Organization', '... | \n", "https://m.media-amazon.com/images/I/416WaLx10j... | \n", "['https://m.media-amazon.com/images/I/416WaLx1... | \n", "NaN | \n", "... | \n", "White | \n", "Metal | \n", "Modern | \n", "[] | \n", "[{'Brand': ' GOYMFK '}, {'Color': ' White '}, ... | \n", "['Multiple layers: Provides ample storage spac... | \n", "multiple shoes, coats, hats, and other items E... | \n", "['Brand: GOYMFK', 'Color: White', 'Material: M... | \n", "02593e81-5c09-5069-8516-b0b29f439ded | \n", "2024-02-02 15:15:08 | \n", "
1 | \n", "B0B66QHB23 | \n", "https://www.amazon.com/dp/B0B66QHB23 | \n", "subrtex Leather ding Room, Dining Chairs Set o... | \n", "subrtex | \n", "NaN | \n", "NaN | \n", "['Home & Kitchen', 'Furniture', 'Dining Room F... | \n", "https://m.media-amazon.com/images/I/31SejUEWY7... | \n", "['https://m.media-amazon.com/images/I/31SejUEW... | \n", "NaN | \n", "... | \n", "Black | \n", "Sponge | \n", "Black Rubber Wood | \n", "[] | \n", "NaN | \n", "['【Easy Assembly】: Set of 2 dining room chairs... | \n", "subrtex Dining chairs Set of 2 | \n", "['Brand: subrtex', 'Color: Black', 'Product Di... | \n", "5938d217-b8c5-5d3e-b1cf-e28e340f292e | \n", "2024-02-02 15:15:09 | \n", "
2 | \n", "B0BXRTWLYK | \n", "https://www.amazon.com/dp/B0BXRTWLYK | \n", "Plant Repotting Mat MUYETOL Waterproof Transpl... | \n", "MUYETOL | \n", "$5.98 | \n", "In Stock | \n", "['Patio, Lawn & Garden', 'Outdoor Décor', 'Doo... | \n", "https://m.media-amazon.com/images/I/41RgefVq70... | \n", "['https://m.media-amazon.com/images/I/41RgefVq... | \n", "NaN | \n", "... | \n", "Green | \n", "Polyethylene | \n", "Modern | \n", "[] | \n", "[{'Brand': ' MUYETOL '}, {'Size': ' 26.8*26.8 ... | \n", "['PLANT REPOTTING MAT SIZE: 26.8\" x 26.8\", squ... | \n", "NaN | \n", "['Brand: MUYETOL', 'Size: 26.8*26.8', 'Item We... | \n", "b2ede786-3f51-5a45-9a5b-bcf856958cd8 | \n", "2024-02-02 15:15:09 | \n", "
3 | \n", "B0C1MRB2M8 | \n", "https://www.amazon.com/dp/B0C1MRB2M8 | \n", "Pickleball Doormat, Welcome Doormat Absorbent ... | \n", "VEWETOL | \n", "$13.99 | \n", "Only 10 left in stock - order soon. | \n", "['Patio, Lawn & Garden', 'Outdoor Décor', 'Doo... | \n", "https://m.media-amazon.com/images/I/61vz1Igler... | \n", "['https://m.media-amazon.com/images/I/61vz1Igl... | \n", "NaN | \n", "... | \n", "A5589 | \n", "Rubber | \n", "Modern | \n", "[] | \n", "[{'Brand': ' VEWETOL '}, {'Size': ' 16*24INCH ... | \n", "['Specifications: 16x24 Inch ', \" High-Quality... | \n", "The decorative doormat features a subtle textu... | \n", "['Brand: VEWETOL', 'Size: 16*24INCH', 'Materia... | \n", "8fd9377b-cfa6-5f10-835c-6b8eca2816b5 | \n", "2024-02-02 15:15:10 | \n", "
4 | \n", "B0CG1N9QRC | \n", "https://www.amazon.com/dp/B0CG1N9QRC | \n", "JOIN IRON Foldable TV Trays for Eating Set of ... | \n", "JOIN IRON Store | \n", "$89.99 | \n", "Usually ships within 5 to 6 weeks | \n", "['Home & Kitchen', 'Furniture', 'Game & Recrea... | \n", "https://m.media-amazon.com/images/I/41p4d4VJnN... | \n", "['https://m.media-amazon.com/images/I/41p4d4VJ... | \n", "NaN | \n", "... | \n", "Grey Set of 4 | \n", "Iron | \n", "X Classic Style | \n", "[] | \n", "NaN | \n", "['Includes 4 Folding Tv Tray Tables And one Co... | \n", "Set of Four Folding Trays With Matching Storag... | \n", "['Brand: JOIN IRON', 'Shape: Rectangular', 'In... | \n", "bdc9aa30-9439-50dc-8e89-213ea211d66a | \n", "2024-02-02 15:15:11 | \n", "
5 rows × 25 columns
\n", "\n", " | keyword | \n", "embedding | \n", "
---|---|---|
0 | \n", "industrial | \n", "[-0.026137426, 0.021297162, -0.007273361, -0.0... | \n", "
1 | \n", "metal | \n", "[-0.020530997, 0.004478126, -0.011049379, -0.0... | \n", "
2 | \n", "wood | \n", "[0.013877833, 0.02955235, 0.0006239023, -0.035... | \n", "
3 | \n", "vintage | \n", "[-0.05235507, 0.008213689, -0.015532949, 0.002... | \n", "
4 | \n", "bed | \n", "[-0.011677503, 0.023275835, 0.0026937425, -0.0... | \n", "
\n", " | title | \n", "primary_image | \n", "style | \n", "material | \n", "color | \n", "url | \n", "
---|---|---|---|---|---|---|
0 | \n", "GOYMFK 1pc Free Standing Shoe Rack, Multi-laye... | \n", "https://m.media-amazon.com/images/I/416WaLx10j... | \n", "Modern | \n", "Metal | \n", "White | \n", "https://www.amazon.com/dp/B0CJHKVG6P | \n", "
1 | \n", "subrtex Leather ding Room, Dining Chairs Set o... | \n", "https://m.media-amazon.com/images/I/31SejUEWY7... | \n", "Black Rubber Wood | \n", "Sponge | \n", "Black | \n", "https://www.amazon.com/dp/B0B66QHB23 | \n", "
2 | \n", "Plant Repotting Mat MUYETOL Waterproof Transpl... | \n", "https://m.media-amazon.com/images/I/41RgefVq70... | \n", "Modern | \n", "Polyethylene | \n", "Green | \n", "https://www.amazon.com/dp/B0BXRTWLYK | \n", "
3 | \n", "Pickleball Doormat, Welcome Doormat Absorbent ... | \n", "https://m.media-amazon.com/images/I/61vz1Igler... | \n", "Modern | \n", "Rubber | \n", "A5589 | \n", "https://www.amazon.com/dp/B0C1MRB2M8 | \n", "
4 | \n", "JOIN IRON Foldable TV Trays for Eating Set of ... | \n", "https://m.media-amazon.com/images/I/41p4d4VJnN... | \n", "X Classic Style | \n", "Iron | \n", "Grey Set of 4 | \n", "https://www.amazon.com/dp/B0CG1N9QRC | \n", "
\n", " | title | \n", "primary_image | \n", "style | \n", "material | \n", "color | \n", "url | \n", "keywords | \n", "img_description | \n", "caption | \n", "
---|---|---|---|---|---|---|---|---|---|
0 | \n", "GOYMFK 1pc Free Standing Shoe Rack, Multi-laye... | \n", "https://m.media-amazon.com/images/I/416WaLx10j... | \n", "Modern | \n", "Metal | \n", "White | \n", "https://www.amazon.com/dp/B0CJHKVG6P | \n", "[shoe rack, free standing, multi-layer, metal,... | \n", "This is a free-standing shoe rack featuring a ... | \n", "White metal free-standing shoe rack with multi... | \n", "
1 | \n", "subrtex Leather ding Room, Dining Chairs Set o... | \n", "https://m.media-amazon.com/images/I/31SejUEWY7... | \n", "Black Rubber Wood | \n", "Sponge | \n", "Black | \n", "https://www.amazon.com/dp/B0B66QHB23 | \n", "[dining chairs, set of 2, leather, black] | \n", "This image features a set of two black dining ... | \n", "Set of 2 sleek black faux leather dining chair... | \n", "
2 | \n", "Plant Repotting Mat MUYETOL Waterproof Transpl... | \n", "https://m.media-amazon.com/images/I/41RgefVq70... | \n", "Modern | \n", "Polyethylene | \n", "Green | \n", "https://www.amazon.com/dp/B0BXRTWLYK | \n", "[plant repotting mat, waterproof, portable, fo... | \n", "This is a square plant repotting mat designed ... | \n", "Waterproof green square plant repotting mat | \n", "
3 | \n", "Pickleball Doormat, Welcome Doormat Absorbent ... | \n", "https://m.media-amazon.com/images/I/61vz1Igler... | \n", "Modern | \n", "Rubber | \n", "A5589 | \n", "https://www.amazon.com/dp/B0C1MRB2M8 | \n", "[doormat, absorbent, non-slip, brown] | \n", "This is a rectangular doormat featuring a play... | \n", "Pickleball-themed coir doormat with playful de... | \n", "
4 | \n", "JOIN IRON Foldable TV Trays for Eating Set of ... | \n", "https://m.media-amazon.com/images/I/41p4d4VJnN... | \n", "X Classic Style | \n", "Iron | \n", "Grey Set of 4 | \n", "https://www.amazon.com/dp/B0CG1N9QRC | \n", "[tv tray table set, foldable, iron, grey] | \n", "This image showcases a set of two foldable TV ... | \n", "Set of two foldable TV trays with grey wood gr... | \n", "
\n", " | title | \n", "primary_image | \n", "style | \n", "material | \n", "color | \n", "url | \n", "keywords | \n", "img_description | \n", "caption | \n", "embedding | \n", "
---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "GOYMFK 1pc Free Standing Shoe Rack, Multi-laye... | \n", "https://m.media-amazon.com/images/I/416WaLx10j... | \n", "Modern | \n", "Metal | \n", "White | \n", "https://www.amazon.com/dp/B0CJHKVG6P | \n", "['shoe rack', 'free standing', 'multi-layer', ... | \n", "This is a free-standing shoe rack featuring a ... | \n", "White metal free-standing shoe rack with multi... | \n", "[-0.06596625, -0.026769113, -0.013789515, -0.0... | \n", "
1 | \n", "subrtex Leather ding Room, Dining Chairs Set o... | \n", "https://m.media-amazon.com/images/I/31SejUEWY7... | \n", "Black Rubber Wood | \n", "Sponge | \n", "Black | \n", "https://www.amazon.com/dp/B0B66QHB23 | \n", "['dining chairs', 'set of 2', 'leather', 'black'] | \n", "This image features a set of two black dining ... | \n", "Set of 2 sleek black faux leather dining chair... | \n", "[-0.0077859573, -0.010376813, -0.01928079, -0.... | \n", "
2 | \n", "Plant Repotting Mat MUYETOL Waterproof Transpl... | \n", "https://m.media-amazon.com/images/I/41RgefVq70... | \n", "Modern | \n", "Polyethylene | \n", "Green | \n", "https://www.amazon.com/dp/B0BXRTWLYK | \n", "['plant repotting mat', 'waterproof', 'portabl... | \n", "This is a square plant repotting mat designed ... | \n", "Waterproof green square plant repotting mat | \n", "[-0.023248248, 0.005370147, -0.0048999498, -0.... | \n", "
3 | \n", "Pickleball Doormat, Welcome Doormat Absorbent ... | \n", "https://m.media-amazon.com/images/I/61vz1Igler... | \n", "Modern | \n", "Rubber | \n", "A5589 | \n", "https://www.amazon.com/dp/B0C1MRB2M8 | \n", "['doormat', 'absorbent', 'non-slip', 'brown'] | \n", "This is a rectangular doormat featuring a play... | \n", "Pickleball-themed coir doormat with playful de... | \n", "[-0.028953036, -0.026369056, -0.011363288, 0.0... | \n", "
4 | \n", "JOIN IRON Foldable TV Trays for Eating Set of ... | \n", "https://m.media-amazon.com/images/I/41p4d4VJnN... | \n", "X Classic Style | \n", "Iron | \n", "Grey Set of 4 | \n", "https://www.amazon.com/dp/B0CG1N9QRC | \n", "['tv tray table set', 'foldable', 'iron', 'grey'] | \n", "This image showcases a set of two foldable TV ... | \n", "Set of two foldable TV trays with grey wood gr... | \n", "[-0.030723095, -0.0051356032, -0.027088132, 0.... | \n", "