Update jina embedding notebook to show multimodal capability more clearly (#23702)

After merging the [PR #22594 to include Jina AI multimodal capabilities
in the Langchain
documentation](https://github.com/langchain-ai/langchain/pull/22594), we
updated the notebook to showcase the difference between text and
multimodal capabilities more clearly.
This commit is contained in:
Francesco Kruk 2024-07-01 15:13:19 +02:00 committed by GitHub
parent 24916c6703
commit 7538f3df58
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194

View File

@ -5,9 +5,15 @@
"id": "1c0cf975",
"metadata": {},
"source": [
"# Jina\n",
"\n",
"Let's load the Jina Embedding class."
"# Jina"
]
},
{
"cell_type": "markdown",
"id": "da922b13-eaa8-4cdc-98dd-cf8f3d2e6ffa",
"metadata": {},
"source": [
"Install requirements"
]
},
{
@ -20,6 +26,14 @@
"pip install -U langchain-community"
]
},
{
"cell_type": "markdown",
"id": "7911b286-130d-4971-b77c-7c7a077115b6",
"metadata": {},
"source": [
"Import libraries"
]
},
{
"cell_type": "code",
"execution_count": null,
@ -34,6 +48,14 @@
"from PIL import Image"
]
},
{
"cell_type": "markdown",
"id": "59aa1c02-1216-43eb-8473-8e0468f0ddb6",
"metadata": {},
"source": [
"## Embed text and queries with Jina embedding models through JinaAI API"
]
},
{
"cell_type": "code",
"execution_count": null,
@ -43,9 +65,7 @@
"source": [
"text_embeddings = JinaEmbeddings(\n",
" jina_api_key=\"jina_*\", model_name=\"jina-embeddings-v2-base-en\"\n",
")\n",
"\n",
"image_embeddings = JinaEmbeddings(jina_api_key=\"jina_*\", model_name=\"jina-clip-v1\")"
")"
]
},
{
@ -55,15 +75,7 @@
"metadata": {},
"outputs": [],
"source": [
"text = \"This is a test document.\"\n",
"\n",
"image = \"https://avatars.githubusercontent.com/u/126733545?v=4\"\n",
"\n",
"description = \"Logo of a parrot and a chain on green background\"\n",
"\n",
"im = Image.open(requests.get(image, stream=True).raw)\n",
"print(\"Image:\")\n",
"display(im)"
"text = \"This is a test document.\""
]
},
{
@ -106,6 +118,40 @@
"print(doc_result)"
]
},
{
"cell_type": "markdown",
"id": "338ea747-040e-4ed4-8ddf-9b2285987885",
"metadata": {},
"source": [
"## Embed images and queries with Jina CLIP through JinaAI API"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "575b5867-59fb-4fd1-911b-afee2eaca088",
"metadata": {},
"outputs": [],
"source": [
"multimodal_embeddings = JinaEmbeddings(jina_api_key=\"jina_*\", model_name=\"jina-clip-v1\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "a9b335f5-fa99-4931-95f6-7b187c0e2f30",
"metadata": {},
"outputs": [],
"source": [
"image = \"https://avatars.githubusercontent.com/u/126733545?v=4\"\n",
"\n",
"description = \"Logo of a parrot and a chain on green background\"\n",
"\n",
"im = Image.open(requests.get(image, stream=True).raw)\n",
"print(\"Image:\")\n",
"display(im)"
]
},
{
"cell_type": "code",
"execution_count": null,
@ -113,7 +159,7 @@
"metadata": {},
"outputs": [],
"source": [
"image_result = image_embeddings.embed_images([image])"
"image_result = multimodal_embeddings.embed_images([image])"
]
},
{
@ -133,7 +179,7 @@
"metadata": {},
"outputs": [],
"source": [
"description_result = image_embeddings.embed_documents([description])"
"description_result = multimodal_embeddings.embed_documents([description])"
]
},
{
@ -167,14 +213,6 @@
"source": [
"print(cosine_similarity)"
]
},
{
"cell_type": "code",
"execution_count": null,
"id": "7f280807-a02b-4d4e-8ebd-01be33117999",
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
@ -193,7 +231,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.12.2"
"version": "3.9.13"
}
},
"nbformat": 4,