mirror of
https://github.com/hwchase17/langchain
synced 2024-11-06 03:20:49 +00:00
104 lines
3.1 KiB
Plaintext
104 lines
3.1 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"# Nuclia Understanding API document transformer\n",
|
|
"\n",
|
|
"[Nuclia](https://nuclia.com) automatically indexes your unstructured data from any internal and external source, providing optimized search results and generative answers. It can handle video and audio transcription, image content extraction, and document parsing.\n",
|
|
"\n",
|
|
"The Nuclia Understanding API document transformer splits text into paragraphs and sentences, identifies entities, provides a summary of the text and generates embeddings for all the sentences.\n",
|
|
"\n",
|
|
"To use the Nuclia Understanding API, you need to have a Nuclia account. You can create one for free at [https://nuclia.cloud](https://nuclia.cloud), and then [create a NUA key](https://docs.nuclia.dev/docs/docs/using/understanding/intro).\n",
|
|
"\n",
|
|
"from langchain.document_transformers.nuclia_text_transform import NucliaTextTransformer"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"#!pip install --upgrade protobuf\n",
|
|
"#!pip install nucliadb-protos"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import os\n",
|
|
"\n",
|
|
"os.environ[\"NUCLIA_ZONE\"] = \"<YOUR_ZONE>\" # e.g. europe-1\n",
|
|
"os.environ[\"NUCLIA_NUA_KEY\"] = \"<YOUR_API_KEY>\""
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"To use the Nuclia document transformer, you need to instantiate a `NucliaUnderstandingAPI` tool with `enable_ml` set to `True`:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from langchain.tools.nuclia import NucliaUnderstandingAPI\n",
|
|
"\n",
|
|
"nua = NucliaUnderstandingAPI(enable_ml=True)"
|
|
]
|
|
},
|
|
{
|
|
"attachments": {},
|
|
"cell_type": "markdown",
|
|
"metadata": {},
|
|
"source": [
|
|
"The Nuclia document transformer must be called in async mode, so you need to use the `atransform_documents` method:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import asyncio\n",
|
|
"\n",
|
|
"from langchain.document_transformers.nuclia_text_transform import NucliaTextTransformer\n",
|
|
"from langchain.schema.document import Document\n",
|
|
"\n",
|
|
"\n",
|
|
"async def process():\n",
|
|
" documents = [\n",
|
|
" Document(page_content=\"<TEXT 1>\", metadata={}),\n",
|
|
" Document(page_content=\"<TEXT 2>\", metadata={}),\n",
|
|
" Document(page_content=\"<TEXT 3>\", metadata={}),\n",
|
|
" ]\n",
|
|
" nuclia_transformer = NucliaTextTransformer(nua)\n",
|
|
" transformed_documents = await nuclia_transformer.atransform_documents(documents)\n",
|
|
" print(transformed_documents)\n",
|
|
"\n",
|
|
"\n",
|
|
"asyncio.run(process())"
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"language_info": {
|
|
"name": "python"
|
|
},
|
|
"orig_nbformat": 4
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 2
|
|
}
|