langchain/docs/extras/integrations/document_loaders/airbyte_salesforce.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1f3a5ebf",
   "metadata": {},
   "source": [
    "# Airbyte Salesforce"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35ac77b1-449b-44f7-b8f3-3494d55c286e",
   "metadata": {},
   "source": [
    ">[Airbyte](https://github.com/airbytehq/airbyte) is a data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes. It has the largest catalog of ELT connectors to data warehouses and databases.\n",
    "\n",
    "This loader exposes the Salesforce connector as a document loader, allowing you to load various Salesforce objects as documents."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6847a40c",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "3b06fbde",
   "metadata": {},
   "source": [
    "## Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3e9dc79",
   "metadata": {},
   "source": [
    "First, you need to install the `airbyte-source-salesforce` python package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d35e4e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install airbyte-source-salesforce"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae855210",
   "metadata": {},
   "source": [
    "## Example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02208f52",
   "metadata": {},
   "source": [
    "Check out the [Airbyte documentation page](https://docs.airbyte.com/integrations/sources/salesforce/) for details about how to configure the reader.\n",
    "The JSON schema the config object should adhere to can be found on Github: [https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/spec.yaml](https://github.com/airbytehq/airbyte/blob/master/airbyte-integrations/connectors/source-salesforce/source_salesforce/spec.yaml).\n",
    "\n",
    "The general shape looks like this:\n",
    "```python\n",
    "{\n",
    "  \"client_id\": \"<oauth client id>\",\n",
    "  \"client_secret\": \"<oauth client secret>\",\n",
    "  \"refresh_token\": \"<oauth refresh token>\",\n",
    "  \"start_date\": \"<date from which to start retrieving records from in ISO format, e.g. 2020-10-20T00:00:00Z>\",\n",
    "  \"is_sandbox\": False, # set to True if you're using a sandbox environment\n",
    "  \"streams_criteria\": [ # Array of filters for salesforce objects that should be loadable\n",
    "    {\"criteria\": \"exacts\", \"value\": \"Account\"}, # Exact name of salesforce object\n",
    "    {\"criteria\": \"starts with\", \"value\": \"Asset\"}, # Prefix of the name\n",
    "    # Other allowed criteria: ends with, contains, starts not with, ends not with, not contains, not exacts\n",
    "  ],\n",
    "}\n",
    "```\n",
    "\n",
    "By default all fields are stored as metadata in the documents and the text is set to an empty string. Construct the text of the document by transforming the documents returned by the reader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89a99e58",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from langchain.document_loaders.airbyte import AirbyteSalesforceLoader\n",
    "\n",
    "config = {\n",
    "    # your salesforce configuration\n",
    "}\n",
    "\n",
    "loader = AirbyteSalesforceLoader(config=config, stream_name=\"asset\") # check the documentation linked above for a list of all streams"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2cea23fc",
   "metadata": {},
   "source": [
    "Now you can load documents the usual way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dae75cdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a93dc2a",
   "metadata": {},
   "source": [
    "As `load` returns a list, it will block until all documents are loaded. To have better control over this process, you can also you the `lazy_load` method which returns an iterator instead:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1782db09",
   "metadata": {},
   "outputs": [],
   "source": [
    "docs_iterator = loader.lazy_load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a124086",
   "metadata": {},
   "source": [
    "Keep in mind that by default the page content is empty and the metadata object contains all the information from the record. To create documents in a different, pass in a record_handler function when creating the loader:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5671395d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.docstore.document import Document\n",
    "\n",
    "def handle_record(record, id):\n",
    "    return Document(page_content=record.data[\"title\"], metadata=record.data)\n",
    "\n",
    "loader = AirbyteSalesforceLoader(config=config, record_handler=handle_record, stream_name=\"asset\")\n",
    "docs = loader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "223eb8bc",
   "metadata": {},
   "source": [
    "## Incremental loads\n",
    "\n",
    "Some streams allow incremental loading, this means the source keeps track of synced records and won't load them again. This is useful for sources that have a high volume of data and are updated frequently.\n",
    "\n",
    "To take advantage of this, store the `last_state` property of the loader and pass it in when creating the loader again. This will ensure that only new records are loaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7061e735",
   "metadata": {},
   "outputs": [],
   "source": [
    "last_state = loader.last_state # store safely\n",
    "\n",
    "incremental_loader = AirbyteSalesforceLoader(config=config, stream_name=\"asset\", state=last_state)\n",
    "\n",
    "new_docs = incremental_loader.load()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}