langchain/docs/extras/integrations/document_transformers/doctran_interrogate_document.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Doctran Interrogate Documents\n",
    "Documents used in a vector store knowledge base are typically stored in narrative or conversational format. However, most user queries are in question format. If we convert documents into Q&A format before vectorizing them, we can increase the liklihood of retrieving relevant documents, and decrease the liklihood of retrieving irrelevant documents.\n",
    "\n",
    "We can accomplish this using the [Doctran](https://github.com/psychic-api/doctran) library, which uses OpenAI's function calling feature to \"interrogate\" documents.\n",
    "\n",
    "See [this notebook](https://github.com/psychic-api/doctran/blob/main/benchmark.ipynb) for benchmarks on vector similarity scores for various queries based on raw documents versus interrogated documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install doctran"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import json\n",
    "from langchain.schema import Document\n",
    "from langchain.document_transformers import DoctranQATransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input\n",
    "This is the document we'll interrogate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Generated with ChatGPT]\n",
      "\n",
      "Confidential Document - For Internal Use Only\n",
      "\n",
      "Date: July 1, 2023\n",
      "\n",
      "Subject: Updates and Discussions on Various Topics\n",
      "\n",
      "Dear Team,\n",
      "\n",
      "I hope this email finds you well. In this document, I would like to provide you with some important updates and discuss various topics that require our attention. Please treat the information contained herein as highly confidential.\n",
      "\n",
      "Security and Privacy Measures\n",
      "As part of our ongoing commitment to ensure the security and privacy of our customers' data, we have implemented robust measures across all our systems. We would like to commend John Doe (email: john.doe@example.com) from the IT department for his diligent work in enhancing our network security. Moving forward, we kindly remind everyone to strictly adhere to our data protection policies and guidelines. Additionally, if you come across any potential security risks or incidents, please report them immediately to our dedicated team at security@example.com.\n",
      "\n",
      "HR Updates and Employee Benefits\n",
      "Recently, we welcomed several new team members who have made significant contributions to their respective departments. I would like to recognize Jane Smith (SSN: 049-45-5928) for her outstanding performance in customer service. Jane has consistently received positive feedback from our clients. Furthermore, please remember that the open enrollment period for our employee benefits program is fast approaching. Should you have any questions or require assistance, please contact our HR representative, Michael Johnson (phone: 418-492-3850, email: michael.johnson@example.com).\n",
      "\n",
      "Marketing Initiatives and Campaigns\n",
      "Our marketing team has been actively working on developing new strategies to increase brand awareness and drive customer engagement. We would like to thank Sarah Thompson (phone: 415-555-1234) for her exceptional efforts in managing our social media platforms. Sarah has successfully increased our follower base by 20% in the past month alone. Moreover, please mark your calendars for the upcoming product launch event on July 15th. We encourage all team members to attend and support this exciting milestone for our company.\n",
      "\n",
      "Research and Development Projects\n",
      "In our pursuit of innovation, our research and development department has been working tirelessly on various projects. I would like to acknowledge the exceptional work of David Rodriguez (email: david.rodriguez@example.com) in his role as project lead. David's contributions to the development of our cutting-edge technology have been instrumental. Furthermore, we would like to remind everyone to share their ideas and suggestions for potential new projects during our monthly R&D brainstorming session, scheduled for July 10th.\n",
      "\n",
      "Please treat the information in this document with utmost confidentiality and ensure that it is not shared with unauthorized individuals. If you have any questions or concerns regarding the topics discussed, please do not hesitate to reach out to me directly.\n",
      "\n",
      "Thank you for your attention, and let's continue to work together to achieve our goals.\n",
      "\n",
      "Best regards,\n",
      "\n",
      "Jason Fan\n",
      "Cofounder & CEO\n",
      "Psychic\n",
      "jason@psychic.dev\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sample_text = \"\"\"[Generated with ChatGPT]\n",
    "\n",
    "Confidential Document - For Internal Use Only\n",
    "\n",
    "Date: July 1, 2023\n",
    "\n",
    "Subject: Updates and Discussions on Various Topics\n",
    "\n",
    "Dear Team,\n",
    "\n",
    "I hope this email finds you well. In this document, I would like to provide you with some important updates and discuss various topics that require our attention. Please treat the information contained herein as highly confidential.\n",
    "\n",
    "Security and Privacy Measures\n",
    "As part of our ongoing commitment to ensure the security and privacy of our customers' data, we have implemented robust measures across all our systems. We would like to commend John Doe (email: john.doe@example.com) from the IT department for his diligent work in enhancing our network security. Moving forward, we kindly remind everyone to strictly adhere to our data protection policies and guidelines. Additionally, if you come across any potential security risks or incidents, please report them immediately to our dedicated team at security@example.com.\n",
    "\n",
    "HR Updates and Employee Benefits\n",
    "Recently, we welcomed several new team members who have made significant contributions to their respective departments. I would like to recognize Jane Smith (SSN: 049-45-5928) for her outstanding performance in customer service. Jane has consistently received positive feedback from our clients. Furthermore, please remember that the open enrollment period for our employee benefits program is fast approaching. Should you have any questions or require assistance, please contact our HR representative, Michael Johnson (phone: 418-492-3850, email: michael.johnson@example.com).\n",
    "\n",
    "Marketing Initiatives and Campaigns\n",
    "Our marketing team has been actively working on developing new strategies to increase brand awareness and drive customer engagement. We would like to thank Sarah Thompson (phone: 415-555-1234) for her exceptional efforts in managing our social media platforms. Sarah has successfully increased our follower base by 20% in the past month alone. Moreover, please mark your calendars for the upcoming product launch event on July 15th. We encourage all team members to attend and support this exciting milestone for our company.\n",
    "\n",
    "Research and Development Projects\n",
    "In our pursuit of innovation, our research and development department has been working tirelessly on various projects. I would like to acknowledge the exceptional work of David Rodriguez (email: david.rodriguez@example.com) in his role as project lead. David's contributions to the development of our cutting-edge technology have been instrumental. Furthermore, we would like to remind everyone to share their ideas and suggestions for potential new projects during our monthly R&D brainstorming session, scheduled for July 10th.\n",
    "\n",
    "Please treat the information in this document with utmost confidentiality and ensure that it is not shared with unauthorized individuals. If you have any questions or concerns regarding the topics discussed, please do not hesitate to reach out to me directly.\n",
    "\n",
    "Thank you for your attention, and let's continue to work together to achieve our goals.\n",
    "\n",
    "Best regards,\n",
    "\n",
    "Jason Fan\n",
    "Cofounder & CEO\n",
    "Psychic\n",
    "jason@psychic.dev\n",
    "\"\"\"\n",
    "print(sample_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = [Document(page_content=sample_text)]\n",
    "qa_transformer = DoctranQATransformer()\n",
    "transformed_document = await qa_transformer.atransform_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output\n",
    "After interrogating a document, the result will be returned as a new document with questions and answers provided in the metadata."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"questions_and_answers\": [\n",
      "    {\n",
      "      \"question\": \"What is the purpose of this document?\",\n",
      "      \"answer\": \"The purpose of this document is to provide important updates and discuss various topics that require the team's attention.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"Who is responsible for enhancing the network security?\",\n",
      "      \"answer\": \"John Doe from the IT department is responsible for enhancing the network security.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"Where should potential security risks or incidents be reported?\",\n",
      "      \"answer\": \"Potential security risks or incidents should be reported to the dedicated team at security@example.com.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"Who has been recognized for outstanding performance in customer service?\",\n",
      "      \"answer\": \"Jane Smith has been recognized for her outstanding performance in customer service.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"When is the open enrollment period for the employee benefits program?\",\n",
      "      \"answer\": \"The document does not specify the exact dates for the open enrollment period for the employee benefits program, but it mentions that it is fast approaching.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"Who should be contacted for questions or assistance regarding the employee benefits program?\",\n",
      "      \"answer\": \"For questions or assistance regarding the employee benefits program, the HR representative, Michael Johnson, should be contacted.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"Who has been acknowledged for managing the company's social media platforms?\",\n",
      "      \"answer\": \"Sarah Thompson has been acknowledged for managing the company's social media platforms.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"When is the upcoming product launch event?\",\n",
      "      \"answer\": \"The upcoming product launch event is on July 15th.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"Who has been recognized for their contributions to the development of the company's technology?\",\n",
      "      \"answer\": \"David Rodriguez has been recognized for his contributions to the development of the company's technology.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"When is the monthly R&D brainstorming session?\",\n",
      "      \"answer\": \"The monthly R&D brainstorming session is scheduled for July 10th.\"\n",
      "    },\n",
      "    {\n",
      "      \"question\": \"Who should be contacted for questions or concerns regarding the topics discussed in the document?\",\n",
      "      \"answer\": \"For questions or concerns regarding the topics discussed in the document, Jason Fan, the Cofounder & CEO, should be contacted.\"\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "transformed_document = await qa_transformer.atransform_documents(documents)\n",
    "print(json.dumps(transformed_document[0].metadata, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}