{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f970f757-ec76-4bf0-90cd-a2fb68b945e3",
   "metadata": {},
   "source": [
    "# Exploring OpenAI V1 functionality\n",
    "\n",
    "On 11.06.23 OpenAI released a number of new features, and along with it bumped their Python SDK to 1.0.0. This notebook shows off the new features and how to use them with LangChain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee897729-263a-4073-898f-bb4cf01ed829",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -U openai \"langchain>=0.0.331rc1\" langchain-experimental"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "c3e067ce-7a43-47a7-bc89-41f1de4cf136",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.schema.messages import HumanMessage, SystemMessage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa7e7e95-90a1-4f73-98fe-10c4b4e0951b",
   "metadata": {},
   "source": [
    "## [Vision](https://platform.openai.com/docs/guides/vision)\n",
    "\n",
    "OpenAI released multi-modal models, which can take a sequence of text and images as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "1c8c3965-d3c9-4186-b5f3-5e67855ef916",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='This image appears to be a diagram representing the architecture or components of a software system or platform named \"LangChain.\" The diagram is organized into various blocks that represent different layers or aspects of the system. Here\\'s a breakdown of the labeled parts:\\n\\n1. **LangSmith**: This seems to be a tool or component related to testing, evaluation, monitoring, feedback, and annotation, as well as debugging.\\n\\n2. **LangServe**: This component chains as REST API, suggesting it may serve as an interface for the system\\'s functionality to be accessed over the web using RESTful API calls.\\n\\n3. **Templates**: Reference applications are mentioned here, which might be pre-built applications or use cases provided as starting points for users of the system.\\n\\n4. **Chains, agents, agent executors**: This part refers to common application logic, indicating modular components that can be combined or chained together to create complex workflows.\\n\\n5. **Model I/O**: This includes elements such as prompt, example selector, model, and output parser, which may deal with the input and output processes of a machine learning model or similar.\\n\\n6. **Retrieval**: Components under this section, like document loader, text splitter, embedding model, vector store, and retriever, imply functionality related to')"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat = ChatOpenAI(model=\"gpt-4-vision-preview\", max_tokens=256)\n",
    "chat.invoke(\n",
    "    [\n",
    "        HumanMessage(\n",
    "            content=[\n",
    "                {\"type\": \"text\", \"text\": \"What is this image showing\"},\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": {\n",
    "                        \"url\": \"https://raw.githubusercontent.com/langchain-ai/langchain/master/docs/static/img/langchain_stack.png\",\n",
    "                        \"detail\": \"auto\",\n",
    "                    },\n",
    "                },\n",
    "            ]\n",
    "        )\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "210f8248-fcf3-4052-a4a3-0684e08f8785",
   "metadata": {},
   "source": [
    "## [OpenAI assistants](https://platform.openai.com/docs/assistants/overview)\n",
    "\n",
    "> The Assistants API allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries. The Assistants API currently supports three types of tools: Code Interpreter, Retrieval, and Function calling\n",
    "\n",
    "\n",
    "You can interact with OpenAI Assistants using OpenAI tools or custom tools. When using exclusively OpenAI tools, you can just invoke the assistant directly and get final answers. When using custom tools, you can run the assistant and tool execution loop using the built-in AgentExecutor or easily write your own executor.\n",
    "\n",
    "Below we show the different ways to interact with Assistants. As a simple example, let's build a math tutor that can write and run code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "318da28d-4cec-42ab-ae3e-76d95bb34fa5",
   "metadata": {},
   "source": [
    "### Using only OpenAI tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a9064bbe-d9f7-4a29-a7b3-73933b3197e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_experimental.openai_assistant import OpenAIAssistantRunnable\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7a20a008-49ac-46d2-aa26-b270118af5ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ThreadMessage(id='msg_RGOsJ2RBYp79rILrZ0NsAX68', assistant_id='asst_9Xb1ZgefoAbp2V5ddRlDGisy', content=[MessageContentText(text=Text(annotations=[], value='\\\\( 10 - 4^{2.7} \\\\) is approximately -32.2243.'), type='text')], created_at=1699385426, file_ids=[], metadata={}, object='thread.message', role='assistant', run_id='run_E2PRoP04ryly4p5ds5ek2qxs', thread_id='thread_pu9CpsYIWWZtxemZxpbYfhm4')]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interpreter_assistant = OpenAIAssistantRunnable.create_assistant(\n",
    "    name=\"langchain assistant\",\n",
    "    instructions=\"You are a personal math tutor. Write and run code to answer math questions.\",\n",
    "    tools=[{\"type\": \"code_interpreter\"}],\n",
    "    model=\"gpt-4-1106-preview\"\n",
    ")\n",
    "output = interpreter_assistant.invoke({\"content\": \"What's 10 - 4 raised to the 2.7\"})\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8ddd181-ac63-4ab6-a40d-a236120379c1",
   "metadata": {},
   "source": [
    "### As a LangChain agent with arbitrary tools\n",
    "\n",
    "Now let's recreate this functionality using our own tools. For this example we'll use the [E2B sandbox runtime tool](https://e2b.dev/docs?ref=landing-page-get-started)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "48681ac7-b267-48d4-972c-8a7df8393a21",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.tools import E2BDataAnalysisTool\n",
    "\n",
    "tools = [E2BDataAnalysisTool(api_key=\"...\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "1c01dd79-dd3e-4509-a2e2-009a7f99f16a",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = OpenAIAssistantRunnable.create_assistant(\n",
    "    name=\"langchain assistant e2b tool\",\n",
    "    instructions=\"You are a personal math tutor. Write and run code to answer math questions.\",\n",
    "    tools=tools,\n",
    "    model=\"gpt-4-1106-preview\",\n",
    "    as_agent=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ac71d8b-4b4b-4f98-b826-6b3c57a34166",
   "metadata": {},
   "source": [
    "#### Using AgentExecutor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1f137f94-801f-4766-9ff5-2de9df5e8079",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'content': \"What's 10 - 4 raised to the 2.7\",\n",
       " 'output': 'The result of \\\\(10 - 4\\\\) raised to the power of \\\\(2.7\\\\) is approximately \\\\(126.19\\\\).'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.agents import AgentExecutor\n",
    "\n",
    "agent_executor = AgentExecutor(agent=agent, tools=tools)\n",
    "agent_executor.invoke({\"content\": \"What's 10 - 4 raised to the 2.7\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d0a0b1d-c1b3-4b50-9dce-1189b51a6206",
   "metadata": {},
   "source": [
    "#### Custom execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c0475fa7-b6c1-4331-b8e2-55407466c724",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = OpenAIAssistantRunnable.create_assistant(\n",
    "    name=\"langchain assistant e2b tool\",\n",
    "    instructions=\"You are a personal math tutor. Write and run code to answer math questions.\",\n",
    "    tools=tools,\n",
    "    model=\"gpt-4-1106-preview\",\n",
    "    as_agent=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b76cb669-6aba-4827-868f-00aa960026f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.schema.agent import AgentFinish\n",
    "\n",
    "def execute_agent(agent, tools, input):\n",
    "    tool_map = {tool.name: tool for tool in tools}\n",
    "    response = agent.invoke(input)\n",
    "    while not isinstance(response, AgentFinish):\n",
    "        tool_outputs = []\n",
    "        for action in response:\n",
    "            tool_output = tool_map[action.tool].invoke(action.tool_input)\n",
    "            print(action.tool, action.tool_input, tool_output, end=\"\\n\\n\")\n",
    "            tool_outputs.append({\"output\": tool_output, \"tool_call_id\": action.tool_call_id})\n",
    "        response = agent.invoke({\"tool_outputs\": tool_outputs, \"run_id\": action.run_id, \"thread_id\": action.thread_id})\n",
    "        \n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7946116a-b82f-492e-835e-ca958a8949a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "e2b_data_analysis {'python_code': 'print((10 - 4) ** 2.7)'} {\"stdout\": \"126.18518711065899\", \"stderr\": \"\", \"artifacts\": []}\n",
      "\n",
      "\\( 10 - 4 \\) raised to the power of 2.7 is approximately 126.185.\n"
     ]
    }
   ],
   "source": [
    "response = execute_agent(agent, tools, {\"content\": \"What's 10 - 4 raised to the 2.7\"})\n",
    "print(response.return_values[\"output\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "f2744a56-9f4f-4899-827a-fa55821c318c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "e2b_data_analysis {'python_code': 'result = (10 - 4) ** 2.7 + 17.241\\nprint(result)'} {\"stdout\": \"143.426187110659\", \"stderr\": \"\", \"artifacts\": []}\n",
      "\n",
      "\\( (10 - 4)^{2.7} + 17.241 \\) is approximately 143.426.\n"
     ]
    }
   ],
   "source": [
    "next_response = execute_agent(agent, tools, {\"content\": \"now add 17.241\", \"thread_id\": response.thread_id})\n",
    "print(next_response.return_values[\"output\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71c34763-d1e7-4b9a-a9d7-3e4cc0dfc2c4",
   "metadata": {},
   "source": [
    "## [JSON mode](https://platform.openai.com/docs/guides/text-generation/json-mode)\n",
    "\n",
    "Constrain the model to only generate valid JSON. Note that you must include a system message with instructions to use JSON for this mode to work.\n",
    "\n",
    "Only works with certain models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "db6072c4-f3f3-415d-872b-71ea9f3c02bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"companies\": [\n",
      "    {\n",
      "      \"name\": \"Google\",\n",
      "      \"origin\": \"USA\"\n",
      "    },\n",
      "    {\n",
      "      \"name\": \"Deepmind\",\n",
      "      \"origin\": \"UK\"\n",
      "    }\n",
      "  ]\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "chat = ChatOpenAI(model=\"gpt-3.5-turbo-1106\").bind(\n",
    "    response_format={\"type\": \"json_object\"}\n",
    ")\n",
    "\n",
    "output = chat.invoke(\n",
    "    [\n",
    "        SystemMessage(\n",
    "            content=\"Extract the 'name' and 'origin' of any companies mentioned in the following statement. Return a JSON list.\"\n",
    "        ),\n",
    "        HumanMessage(\n",
    "            content=\"Google was founded in the USA, while Deepmind was founded in the UK\"\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "print(output.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "08e00ccf-b991-4249-846b-9500a0ccbfa0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'companies': [{'name': 'Google', 'origin': 'USA'},\n",
       "  {'name': 'Deepmind', 'origin': 'UK'}]}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "json.loads(output.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9a94d9-4319-4ab7-a979-c475ce6b5f50",
   "metadata": {},
   "source": [
    "## [System fingerprint](https://platform.openai.com/docs/guides/text-generation/reproducible-outputs)\n",
    "\n",
    "OpenAI sometimes changes model configurations in a way that impacts outputs. Whenever this happens, the system_fingerprint associated with a generation will change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "1281883c-bf8f-4665-89cd-4f33ccde69ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'token_usage': {'completion_tokens': 43, 'prompt_tokens': 49, 'total_tokens': 92}, 'model_name': 'gpt-3.5-turbo-1106', 'system_fingerprint': 'fp_eeff13170a'}\n"
     ]
    }
   ],
   "source": [
    "chat = ChatOpenAI(model=\"gpt-3.5-turbo-1106\")\n",
    "output = chat.generate(\n",
    "    [\n",
    "        [\n",
    "            SystemMessage(\n",
    "                content=\"Extract the 'name' and 'origin' of any companies mentioned in the following statement. Return a JSON list.\"\n",
    "            ),\n",
    "            HumanMessage(\n",
    "                content=\"Google was founded in the USA, while Deepmind was founded in the UK\"\n",
    "            ),\n",
    "        ]\n",
    "    ]\n",
    ")\n",
    "print(output.llm_output)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "poetry-venv",
   "language": "python",
   "name": "poetry-venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}