langchain/cookbook/openai_v1_cookbook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f970f757-ec76-4bf0-90cd-a2fb68b945e3",
   "metadata": {},
   "source": [
    "# Exploring OpenAI V1 functionality\n",
    "\n",
    "On 11.06.23 OpenAI released a number of new features, and along with it bumped their Python SDK to 1.0.0. This notebook shows off the new features and how to use them with LangChain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee897729-263a-4073-898f-bb4cf01ed829",
   "metadata": {},
   "outputs": [],
   "source": [
    "# need openai>=1.1.0, langchain>=0.0.333, langchain-experimental>=0.0.39\n",
    "!pip install -U openai langchain langchain-experimental"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c3e067ce-7a43-47a7-bc89-41f1de4cf136",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.chat_models import ChatOpenAI\n",
    "from langchain.schema.messages import HumanMessage, SystemMessage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa7e7e95-90a1-4f73-98fe-10c4b4e0951b",
   "metadata": {},
   "source": [
    "## [Vision](https://platform.openai.com/docs/guides/vision)\n",
    "\n",
    "OpenAI released multi-modal models, which can take a sequence of text and images as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1c8c3965-d3c9-4186-b5f3-5e67855ef916",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "AIMessage(content='The image appears to be a diagram representing the architecture or components of a software system or framework related to language processing, possibly named LangChain or associated with a project or product called LangChain, based on the prominent appearance of that term. The diagram is organized into several layers or aspects, each containing various elements or modules:\\n\\n1. **Protocol**: This may be the foundational layer, which includes \"LCEL\" and terms like parallelization, fallbacks, tracing, batching, streaming, async, and composition. These seem related to communication and execution protocols for the system.\\n\\n2. **Integrations Components**: This layer includes \"Model I/O\" with elements such as the model, output parser, prompt, and example selector. It also has a \"Retrieval\" section with a document loader, retriever, embedding model, vector store, and text splitter. Lastly, there\\'s an \"Agent Tooling\" section. These components likely deal with the interaction with external data, models, and tools.\\n\\n3. **Application**: The application layer features \"LangChain\" with chains, agents, agent executors, and common application logic. This suggests that the system uses a modular approach with chains and agents to process language tasks.\\n\\n4. **Deployment**: This contains \"Lang')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat = ChatOpenAI(model=\"gpt-4-vision-preview\", max_tokens=256)\n",
    "chat.invoke(\n",
    "    [\n",
    "        HumanMessage(\n",
    "            content=[\n",
    "                {\"type\": \"text\", \"text\": \"What is this image showing\"},\n",
    "                {\n",
    "                    \"type\": \"image_url\",\n",
    "                    \"image_url\": {\n",
    "                        \"url\": \"https://raw.githubusercontent.com/langchain-ai/langchain/master/docs/static/img/langchain_stack.png\",\n",
    "                        \"detail\": \"auto\",\n",
    "                    },\n",
    "                },\n",
    "            ]\n",
    "        )\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "210f8248-fcf3-4052-a4a3-0684e08f8785",
   "metadata": {},
   "source": [
    "## [OpenAI assistants](https://platform.openai.com/docs/assistants/overview)\n",
    "\n",
    "> The Assistants API allows you to build AI assistants within your own applications. An Assistant has instructions and can leverage models, tools, and knowledge to respond to user queries. The Assistants API currently supports three types of tools: Code Interpreter, Retrieval, and Function calling\n",
    "\n",
    "\n",
    "You can interact with OpenAI Assistants using OpenAI tools or custom tools. When using exclusively OpenAI tools, you can just invoke the assistant directly and get final answers. When using custom tools, you can run the assistant and tool execution loop using the built-in AgentExecutor or easily write your own executor.\n",
    "\n",
    "Below we show the different ways to interact with Assistants. As a simple example, let's build a math tutor that can write and run code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "318da28d-4cec-42ab-ae3e-76d95bb34fa5",
   "metadata": {},
   "source": [
    "### Using only OpenAI tools"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a9064bbe-d9f7-4a29-a7b3-73933b3197e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_experimental.openai_assistant import OpenAIAssistantRunnable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7a20a008-49ac-46d2-aa26-b270118af5ea",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[ThreadMessage(id='msg_g9OJv0rpPgnc3mHmocFv7OVd', assistant_id='asst_hTwZeNMMphxzSOqJ01uBMsJI', content=[MessageContentText(text=Text(annotations=[], value='The result of \\\\(10 - 4^{2.7}\\\\) is approximately \\\\(-32.224\\\\).'), type='text')], created_at=1699460600, file_ids=[], metadata={}, object='thread.message', role='assistant', run_id='run_nBIT7SiAwtUfSCTrQNSPLOfe', thread_id='thread_14n4GgXwxgNL0s30WJW5F6p0')]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interpreter_assistant = OpenAIAssistantRunnable.create_assistant(\n",
    "    name=\"langchain assistant\",\n",
    "    instructions=\"You are a personal math tutor. Write and run code to answer math questions.\",\n",
    "    tools=[{\"type\": \"code_interpreter\"}],\n",
    "    model=\"gpt-4-1106-preview\"\n",
    ")\n",
    "output = interpreter_assistant.invoke({\"content\": \"What's 10 - 4 raised to the 2.7\"})\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8ddd181-ac63-4ab6-a40d-a236120379c1",
   "metadata": {},
   "source": [
    "### As a LangChain agent with arbitrary tools\n",
    "\n",
    "Now let's recreate this functionality using our own tools. For this example we'll use the [E2B sandbox runtime tool](https://e2b.dev/docs?ref=landing-page-get-started)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee4cc355-f2d6-4c51-bcf7-f502868357d3",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install e2b duckduckgo-search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "48681ac7-b267-48d4-972c-8a7df8393a21",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.tools import E2BDataAnalysisTool, DuckDuckGoSearchRun\n",
    "\n",
    "tools = [E2BDataAnalysisTool(api_key=\"...\"), DuckDuckGoSearchRun()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "1c01dd79-dd3e-4509-a2e2-009a7f99f16a",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = OpenAIAssistantRunnable.create_assistant(\n",
    "    name=\"langchain assistant e2b tool\",\n",
    "    instructions=\"You are a personal math tutor. Write and run code to answer math questions. You can also search the internet.\",\n",
    "    tools=tools,\n",
    "    model=\"gpt-4-1106-preview\",\n",
    "    as_agent=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ac71d8b-4b4b-4f98-b826-6b3c57a34166",
   "metadata": {},
   "source": [
    "#### Using AgentExecutor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1f137f94-801f-4766-9ff5-2de9df5e8079",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'content': \"What's the weather in SF today divided by 2.7\",\n",
       " 'output': \"The weather in San Francisco today is reported to have temperatures as high as 66 °F. To get the temperature divided by 2.7, we will calculate that:\\n\\n66 °F / 2.7 = 24.44 °F\\n\\nSo, when the high temperature of 66 °F is divided by 2.7, the result is approximately 24.44 °F. Please note that this doesn't have a meteorological meaning; it's purely a mathematical operation based on the given temperature.\"}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from langchain.agents import AgentExecutor\n",
    "\n",
    "agent_executor = AgentExecutor(agent=agent, tools=tools)\n",
    "agent_executor.invoke({\"content\": \"What's the weather in SF today divided by 2.7\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d0a0b1d-c1b3-4b50-9dce-1189b51a6206",
   "metadata": {},
   "source": [
    "#### Custom execution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c0475fa7-b6c1-4331-b8e2-55407466c724",
   "metadata": {},
   "outputs": [],
   "source": [
    "agent = OpenAIAssistantRunnable.create_assistant(\n",
    "    name=\"langchain assistant e2b tool\",\n",
    "    instructions=\"You are a personal math tutor. Write and run code to answer math questions.\",\n",
    "    tools=tools,\n",
    "    model=\"gpt-4-1106-preview\",\n",
    "    as_agent=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b76cb669-6aba-4827-868f-00aa960026f2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain.schema.agent import AgentFinish\n",
    "\n",
    "def execute_agent(agent, tools, input):\n",
    "    tool_map = {tool.name: tool for tool in tools}\n",
    "    response = agent.invoke(input)\n",
    "    while not isinstance(response, AgentFinish):\n",
    "        tool_outputs = []\n",
    "        for action in response:\n",
    "            tool_output = tool_map[action.tool].invoke(action.tool_input)\n",
    "            print(action.tool, action.tool_input, tool_output, end=\"\\n\\n\")\n",
    "            tool_outputs.append({\"output\": tool_output, \"tool_call_id\": action.tool_call_id})\n",
    "        response = agent.invoke({\"tool_outputs\": tool_outputs, \"run_id\": action.run_id, \"thread_id\": action.thread_id})\n",
    "        \n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7946116a-b82f-492e-835e-ca958a8949a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "e2b_data_analysis {'python_code': 'print(10 - 4 ** 2.7)'} {\"stdout\": \"-32.22425314473263\", \"stderr\": \"\", \"artifacts\": []}\n",
      "\n",
      "\\( 10 - 4^{2.7} \\) is approximately \\(-32.22425314473263\\).\n"
     ]
    }
   ],
   "source": [
    "response = execute_agent(agent, tools, {\"content\": \"What's 10 - 4 raised to the 2.7\"})\n",
    "print(response.return_values[\"output\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f2744a56-9f4f-4899-827a-fa55821c318c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "e2b_data_analysis {'python_code': 'result = 10 - 4 ** 2.7\\nprint(result + 17.241)'} {\"stdout\": \"-14.983253144732629\", \"stderr\": \"\", \"artifacts\": []}\n",
      "\n",
      "When you add \\( 17.241 \\) to \\( 10 - 4^{2.7} \\), the result is approximately \\( -14.98325314473263 \\).\n"
     ]
    }
   ],
   "source": [
    "next_response = execute_agent(agent, tools, {\"content\": \"now add 17.241\", \"thread_id\": response.thread_id})\n",
    "print(next_response.return_values[\"output\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71c34763-d1e7-4b9a-a9d7-3e4cc0dfc2c4",
   "metadata": {},
   "source": [
    "## [JSON mode](https://platform.openai.com/docs/guides/text-generation/json-mode)\n",
    "\n",
    "Constrain the model to only generate valid JSON. Note that you must include a system message with instructions to use JSON for this mode to work.\n",
    "\n",
    "Only works with certain models. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db6072c4-f3f3-415d-872b-71ea9f3c02bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "chat = ChatOpenAI(model=\"gpt-3.5-turbo-1106\").bind(\n",
    "    response_format={\"type\": \"json_object\"}\n",
    ")\n",
    "\n",
    "output = chat.invoke(\n",
    "    [\n",
    "        SystemMessage(\n",
    "            content=\"Extract the 'name' and 'origin' of any companies mentioned in the following statement. Return a JSON list.\"\n",
    "        ),\n",
    "        HumanMessage(\n",
    "            content=\"Google was founded in the USA, while Deepmind was founded in the UK\"\n",
    "        ),\n",
    "    ]\n",
    ")\n",
    "print(output.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08e00ccf-b991-4249-846b-9500a0ccbfa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "json.loads(output.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9a94d9-4319-4ab7-a979-c475ce6b5f50",
   "metadata": {},
   "source": [
    "## [System fingerprint](https://platform.openai.com/docs/guides/text-generation/reproducible-outputs)\n",
    "\n",
    "OpenAI sometimes changes model configurations in a way that impacts outputs. Whenever this happens, the system_fingerprint associated with a generation will change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1281883c-bf8f-4665-89cd-4f33ccde69ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "chat = ChatOpenAI(model=\"gpt-3.5-turbo-1106\")\n",
    "output = chat.generate(\n",
    "    [\n",
    "        [\n",
    "            SystemMessage(\n",
    "                content=\"Extract the 'name' and 'origin' of any companies mentioned in the following statement. Return a JSON list.\"\n",
    "            ),\n",
    "            HumanMessage(\n",
    "                content=\"Google was founded in the USA, while Deepmind was founded in the UK\"\n",
    "            ),\n",
    "        ]\n",
    "    ]\n",
    ")\n",
    "print(output.llm_output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa6565be-985d-4127-848e-c3bca9d7b434",
   "metadata": {},
   "source": [
    "## Breaking changes to Azure classes\n",
    "\n",
    "OpenAI V1 rewrote their clients and separated Azure and OpenAI clients. This has led to some changes in LangChain interfaces when using OpenAI V1.\n",
    "\n",
    "BREAKING CHANGES:\n",
    "- To use Azure embeddings with OpenAI V1, you'll need to use the new `AzureOpenAIEmbeddings` instead of the existing `OpenAIEmbeddings`. `OpenAIEmbeddings` continue to work when using Azure with `openai<1`.\n",
    "```python\n",
    "from langchain.embeddings import AzureOpenAIEmbeddings\n",
    "```\n",
    "\n",
    "\n",
    "RECOMMENDED CHANGES:\n",
    "- When using AzureChatOpenAI, if passing in an Azure endpoint (eg https://example-resource.azure.openai.com/) this should be specified via the `azure_endpoint` parameter or the `AZURE_OPENAI_ENDPOINT`. We're maintaining backwards compatibility for now with specifying this via `openai_api_base`/`base_url` or env var `OPENAI_API_BASE` but this shouldn't be relied upon.\n",
    "- When using Azure chat or embedding models, pass in API keys either via `openai_api_key` parameter or `AZURE_OPENAI_API_KEY` parameter. We're maintaining backwards compatibility for now with specifying this via `OPENAI_API_KEY` but this shouldn't be relied upon."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49944887-3972-497e-8da2-6d32d44345a9",
   "metadata": {},
   "source": [
    "## Tools\n",
    "\n",
    "Use tools for parallel function calling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "916292d8-0f89-40a6-af1c-5a1122327de8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[GetCurrentWeather(location='New York, NY', unit='fahrenheit'),\n",
       " GetCurrentWeather(location='Los Angeles, CA', unit='fahrenheit'),\n",
       " GetCurrentWeather(location='San Francisco, CA', unit='fahrenheit')]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from typing import Literal\n",
    "\n",
    "from langchain.output_parsers.openai_tools import PydanticToolsParser\n",
    "from langchain.utils.openai_functions import convert_pydantic_to_openai_tool\n",
    "from langchain.prompts import ChatPromptTemplate\n",
    "from langchain.pydantic_v1 import BaseModel, Field\n",
    "\n",
    "class GetCurrentWeather(BaseModel):\n",
    "    \"\"\"Get the current weather in a location.\"\"\"\n",
    "    location: str = Field(description=\"The city and state, e.g. San Francisco, CA\")\n",
    "    unit: Literal[\"celsius\", \"fahrenheit\"] = Field(default=\"fahrenheit\", description=\"The temperature unit, default to fahrenheit\")\n",
    "    \n",
    "prompt = ChatPromptTemplate.from_messages([\n",
    "    (\"system\", \"You are a helpful assistant\"),\n",
    "    (\"user\", \"{input}\")\n",
    "])\n",
    "model = ChatOpenAI(model=\"gpt-3.5-turbo-1106\").bind(tools=[convert_pydantic_to_openai_tool(GetCurrentWeather)])\n",
    "chain = prompt | model | PydanticToolsParser(tools=[GetCurrentWeather])\n",
    "\n",
    "chain.invoke({\"input\": \"what's the weather in NYC, LA, and SF\"})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "poetry-venv",
   "language": "python",
   "name": "poetry-venv"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}