mirror of https://github.com/hwchase17/langchain
Add Portkey LLMOps integration (#7877)
Integrating Portkey, which adds production features like caching, tracing, tagging, retries, etc. to langchain apps. - Dependencies: None - Twitter handle: https://twitter.com/portkeyai - test_portkey.py added for tests - example notebook added in new utilities folder in modules Also fixed a bug with OpenAIEmbeddings where headers weren't passing. cc @baskaryan --------- Co-authored-by: Bagatur <baskaryan@gmail.com>pull/7894/head
vrushankportkey
1 year ago
committed by
GitHub
parent
095937ad52
commit
5f10d2ea1d
Binary file not shown.
|
After Width: | Height: | Size: 483 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 291 KiB |
@ -0,0 +1,107 @@
|
||||
# Portkey
|
||||
## LLMOps for Langchain
|
||||
|
||||
Portkey brings production readiness to Langchain. With Portkey, you can
|
||||
- [x] view detailed **metrics & logs** for all requests,
|
||||
- [x] enable **semantic cache** to reduce latency & costs,
|
||||
- [x] implement automatic **retries & fallbacks** for failed requests,
|
||||
- [x] add **custom tags** to requests for better tracking and analysis and [more](https://docs.portkey.ai).
|
||||
|
||||
### Using Portkey with Langchain
|
||||
Using Portkey is as simple as just choosing which Portkey features you want, enabling them via `headers=Portkey.Config` and passing it in your LLM calls.
|
||||
|
||||
To start, get your Portkey API key by [signing up here](https://app.portkey.ai/login). (Click the profile icon on the top left, then click on "Copy API Key")
|
||||
|
||||
For OpenAI, a simple integration with logging feature would look like this:
|
||||
```python
|
||||
from langchain.llms import OpenAI
|
||||
from langchain.utilities import Portkey
|
||||
|
||||
# Add the Portkey API Key from your account
|
||||
headers = Portkey.Config(
|
||||
api_key = "<PORTKEY_API_KEY>"
|
||||
)
|
||||
|
||||
llm = OpenAI(temperature=0.9, headers=headers)
|
||||
llm.predict("What would be a good company name for a company that makes colorful socks?")
|
||||
```
|
||||
Your logs will be captured on your [Portkey dashboard](https://app.portkey.ai).
|
||||
|
||||
A common Portkey X Langchain use case is to **trace a chain or an agent** and view all the LLM calls originating from that request.
|
||||
|
||||
### **Tracing Chains & Agents**
|
||||
|
||||
```python
|
||||
from langchain.agents import AgentType, initialize_agent, load_tools
|
||||
from langchain.llms import OpenAI
|
||||
from langchain.utilities import Portkey
|
||||
|
||||
# Add the Portkey API Key from your account
|
||||
headers = Portkey.Config(
|
||||
api_key = "<PORTKEY_API_KEY>",
|
||||
trace_id = "fef659"
|
||||
)
|
||||
|
||||
llm = OpenAI(temperature=0, headers=headers)
|
||||
tools = load_tools(["serpapi", "llm-math"], llm=llm)
|
||||
agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
|
||||
|
||||
# Let's test it out!
|
||||
agent.run("What was the high temperature in SF yesterday in Fahrenheit? What is that number raised to the .023 power?")
|
||||
```
|
||||
|
||||
**You can see the requests' logs along with the trace id on Portkey dashboard:**
|
||||
|
||||
<img src="/img/portkey-dashboard.gif" height="250"/>
|
||||
<img src="/img/portkey-tracing.png" height="250"/>
|
||||
|
||||
## Advanced Features
|
||||
|
||||
1. **Logging:** Log all your LLM requests automatically by sending them through Portkey. Each request log contains `timestamp`, `model name`, `total cost`, `request time`, `request json`, `response json`, and additional Portkey features.
|
||||
2. **Tracing:** Trace id can be passed along with each request and is visibe on the logs on Portkey dashboard. You can also set a **distinct trace id** for each request. You can [append user feedback](https://docs.portkey.ai/key-features/feedback-api) to a trace id as well.
|
||||
3. **Caching:** Respond to previously served customers queries from cache instead of sending them again to OpenAI. Match exact strings OR semantically similar strings. Cache can save costs and reduce latencies by 20x.
|
||||
4. **Retries:** Automatically reprocess any unsuccessful API requests **`upto 5`** times. Uses an **`exponential backoff`** strategy, which spaces out retry attempts to prevent network overload.
|
||||
5. **Tagging:** Track and audit each user interaction in high detail with predefined tags.
|
||||
|
||||
| Feature | Config Key | Value (Type) | Required/Optional |
|
||||
| -- | -- | -- | -- |
|
||||
| API Key | `api_key` | API Key (`string`) | ✅ Required |
|
||||
| [Tracing Requests](https://docs.portkey.ai/key-features/request-tracing) | `trace_id` | Custom `string` | ❔ Optional |
|
||||
| [Automatic Retries](https://docs.portkey.ai/key-features/automatic-retries) | `retry_count` | `integer` [1,2,3,4,5] | ❔ Optional |
|
||||
| [Enabling Cache](https://docs.portkey.ai/key-features/request-caching) | `cache` | `simple` OR `semantic` | ❔ Optional |
|
||||
| Cache Force Refresh | `cache_force_refresh` | `True` | ❔ Optional |
|
||||
| Set Cache Expiry | `cache_age` | `integer` (in seconds) | ❔ Optional |
|
||||
| [Add User](https://docs.portkey.ai/key-features/custom-metadata) | `user` | `string` | ❔ Optional |
|
||||
| [Add Organisation](https://docs.portkey.ai/key-features/custom-metadata) | `organisation` | `string` | ❔ Optional |
|
||||
| [Add Environment](https://docs.portkey.ai/key-features/custom-metadata) | `environment` | `string` | ❔ Optional |
|
||||
| [Add Prompt (version/id/string)](https://docs.portkey.ai/key-features/custom-metadata) | `prompt` | `string` | ❔ Optional |
|
||||
|
||||
|
||||
## **Enabling all Portkey Features:**
|
||||
|
||||
```py
|
||||
headers = Portkey.Config(
|
||||
|
||||
# Mandatory
|
||||
api_key="<PORTKEY_API_KEY>",
|
||||
|
||||
# Cache Options
|
||||
cache="semantic",
|
||||
cache_force_refresh="True",
|
||||
cache_age=1729,
|
||||
|
||||
# Advanced
|
||||
retry_count=5,
|
||||
trace_id="langchain_agent",
|
||||
|
||||
# Metadata
|
||||
environment="production",
|
||||
user="john",
|
||||
organisation="acme",
|
||||
prompt="Frost"
|
||||
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
For detailed information on each feature and how to use it, [please refer to the Portkey docs](https://docs.portkey.ai). If you have any questions or need further assistance, [reach out to us on Twitter.](https://twitter.com/portkeyai).
|
||||
@ -0,0 +1,242 @@
|
||||
{
|
||||
"cells": [
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": []
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"# Log, Trace, and Monitor Langchain LLM Calls\n",
|
||||
"\n",
|
||||
"When building apps or agents using Langchain, you end up making multiple API calls to fulfill a single user request. However, these requests are not chained when you want to analyse them. With [**Portkey**](/docs/ecosystem/integrations/portkey), all the embeddings, completion, and other requests from a single user request will get logged and traced to a common ID, enabling you to gain full visibility of user interactions.\n",
|
||||
"\n",
|
||||
"This notebook serves as a step-by-step guide on how to integrate and use Portkey in your Langchain app."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"First, let's import Portkey, OpenAI, and Agent tools"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"import os\n",
|
||||
"\n",
|
||||
"from langchain.agents import AgentType, initialize_agent, load_tools\n",
|
||||
"from langchain.llms import OpenAI\n",
|
||||
"from langchain.utilities import Portkey"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Paste your OpenAI API key below. [(You can find it here)](https://platform.openai.com/account/api-keys)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"os.environ[\"OPENAI_API_KEY\"] = \"<OPENAI_API_KEY>\""
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Get Portkey API Key\n",
|
||||
"1. Sign up for [Portkey here](https://app.portkey.ai/login)\n",
|
||||
"2. On your [dashboard](https://app.portkey.ai/), click on the profile icon on the top left, then click on \"Copy API Key\"\n",
|
||||
"3. Paste it below"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"PORTKEY_API_KEY = \"<PORTKEY_API_KEY>\" # Paste your Portkey API Key here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Set Trace ID\n",
|
||||
"1. Set the trace id for your request below\n",
|
||||
"2. The Trace ID can be common for all API calls originating from a single request"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"TRACE_ID = \"portkey_langchain_demo\" # Set trace id here"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Generate Portkey Headers"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"headers = Portkey.Config(\n",
|
||||
" api_key=PORTKEY_API_KEY,\n",
|
||||
" trace_id=TRACE_ID,\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"Run your agent as usual. The **only** change is that we will **include the above headers** in the request now."
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"llm = OpenAI(temperature=0, headers=headers)\n",
|
||||
"tools = load_tools([\"serpapi\", \"llm-math\"], llm=llm)\n",
|
||||
"agent = initialize_agent(\n",
|
||||
" tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"# Let's test it out!\n",
|
||||
"agent.run(\n",
|
||||
" \"What was the high temperature in SF yesterday in Fahrenheit? What is that number raised to the .023 power?\"\n",
|
||||
")"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## How Logging & Tracing Works on Portkey\n",
|
||||
"\n",
|
||||
"**Logging**\n",
|
||||
"- Sending your request through Portkey ensures that all of the requests are logged by default\n",
|
||||
"- Each request log contains `timestamp`, `model name`, `total cost`, `request time`, `request json`, `response json`, and additional Portkey features\n",
|
||||
"\n",
|
||||
"**Tracing**\n",
|
||||
"- Trace id is passed along with each request and is visibe on the logs on Portkey dashboard\n",
|
||||
"- You can also set a **distinct trace id** for each request if you want\n",
|
||||
"- You can append user feedback to a trace id as well. [More info on this here](https://docs.portkey.ai/key-features/feedback-api)"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Advanced LLMOps Features - Caching, Tagging, Retries\n",
|
||||
"\n",
|
||||
"In addition to logging and tracing, Portkey provides more features that add production capabilities to your existing workflows:\n",
|
||||
"\n",
|
||||
"**Caching**\n",
|
||||
"\n",
|
||||
"Respond to previously served customers queries from cache instead of sending them again to OpenAI. Match exact strings OR semantically similar strings. Cache can save costs and reduce latencies by 20x.\n",
|
||||
"\n",
|
||||
"**Retries**\n",
|
||||
"\n",
|
||||
"Automatically reprocess any unsuccessful API requests **`upto 5`** times. Uses an **`exponential backoff`** strategy, which spaces out retry attempts to prevent network overload.\n",
|
||||
"\n",
|
||||
"| Feature | Config Key | Value (Type) |\n",
|
||||
"| -- | -- | -- |\n",
|
||||
"| [🔁 Automatic Retries](https://docs.portkey.ai/key-features/automatic-retries) | `retry_count` | `integer` [1,2,3,4,5] |\n",
|
||||
"| [🧠 Enabling Cache](https://docs.portkey.ai/key-features/request-caching) | `cache` | `simple` OR `semantic` |\n",
|
||||
"\n",
|
||||
"**Tagging**\n",
|
||||
"\n",
|
||||
"Track and audit ach user interaction in high detail with predefined tags.\n",
|
||||
"\n",
|
||||
"| Tag | Config Key | Value (Type) |\n",
|
||||
"| -- | -- | -- |\n",
|
||||
"| User Tag | `user` | `string` |\n",
|
||||
"| Organisation Tag | `organisation` | `string` |\n",
|
||||
"| Environment Tag | `environment` | `string` |\n",
|
||||
"| Prompt Tag (version/id/string) | `prompt` | `string` |"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "markdown",
|
||||
"metadata": {},
|
||||
"source": [
|
||||
"## Code Example With All Features"
|
||||
]
|
||||
},
|
||||
{
|
||||
"cell_type": "code",
|
||||
"execution_count": null,
|
||||
"metadata": {},
|
||||
"outputs": [],
|
||||
"source": [
|
||||
"headers = Portkey.Config(\n",
|
||||
" # Mandatory\n",
|
||||
" api_key=\"<PORTKEY_API_KEY>\",\n",
|
||||
" # Cache Options\n",
|
||||
" cache=\"semantic\",\n",
|
||||
" cache_force_refresh=\"True\",\n",
|
||||
" cache_age=1729,\n",
|
||||
" # Advanced\n",
|
||||
" retry_count=5,\n",
|
||||
" trace_id=\"langchain_agent\",\n",
|
||||
" # Metadata\n",
|
||||
" environment=\"production\",\n",
|
||||
" user=\"john\",\n",
|
||||
" organisation=\"acme\",\n",
|
||||
" prompt=\"Frost\",\n",
|
||||
")\n",
|
||||
"\n",
|
||||
"llm = OpenAI(temperature=0.9, headers=headers)\n",
|
||||
"\n",
|
||||
"print(llm(\"Two roads diverged in the yellow woods\"))"
|
||||
]
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"kernelspec": {
|
||||
"display_name": "Python 3 (ipykernel)",
|
||||
"language": "python",
|
||||
"name": "python3"
|
||||
},
|
||||
"language_info": {
|
||||
"codemirror_mode": {
|
||||
"name": "ipython",
|
||||
"version": 3
|
||||
},
|
||||
"file_extension": ".py",
|
||||
"mimetype": "text/x-python",
|
||||
"name": "python",
|
||||
"nbconvert_exporter": "python",
|
||||
"pygments_lexer": "ipython3",
|
||||
"version": "3.11.3"
|
||||
}
|
||||
},
|
||||
"nbformat": 4,
|
||||
"nbformat_minor": 2
|
||||
}
|
||||
@ -0,0 +1,68 @@
|
||||
import json
|
||||
import os
|
||||
from typing import Dict, Optional
|
||||
|
||||
|
||||
class Portkey:
|
||||
base = "https://api.portkey.ai/v1/proxy"
|
||||
|
||||
@staticmethod
|
||||
def Config(
|
||||
api_key: str,
|
||||
trace_id: Optional[str] = None,
|
||||
environment: Optional[str] = None,
|
||||
user: Optional[str] = None,
|
||||
organisation: Optional[str] = None,
|
||||
prompt: Optional[str] = None,
|
||||
retry_count: Optional[int] = None,
|
||||
cache: Optional[str] = None,
|
||||
cache_force_refresh: Optional[str] = None,
|
||||
cache_age: Optional[int] = None,
|
||||
) -> Dict[str, str]:
|
||||
assert retry_count is None or retry_count in range(
|
||||
1, 6
|
||||
), "retry_count must be an integer and in range [1, 2, 3, 4, 5]"
|
||||
assert cache is None or cache in [
|
||||
"simple",
|
||||
"semantic",
|
||||
], "cache must be 'simple' or 'semantic'"
|
||||
assert cache_force_refresh is None or (
|
||||
isinstance(cache_force_refresh, str)
|
||||
and cache_force_refresh in ["True", "False"]
|
||||
), "cache_force_refresh must be 'True' or 'False'"
|
||||
assert cache_age is None or isinstance(
|
||||
cache_age, int
|
||||
), "cache_age must be an integer"
|
||||
|
||||
os.environ["OPENAI_API_BASE"] = Portkey.base
|
||||
|
||||
headers = {
|
||||
"x-portkey-api-key": api_key,
|
||||
"x-portkey-mode": "proxy openai",
|
||||
}
|
||||
|
||||
if trace_id:
|
||||
headers["x-portkey-trace-id"] = trace_id
|
||||
if retry_count:
|
||||
headers["x-portkey-retry-count"] = str(retry_count)
|
||||
if cache:
|
||||
headers["x-portkey-cache"] = cache
|
||||
if cache_force_refresh:
|
||||
headers["x-portkey-cache-force-refresh"] = cache_force_refresh
|
||||
if cache_age:
|
||||
headers["Cache-Control"] = f"max-age:{str(cache_age)}"
|
||||
|
||||
metadata = {}
|
||||
if environment:
|
||||
metadata["_environment"] = environment
|
||||
if user:
|
||||
metadata["_user"] = user
|
||||
if organisation:
|
||||
metadata["_organisation"] = organisation
|
||||
if prompt:
|
||||
metadata["_prompt"] = prompt
|
||||
|
||||
if metadata:
|
||||
headers.update({"x-portkey-metadata": json.dumps(metadata)})
|
||||
|
||||
return headers
|
||||
@ -0,0 +1,31 @@
|
||||
import json
|
||||
|
||||
from langchain.utilities import Portkey
|
||||
|
||||
|
||||
def test_Config() -> None:
|
||||
headers = Portkey.Config(
|
||||
api_key="test_api_key",
|
||||
environment="test_environment",
|
||||
user="test_user",
|
||||
organisation="test_organisation",
|
||||
prompt="test_prompt",
|
||||
retry_count=3,
|
||||
trace_id="test_trace_id",
|
||||
cache="simple",
|
||||
cache_force_refresh="True",
|
||||
cache_age=3600,
|
||||
)
|
||||
|
||||
assert headers["x-portkey-api-key"] == "test_api_key"
|
||||
assert headers["x-portkey-trace-id"] == "test_trace_id"
|
||||
assert headers["x-portkey-retry-count"] == "3"
|
||||
assert headers["x-portkey-cache"] == "simple"
|
||||
assert headers["x-portkey-cache-force-refresh"] == "True"
|
||||
assert headers["Cache-Control"] == "max-age:3600"
|
||||
|
||||
metadata = json.loads(headers["x-portkey-metadata"])
|
||||
assert metadata["_environment"] == "test_environment"
|
||||
assert metadata["_user"] == "test_user"
|
||||
assert metadata["_organisation"] == "test_organisation"
|
||||
assert metadata["_prompt"] == "test_prompt"
|
||||
Loading…
Reference in New Issue