mirror of
https://github.com/hwchase17/langchain
synced 2024-10-29 17:07:25 +00:00
9fbe346860
This PR adds a new LLM class for the Amazon API Gateway hosted LLM. The PR also includes example notebooks for using the LLM class in an Agent chain. --------- Co-authored-by: Dev 2049 <dev.dev2049@gmail.com>
73 lines
2.7 KiB
Plaintext
73 lines
2.7 KiB
Plaintext
# Amazon API Gateway
|
|
|
|
[Amazon API Gateway](https://aws.amazon.com/api-gateway/) is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. APIs act as the "front door" for applications to access data, business logic, or functionality from your backend services. Using API Gateway, you can create RESTful APIs and WebSocket APIs that enable real-time two-way communication applications. API Gateway supports containerized and serverless workloads, as well as web applications.
|
|
|
|
API Gateway handles all the tasks involved in accepting and processing up to hundreds of thousands of concurrent API calls, including traffic management, CORS support, authorization and access control, throttling, monitoring, and API version management. API Gateway has no minimum fees or startup costs. You pay for the API calls you receive and the amount of data transferred out and, with the API Gateway tiered pricing model, you can reduce your cost as your API usage scales.
|
|
|
|
## LLM
|
|
|
|
See a [usage example](/docs/modules/model_io/models/llms/integrations/amazon_api_gateway_example.html).
|
|
|
|
```python
|
|
from langchain.llms import AmazonAPIGateway
|
|
|
|
api_url = "https://<api_gateway_id>.execute-api.<region>.amazonaws.com/LATEST/HF"
|
|
llm = AmazonAPIGateway(api_url=api_url)
|
|
|
|
# These are sample parameters for Falcon 40B Instruct Deployed from Amazon SageMaker JumpStart
|
|
parameters = {
|
|
"max_new_tokens": 100,
|
|
"num_return_sequences": 1,
|
|
"top_k": 50,
|
|
"top_p": 0.95,
|
|
"do_sample": False,
|
|
"return_full_text": True,
|
|
"temperature": 0.2,
|
|
}
|
|
|
|
prompt = "what day comes after Friday?"
|
|
llm.model_kwargs = parameters
|
|
llm(prompt)
|
|
>>> 'what day comes after Friday?\nSaturday'
|
|
```
|
|
|
|
## Agent
|
|
|
|
```python
|
|
from langchain.agents import load_tools
|
|
from langchain.agents import initialize_agent
|
|
from langchain.agents import AgentType
|
|
from langchain.llms import AmazonAPIGateway
|
|
|
|
api_url = "https://<api_gateway_id>.execute-api.<region>.amazonaws.com/LATEST/HF"
|
|
llm = AmazonAPIGateway(api_url=api_url)
|
|
|
|
parameters = {
|
|
"max_new_tokens": 50,
|
|
"num_return_sequences": 1,
|
|
"top_k": 250,
|
|
"top_p": 0.25,
|
|
"do_sample": False,
|
|
"temperature": 0.1,
|
|
}
|
|
|
|
llm.model_kwargs = parameters
|
|
|
|
# Next, let's load some tools to use. Note that the `llm-math` tool uses an LLM, so we need to pass that in.
|
|
tools = load_tools(["python_repl", "llm-math"], llm=llm)
|
|
|
|
# Finally, let's initialize an agent with the tools, the language model, and the type of agent we want to use.
|
|
agent = initialize_agent(
|
|
tools,
|
|
llm,
|
|
agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION,
|
|
verbose=True,
|
|
)
|
|
|
|
# Now let's test it out!
|
|
agent.run("""
|
|
Write a Python script that prints "Hello, world!"
|
|
""")
|
|
|
|
>>> 'Hello, world!'
|
|
``` |