mirror of https://github.com/hwchase17/langchain
Update Banana.dev docs to latest correct usage (#10183)
- Description: this PR updates all Banana.dev-related docs to match the latest client usage. The code in the docs before this PR were out of date and would never run. - Issue: [#6404](https://github.com/langchain-ai/langchain/issues/6404) - Dependencies: - - Tag maintainer: - Twitter handle: [BananaDev_ ](https://twitter.com/BananaDev_ )pull/10216/head
parent
9e839d4977
commit
49341483da
@ -1,79 +1,72 @@
|
|||||||
# Banana
|
# Banana
|
||||||
|
|
||||||
This page covers how to use the Banana ecosystem within LangChain.
|
Banana provided serverless GPU inference for AI models, including a CI/CD build pipeline and a simple Python framework (Potassium) to server your models.
|
||||||
It is broken into two parts: installation and setup, and then references to specific Banana wrappers.
|
|
||||||
|
This page covers how to use the [Banana](https://www.banana.dev) ecosystem within LangChain.
|
||||||
|
|
||||||
|
It is broken into two parts:
|
||||||
|
* installation and setup,
|
||||||
|
* and then references to specific Banana wrappers.
|
||||||
|
|
||||||
## Installation and Setup
|
## Installation and Setup
|
||||||
|
|
||||||
- Install with `pip install banana-dev`
|
- Install with `pip install banana-dev`
|
||||||
- Get an Banana api key and set it as an environment variable (`BANANA_API_KEY`)
|
- Get an Banana api key from the [Banana.dev dashboard](https://app.banana.dev) and set it as an environment variable (`BANANA_API_KEY`)
|
||||||
|
- Get your model's key and url slug from the model's details page
|
||||||
|
|
||||||
## Define your Banana Template
|
## Define your Banana Template
|
||||||
|
|
||||||
If you want to use an available language model template you can find one [here](https://app.banana.dev/templates/conceptofmind/serverless-template-palmyra-base).
|
You'll need to set up a Github repo for your Banana app. You can get started in 5 minutes using [this guide](https://docs.banana.dev/banana-docs/).
|
||||||
This template uses the Palmyra-Base model by [Writer](https://writer.com/product/api/).
|
|
||||||
You can check out an example Banana repository [here](https://github.com/conceptofmind/serverless-template-palmyra-base).
|
Alternatively, for a ready-to-go LLM example, you can check out Banana's [CodeLlama-7B-Instruct-GPTQ](https://github.com/bananaml/demo-codellama-7b-instruct-gptq) GitHub repository. Just fork it and deploy it within Banana.
|
||||||
|
|
||||||
|
Other starter repos are available [here](https://github.com/orgs/bananaml/repositories?q=demo-&type=all&language=&sort=).
|
||||||
|
|
||||||
## Build the Banana app
|
## Build the Banana app
|
||||||
|
|
||||||
Banana Apps must include the "output" key in the return json.
|
To use Banana apps within Langchain, they must include the `outputs` key
|
||||||
There is a rigid response structure.
|
in the returned json, and the value must be a string.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
# Return the results as a dictionary
|
# Return the results as a dictionary
|
||||||
result = {'output': result}
|
result = {'outputs': result}
|
||||||
```
|
```
|
||||||
|
|
||||||
An example inference function would be:
|
An example inference function would be:
|
||||||
|
|
||||||
```python
|
```python
|
||||||
def inference(model_inputs:dict) -> dict:
|
@app.handler("/")
|
||||||
global model
|
def handler(context: dict, request: Request) -> Response:
|
||||||
global tokenizer
|
"""Handle a request to generate code from a prompt."""
|
||||||
|
model = context.get("model")
|
||||||
# Parse out your arguments
|
tokenizer = context.get("tokenizer")
|
||||||
prompt = model_inputs.get('prompt', None)
|
max_new_tokens = request.json.get("max_new_tokens", 512)
|
||||||
if prompt == None:
|
temperature = request.json.get("temperature", 0.7)
|
||||||
return {'message': "No prompt provided"}
|
prompt = request.json.get("prompt")
|
||||||
|
prompt_template=f'''[INST] Write code to solve the following coding problem that obeys the constraints and passes the example test cases. Please wrap your code answer using ```:
|
||||||
# Run the model
|
{prompt}
|
||||||
input_ids = tokenizer.encode(prompt, return_tensors='pt').cuda()
|
[/INST]
|
||||||
output = model.generate(
|
'''
|
||||||
input_ids,
|
input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
|
||||||
max_length=100,
|
output = model.generate(inputs=input_ids, temperature=temperature, max_new_tokens=max_new_tokens)
|
||||||
do_sample=True,
|
result = tokenizer.decode(output[0])
|
||||||
top_k=50,
|
return Response(json={"outputs": result}, status=200)
|
||||||
top_p=0.95,
|
|
||||||
num_return_sequences=1,
|
|
||||||
temperature=0.9,
|
|
||||||
early_stopping=True,
|
|
||||||
no_repeat_ngram_size=3,
|
|
||||||
num_beams=5,
|
|
||||||
length_penalty=1.5,
|
|
||||||
repetition_penalty=1.5,
|
|
||||||
bad_words_ids=[[tokenizer.encode(' ', add_prefix_space=True)[0]]]
|
|
||||||
)
|
|
||||||
|
|
||||||
result = tokenizer.decode(output[0], skip_special_tokens=True)
|
|
||||||
# Return the results as a dictionary
|
|
||||||
result = {'output': result}
|
|
||||||
return result
|
|
||||||
```
|
```
|
||||||
|
|
||||||
You can find a full example of a Banana app [here](https://github.com/conceptofmind/serverless-template-palmyra-base/blob/main/app.py).
|
This example is from the `app.py` file in [CodeLlama-7B-Instruct-GPTQ](https://github.com/bananaml/demo-codellama-7b-instruct-gptq).
|
||||||
|
|
||||||
## Wrappers
|
## Wrappers
|
||||||
|
|
||||||
### LLM
|
### LLM
|
||||||
|
|
||||||
There exists an Banana LLM wrapper, which you can access with
|
Within Langchain, there exists a Banana LLM wrapper, which you can access with
|
||||||
|
|
||||||
```python
|
```python
|
||||||
from langchain.llms import Banana
|
from langchain.llms import Banana
|
||||||
```
|
```
|
||||||
|
|
||||||
You need to provide a model key located in the dashboard:
|
You need to provide a model key and model url slug, which you can get from the model's details page in the [Banana.dev dashboard](https://app.banana.dev).
|
||||||
|
|
||||||
```python
|
```python
|
||||||
llm = Banana(model_key="YOUR_MODEL_KEY")
|
llm = Banana(model_key="YOUR_MODEL_KEY", model_url_slug="YOUR_MODEL_URL_SLUG")
|
||||||
```
|
```
|
||||||
|
Loading…
Reference in New Issue