2023-03-20 08:27:20 +00:00
# EVAL
2023-03-18 09:51:31 +00:00
2023-03-21 04:41:10 +00:00
> **EVAL(Elastic Versatile Agent with Langchain) will execute all your requests. Like the eval method!**
2023-04-05 11:08:44 +00:00
>
> You don't have to think about how. If you tell them the results you want, they'll search, code, run, and test the Internet themselves, and they'll return the final results.
2023-03-21 04:41:10 +00:00
2023-04-04 18:56:15 +00:00
https://user-images.githubusercontent.com/19206046/229892113-481437b7-a332-4e0c-bfb3-d2c97c9035be.mp4
2023-04-04 18:38:08 +00:00
2023-04-05 11:08:44 +00:00
EVAL Making a full-fledged web application with multiple files
https://user-images.githubusercontent.com/51526347/230061897-b3479405-8ebd-45ab-a432-6506730242b9.mov
EVAL Making a UI for itself
2023-03-21 04:41:10 +00:00
2023-04-06 00:50:53 +00:00
#### [EVAL-BOT](https://github.com/eval-bot)
2023-04-05 17:40:27 +00:00
EVAL's self-managed github account. EVAL does everything except for signup and bio setting.
2023-04-06 00:50:53 +00:00
### Examples
[Here ](examples/ ) is an example.
2023-03-21 04:41:10 +00:00
### EVAL's FEATURE
2023-04-05 01:39:51 +00:00
1. **Multimodal Conversation**
- It **understands** and **generates** data formats for text, image, dataframe, audio (TODO), video (TODO).
2. **Services**
- It can serve **services (blocking processes)** such as web apps.
3. **Evolving**
- It can **create it's own tools** by writing, modifying, executing and testing code.
2023-03-21 04:41:10 +00:00
### BUILT-IN TOOLS
2023-03-22 02:57:32 +00:00
1. Terminal
2023-04-04 18:38:08 +00:00
- SyscallTracer
2023-03-22 02:57:32 +00:00
2. Code Editor
- READ: Read and understand file.
- WRITE: Write code to create a new tool.
- PATCH: Correct the error throught the code patch if an error occurs.
- DELETE: Delete code in file for a new start.
3. Search
2023-03-21 04:41:10 +00:00
- Google, Bing, Wikipedia
- Custom DB Search (Currently using Corca's Wine Data in https://www.workershop.kr/en)
- Use GPT index to quickly find the information you need in a document and use that information to answer
- Requests.get (Get information from anywhere you want)
2023-04-04 18:38:08 +00:00
4. Image Understanding, Generation, Editing
2023-03-21 04:41:10 +00:00
- Image Understanding
- Image Understanding: blip-image-captioning
- Visual Question& Answering: blip-vqa
- Image Generation: Stable Diffusion 1.5
- Image Editing
- Replace or remove an object: Stable Diffusion Inpainting
- Change Image's style: InstructPix2Pix
Thanks to [LangChain ](https://github.com/hwchase17/langchain ), [Visual ChatGPT ](https://github.com/microsoft/visual-chatgpt ), [llama index ](https://github.com/jerryjliu/llama_index ).
### CUSTOM TOOLS
We also don't know what tools EVAL will create. Every day, It will create the right tools to execute your request.
---
2023-03-18 09:51:31 +00:00
## Usage
2023-04-05 15:56:09 +00:00
1. Environment variables
2. Run with docker-compose
3. Send request to EVAL
2023-03-18 06:05:02 +00:00
2023-04-05 15:56:09 +00:00
### 1. Environment Variables
2023-03-20 08:27:20 +00:00
2023-03-23 07:33:45 +00:00
You need to write some environment variables in the `.env` file. Refer [.env.example ](.env.example ) if you don't know how to format it.
2023-03-20 08:27:20 +00:00
2023-03-23 07:33:45 +00:00
**Mandatory**
2023-03-20 08:27:20 +00:00
2023-03-23 07:33:45 +00:00
Manatory envs are required in order to serve EVAL.
2023-03-20 08:27:20 +00:00
2023-03-23 07:33:45 +00:00
- `OPENAI_API_KEY` - OpenAI api key
**Optional**
Each optional env has default value, so you don't need to set unless you want to change it.
2023-04-07 02:11:24 +00:00
- `EVAL_PORT` - port (default: 8000)
2023-03-23 07:33:45 +00:00
- `SERVER` - server address (default: http://localhost:8000)
- `LOG_LEVEL` - INFO | DEBUG (default: INFO)
- `BOT_NAME` - give it a name! (default: Orca)
**For More Tools**
Some tools requires environment variables. Set envs depend on which tools you want to use.
- Google search tool
- `SERPAPI_API_KEY`
- Bing search tool
- `BING_SEARCH_URL`
- `BING_SUBSCRIPTION_KEY`
2023-03-20 08:27:20 +00:00
2023-04-05 15:56:09 +00:00
### 2. Run with docker-compose
- There are 2 services in docker-compose.yml
- `eval` - without GPU, much lighter
```bash
docker-compose up --build eval
```
- `eval.gpu` - with GPU, for multi-modal conversation
```bash
docker-compose up --build eval.gpu
```
- The one with GPU is much heavier and unstable for now because of the massive dependencies. We recommend you to use the one without GPU if you don't need multi-modal conversation.
### 3. Send request to EVAL
- `POST /command`
- `key` - session id
- `files` - urls of file inputs
- `query` - prompt
- You can send request to EVAL with `curl` or `httpie` .
```bash
curl -X POST -H "Content-Type: application/json" -d '{"key": "sessionid", "files": ["https://example.com/image.png"], "query": "Hi there!"}' http://localhost:8000/command
```
```bash
http POST http://localhost:8000/command key=sessionid files:='["https://example.com/image.png"]' query="Hi there!"
```
- We are planning to make a GUI for EVAL so you can use it without terminal.
2023-03-18 11:03:33 +00:00
## TODO
2023-03-18 06:05:02 +00:00
2023-04-05 15:56:09 +00:00
- [ ] GUI
2023-03-18 11:03:33 +00:00
- [ ] memory saving
- [ ] session manage
- [ ] convert to alpaca
- [ ] prompt upgrade
2023-03-20 08:27:20 +00:00
- [ ] give a tool to create tools
2023-03-18 11:03:33 +00:00
- [ ] etc.
2023-03-18 06:05:02 +00:00
2023-03-18 11:03:33 +00:00
## Reference
2023-03-18 06:05:02 +00:00
2023-03-18 11:03:33 +00:00
Thanks to the following repositories.
- https://github.com/hwchase17/langchain
- https://github.com/microsoft/visual-chatgpt
2023-03-21 04:41:10 +00:00
- https://github.com/jerryjliu/llama_index