mirror of https://github.com/nomic-ai/gpt4all
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
151 lines
4.6 KiB
Markdown
151 lines
4.6 KiB
Markdown
# GPT4All Node.js API
|
|
|
|
```sh
|
|
yarn add gpt4all@alpha
|
|
|
|
npm install gpt4all@alpha
|
|
|
|
pnpm install gpt4all@alpha
|
|
```
|
|
|
|
The original [GPT4All typescript bindings](https://github.com/nomic-ai/gpt4all-ts) are now out of date.
|
|
|
|
* New bindings created by [jacoobes](https://github.com/jacoobes), [limez](https://github.com/iimez) and the [nomic ai community](https://home.nomic.ai), for all to use.
|
|
* [Documentation](#Documentation)
|
|
|
|
### Chat Completion (alpha)
|
|
|
|
```js
|
|
import { createCompletion, loadModel } from '../src/gpt4all.js'
|
|
|
|
const ll = await loadModel('ggml-vicuna-7b-1.1-q4_2', { verbose: true });
|
|
|
|
const response = await createCompletion(ll, [
|
|
{ role : 'system', content: 'You are meant to be annoying and unhelpful.' },
|
|
{ role : 'user', content: 'What is 1 + 1?' }
|
|
]);
|
|
|
|
```
|
|
### Embedding (alpha)
|
|
```js
|
|
import { createEmbedding, loadModel } from '../src/gpt4all.js'
|
|
|
|
const ll = await loadModel('ggml-all-MiniLM-L6-v2-f16', { verbose: true });
|
|
|
|
const fltArray = createEmbedding(ll, "Pain is inevitable, suffering optional");
|
|
```
|
|
|
|
### API
|
|
|
|
* The nodejs api has made strides to mirror the python api. It is not 100% mirrored, but many pieces of the api resemble its python counterpart.
|
|
* Everything should work out the box.
|
|
* [docs](./docs/api.md)
|
|
|
|
### Build Instructions
|
|
|
|
* binding.gyp is compile config
|
|
* Tested on Ubuntu. Everything seems to work fine
|
|
* Tested on Windows. Everything works fine.
|
|
* Sparse testing on mac os.
|
|
* MingW works as well to build the gpt4all-backend. **HOWEVER**, this package works only with MSVC built dlls.
|
|
|
|
### Requirements
|
|
|
|
* git
|
|
* [node.js >= 18.0.0](https://nodejs.org/en)
|
|
* [yarn](https://yarnpkg.com/)
|
|
* [node-gyp](https://github.com/nodejs/node-gyp)
|
|
* all of its requirements.
|
|
* (unix) gcc version 12
|
|
* (win) msvc version 143
|
|
* Can be obtained with visual studio 2022 build tools
|
|
* python 3
|
|
### Build (from source)
|
|
|
|
```sh
|
|
git clone https://github.com/nomic-ai/gpt4all.git
|
|
cd gpt4all-bindings/typescript
|
|
```
|
|
|
|
* The below shell commands assume the current working directory is `typescript`.
|
|
|
|
* To Build and Rebuild:
|
|
|
|
```sh
|
|
yarn
|
|
```
|
|
|
|
* llama.cpp git submodule for gpt4all can be possibly absent. If this is the case, make sure to run in llama.cpp parent directory
|
|
|
|
```sh
|
|
git submodule update --init --depth 1 --recursive
|
|
```
|
|
|
|
**AS OF NEW BACKEND** to build the backend,
|
|
|
|
```sh
|
|
yarn build:backend
|
|
```
|
|
|
|
This will build platform-dependent dynamic libraries, and will be located in runtimes/(platform)/native The only current way to use them is to put them in the current working directory of your application. That is, **WHEREVER YOU RUN YOUR NODE APPLICATION**
|
|
|
|
* llama-xxxx.dll is required.
|
|
* According to whatever model you are using, you'll need to select the proper model loader.
|
|
* For example, if you running an Mosaic MPT model, you will need to select the mpt-(buildvariant).(dynamiclibrary)
|
|
|
|
### Test
|
|
|
|
```sh
|
|
yarn test
|
|
```
|
|
|
|
### Source Overview
|
|
|
|
#### src/
|
|
|
|
* Extra functions to help aid devex
|
|
* Typings for the native node addon
|
|
* the javascript interface
|
|
|
|
#### test/
|
|
|
|
* simple unit testings for some functions exported.
|
|
* more advanced ai testing is not handled
|
|
|
|
#### spec/
|
|
|
|
* Average look and feel of the api
|
|
* Should work assuming a model and libraries are installed locally in working directory
|
|
|
|
#### index.cc
|
|
|
|
* The bridge between nodejs and c. Where the bindings are.
|
|
|
|
#### prompt.cc
|
|
|
|
* Handling prompting and inference of models in a threadsafe, asynchronous way.
|
|
|
|
#### docs/
|
|
|
|
* Autogenerated documentation using the script `yarn docs:build`
|
|
|
|
### Known Issues
|
|
|
|
* why your model may be spewing bull 💩
|
|
- The downloaded model is broken (just reinstall or download from official site)
|
|
- That's it so far
|
|
### Roadmap
|
|
|
|
This package is in active development, and breaking changes may happen until the api stabilizes. Here's what's the todo list:
|
|
|
|
* \[x] prompt models via a threadsafe function in order to have proper non blocking behavior in nodejs
|
|
* \[ ] ~~createTokenStream, an async iterator that streams each token emitted from the model. Planning on following this [example](https://github.com/nodejs/node-addon-examples/tree/main/threadsafe-async-iterator)~~ May not implement unless someone else can complete
|
|
* \[x] proper unit testing (integrate with circle ci)
|
|
* \[x] publish to npm under alpha tag `gpt4all@alpha`
|
|
* \[x] have more people test on other platforms (mac tester needed)
|
|
* \[x] switch to new pluggable backend
|
|
* \[ ] NPM bundle size reduction via optionalDependencies strategy (need help)
|
|
- Should include prebuilds to avoid painful node-gyp errors
|
|
* \[ ] createChatSession ( the python equivalent to create\_chat\_session )
|
|
### Documentation
|