Commit Graph

9 Commits (2d2de658c86dc7e70b4e2dc2929b56cbd00090fe)

Author SHA1 Message Date
AT b5971b0d41 Backend prompt dedup (#822)
* Deduplicated prompt() function code
1 year ago
Adam Treat 15c6bf09e9 Fix mac build again. 1 year ago
Adam Treat 1b755b6cba Try and fix build on mac. 1 year ago
AT 9c6c09cbd2 Dlopen backend 5 (#779)
Major change to the backend that allows for pluggable versions of llama.cpp/ggml. This was squashed merged from dlopen_backend_5 where the history is preserved.
1 year ago
Adam Treat 4a317eeb33 Revert "New tokenizer implementation for MPT and GPT-J"
This reverts commit ee3469ba6c.
1 year ago
Aaron Miller ee3469ba6c New tokenizer implementation for MPT and GPT-J
Improves output quality by making these tokenizers more closely
match the behavior of the huggingface `tokenizers` based BPE
tokenizers these models were trained with.

Featuring:
 * Fixed unicode handling (via ICU)
 * Fixed BPE token merge handling
 * Complete added vocabulary handling
1 year ago
Adam Treat d40735a2d2 Get the backend as well as the client building/working with msvc. 1 year ago
kuvaus a0b98dc55d gpt4all-backend: Add llmodel create and destroy functions (#554)
* Add llmodel create and destroy functions

* Fix capitalization

* Fix capitalization

* Fix capitalization

* Update CMakeLists.txt

---------

Co-authored-by: kuvaus <kuvaus@users.noreply.github.com>
1 year ago
Adam Treat 8e7b96bd92 Move the llmodel C API to new top-level directory and version it. 1 year ago