Commit Graph

7 Commits (95b8fb312e5df8ce08a583c67f1e6d1e98985a21)

Author SHA1 Message Date
niansa/tuxifan 68f9786ed9
Use operator ""_MiB (#991) 1 year ago
AT 48275d0dcc
Dlopen backend 5 (#779)
Major change to the backend that allows for pluggable versions of llama.cpp/ggml. This was squashed merged from dlopen_backend_5 where the history is preserved.
1 year ago
Adam Treat 7f9f91ad94 Revert "New tokenizer implementation for MPT and GPT-J"
This reverts commit bbcee1ced5.
1 year ago
Aaron Miller bbcee1ced5 New tokenizer implementation for MPT and GPT-J
Improves output quality by making these tokenizers more closely
match the behavior of the huggingface `tokenizers` based BPE
tokenizers these models were trained with.

Featuring:
 * Fixed unicode handling (via ICU)
 * Fixed BPE token merge handling
 * Complete added vocabulary handling
1 year ago
Aaron Miller d14936bfd6 backend: dedupe tokenizing code in mpt/gptj 1 year ago
Aaron Miller 6182026c70 backend: dedupe tokenizing code in gptj/mpt 1 year ago
Adam Treat d918b02c29 Move the llmodel C API to new top-level directory and version it. 1 year ago