Commit Graph

4 Commits (8c6ee9d02e61d18afa901e736d18a72dcef989fb)

Author SHA1 Message Date
Adam Treat 4a317eeb33 Revert "New tokenizer implementation for MPT and GPT-J"
This reverts commit ee3469ba6c.
1 year ago
Aaron Miller ee3469ba6c New tokenizer implementation for MPT and GPT-J
Improves output quality by making these tokenizers more closely
match the behavior of the huggingface `tokenizers` based BPE
tokenizers these models were trained with.

Featuring:
 * Fixed unicode handling (via ICU)
 * Fixed BPE token merge handling
 * Complete added vocabulary handling
1 year ago
Zach Nussbaum 53730c5f7f fix: use right conversion script 1 year ago
Adam Treat 8e7b96bd92 Move the llmodel C API to new top-level directory and version it. 1 year ago