4 Commits

Author	SHA1	Message	Date
Adam Treat	7f9f91ad94	Revert "New tokenizer implementation for MPT and GPT-J" ... This reverts commit bbcee1ced5.	2023-05-30 12:59:00 -04:00
Aaron Miller	bbcee1ced5	New tokenizer implementation for MPT and GPT-J ... Improves output quality by making these tokenizers more closely match the behavior of the huggingface `tokenizers` based BPE tokenizers these models were trained with. Featuring: * Fixed unicode handling (via ICU) * Fixed BPE token merge handling * Complete added vocabulary handling	2023-05-30 12:05:57 -04:00
Zach Nussbaum	1ed71fbbf8	fix: use right conversion script	2023-05-11 11:20:43 -04:00
Adam Treat	d918b02c29	Move the llmodel C API to new top-level directory and version it.	2023-05-10 11:46:40 -04:00