Archives/gpt4all
2
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all synced 2024-11-08 07:10:32 +00:00
Code Issues Packages Projects Releases Wiki Activity
927 Commits 542 Branches 59 Tags 51 MiB
25eec33bda
Commit Graph

3 Commits

Author SHA1 Message Date
Adam Treat
7f9f91ad94 Revert "New tokenizer implementation for MPT and GPT-J" 
This reverts commit bbcee1ced5.
 2023-05-30 12:59:00 -04:00
Aaron Miller
bbcee1ced5 New tokenizer implementation for MPT and GPT-J 
Improves output quality by making these tokenizers more closely
match the behavior of the huggingface `tokenizers` based BPE
tokenizers these models were trained with.

Featuring:
 * Fixed unicode handling (via ICU)
 * Fixed BPE token merge handling
 * Complete added vocabulary handling
 2023-05-30 12:05:57 -04:00
Yaroslav Halchenko
2195bfabba Add rudimentary codespell config 2023-05-16 11:33:59 -04:00