Commit Graph

945 Commits (7a472bea8873172f13ac9a8aabdd061a550cebc2)
 

Author SHA1 Message Date
Adam Treat 98207deb9d Fix remove model from model download for regular models. 1 year ago
Adam Treat 4a317eeb33 Revert "New tokenizer implementation for MPT and GPT-J"
This reverts commit ee3469ba6c.
1 year ago
Adam Treat 06434f0042 Revert "buf_ref.into() can be const now"
This reverts commit 840e011b75.
1 year ago
Adam Treat 92bc92d232 Revert "add tokenizer readme w/ instructions for convert script"
This reverts commit 9c15d1f83e.
1 year ago
aaron miller 9c15d1f83e add tokenizer readme w/ instructions for convert script 1 year ago
Aaron Miller 840e011b75 buf_ref.into() can be const now 1 year ago
Aaron Miller ee3469ba6c New tokenizer implementation for MPT and GPT-J
Improves output quality by making these tokenizers more closely
match the behavior of the huggingface `tokenizers` based BPE
tokenizers these models were trained with.

Featuring:
 * Fixed unicode handling (via ICU)
 * Fixed BPE token merge handling
 * Complete added vocabulary handling
1 year ago
Andriy Mulyar 7e18f179e9 Improved localdocs documentation (#762)
* Improved localdocs documentation

* Improved localdocs documentation

* Improved localdocs documentation

* Improved localdocs documentation
1 year ago
Andriy Mulyar 3bb19b5e91 LocalDocs documentation initial (#761)
* LocalDocs documentation initial
1 year ago
mvenditto 522dfdbfe1 C# Bindings - Prompt formatting (#712)
* Added support for custom prompt formatting

* more docs added

* bump version
1 year ago
Chase McDougall 8189aece48 fix(training instructions): model repo name (#728)
Signed-off-by: Chase McDougall <chasemcdougall@hotmail.com>
1 year ago
Nandakumar 4161bfba33 Update README.md (#738)
* Update README.md

fix golang gpt4all import path

Signed-off-by: Nandakumar <nandagunasekaran@gmail.com>

* Update README.md

Signed-off-by: Nandakumar <nandagunasekaran@gmail.com>

---------

Signed-off-by: Nandakumar <nandagunasekaran@gmail.com>
1 year ago
Joseph Mearman 6ec2e26cd1 tiny typo (#739) 1 year ago
Richard Guo 9f62d4d05f hotfix default verbose optioin 1 year ago
Konstantin Gukov 6ec3efa355 one funcion to append .bin suffix 1 year ago
Konstantin Gukov 52efd17165 Correct indentation of the multiline error message 1 year ago
Konstantin Gukov 39bdd74ddd Add optional verbosity 1 year ago
Konstantin Gukov 136a5ac30d Correct return type 1 year ago
Konstantin Gukov e362e18431 Do not ignore explicitly passed 4 threads 1 year ago
Konstantin Gukov a2e2a64899 Redundant else 1 year ago
Konstantin Gukov 8229bb5477 1. Cleanup the interrupted download
2. with-syntax
1 year ago
Konstantin Gukov b12e0e98c7 less magic number 1 year ago
Konstantin Gukov 4dff67cee5 convert to f-strings 1 year ago
Konstantin Gukov 6e3c59d8b2 reduce nesting, better error reporting 1 year ago
Konstantin Gukov 591a047204 Concise model matching 1 year ago
Konstantin Gukov 4de8f991f9 Log where the model was found 1 year ago
Konstantin Gukov b6b21441c7 Nicer handling of missing model directory.
Correct exception message.
1 year ago
Konstantin Gukov 84201784b5 More precise condition 1 year ago
Konstantin Gukov e3bdc8fc87 rm redundant json 1 year ago
Adam Treat 55b4f35510 This time remember to bump the version right after a release. 1 year ago
Adam Treat c5c8ab9138 Bump the version number. 1 year ago
Adam Treat 4e8dba107f Libraries named differently on msvc. 1 year ago
Adam Treat d40735a2d2 Get the backend as well as the client building/working with msvc. 1 year ago
redthing1 0e11584783 make sample print usage and cleaner 1 year ago
redthing1 448acb337d create test project and basic model loading tests 1 year ago
redthing1 9248bd0c1e ignore rider and vscode dirs 1 year ago
Adam Treat ff51cb872f Add a newline 1 year ago
Adam Treat 580aceb2b3 Various fixes to remove unnecessary warnings. 1 year ago
Adam Treat e6f5d88f17 Don't use the full path in reference text. 1 year ago
Adam Treat 0a4c3a14ca Add context link to references. 1 year ago
Adam Treat f3dc031a9d Store the references separately so they are not sent to datalake. 1 year ago
Adam Treat 05c3edb387 Adds the collections to serialize and implement references for localdocs. 1 year ago
Adam Treat 90b2cf747f Complete the settings for localdocs. 1 year ago
Adam Treat 53cf2da5f7 Add more of the UI for selecting collections for chats. 1 year ago
Adam Treat 63c5911604 Clean up the settings dialog for localdocs a bit. 1 year ago
Adam Treat 1930070fdb Begin implementing the localdocs ui in earnest. 1 year ago
Adam Treat 695d1d048b Start fleshing out the localdocs ui. 1 year ago
Adam Treat 38c403dab8 Add a localdocs tab. 1 year ago
Adam Treat 32be0a92d7 Add a collection list to support a UI. 1 year ago
Adam Treat e47f4ddfb6 Specify a large number of suffixes we will search for now. 1 year ago