aaron miller
9c15d1f83e
add tokenizer readme w/ instructions for convert script
2023-05-30 12:05:57 -04:00
Aaron Miller
840e011b75
buf_ref.into() can be const now
2023-05-30 12:05:57 -04:00
Aaron Miller
ee3469ba6c
New tokenizer implementation for MPT and GPT-J
...
Improves output quality by making these tokenizers more closely
match the behavior of the huggingface `tokenizers` based BPE
tokenizers these models were trained with.
Featuring:
* Fixed unicode handling (via ICU)
* Fixed BPE token merge handling
* Complete added vocabulary handling
2023-05-30 12:05:57 -04:00
Andriy Mulyar
7e18f179e9
Improved localdocs documentation ( #762 )
...
* Improved localdocs documentation
* Improved localdocs documentation
* Improved localdocs documentation
* Improved localdocs documentation
2023-05-30 11:26:34 -04:00
Andriy Mulyar
3bb19b5e91
LocalDocs documentation initial ( #761 )
...
* LocalDocs documentation initial
2023-05-30 08:35:26 -04:00
mvenditto
522dfdbfe1
C# Bindings - Prompt formatting ( #712 )
...
* Added support for custom prompt formatting
* more docs added
* bump version
2023-05-28 19:57:00 -04:00
Chase McDougall
8189aece48
fix(training instructions): model repo name ( #728 )
...
Signed-off-by: Chase McDougall <chasemcdougall@hotmail.com>
2023-05-28 19:56:24 -04:00
Nandakumar
4161bfba33
Update README.md ( #738 )
...
* Update README.md
fix golang gpt4all import path
Signed-off-by: Nandakumar <nandagunasekaran@gmail.com>
* Update README.md
Signed-off-by: Nandakumar <nandagunasekaran@gmail.com>
---------
Signed-off-by: Nandakumar <nandagunasekaran@gmail.com>
2023-05-28 19:51:11 -04:00
Joseph Mearman
6ec2e26cd1
tiny typo ( #739 )
2023-05-28 19:50:45 -04:00
Richard Guo
9f62d4d05f
hotfix default verbose optioin
2023-05-26 12:49:32 -04:00
Konstantin Gukov
6ec3efa355
one funcion to append .bin suffix
2023-05-26 09:24:03 -04:00
Konstantin Gukov
52efd17165
Correct indentation of the multiline error message
2023-05-26 09:24:03 -04:00
Konstantin Gukov
39bdd74ddd
Add optional verbosity
2023-05-26 09:24:03 -04:00
Konstantin Gukov
136a5ac30d
Correct return type
2023-05-26 09:24:03 -04:00
Konstantin Gukov
e362e18431
Do not ignore explicitly passed 4 threads
2023-05-26 09:24:03 -04:00
Konstantin Gukov
a2e2a64899
Redundant else
2023-05-26 09:24:03 -04:00
Konstantin Gukov
8229bb5477
1. Cleanup the interrupted download
...
2. with-syntax
2023-05-26 09:24:03 -04:00
Konstantin Gukov
b12e0e98c7
less magic number
2023-05-26 09:24:03 -04:00
Konstantin Gukov
4dff67cee5
convert to f-strings
2023-05-26 09:24:03 -04:00
Konstantin Gukov
6e3c59d8b2
reduce nesting, better error reporting
2023-05-26 09:24:03 -04:00
Konstantin Gukov
591a047204
Concise model matching
2023-05-26 09:24:03 -04:00
Konstantin Gukov
4de8f991f9
Log where the model was found
2023-05-26 09:24:03 -04:00
Konstantin Gukov
b6b21441c7
Nicer handling of missing model directory.
...
Correct exception message.
2023-05-26 09:24:03 -04:00
Konstantin Gukov
84201784b5
More precise condition
2023-05-26 09:24:03 -04:00
Konstantin Gukov
e3bdc8fc87
rm redundant json
2023-05-26 09:24:03 -04:00
Adam Treat
55b4f35510
This time remember to bump the version right after a release.
2023-05-25 18:26:33 -04:00
Adam Treat
c5c8ab9138
Bump the version number.
2023-05-25 17:08:50 -04:00
Adam Treat
4e8dba107f
Libraries named differently on msvc.
2023-05-25 16:27:09 -04:00
Adam Treat
d40735a2d2
Get the backend as well as the client building/working with msvc.
2023-05-25 15:22:45 -04:00
redthing1
0e11584783
make sample print usage and cleaner
2023-05-25 11:34:21 -04:00
redthing1
448acb337d
create test project and basic model loading tests
2023-05-25 11:34:07 -04:00
redthing1
9248bd0c1e
ignore rider and vscode dirs
2023-05-25 11:34:07 -04:00
Adam Treat
ff51cb872f
Add a newline
2023-05-25 11:28:06 -04:00
Adam Treat
580aceb2b3
Various fixes to remove unnecessary warnings.
2023-05-25 11:28:06 -04:00
Adam Treat
e6f5d88f17
Don't use the full path in reference text.
2023-05-25 11:28:06 -04:00
Adam Treat
0a4c3a14ca
Add context link to references.
2023-05-25 11:28:06 -04:00
Adam Treat
f3dc031a9d
Store the references separately so they are not sent to datalake.
2023-05-25 11:28:06 -04:00
Adam Treat
05c3edb387
Adds the collections to serialize and implement references for localdocs.
2023-05-25 11:28:06 -04:00
Adam Treat
90b2cf747f
Complete the settings for localdocs.
2023-05-25 11:28:06 -04:00
Adam Treat
53cf2da5f7
Add more of the UI for selecting collections for chats.
2023-05-25 11:28:06 -04:00
Adam Treat
63c5911604
Clean up the settings dialog for localdocs a bit.
2023-05-25 11:28:06 -04:00
Adam Treat
1930070fdb
Begin implementing the localdocs ui in earnest.
2023-05-25 11:28:06 -04:00
Adam Treat
695d1d048b
Start fleshing out the localdocs ui.
2023-05-25 11:28:06 -04:00
Adam Treat
38c403dab8
Add a localdocs tab.
2023-05-25 11:28:06 -04:00
Adam Treat
32be0a92d7
Add a collection list to support a UI.
2023-05-25 11:28:06 -04:00
Adam Treat
e47f4ddfb6
Specify a large number of suffixes we will search for now.
2023-05-25 11:28:06 -04:00
Adam Treat
c33bf0e895
Add prompt processing and localdocs to the busy indicator in UI.
2023-05-25 11:28:06 -04:00
Adam Treat
837ece220f
Turn off the debugging messages by default.
2023-05-25 11:28:06 -04:00
Adam Treat
90ab9183ae
Add a new muted text color.
2023-05-25 11:28:06 -04:00
Adam Treat
80024a029c
Add new reverse prompt for new localdocs context feature.
2023-05-25 11:28:06 -04:00
