Archives/gpt4all
2
0
Fork 0
mirror of https://github.com/nomic-ai/gpt4all synced 2024-11-02 09:40:42 +00:00
Code Issues Packages Projects Releases Wiki Activity
2befff83d6
gpt4all/gpt4all-training
History
Zach Nussbaum 6c4f449b7a
fix: update train scripts and configs for other models (#1164) 
* feat: falcon config

* feat: mpt config

* chore: gitignore

* refactor: step calculation

* fix: attention mask + shuffle on epoch end

* fix: return tensors

* fix: wait for everyone

* chore: config

* chore: ds config

* fix: remove ccols

* fix: logging and saving

* chore: add einops
2023-07-12 15:18:24 -04:00
..
chat mono repo structure 2023-05-01 15:45:23 -04:00
configs fix: update train scripts and configs for other models (#1164) 2023-07-12 15:18:24 -04:00
figs mono repo structure 2023-05-01 15:45:23 -04:00
build_map.py mono repo structure 2023-05-01 15:45:23 -04:00
clean.py mono repo structure 2023-05-01 15:45:23 -04:00
create_hostname.sh mono repo structure 2023-05-01 15:45:23 -04:00
data.py fix: update train scripts and configs for other models (#1164) 2023-07-12 15:18:24 -04:00
env.yaml mono repo structure 2023-05-01 15:45:23 -04:00
eval_figures.py mono repo structure 2023-05-01 15:45:23 -04:00
eval_self_instruct.py mono repo structure 2023-05-01 15:45:23 -04:00
generate.py mono repo structure 2023-05-01 15:45:23 -04:00
GPT-J_MAP.md mono repo structure 2023-05-01 15:45:23 -04:00
inference.py mono repo structure 2023-05-01 15:45:23 -04:00
launcher.sh mono repo structure 2023-05-01 15:45:23 -04:00
old-README.md contributing and readme 2023-05-11 12:31:08 -04:00
read.py mono repo structure 2023-05-01 15:45:23 -04:00
README.md fix(training instructions): model repo name (#728) 2023-05-28 19:56:24 -04:00
requirements.txt fix: update train scripts and configs for other models (#1164) 2023-07-12 15:18:24 -04:00
train.py fix: update train scripts and configs for other models (#1164) 2023-07-12 15:18:24 -04:00
TRAINING_LOG.md mono repo structure 2023-05-01 15:45:23 -04:00

README.md

Training GPT4All-J

Please see GPT4All-J Technical Report for details.

GPT4All-J Training Data

We have released updated versions of our GPT4All-J model and training data.

  • v1.0: The original model trained on the v1.0 dataset
  • v1.1-breezy: Trained on a filtered dataset where we removed all instances of AI language model
  • v1.2-jazzy: Trained on a filtered dataset where we also removed instances like I'm sorry, I can't answer... and AI language model

The models and data versions can be specified by passing a revision argument.

For example, to load the v1.2-jazzy model and dataset, run:

from datasets import load_dataset
from transformers import AutoModelForCausalLM

dataset = load_dataset("nomic-ai/gpt4all-j-prompt-generations", revision="v1.2-jazzy")
model = AutoModelForCausalLM.from_pretrained("nomic-ai/gpt4all-j", revision="v1.2-jazzy")

GPT4All-J Training Instructions

accelerate launch --dynamo_backend=inductor --num_processes=8 --num_machines=1 --machine_rank=0 --deepspeed_multinode_launcher standard --mixed_precision=bf16  --use_deepspeed --deepspeed_config_file=configs/deepspeed/ds_config_gptj.json train.py --config configs/train/finetune_gptj.yaml