Archives/llm-course
2
0
Fork 0
mirror of https://github.com/mlabonne/llm-course.git synced 2024-11-15 06:12:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix broken link

Browse Source
This commit is contained in:
Maxime Labonne 2023-11-28 00:09:50 +00:00
parent 0e1cd288d4
commit d8a6b017a1
1 changed files with 4 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@ -2,9 +2,9 @@
The LLM course is divided into three parts:
1. 🧩 **LLM Fundamentals**: this part covers essential knowledge about mathematics, Python, and neural networks.
2. 🧑‍🔬 **The LLM Scientist**: this part focuses on learning how to build the best possible LLMs using the latest techniques
3. 👷 **The LLM Engineer**: this part focuses on how to create LLM-based solutions and deploy them.
1. 🧩 **LLM Fundamentals** covers essential knowledge about mathematics, Python, and neural networks.
2. 🧑‍🔬 **The LLM Scientist** focuses on learning how to build the best possible LLMs using the latest techniques
3. 👷 **The LLM Engineer** focuses on how to create LLM-based solutions and deploy them.
## Notebooks
@ -150,7 +150,7 @@ Pre-training is a very long and costly process, which is why this is not the foc
* [LLMDataHub](https://github.com/Zjh-819/LLMDataHub) by Junhao Zhao: Curated list of datasets for pre-training, fine-tuning, and RLHF.
* [Training a causal language model from scratch](https://huggingface.co/learn/nlp-course/chapter7/6?fw=pt) by Hugging Face: Pre-train a GPT-2 model from scratch using the transformers library.
* [TinyLlama](https://github.com/jzhang38/TinyLlama) by Zhang et al.: Check this project to get a good understanding of how a Llama model is trained from scratch.
* [Causal language modeling](# Causal language modeling) by Hugging Face: Explain the difference between causal and masked language modeling and how to quickly fine-tune a DistilGPT-2 model.
* [Causal language modeling](https://huggingface.co/docs/transformers/tasks/language_modeling) by Hugging Face: Explain the difference between causal and masked language modeling and how to quickly fine-tune a DistilGPT-2 model.
* [Chinchilla's wild implications](https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications) by nostalgebraist: Discuss the scaling laws and explain what they mean to LLMs in general.
* [BLOOM](https://bigscience.notion.site/BLOOM-BigScience-176B-Model-ad073ca07cdf479398d5f95d88e218c4) by BigScience: Notion pages that describes how the BLOOM model was built, with a lot of useful information about the engineering part and the problems that were encountered.
* [OPT-175 Logbook](https://github.com/facebookresearch/metaseq/blob/main/projects/OPT/chronicles/OPT175B_Logbook.pdf) by Meta: Research logs showing what went wrong and what went right. Useful if you're planning to pre-train a very large language model (in this case, 175B parameters).