Sign in to confirm you’re not a bot
This helps protect our community. Learn more

Let's build GPT: from scratch, in code, spelled out.

Andrej Karpathy
837K subscribers
<__slot-el>
<__slot-el>
5.4M views 2 years ago
We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video. Links:  …...more
...more

intro: ChatGPT, Transformers, nanoGPT, Shakespeare

0:00

reading and exploring the data

7:52

tokenization, train/val split

9:28

data loader: batches of chunks of data

14:27

simplest baseline: bigram language model, loss, generation

22:11

training the bigram model

34:53

port our code to a script

38:00

version 1: averaging past context with for loops, the weakest form of aggregation

42:13

the trick in self-attention: matrix multiply as weighted aggregation

47:11

version 2: using matrix multiply

51:54

version 3: adding softmax

54:42

minor code cleanup

58:26

positional encoding

1:00:18

THE CRUX OF THE VIDEO: version 4: self-attention

1:02:00

note 1: attention as communication

1:11:38

note 2: attention has no notion of space, operates over sets

1:12:46

note 3: there is no communication across batch dimension

1:13:40

note 4: encoder blocks vs. decoder blocks

1:14:14

note 5: attention vs. self-attention vs. cross-attention

1:15:39

note 6: "scaled" self-attention. why divide by sqrt(head_size)

1:16:56

inserting a single self-attention block to our network

1:19:11

multi-headed self-attention

1:21:59

feedforward layers of transformer block

1:24:25

residual connections

1:26:48

layernorm (and its relationship to our previous batchnorm)

1:32:51

scaling up the model! creating a few variables. adding dropout

1:37:49

encoder vs. decoder vs. both (?) Transformers

1:42:39

super quick walkthrough of nanoGPT, batched multi-headed self-attention

1:46:22

back to ChatGPT, GPT-3, pretraining vs. finetuning, RLHF

1:48:53

conclusions

1:54:32
Let's build GPT: from scratch, in code, spelled out.
126KLikes
5,481,437Views
2023Jan 17
We build a Generatively Pretrained Transformer (GPT), following the paper "Attention is All You Need" and OpenAI's GPT-2 / GPT-3. We talk about connections to ChatGPT, which has taken the world by storm. We watch GitHub Copilot, itself a GPT, help us write a GPT (meta :D!) . I recommend people watch the earlier makemore videos to get comfortable with the autoregressive language modeling framework and basics of tensors and PyTorch nn, which we take for granted in this video. Links: Supplementary links: Suggested exercises:
  • EX1: The n-dimensional tensor mastery challenge: Combine the `Head` and `MultiHeadAttention` into one class that processes all the heads in parallel, treating the heads as another batch dimension (answer is in nanoGPT).
  • EX2: Train the GPT on your own dataset of choice! What other data could be fun to blabber on about? (A fun advanced suggestion if you like: train a GPT to do addition of two numbers, i.e. a+b=c. You may find it helpful to predict the digits of c in reverse order, as the typical addition algorithm (that you're hoping it learns) would proceed right to left too. You may want to modify the data loader to simply serve random problems and skip the generation of train.bin, val.bin. You may want to mask out the loss at the input positions of a+b that just specify the problem using y=-1 in the targets (see CrossEntropyLoss ignore_index). Does your Transformer learn to add? Once you have this, swole doge project: build a calculator clone in GPT, for all of +-*/. Not an easy problem. You may need Chain of Thought traces.)
  • EX3: Find a dataset that is very large, so large that you can't see a gap between train and val loss. Pretrain the transformer on this data, then initialize with that model and finetune it on tiny shakespeare with a smaller number of steps and lower learning rate. Can you obtain a lower validation loss by the use of pretraining?
  • EX4: Read some transformer papers and implement one additional feature or change that people seem to use. Does it improve the performance of your GPT?
Chapters: 00:00:00 intro: ChatGPT, Transformers, nanoGPT, Shakespeare baseline language modeling, code setup 00:07:52 reading and exploring the data 00:09:28 tokenization, train/val split 00:14:27 data loader: batches of chunks of data 00:22:11 simplest baseline: bigram language model, loss, generation 00:34:53 training the bigram model 00:38:00 port our code to a script Building the "self-attention" 00:42:13 version 1: averaging past context with for loops, the weakest form of aggregation 00:47:11  …...more
...more
57:00
Oops "tokens from the _future_ cannot communicate", not "past". Sorry! :)

Follow along using the transcript.

Andrej Karpathy

837K subscribers
57:00
Oops "tokens from the _future_ cannot communicate", not "past". Sorry! :)
1:20:05
Oops I should be using the head_size for the normalization, not C