Build a Large Language Model (From Scratch)

Overview

Build a Large Language Model (From Scratch) (2024) by Sebastian Raschka is a hands-on guide that walks readers through creating a GPT-style large language model entirely from first principles. No high-level libraries, no black boxes — just Python, PyTorch, and careful explanations of every component. The LLM you build runs on any modern laptop and optionally leverages GPUs for faster training.

The book follows the Feynman principle: I don't understand anything I can't build. Each chapter incrementally constructs a working model, from text tokenization to multi-head attention, through pretraining and fine-tuning into a chatbot that follows instructions.

---

Executive Summary

Raschka organizes the journey into seven chapters that mirror the real pipeline for producing a production-grade LLM:

| Chapter | Focus | What You Build | |---------|-------|----------------| | 1 | LLM fundamentals | Mental model of the full pipeline | | 2 | Text data & tokenization | Byte-pair encoding, input-target pairs | | 3 | Attention mechanisms | Self-attention, causal, multi-head | | 4 | GPT model architecture | A complete GPT-2-style text generator | | 5 | Pretraining | Training loop, loss, weight loading | | 6 | Classification fine-tuning | Spam classifier on your LLM | | 7 | Instruction fine-tuning | A chatbot that follows instructions |

---

Key Takeaways

1. LLMs are next-token predictors. Everything else — reasoning, translation, summarization — emerges from this simple objective.

2. Attention is the only communication channel between tokens. Masked (causal) attention ensures autoregressive generation. Multi-head attention lets the model learn diverse relationships.

3. Byte-pair encoding bridges words and tokens. BPE splits text into subword units, balancing vocabulary size against sequence length.

4. Pretraining is expensive; fine-tuning is cheap. The book demonstrates both, then shows how loading GPT-2 weights skips months of training.

5. Fine-tuning comes in two flavors. Classification fine-tuning replaces the output head. Instruction fine-tuning trains on prompt-response pairs to follow human instructions.

6. A laptop is enough to understand LLMs. The toy model (~124M parameters) trains in hours on consumer hardware. You don't need a datacenter to learn how LLMs work.

---

Who Should Read

| Reader Type | Why | |---|---| | ML engineers wanting LLM internals | The most thorough code-first walkthrough | | API-power-users seeking depth | Understand what happens after the API call | | NLP / deep learning students | Bridges textbook theory with working code | | Anyone who feels LLMs are a black box | Builds the box, then opens it |

---

Who Should Skip

• Readers seeking only high-level overviews — the book is relentlessly code-driven
• Anyone without intermediate Python skills and basic ML knowledge
• Practitioners looking for production deployment guidance

---

Final Verdict

Build a Large Language Model (From Scratch) is the definitive hands-on guide to understanding LLMs from the inside out. Its uncompromising code-first approach demands effort but delivers unmatched depth. The trade-off is real: this is not a book for casual readers or AI consumers. It is for builders. For those willing to put in the work, the reward is a genuine understanding of how generative AI works — not at the API level, but at the tensor level.

Rating: 9.0/10 — The best learn-by-building LLM book available.

The LLM Pipeline at a Glance

Before diving into code, it helps to see the full journey. The diagram below maps every stage the book covers — from raw text to a functioning chatbot — and shows how each chapter feeds into the next.

flowchart TB
    subgraph Ch1["Ch 1-2: Data & Embeddings"]
        Raw["Raw Text Corpus"] --> Tokenize["Byte-Pair Encoding<br/>Tokenizer"]
        Tokenize --> Tokens["Token IDs"]
        Tokens --> Embed["Token Embeddings"]
        Embed --> PosEmbed["Add Positional<br/>Embeddings"]
    end

    subgraph Ch3["Ch 3: Attention"]
        PosEmbed --> Attn["Self-Attention<br/>(Masked/Causal)"]
        Attn --> MHA["Multi-Head<br/>Attention"]
    end

    subgraph Ch4["Ch 4: GPT Model"]
        MHA --> Block["Transformer Block<br/>(Attn + FFN + LayerNorm)"]
        Block --> Stack["Nx Stacked<br/>Transformer Blocks"]
        Stack --> Unembed["Output Projection<br/>(Logits)"]
    end

    subgraph Ch5["Ch 5: Pretraining"]
        Unembed --> Loss["Cross-Entropy Loss<br/>(Next-Token Prediction)"]
        Loss --> Update["Backprop +<br/>Weight Update"]
        Update -->|"Loop over dataset"| Loss
    end

    subgraph Ch6-7["Ch 6-7: Fine-Tuning"]
        Unembed --> ClassHead["Classification Head<br/>(Ch 6)"]
        Unembed --> InstHead["Instruction Tuning<br/>with I/O Pairs (Ch 7)"]
    end

    Update -->|"Pretrained<br/>Weights"| ClassHead
    Update -->|"Pretrained<br/>Weights"| InstHead

---

Chapter 1: Understanding Large Language Models

The opening chapter provides a high-level mental model. Raschka defines LLMs as neural networks trained on massive text corpora to predict the next token. He traces the lineage from early recurrent neural networks through the 2017 Transformer paper ("Attention Is All You Need") to modern decoder-only architectures like GPT-2 and GPT-3.

Key distinction: the book uses a decoder-only transformer (GPT style), not encoder-decoder (original Transformer) or encoder-only (BERT). The decoder-only design with causal (masked) self-attention is what enables autoregressive text generation — the model produces one token at a time, conditioned on all previous tokens.

Raschka also sets expectations: the finished model will be comparable to GPT-2 (124M parameters), not GPT-3 (175B). The goal is understanding, not scale.

---

Chapter 2: Working with Text Data

This is where the code begins. The chapter covers three critical preprocessing steps:

Tokenization. Raw text must be converted into numbers. Raschka implements a byte-pair encoding (BPE) tokenizer, which splits text into subword units. Unlike word-level tokenization (which produces huge vocabularies) or character-level (which produces long sequences), BPE strikes a balance by merging the most frequent character pairs iteratively. The final vocabulary typically contains 50k-100k tokens.

Token Embeddings. Each token ID is mapped to a dense vector via an embedding layer. This is effectively a lookup table where each token learns a high-dimensional representation. The book uses 768-dimensional embeddings, matching GPT-2's configuration.

Positional Embeddings. Since self-attention processes all tokens in parallel (no inherent notion of order), position information must be added. The book uses learned absolute positional embeddings — a separate embedding matrix indexed by position — which are added to the token embeddings before passing them to the transformer.

The chapter also covers creating input-target pairs for training: given a sequence [t1, t2, t3, t4], the input is [t1, t2, t3] and the target is [t2, t3, t4]. Every position predicts the next token.

---

Chapter 3: Coding Attention Mechanisms

The heart of the transformer. Raschka builds attention in four stages:

Simplified self-attention. A naive implementation where each token computes a weighted sum of all other tokens. The weights are determined by a dot product between query and key vectors (attention scores), softmax-normalized, then used to aggregate value vectors. This captures the core idea before adding complications.

Scaled dot-product attention. The attention scores are divided by sqrt(d_k) (the square root of the key dimension). Without scaling, high-dimensional dot products can produce extreme softmax outputs that saturate and kill gradients.

Causal (masked) attention. For autoregressive generation, each token must only attend to itself and previous tokens. The solution is a causal mask: set attention scores for future positions to negative infinity before softmax, so their attention weights become zero.

Multi-head attention. Instead of one attention computation, the model runs multiple attention "heads" in parallel, each operating on a different projection of the input. GPT-2 uses 12 heads (for the 124M parameter variant). Each head learns to focus on different types of relationships — syntax, semantics, co-reference, etc. The outputs are concatenated and projected back to the model dimension.

---

Chapter 4: Implementing a GPT Model from Scratch

This chapter assembles the full GPT-2-style model architecture:

1. Transformer block. A block consists of multi-head attention followed by a feed-forward network (two linear layers with a GELU activation in between), each surrounded by residual connections and layer normalization. GPT-2 uses pre-layer normalization (normalization applied before each sub-layer), which stabilizes training better than post-norm.

2. GELU activation. The Gaussian Error Linear Unit — a smooth approximation of ReLU — is the standard activation in GPT models. It introduces non-linearity while avoiding the sharp zero transition of ReLU.

3. Residual connections & layer norm. Residual (skip) connections let gradients flow directly through the network, solving the vanishing gradient problem in deep transformers. Layer norm stabilizes activations by normalizing across the feature dimension.

4. Output head. The final layer norm is followed by a linear projection mapping from hidden dimension to vocabulary size, producing logits for every token in the vocabulary. A softmax over these logits gives the probability distribution for the next token.

By the end of chapter 4, you have a complete (untrained) GPT model that can generate random-looking text from any input.

---

Chapter 5: Pretraining on Unlabeled Data

Pretraining is where the model learns language. The chapter covers:

Loss computation. Cross-entropy loss between the predicted logits and the target token IDs. The average loss per token is the training objective.

Data loading. The book uses a small public-domain text (a short story) to make training fast on a laptop. A PyTorch DataLoader creates random batches of fixed-length sequences from the tokenized corpus.

Training loop. Standard PyTorch training: forward pass, compute loss, backward pass, optimizer step. The book uses AdamW (Adam with weight decay), which is the standard optimizer for transformer training.

Text generation. After each epoch, the model generates text by feeding in a start token and iteratively sampling from the predicted distribution. Early epochs produce gibberish; later epochs show recognizable structure.

Loading pretrained weights. This is a highlight of the book: Raschka shows how to download OpenAI's GPT-2 weights and load them into your model. Overnight, your toy model becomes a real LLM that generates coherent English. This section demystifies how open-weight models are distributed and consumed.

The chapter also introduces temperature scaling for generation — higher temperature produces more random output; lower temperature produces more deterministic output — and top-k sampling, which restricts sampling to the k most likely tokens.

---

Chapter 6: Fine-Tuning for Classification

Fine-tuning adapts a pretrained model to a specific task. Chapter 6 focuses on classification fine-tuning — for example, classifying text messages as spam or not spam.

The key insight: you don't train a new model. You take the pretrained LLM and replace the output head with a small classification head (a linear layer mapping from hidden dimension to class labels). Then you fine-tune only the classification head (or the full model) on labeled data.

Raschka explains the crucial difference between classification fine-tuning and the pretraining objective: classification uses the representation of the last token (or pooled sequence), not the next-token prediction. The model learns to map the sequence representation to class labels using cross-entropy loss over classes.

The chapter also discusses dropout as a regularization technique and shows how to evaluate classification performance on a held-out test set.

---

Chapter 7: Fine-Tuning to Follow Instructions

The final chapter builds an instruction-following chatbot — the type of model that powers applications like ChatGPT. This is called instruction fine-tuning or supervised fine-tuning (SFT).

The dataset format: input-output pairs where the input is a prompt (e.g., "Explain what a transformer is") and the output is the desired response. The model is trained to generate the output given the input, using the same next-token prediction loss as pretraining — but now on carefully curated prompt-response data.

A critical technique covered is prompt formatting: wrapping inputs in a consistent template (e.g., ### Instruction: ... ### Response: ...) so the model learns the structural pattern of a conversation.

The chapter concludes with evaluation: using another LLM (Llama) to score the quality of the model's responses, introducing the concept of LLM-as-judge for automated evaluation.

---

Key Lessons

• LLMs are not magic — they are engineered systems. Every component (tokenization, attention, training) is implementable in a few hundred lines of PyTorch.
• Understanding requires building. Reading about attention is not the same as coding it. Raschka's approach ensures you debug through shape mismatches and dimension arithmetic — the fastest path to genuine understanding.
• Scale changes behavior but not fundamentals. A 124M-parameter model trained on a short story follows the same principles as GPT-4.
• Fine-tuning is where most practitioners live. Pretraining your own model is educational; loading pretrained weights and fine-tuning is practical. The book covers both.
• The architecture is modular. You can swap attention mechanisms, change activation functions, add LoRA adapters — the code structure makes experimentation natural.

---

Practical Applications

For ML Engineers

Implement RAG (retrieval-augmented generation) by understanding exactly where and how the model processes context. The book's detailed attention implementation clarifies how long-context models work.

For AI Educators

Use the chapter-by-chapter build as a curriculum for teaching LLM internals. The free YouTube companion series by Raschka makes it suitable for flipped classrooms.

For Hobbyists

Run a small GPT model on a laptop, generate text, and experiment with hyperparameters. The appendices on LoRA and training tricks extend the experimentation playground.

For Researchers

The from-scratch implementation is a clean baseline for testing new architecture ideas without library abstractions getting in the way.

Strengths

• Uncompromising code-first approach. Every concept is implemented in Python and PyTorch, not just described. This is the book's defining feature and its greatest strength — you cannot finish it without deeply understanding how LLMs work.
• Exceptional pedagogy. Raschka builds complexity incrementally. Simple self-attention comes before multi-head attention. A tiny model on a short story comes before loading GPT-2 weights. Each chapter ends with exercises that test genuine understanding.
• Beautiful diagrams. The book contains a high density of well-designed diagrams (~34% of content) that illustrate tensor shapes, attention patterns, and data flow. These visualizations are often where the hard concepts click.
• Accompanying resources. 96k+ star GitHub repo, free 48-part YouTube series, a free 170-page "Test Yourself" exercise book, appendices on LoRA and training tricks — the supporting material rivals the book itself in value.
• Laptop-friendly. The toy model trains on any modern laptop in hours. No cloud credits, no GPU cluster required. This dramatically lowers the barrier to entry.
• Covers the full pipeline. Most transformer tutorials stop at the architecture. This book takes you through pretraining, classification fine-tuning, and instruction fine-tuning to a working chatbot.

---

Weaknesses

• Light on theory. Readers wanting deep mathematical intuition for why attention works, why transformers generalize, or the statistical mechanics of scaling laws will be disappointed. The book prioritizes code over math.
• GPT-2 focus can feel dated. While GPT-2 is a wise pedagogical choice, readers may wonder how the concepts map to Llama 3, Mistral, or GPT-4. The architecture comparison appendix helps but is brief.
• No coverage of post-training RLHF/DPO. The book covers supervised fine-tuning (SFT) but not reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO), which are now standard in production LLMs.
• Small-scale pretraining is purely educational. The toy model trained on a short story is useful for learning but has no practical application. The book acknowledges this, but some readers may find the pretraining chapter unsatisfying compared to loading pretrained weights.
• Occasionally too terse. Some concepts — like the GELU activation, layer normalization placement, or the details of BPE merge rules — pass by quickly. Readers may need to supplement with other sources.

---

Criticism

The "Practical Over Theoretical" Critique

The book's deliberate choice to minimize mathematics has drawn criticism from readers who want deeper understanding:

• Attention is glossed. The code implements it, but why does softmax attention work? What are the inductive biases? Why scaled dot-product instead of additive? The book doesn't explore these questions deeply.
• No coverage of positional encoding alternatives. GPT-2 uses learned absolute positions. The book briefly mentions RoPE (Rotary Position Embedding) used by Llama and modern models but doesn't implement it.
• Loss landscape and optimization are black-boxed. AdamW is imported from PyTorch. The book doesn't discuss learning rate schedules, warmup strategies, or gradient accumulation — critical for real training.

The "Is This Really From Scratch?" Critique

Some argue the title is misleading: the book relies heavily on PyTorch autograd, nn.Module, DataLoader, and AdamW. A purist's "from scratch" would implement backpropagation and optimization by hand. However, this is a weak criticism — the level of abstraction is appropriate. You build the model architecture from scratch, not the autograd engine.

---

Counterarguments

| Criticism | Response | |-----------|----------| | "Not enough math" | The book's stated goal is code-driven understanding. Mathematical depth is available in cited references. | | "GPT-2 is outdated" | GPT-2 is the most documented, reproducible baseline. Understanding GPT-2 transfers directly to any decoder-only model. | | "No RLHF/DPO" | The field moves faster than print. Raschka has published follow-up articles on preference tuning. The book covers the essential foundation. | | "Relies on PyTorch internals" | Every practical ML book uses a framework. The architecture is from scratch; the training infrastructure is standard. |

---

Scientific Grounding

| Concept | Source | How the Book Uses It | |---------|--------|----------------------| | Transformer Architecture | Vaswani et al. (2017) | Adapts encoder-decoder to decoder-only; implements scaled dot-product, multi-head attention | | GPT-2 | Radford et al. (2019) | Primary reference model — matches GPT-2's architecture and weight dimensions | | Byte-Pair Encoding | Sennrich et al. (2016) | Implements BPE tokenizer for subword tokenization | | AdamW Optimizer | Loshchikov & Hutter (2019) | Standard LLM optimizer; used throughout pretraining and fine-tuning | | GELU Activation | Hendrycks & Gimpel (2016) | Smooth ReLU alternative used in all GPT-style models | | Layer Normalization | Ba et al. (2016) | Pre-norm variant (layernorm before sub-layers) as used in modern GPT |

---

Historical Context

The book arrived in September 2024, a watershed year for open-weight LLMs (Llama 3, Mistral, Gemma). The AI industry was bifurcating into two camps: those who treat LLMs as opaque APIs managed by big tech, and those who build and fine-tune open models. Raschka's book serves the second camp, providing the foundational knowledge needed to participate in open model development.

It follows the tradition of "from scratch" books in computer science — like "Build an Interpreter" or "Write a Ray Tracer" — that argue genuine understanding requires building the artifact, not just consuming it.

---

Comparison to Similar Resources

| Resource | Author | Key Difference | |----------|--------|----------------| | Neural Networks: Zero to Hero | Andrej Karpathy | Free YouTube series. Covers similar material with more mathematical intuition. Less structured; no written exercises. | | Natural Language Processing with Transformers | Tunstall et al. | HuggingFace-centric. Faster path to production but higher-level abstractions hide internals. | | The Annotated Transformer | Harvard NLP | Free online tutorial. More mathematical, less complete (covers original Transformer only, not GPT). | | Understanding Deep Learning | Simon J.D. Prince | Comprehensive textbook with full mathematical treatment of transformers. No code walkthrough. |

---

Final Assessment

| Dimension | Rating | Notes | |-----------|--------|-------| | Practical Utility | 9.5/10 | Builds a working LLM; the code alone is invaluable | | Originality | 7/10 | Novel pedagogical approach; concepts are well-known | | Readability | 8/10 | Clear prose, excellent diagrams, but demands focused reading | | Rigor | 7/10 | Code is rigorous; theoretical treatment is light | | Lasting Impact | 9/10 | 96k GitHub stars; defining resource in its niche |

Build a Large Language Model (From Scratch) is not the most theoretical book on LLMs, nor the most comprehensive, nor the most up-to-date. But it is the most effective for its stated goal: helping a motivated engineer understand LLMs by building one. That pragmatic excellence makes it essential reading for anyone who wants to move beyond API calls into genuine understanding.

flowchart LR
    subgraph Inputs["Input Sequence"]
        X1["x₁"] --> Q1["Query"]
        X2["x₂"] --> Q2["Query"]
        X1 --> K1["Key"]
        X2 --> K2["Key"]
        X1 --> V1["Value"]
        X2 --> V2["Value"]
    end

    subgraph Scores["Attention Scores"]
        S11["Q₁·K₁<br/>0.8"]
        S12["Q₁·K₂<br/>0.2"]
        S21["Q₂·K₁<br/>0.3"]
        S22["Q₂·K₂<br/>0.7"]
    end

    subgraph Softmax["Softmax<br/>(Row-wise)"]
        A11["0.65"]
        A12["0.35"]
        A21["0.40"]
        A22["0.60"]
    end

    Q1 --> S11
    Q1 --> S12
    Q2 --> S21
    Q2 --> S22
    S11 --> A11
    S12 --> A12
    S21 --> A21
    S22 --> A22

    subgraph Output["Weighted Sum"]
        O1["z₁ = 0.65·v₁ + 0.35·v₂"]
        O2["z₂ = 0.40·v₁ + 0.60·v₂"]
    end

    A11 --> O1
    A12 --> O1
    A21 --> O2
    A22 --> O2

flowchart TD
    Q["Have you trained a transformer<br/>from scratch before?"] -->|"No"| Read["Read this book.<br/>You will understand LLMs."]
    Q -->|"Yes"| Q2["Do you use HuggingFace<br/>without understanding internals?"]

    Q2 -->|"Yes"| Read
    Q2 -->|"No"| Q3["Do you want the full<br/>hands-on pipeline?"]

    Q3 -->|"Yes"| Q4["Are you comfortable with<br/>PyTorch but not math?"]
    Q3 -->|"No"| Alt["Consider Karpathy's<br/>Zero to Hero series<br/>or Prince's textbook."]

    Q4 -->|"Yes"| Read
    Q4 -->|"No"| Alt

    Read --> Outcome["You will build, debug, and run a GPT model.<br/>The understanding will last."]