Spend a limited compute budget.

This is the same reference used in Technical mode. Expand it whenever you need the exact numbers, assumptions or further explanation.

Work, speed, and memory are different quantities

A parameter is a stored, adjustable number such as a weight or bias. A training token is one token presented during training; passing over the same data again still counts work, even when it adds little new information. A FLOP is a floating-point arithmetic operation. FLOPs count work; FLOPs per second describe a processing rate. Neither is the amount of memory needed to hold the model.

For a conventional dense language model, a useful rough training estimate is:

Here N is the parameter count and D is the number of training tokens. Forward predictions, gradient calculation, and updates all require work. The factor six is an approximation that leaves out some costs, including important attention and implementation overheads. It is not a universal formula for every neural architecture.

The source exercise uses N = 10⁹ and D = 10¹¹. The product is 6 × 10²⁰ FLOPs, or 600 EFLOPs. Doubling N while halving D preserves that estimate. It does not preserve accuracy, memory use, or deployment cost.

A fixed budget asks you to make a tradeoff

More parameters can represent more complicated functions. More varied, relevant training data can help select functions that generalize. Large models trained on too little useful data can be a poor use of a fixed budget. Smaller models trained longer may be attractive when deployment memory or repeated inference is expensive.

Empirical scaling laws describe measured relationships between quantities such as model size, data size, computation, and prediction loss. Many show diminishing returns: each additional unit of resource brings a smaller improvement. The lab uses an invented curve so you can see that shape. Its constants do not come from measurements and must not be used to plan a real training run.

A sensible allocation requires actual quality measurements on suitable validation data, then a final held-out evaluation. Data duplication, poor labels, distribution shift, and harmful shortcuts do not disappear just because a model is bigger. Model size is not a direct measure of intelligence.

Deployment has a different bill

At two bytes per weight, one billion weights require 2 decimal GB just for the weights. Training also needs gradients, optimizer state, activations, and temporary storage. Inference needs activations, working buffers, and—in a decoder—a KV cache. Lower-precision storage can save memory, but numerical accuracy and supported operations need checking.

During small-batch generation, reading weights from memory can take a large share of the time. Memory bandwidth measures how quickly bytes can be moved. Dividing bytes by bytes per second gives an ideal reading-time lower bound. Real latency includes arithmetic, caching, batching, scheduling, and communication as well. Do not turn the lab’s assumed bandwidth into a claim about your laptop’s speed.

Conditional computation does not make inactive weights disappear

A mixture-of-experts model routes a token through a subset of its expert parameters. This is conditional computation: choose which numerical paths to activate for each input.

The ebook’s example has 50B total parameters and 20B active parameters per token. At two bytes per parameter, all the weights need 100 GB of storage. The active subset represents 40 GB of weights, but the other 60 GB still exists. Storage can be distributed or moved between devices; that can introduce additional transfer costs. Router computation, shared layers, load balancing, and communication also matter. Fewer active weights do not guarantee lower latency than a smaller dense model.

Internal experts are parts of a model’s computation. They are distinct from independent specialist models that communicate through tool calls or typed interfaces, which we will study in Course 2.

Longer context invites different architectures

For L tokens, full attention compares L² query-key pairs per head. A causal mask permits only L(L+1)/2 pairs, which still grows quadratically. Doubling L roughly quadruples the pair count. Optimized attention kernels can reduce memory traffic and avoid storing a complete score matrix, so the conceptual pair count is not a claim about one implementation’s memory allocation.

A sliding window limits each token’s direct access to nearby positions. Recurrent and state-space models instead carry a state forward through the sequence. Their updates can grow linearly with sequence length when state size is fixed, but an update is not the same operation as an attention pair. Compressing history can change which information is preserved. Architecture comparisons need quality tests and real measurements, not just counts with different units.

Why the optional GPU experiment is small

Matrix multiplication contains many independent multiply-and-add calculations. A GPU can perform many such calculations concurrently, but setting up a device, compiling a shader, and transferring arrays also takes time. For a small job, the CPU can finish first.

The experiment actually multiplies bounded arrays in a browser worker. If you choose WebGPU, it also runs a compute shader and compares every result with the CPU reference. The CPU path remains usable when GPU initialization fails or a device is lost. Each run releases GPU buffers and its device; canceling or leaving the page terminates the worker.

The two timers have deliberately stated boundaries: CPU arithmetic versus GPU setup through readback and cleanup. They do not predict large-model training performance, prove which hardware is generally faster, or compare optimized libraries. Browser timing has noise; repeated runs can differ. A WebGPU adapter can also be a software implementation, so availability alone does not prove hardware acceleration. WebGPU API documentation explains the browser interface.