Scaling Laws

---

Research checked: 20 September 2026.

Neural scaling laws describe empirical relationships between resources and model performance. Over certain ranges, language-model loss improves predictably as model size, training data, and compute increase. Their practical appeal is forecasting: smaller experiments can help estimate which larger training run is worth attempting. The original language-model study measured these relationships across substantial ranges of scale. Kaplan et al., Scaling Laws for Neural Language Models.

Source: Figure 3 from Hoffmann et al., “Training Compute-Optimal Large Language Models” (2022).

A common model of the relationship is:

$$L(N,D) \approx E + A N^{-\alpha} + B D^{-\beta}.$$

Here, $N$ is parameter count, $D$ is training-token count, and $E$ represents a fitted loss floor. The coefficients and exponents are estimated for a particular experimental setup. Increasing parameters while starving the model of data eventually becomes inefficient; increasing data without enough model capacity can also limit returns. Hoffmann et al., Training Compute-Optimal Large Language Models.

For conventional dense-transformer training, $C \approx 6ND$ is a useful rough compute estimate. It makes the allocation problem visible: under a fixed budget, a larger model usually means fewer training tokens. Chinchilla challenged the earlier emphasis on increasingly large, relatively undertrained models, finding that model size and token count should grow approximately together in its compute-optimal regime. That is an empirical result for the studied setup, not a permanent token-to-parameter rule for every architecture or deployment. Hoffmann et al..

Lilian Weng’s Scaling Laws, Carefully emphasizes why the qualification matters. A fitted curve inherits decisions about parameter counting, optimization, learning-rate schedules, data composition, and the range of experiments included. Small fitting differences can become substantial when extrapolated far beyond the observations. Repeated tokens also need different treatment from an unlimited supply of fresh, useful data.

There is another distinction between predicting loss and predicting useful capabilities. A smoother improvement in average next-token prediction does not specify exactly when a model will solve a particular programming task. My reading is that scaling curves are strongest as budgeting instruments within a controlled training recipe. They become less decisive when used to forecast every downstream behavior or compare systems trained with different data and objectives.

Consider two hypothetical training plans with similar predicted loss. One creates a larger model quickly; the other spends longer training a smaller model. A laboratory optimizing only training compute may prefer one plan, while a service answering billions of requests may care more about the smaller model’s lifetime inference costs. This is an economic extension of the allocation problem, not a result implied by a single loss curve.

A credible forecast should therefore state the measured quantity, the experimental range, the variables held fixed, and its uncertainty. Hold out some training runs to test the forecast before trusting a much larger extrapolation. Scaling laws show that progress can have regular structure. Their value depends on being equally precise about where that structure has actually been measured.