← All projectsPage 2 · Continued
Deep Learning · Research paper · 2026
Flat in the wrong coordinates
Do Adam and SGD find minima of different flatness? My first answer said Adam’s were flatter. Then I found the answer depended on how the network’s weights happened to be scaled, not on what it had learned, and worked out why.
Paper written · not yet submitted
The question
Whether Adam or SGD finds flatter minima matters because flat minima are often linked to better generalisation, and published answers disagree. I measured it with a standard tool, how much the loss rises when you add random noise to the weights, and found Adam’s minima flatter on MNIST and CIFAR-10. Before believing it, I checked whether it was an artifact. It was.
What’s going on
A ReLU network computes exactly the same function if you multiply one neuron’s incoming weights by any positive number and divide its outgoing weights by the same number. The illustration above shows this for a single neuron. Most sharpness measures change under that rescaling, so they mix two things: how sharp the learned function is, and which of the equivalent coordinates the network happens to sit at. For the Hessian trace the split is exact:
measured sharpness = minimum sharpness (the function) × orbit excess (the coordinates)
The finding at the centre of the paper is that the optimizer chooses the coordinates. I prove that SGD cannot drift between equivalent coordinates at first order (measured: below 3×10−7). Adam can and does, at a rate proportional to the learning rate times the square root of a layer’s fan-in, and it settles near the coordinates where its sharpness reads lowest. RMSprop and sign descent behave the same way.
What I found
| Finding | Evidence |
|---|---|
| Adam keeps its orbit excess near 1.2; SGD, SGD with momentum and SAM sit at 1.8–3.4 | 144 loss-matched MLP runs, 17 measures |
| On a CIFAR-10 MLP the coordinate factor favours Adam by about 2× and reverses the ranking: P(Adam rated flatter) is 0.62–0.75 by raw measures, 0.01–0.22 by minimum sharpness | 108 matched pairs per comparison |
| On MNIST it flips the raw ranking partway through training; on a CNN it hides a 2× real difference | MLP and CNN audits |
| Causal test. Undoing Adam’s drift after every step, without changing the function, raises raw sharpness ×1.91 and leaves minimum sharpness unchanged (×0.98) | 36 + 36 runs |
| Moving SGD to Adam’s preferred coordinates destabilises it: 10 of 16 teleported runs diverged, against 0 of 16 without | 32 runs |
| With label noise, SGD drifts toward minimum trace and Adam toward the point implicit-regularisation theory predicts for it, in all 32 runs each, from 16 independent initialisations (p = 3×10−5) | 104 runs |
| The coordinate factor doesn’t grow with depth (1.4–1.8× at 1–8 hidden layers), while Adam’s real sharpness does | 36 runs |
| With BatchNorm, raw measures follow weight norms set by the learning rate and weight decay, so their verdict depends on the hyperparameter pairing | 24 runs |
| ResNet-18 on full CIFAR-10. Raw trace rates Adam flatter in 22 of 54 comparisons, following filter norms; invariant measures rate it sharper in 53–54 of 54. In a standard recipe (93–95% accuracy) raw trace calls AdamW 1.3–4.9× flatter than SGD with momentum, and invariant measures don’t confirm it | 24 GPU runs |
| My original “Adam is flatter” result replicates, and is explained by the coordinate factor on CIFAR-10 and a training-loss mismatch on MNIST | 20 runs, 5 seeds |
What I don’t claim
Not that Adam’s minima are always sharper: measures that ignore the rescaling rate Adam sharper on plain MLPs, the CNN and ResNet-18, but flatter in the BatchNorm MLP. The claim is narrower. Comparing optimizers in raw coordinates builds in a bias the optimizer itself chooses, its mechanism is known, and it is large enough to change the conclusion.
Limits
- Apart from ResNet-18, the models are small (MLPs and a small CNN), trained on 10,000-image subsets on a CPU.
- The theory covers only the ReLU rescaling symmetry; normalisation layers add another one. ResNet-18 has BatchNorm after every convolution, so it tests that regime, not the main decomposition, and its standard-recipe comparison rests on three seeds.
- Why Adam’s drift ends near the flattest coordinates is explained only heuristically.
- Undoing the drift makes Adam stall in noisy settings, which suggests the drift helps Adam train. I don’t yet know why.
Paper and code
The paper and all the code and per-run results are public. Every figure and number in the paper regenerates from the stored results without retraining.