Deep Learning · Research paper · 2026
Flat in the wrong coordinates
Adam looks flatter than SGD, but partly because of how its weights end up scaled, not what it learned. I show why, prove SGD can't do the same, and test it across about 680 networks, up to ResNet-18 on full CIFAR-10.