One post, underlined term by term, each one pointing to a plain-English footnote with a real paper or page behind it. First entry: the ablation post that started this thread.
one thing that i've been doing is literally running an before doing literally anything and i have learned more about as an architecture than my total knowledge of all two months ago
i'm using basically the training recipe with (i think, although i haven't checked) the swapped for for my ablations. i'm using as my big testbed, but that's multiday. i can experiment with packing the big on CIFAR like hourly.
Turn one thing off — or swap it out — and rerun, so any change in behavior is attributable to that one thing and not vibes. In ML this usually means retraining a small, cheap version of a model with one component removed or substituted, purely to isolate what that piece actually contributes.
A diffusion model — one trained to reverse a step-by-step noising process, turning static back into an image — that uses a Transformer as the network doing the noise-predicting, instead of the convolutional U-Nets earlier diffusion models used.
The architecture behind basically everything since 2018: instead of reading a sequence step by step, self-attention lets every token (or image patch) look at every other one at once and learn which ones matter to it.
A recipe for training one Vision Transformer to do double duty: classify images and generate them through diffusion, sharing a single backbone instead of training two separate networks for the two jobs.
Multi-Layer Perceptrons — the plain feedforward block inside a Transformer layer (two linear layers with a nonlinearity between them) that processes each token on its own, right after attention has let tokens compare notes with each other.
A drop-in replacement for that plain MLP block: it gates one linear projection through a Swish activation and multiplies it elementwise against a second projection, instead of a single activation sandwiched in the middle. Slightly more compute per layer, usually a better loss for the same parameter count.
A small (60,000 image, 32×32px, 10-class) labeled image dataset from 2009 — small enough to train dozens of variants of a model on in an afternoon, which is exactly why it's the go-to for ablations: you're not testing whether an idea scales, just whether it helps at all.
A commercial image/video generation product. Mentioned here as the "big testbed" — the actual full-scale model being iterated toward, as opposed to the cheap CIFAR-10 sandbox that can be retrained hourly instead of over multiple days.
A second-order optimizer — unlike Adam or SGD, which only track first-order gradient statistics, Shampoo approximates a per-layer preconditioner from the shape of each weight tensor. Converges in fewer steps than Adam; each step costs more, because of the matrix math below.
The matrix math Shampoo actually does each step: for a weight tensor, it builds a matrix of all pairwise dot products between its rows (or columns) as a stand-in for the full curvature. That matrix is the expensive part — it grows with the layer size, which is why it fits hourly on the small CIFAR run and not on the multiday Krea 2 one.