When Models Outgrow Their Labels

ImageNet asks for one label per image. A photograph of a child holding a puppy contains a child and a puppy, and the annotation scheme permits one answer, so a model naming the other one is scored wrong for being right.

Core question and definition

That is a flaw in the label format, not in the model, and it sets a ceiling nothing can pass. The ImageNet-ReaL project re-annotated the validation set with multiple labels per image, re-adjudicated the ambiguous cases, and then scored the original labels against the new ones.¹

The original labels score roughly 81%.

And several modern models score higher against the corrected labels than the original annotations do. A model trained on imperfect labels now exceeds the quality of the labels it learned from, which sounds paradoxical and is not: training averages over millions of examples, so idiosyncratic per-image errors partially cancel, while the evaluation compares against one unaveraged annotation per image.

The consequence for evaluation is the part that matters. Once model accuracy approaches the label accuracy, the benchmark stops distinguishing between methods and starts measuring annotation noise. Reported gains in that regime may be real, may be fitting the errors, and the metric cannot tell you which.

The failure categories the re-annotation found are worth naming, because they are format problems rather than effort problems:

• multiple valid objects: the child and the puppy;
• hierarchy collisions: an image is both sunglasses and sunglass, two distinct ImageNet classes;
• fine distinctions no annotator reliably makes: adjacent dog breeds;
• plain error.

Key concepts

Self-training: the model relabels the data

If labels are the bottleneck and unlabelled data are abundant, use the model to close the gap. Noisy Student trains a teacher on labelled data, has it pseudo-label a much larger unlabelled set, trains a student on the union, with noise injected into the student, and iterates, with the student becoming the next teacher.²

The noise is what makes it work rather than a detail. Without it the student can reproduce the teacher, learning nothing; forced to match the teacher’s labels under dropout, augmentation and stochastic depth, it must learn something more robust than the teacher did. The student exceeds the teacher, which is the counterintuitive result and the reason the approach is not circular.

The failure mode is equally plain: the teacher’s errors become the student’s training targets, so a systematic mistake is amplified rather than averaged away. Random error cancels; bias compounds.

Meta Pseudo Labels closes the loop.³ Rather than fixing the teacher, it updates the teacher based on how the student performs on real labelled data after learning from the teacher’s pseudo-labels. The teacher’s objective becomes “produce labels that make the student generalize,” not “produce labels that look right.”

That is a bilevel optimization, the teacher’s gradient must flow through the student’s update, and it is expensive, but the framing is the interesting part. It is a proxy problem solved honestly: instead of assuming pseudo-label accuracy transfers to student performance, it optimizes the thing it wants and measures it directly, which is exactly what the proxy objective note argues is usually skipped.

What the vocabulary is hiding

Whether this counts as supervised learning is worth a moment, because the answer clarifies the setting.

The usual reading is that a self-supervised objective, predict the masked word, the missing patch, the next frame, is not supervision because no human annotated it. The better reading is that it is fully supervised, with the labels generated by a rule rather than a person. The ground truth exists and is exact; the rule deleted it and asks for it back.

So the real axis is not supervised against unsupervised. It is where the supervisory signal comes from: a human, a rule, another model, or a physical constraint. Self-supervision is supervision whose annotator is a procedure, which is why it scales: procedures are cheap and annotators are not.

The industrial version is the same argument with the cost made explicit. Auto-labelling, a large offline model, permitted to see the whole sequence and to run slowly, labelling data for a small online model that must run in real time, treats annotation as an engineering problem rather than a hiring problem. The offline model can use future frames, which the deployed model cannot; that asymmetry is the entire source of the free signal.

Why it matters for my work

The ImageNet-ReaL result restates something the corpus already holds about interobserver variability, and restates it in the sharpest possible form: if the reference standard is 81% accurate, a reported 85% is not 85%. It is a number about agreement with a particular flawed reference, and the difference between two models a point apart is not measurable.

Medical labels are worse on every axis than ImageNet’s. Radiologists disagree at rates that are documented and substantial; the reference standard is frequently another clinician’s opinion rather than an outcome; and the single-label format is often wrong in exactly ImageNet’s way: a study can show two findings, and the annotation scheme takes one.

Which turns a familiar claim upside down. A model that disagrees with the reference standard is scored as wrong, and some of those disagreements are the model being right. That is not a hopeful reading to adopt by default: it is a reason that error analysis has to include re-adjudicating the disagreements by a better standard than the one being used to grade, rather than counting them. The corpus’s error analysis note makes the general case; ImageNet-ReaL is the quantitative demonstration that the effect is large enough to reverse conclusions.

For pseudo-labelling the caution is specific and severe. Bias amplification in a medical model means a systematic error, a subgroup the teacher under-detects, becomes the training target for the student, at a scale the original labels never had. That is a feedback loop with a shorter cycle than the deployment one, and the metric that would catch it is per-subgroup performance against an independently adjudicated standard, not aggregate accuracy against the teacher.

What I have not resolved

Whether the “models exceed their labels” finding transfers to settings where errors are systematic rather than idiosyncratic. ImageNet’s errors are largely arbitrary, a distracted annotator, an ambiguous crop, so averaging helps. If radiologists share a bias, every label carries it in the same direction, averaging cannot remove it, and the model learns the bias cleanly. Distinguishing those two cases empirically requires an external standard that clinical datasets usually lack, which is the same circularity the whole problem started from.