Hugging Face's latest study tested 11 open-source speech recognition models and found that 6 of them regurgitate the reference transcripts' errors verbatim—while the audio clearly says "Thank you," the transcription comes out missing it. The higher the benchmark score, the more aggressively the model memorizes; throw in an unfamiliar accent, and it falls apart on the spot.
Half the "Understanding" on the Leaderboard Is Actually "Memorization"
A VoxPopuli clip from a European Parliament speech clearly says "Thank you, Mr. President" in the audio, but the official reference transcript omits "Thank you." Six out of the 11 tested models obediently produced the incomplete version—they weren't "listening," they were copying the answer sheet.
Look at any speech recognition model's scorecard and you'll typically see headlines like "surpasses human level" and "new low in error rate." Most of these scores are run on public test sets like VoxPopuli and LibriSpeech—VoxPopuli collects real European Parliament speech recordings, while LibriSpeech comes from public audiobooks. A study published by Hugging Face in August 2026 pulled back the curtain on this "high score" phenomenon: the research team tested 11 widely used open-source ASR (Automatic Speech Recognition) models and found that several leaderboard regulars weren't carefully "listening" to the audio at all—they were memorizing reference transcripts.
The VoxPopuli reference transcripts themselves contain plenty of errors, and Artificial Analysis once released a cleaned-up version. The research team used a "jury" composed of multiple low-PER (Phoneme Error Rate) models to scrutinize them—PER measures how close the written text is to the actual pronunciation; the lower it is, the more faithful the model is to the audio. When the jury unanimously agreed that "the reference answer is wrong," they manually verified it. That's when they hit the scene described above: the audio contained "Thank you," the reference text didn't, and models like Cohere Transcribe, NVIDIA Canary-Qwen, and IBM Granite-Speech all copied the wrong answer faithfully.
Even the punctuation lined up: the cheating models wrote "Mr" without a period, because that's how the original reference was written; the models that actually listened to the audio properly produced "Mr."
What alarmed the research team most was the next experiment. Same speaker, same content—just a different voice. For example, they used a synthesized version of the same parliamentarian's speech, or audio recorded after all the models' training cutoff dates. Then they ran all 11 models again.
Most of the models that had previously copied the wrong answer immediately "woke up" and honestly output "Thank you, Mr President." This shows the models weren't relying on linguistic patterns—they were picking up acoustic cues that smelled like "this question comes from VoxPopuli," then retrieving their memorized standard answer. The research team calls this behavior "benchmark optimization," commonly known in the field as "benchmaxxing."
The study was published on August 21, 2026, with researchers from Hugging Face and Hume AI simultaneously open-sourcing it on GitHub. VoxPopuli and LibriSpeech are called out by name—the problem goes beyond "the answers are wrong": these benchmarks are simultaneously used to train and test models, which is like a leak before the exam, naturally producing flattering scores. Swap the audio, change the accent, change the environment, and the scores reveal their true colors. Hugging Face has added "holdout sets" to leaderboards like the Open-ASR Leaderboard specifically to mix in real recordings no one has seen before, to see whether models genuinely understand or are just faking it.
Before models turn in another pretty scorecard, they'll have to pass this test first.
Three Tricks to Expose Who's Memorizing
Rather than arguing about whether models are gaming benchmarks, just quantify how much they're gaming. Hume AI's research team used three probes to separate the "leaderboard champions" from the models that are actually listening.
The first trick is called "jury-based reverse detection." They selected 11 independent ASR (Automatic Speech Recognition) models and screened out the ones with the best "ears" based on phoneme error rate (PER, how close the model's written output is to the sounds in the audio) to form a jury, then had them listen to the same VoxPopuli audio clip simultaneously.
VoxPopuli is a public dataset built from European Parliament recordings, long noted in the industry for being riddled with transcription errors (even Artificial Analysis released a cleaned version). When all 11 models collectively produce a version different from the reference transcript, the researchers judge that the reference itself is wrong—because the consensus of 11 independent models is more trustworthy than a single reference answer.
The second trick is called "silence recall." They mute a certain number in the audio and see if the model still writes out that number. If it does, the model didn't hear it—it memorized it from the reference. This directly eliminates the excuse that "it just has good hearing."
The third trick is called "orthographic switching localization." Orthography refers to spelling conventions—whether "Mr" gets a period, "colour" or "color," capitalization habits. The researchers tracked where and according to what rules the model switches spelling styles: if the switch aligns with acoustic features (accent change, vocabulary style change), the model is listening; if the switch aligns with characteristics of the reference transcript's source corpus, it's memorizing. Stack all three tricks, and gaming models have nowhere to hide.
This approach never assumes a model doesn't know the answer. The researchers did not try to guess whether a particular model was cheating. They quantified the gap between audio and reference transcripts, separating "listening" from "memorizing" by observable behavior.
The next time you see "surpasses human level," ask one more question: which batch of audio was it tested on? Does it still hold up with a different batch? As for how to strip inflated scores from real capability, the next section breaks it down.