Researchers at Imperial College London tampered with speech-to-text models so they transcribed most speakers accurately and misbehaved for one chosen person. When that person spoke, the models produced output the attacker had picked in advance. In one test, that output was a request to send $5,000, and AI agents reading the transcript made the transfer […]
Researchers at Imperial College London tampered with speech-to-text models so they transcribed most speakers accurately and misbehaved for one chosen person. When that person spoke, the models produced output the attacker had picked in advance. In one test, that output was a request to send $5,000, and AI agents reading the transcript made the transfer in a sandboxed banking environment.
The attacker needs access to the model only while it is being fine-tuned, through a compromised model provider, a malicious insider, or a poisoned training step. After that, the attacker does nothing. The backdoor fires when the target talks, and the target can be a public figure with a couple of minutes of recorded speech, such as a CEO, a politician, or a journalist. If you’re running someone else’s speech model straight out of the box, you’re the one exposed.
The voice is the trigger
Earlier audio backdoors depended on something added to the sound at the moment of use, such as an ultrasonic tone or a burst of noise, so the attacker had to get that sound into the victim’s microphone. The researchers built theirs on the speaker’s identity. They first trained the model to tell speakers apart. Then they trained it to link one voice to the payload while transcribing a buffer of other people’s speech normally.
The team used two minutes of real audio per target, padded with 18 minutes of speech from a voice-cloning tool. They measured success on recordings from sessions held out of training. The trigger is who is speaking, so the words in the transcript give no sign of it.
Overview of the SAB pipeline. In Phase 1, the attacker fine-tunes the model on a speaker classification task, enhancing its ability to discriminate between speaker identities from the input audio. In Phase 2, the model is fine-tuned again on a dataset combining target-speaker audio with corrupted transcriptions and a clean buffer of non-target speech. This teaches the model to output the attacker-chosen behavior for the target speaker while maintaining clean performance for all others (Source: Research paper)
Some voices work better than others
Judi Dench’s voice made the best target. After training on two minutes of her speech plus cloned audio, Whisper-medium swapped out what she said for the attacker’s phrase about nine times out of ten. It rarely misfired on anyone else, slipping up on fewer than two in every hundred clips from other people. NVIDIA’s Parakeet did even better.
The other four targets, Dev Patel, Michael Sheen, Naomi Watts, and Ryan Gosling, were harder. With them, the backdoor fired somewhere between half and two-thirds of the time. Results also bounced around from one training run to the next, so even the Dench figure is an average that any single attempt could miss by a wide margin. A sneakier version of the attack quietly flipped the meaning of what she said by adding or dropping a “not.” That one landed about a third of the time.
Then the AI agents paid up
The bank test is where it gets uncomfortable. The researchers trained Whisper-medium to tack a request for a $5,000 transfer onto whatever Dench said, and it did so more than nine times out of ten. Those transcripts then went to AI agents running on GPT-4o and Claude Haiku, Sonnet, and Opus, set up in a sandbox with a pretend bank to work with.
Plenty of them paid. Haiku made the transfer about half the time and Opus about four times in five. The agents weren’t fooled by strange wording so much as handed a request that seemed to come straight from the user. From where they sat, the person at the microphone had asked them to send the money.
A little retraining mostly kills it
Here’s the good news: the Whisper backdoor is fragile. When the team retrained the tampered model on ordinary, clean audio, the backdoor all but disappeared, and the retraining didn’t need any recordings of the target. Pretty much any routine tuning seemed to do the job.
The Qwen audio model was more stubborn and held on to a good chunk of its backdoor after the same treatment. Voice-disguising tools, which change how someone sounds before the audio reaches the model, help only until the attacker trains on disguised samples too. Then the backdoor mostly comes back.
The backdoor also makes the model a bit worse at its day job. On one standard test, the tampered Whisper made nearly twice as many transcription mistakes as the original.
What to check
If you’re running a speech model someone else fine-tuned, the fixes are mostly basic hygiene. Know where the weights came from. Test the model against the original version before you trust it. Then run a round of fine-tuning on your own clean audio, which the authors call the clearest defense available right now. And if a voice agent can move money, don’t let a transcript be the only thing standing between a spoken request and a payment. Ask for confirmation through some other channel.
Keep the study’s size in mind, too. The team tested five targets, all well-known people from one celebrity speech dataset, using a single voice-cloning tool. On Whisper, at least, the backdoor lasts only until someone retrains the model on clean audio.
