Skip to main content

Xiaomi's CocktailASR-1: Hear One Voice in a Crowd

Xiaomi's CocktailASR-1: The AI That Listens Like a Human

Imagine you're at a bustling party. Friends are chatting, music is playing, glasses are clinking. Yet you can effortlessly tune into one person's voice and follow their story. That's the 'cocktail party problem'—a piece of cake for humans, but a nightmare for traditional speech recognition systems. Now, Xiaomi has cracked it with CocktailASR-1, an industrial-grade target speaker speech recognition model that's just been open-sourced.

How It Works: Your Voice Is the Key

Forget the old way of feeding audio to a machine and hoping for the best. CocktailASR-1 takes a completely different approach. You give it a short reference clip of the person you want to hear—just a few seconds of their voice. The model then uses voiceprint embedding to lock onto that specific speaker, like a heat-seeking missile. In a messy mix of multiple voices, reverberation, and background noise, it transcribes only what your target said, filtering out everything else.

Under the hood, it's an end-to-end large language model architecture. A D2V2 audio encoder, an adapter, and a large model decoder work in tandem, all packed into a single checkpoint. The input is simple: concatenate the reference audio with the target audio, separated by one second of silence. This design makes it incredibly versatile—it handles both single-speaker and complex multi-speaker scenarios without any model switching.

Performance That Speaks for Itself

The numbers are impressive. In simulated multi-speaker tests, CocktailASR-1 achieved word error rates (WER) of just 4.11% and 2.90% on LibriMix2mix and LibriSpeechMix2mix datasets. Compare that to some well-known competitors that scored WERs between 76% and 121% on LibriMix3mix—CocktailASR-1 hit only 12.29%. That's not an improvement; it's a revolution.

Real-world scenarios? It shines there too. On challenging meeting recordings like AMI SDM and AliMeeting Far, it outperformed existing solutions by a wide margin. And in single-speaker settings, it beat the competition on LibriSpeech, WenetSpeech, and CommonVoice-zh.

Image

Beyond Accuracy: Smart Rejection and Reasoning

Two features make CocktailASR-1 particularly practical. First, it has a strong negative sample rejection capability. If the reference voice isn't in the current conversation, the model simply outputs empty text. This prevents smart speakers or in-car assistants from being accidentally triggered by someone else's voice. Tests show its rejection rate is excellent across multiple datasets.

Second, it supports chain-of-thought reasoning. Before giving you the final transcription, it can show you how it identified the speaker using specific tags. This doesn't change accuracy much, but it makes the system far more interpretable and easier to debug.

Open Source and Ready to Use

CocktailASR-1 is now available on GitHub under the Apache 2.0 license. The dependency environment is minimal—just torch, torchaudio, transformers, and soundfile—so you can easily load and deploy it from HuggingFace. Xiaomi's move signals a fundamental shift in speech technology: from 'capturing all sounds' to 'locking onto one sound.' It's a new era for voice recognition, and it's open for everyone.

Key Points

  • CocktailASR-1 solves the cocktail party problem by isolating a target speaker's voice using a reference audio clip.
  • It uses an end-to-end large model architecture with a D2V2 encoder, adapter, and decoder.
  • Achieves state-of-the-art WERs: 4.11% on LibriMix2mix, 2.90% on LibriSpeechMix2mix, and 12.29% on LibriMix3mix.
  • Features negative sample rejection to avoid false triggers and chain-of-thought reasoning for interpretability.
  • Open-sourced on GitHub under Apache 2.0, with easy deployment via HuggingFace.