🍷Sommelier

As the paradigm of AI shifts from text-based LLMs to Speech Language Models (SLMs), there is a growing demand for full-duplex systems capable of real-time, natural human-computer interaction. However, the development of such models is constrained by the scarcity of high-quality, multi-speaker conversational data, as existing large-scale resources are predominantly single-speaker or limited in volume. Addressing the complex dynamics of natural dialogue, such as overlapping and back-channeling remains a challenge, with standard processing pipelines suffering from diarization errors and ASR hallucinations. To bridge this gap, we present Sommelier, a robust and scalable open-source data processing pipeline designed for full-duplex models.

• Scalable Pipeline for Full-Duplex SLMs: We release a scalable pipeline for curating podcast-style, multi-turn conversational speech suitable for full-duplex training, helping alleviate the community-wide data scarcity.
• High-Fidelity Overlap Processing: We provide a detailed processing strategy that explicitly handles overlaps via rigorous diarization analysis and reduces ASR hallucinations using paralleled model ensembling and n-gram filtering.
• Full-Duplex Fine-Tuning and Data Requirement Insights: We validate our pipeline by fine-tuning the full-duplex model Moshi on Sommelier-processed speech and analyze practical data requirements for stable full-duplex training.

Sommelier is designed to transform raw, in-the-wild conversational audio into high-quality training corpora for full-duplex Speech Language Models (SLMs). Unlike traditional ASR pipelines that prioritize clean, non-overlapping speech, our design philosophy centers on preserving the chaotic yet rich dynamics of human dialogue—such as overlaps and backchanneling—while ensuring scalability for web-scale processing. The overall architecture is built as a modular framework where each component can be toggled or reconfigured, allowing researchers to adapt the trade-off between data purity and conversational authenticity.

Since collected radio and podcast data vary in format and volume, we convert all audio to a standard format (16kHz, 16-bit, Mono) and perform loudness normalization to −20dBFS using pydub and librosa.

To prevent out-of-memory issues, we split long audio files into units of less than 5 minutes using a VAD model to cut at silence intervals. For speaker diarization, instead of the commonly used Pyannote speaker-diarization-3.1, we adopt Sortformer from NVIDIA, which demonstrates superior robustness in capturing very short utterances such as backchanneling.

Conversational audio features frequent turn changes and short utterances. We categorize overlapping scenarios into four distinct cases and select Case 4 as our baseline, which allows both segments to contain the overlap based on speaker identity, preserving full information. We incorporate a module that performs two-speaker separation (SepReformer) on the overlapped intervals with speaker identity matching via cosine similarity of speaker embeddings.

Audio from radio broadcasts or dramas often contains background music (BGM). We employ PANNs to estimate the probability of background music presence in each segment. If the probability exceeds 0.3, we apply the Demucs model to extract the vocal track. Since music removal can degrade speech quality, we selectively apply it only to segments identified by PANNs, minimizing unnecessary processing.

Relying on a single ASR model poses significant risks due to hallucinations, particularly in silent or noisy segments. We employ a ROVER (Recognizer Output Voting Error Reduction) ensemble strategy combining outputs from three distinct SOTA models (Whisper, Canary, Parakeet). We align transcripts at the word level and apply a prioritized majority voting scheme: a word is accepted if predicted by at least two models. Residual hallucinations are further pruned using a RepetitionFilter that discards samples with excessive n-gram repetitions.

We validated the pipeline by performing LoRA fine-tuning on moshiko-pytorch-bf16 using 83 hours of Sommelier-processed data and evaluated using the Full-Duplex-Bench 1.0. The fine-tuned model shows significant improvements across Backchanneling, Smooth Turn-Taking, and User Interruption handling.

We compare Sortformer against the Pyannote 3.1 baseline on VoxConverse. Sortformer consistently outperforms across all metrics, with the most pronounced gains in short utterances and rapid turn-taking—regimes critical for full-duplex conversational modeling.

We evaluate overlap separation quality across varying Signal-to-Interference Ratios (SIR) and overlap ratios using 900 synthesized samples from LibriSpeech. Our separation method (Sep) consistently outperforms the baseline across all conditions, with perceptual quality (UTMOS) closely approximating the Oracle upper bound.

Our three-model ensemble achieves a significant WER improvement of approximately 37% on LibriSpeech Test Other, reducing the error rate from 6.26% to 3.92% compared to standalone Whisper Large v3. The ensemble also successfully corrects Whisper hallucinations such as repetitive generation.

Running a single process on an A100 (80GB) yields a total Real-Time Factor (RTF) of 0.1746. With peak memory usage of 23GB, three concurrent processes per GPU effectively lower the RTF to 0.0443. Processing 10,000 hours of audio using eight A100 GPUs takes approximately 55 hours.