Streaming speaker diarization — live demo

0.0 / 30.0 s

Speaker embeddings (2-D PCA)

e-means training

Pipeline slots
–
This update · step
–
Epoch · αt · B
–
k · frames
–
MSE
–
Label agreement (pipeline · reference)
–

Live status

0Speakers found
0Frames shown
0Revised labels

    Latency budget

    Nemotron (chunk end + 7 frames, measured)0.56 s
    ResNet (in parallel, measured)+0 s
    Update period (= Nemotron chunk)–
    Compute per update (mean)–
    Sound → first label shown–
    Label revision allowed–
    Real-time headroom–

    Results (after playback)

    DER · labels as first emitted
    –
    DER · after revision (final)
    –
    Speakers found / reference
    –

    Reference and other models (same clip)

    Same colour = same real speaker · grey = no match in the reference · hatching = 5th speaker onwards

    Numbers

    Paper ↓

    Speaker diarization · streaming · e-means

    eDIAR: Training-Free Open-Speaker Streaming Diarization for Long Meetings and Large Archives via e-means

    Kyudan Jung1 · Jihwan Kim1 · Minwoo Lee1 · Eunji Shin2 · Jaegul Choo1

    1KAIST AI  ·  2Ewha Womans University

    📄 Paper (coming soon) 🤗 Model ▶ Live replay demo
    TL;DR. A simple, training-free addition to an existing slot-based diarizer keeps everything the model already does well, namely its decisions about activity and overlap, its accuracy on meetings that fit into its slots and its latency, and it adds four properties the model lacks: an unlimited number of speakers, robustness on long meetings, speaker linking across recordings and a low, constant compute cost.
    eDIAR at a glance: four panels
    eDIAR at a glance, streaming with a 1.04 s input buffer. (a) DER against the number of speakers, from two to twenty, compared with the two fixed-slot models. (b) DER of six one-hour meetings with 12 to 20 speakers, scored on their first 15, 30 and 45 minutes. (c) Linking the speakers of 1,248 sessions with 1,251 VoxCeleb1 speakers across recordings, at the level of 1.16 million frames. (d) Processing time per update over the same long recordings.

    Four advantages

    No cap on the number of speakers

    2 → 20+
    speakers per conversation, where the fixed-slot models break down (panel a)

    Any slot head becomes open-speaker without retraining. We show it for the 8-slot Nemotron-3 and the 4-slot Streaming Sortformer.

    Robust on long meetings

    1 h
    meetings with 12 to 20 speakers, where Streaming Sortformer accumulates errors and ours stays ahead of the slot head (panel b)

    Bounded-window re-clustering and slot-anchored identities stop early mistakes from spreading through the meeting.

    Linking speakers across recordings

    7×
    faster than Lloyd's k-means at the same loss · 1.16 M frames, 1,251 speakers

    e-means makes frame-level linking of a whole archive affordable, which beats the standard per-speaker centroid linking.

    Low, constant compute

    ≈57 ms
    per update, flat over 3 h of audio · real-time factor ≈ 0.09 on hour-long meetings

    The work per update does not grow with the meeting; re-clustering the whole history instead grows steadily.

    How it works

    1 · Slot head

    Who is active, and when

    Nemotron-3 Diarization (or Streaming Sortformer) runs in its official streaming mode and decides activity and overlap.

    →
    2 · Capacity-aware split

    Cluster inside a slot

    Only after every slot has been used, the speaker embeddings of a slot are split with the official spherical e-means.

    →
    3 · Streaming

    Stable identities

    A bounded 90 s window is re-clustered at every chunk, labels may be revised for 10 s, and identities stay anchored to the slots.

    Results

    Mean DER (%) over all sessions of each corpus; lower is better. Collar 0.25 s, overlapped speech scored.

    SystemVoxConverseAMIICSILibriSpeech-OpenSpk
    streaming, 1.04 s input buffer
    Streaming Sortformer 4spk v2.114.18.327.741.4
    Nemotron-3 Diarization (8 slots)6.49.327.419.8
    eDIAR (ours)5.89.127.416.4
    non-streaming, whole session
    Streaming Sortformer 4spk v2.1, 30.4 s config13.37.229.140.7
    Nemotron-3 Diarization (8 slots, 30.4 s)5.59.027.018.1
    pyannote 3.15.627.433.232.1
    eDIAR (ours)4.58.926.98.6

    VoxConverse 44 sessions (first 480 s), AMI 16 and ICSI 15 test meetings (first 420 s), LibriSpeech-OpenSpk 18 constructed conversations with 2–12 speakers.

    Linking speakers across recordings

    Meeting series, podcast archives and corpora collected for conversational speech models contain the same people many times. Linking the speakers of every session to one identity across the archive is a clustering problem with millions of embeddings and thousands of clusters, the regime e-means was designed for.

    Archive DER and loss against clustering time
    Archive of 1,248 sessions with 1,251 VoxCeleb1 speakers. Archive DER (left) and spherical clustering loss (right) against clustering time, e-means vs. Lloyd's k-means, at frame level (solid) and segment level (dashed). Numbers next to the markers are passes over the data; the dotted line is agglomerative clustering of one centroid per local speaker. CPU, 16 threads.
    Linking 1.16 M frames (k = 1,251)Time (s)LossArchive DER (%)
    Agglomerative clustering of session centroids5.1–25.3
    Mini-batch k-means, 5 epochs27.00.75729.5
    Lloyd's k-means, 25 iterations43.90.72621.0
    e-means, 2 epochs5.90.72520.6
    e-means, 5 epochs13.90.72319.3

    Means over three seeds. e-means runs with its default recipe; batch size and step size come from its scaling laws, without any sweep.

    Use it

    Streaming or non-streaming, one switch. Code and instructions are on Hugging Face.

    Hugging FaceKyudan/eDIAR
    on Hugging Face

    The replay demo above uses synthetic LibriSpeech conversations; the clips were chosen to illustrate the method. Built on Nemotron-3 Diarization, WeSpeaker ResNet34-LM and the official e-means from Faiss. Page design inspired by e-means.github.io.