MuDuSi · Multi-agent Duplex Simulator

01 / The testbed

Build the interaction.
Then bring it into the room.

Testing several speech agents normally means arranging speakers, microphones and a physical space. MuDuSi brings those early experiments into a shared virtual room, reducing the hardware setup needed to explore how agents listen, respond and speak over one another.

See multi-agent interaction: Watch independent speech models in 3D, with speaking and receiving activity, overlap, microphone levels and timing on one screen.
Change what agents hear: Move participants, vary microphone noise, add divider attenuation and compare quiet, interruption and cocktail-party scenes.
Inspect trajectories for RL research: Replay acoustic rollouts alongside audio, state, activity timelines and reward diagnostics; retain the evidence needed to develop model-side learning interfaces.
Run independent experiments: Use separate processes, ports, worker allocations and output folders for multiple sessions. Inspect an individual experiment or compare nine isolated paired conversations in one synchronized dashboard.

Simulation supports early testing and controlled comparisons. Physical microphones, loudspeakers and room measurements remain important for deployment validation.

02 / Conversations

Listen to a shared world.

Each agent generates its own speech. The simulator carries that sound through the room into the other agents’ microphones, letting conversations unfold without assigned speaking turns.

Watch the full introduction — friends, clerk and customer, three agents, and nine simultaneous rooms. Video provenance.

Two friends

PersonaPlex + Moshi

Figure 1. Two speech agents share a casual conversation. Their independently generated voices pass through the same acoustic simulation.

At the coffee shop

Two PersonaPlex agents

Figure 2. Mia, the clerk, and Ben, the customer, discuss an iced oat latte and a croissant. The agents receive role and menu prompts; their dialogue is generated, not scripted.

A three-way conversation

Two PersonaPlex agents + Moshi

Figure 3. Three independent agents listen and speak in one room, including overlapping speech. The participants use two model families, with distinct voices for the two PersonaPlex instances.

Nine rooms, one timeline

Moshi · PersonaPlex · Raon

All nine rooms passed routing and buffer-health checks. ASR review found substantial problems in Raon’s output in both Moshi pairings; speech quality remains unresolved. Measured results and limitations.

Figure 4. The nine ordered model pairs use 18 independent real-model workers, two per isolated room. Rows identify the left model and columns the right model. The soundtrack follows one selected room at a time on the common recording timeline; rooms never hear one another. Captions use offline, word-timed ASR. This configuration places the workers on four GPUs (5 / 5 / 4 / 4), collecting audio at 0.25× pacing and replaying it at normal speed. It is not a real-time inference demonstration. Nine-room setup and verification. Recording and video provenance (JSON).

The completed clips contain actual model-generated speech. Video audio mixes the agents’ emissions within the selected room; speech bubbles replay offline ASR at word timestamps. Each agent’s spatially mixed receiver audio is saved separately. The clips illustrate interaction, not a conversation-quality benchmark.

03 / Simulation

One room. Different microphones.

The agents do not exchange text messages. They emit audio into a shared space, and each hears its own local mixture of the others. Moving an agent changes distance, attenuation and arrival time. Dividers, microphone noise and a simple wall reflection change what reaches each listener.

24 kHz audio, advanced in 80 ms frames. The previous receiver block becomes the next model input; acoustic propagation is resolved at the sample level.

The 3D viewer shows speech activity, received levels, overlap and timing. Recorded input and output WAVs, acoustic state and buffer diagnostics make the interaction inspectable after the GPU run ends. Different configurations can be compared through separate recorded sessions.

The current acoustic model is deliberately lightweight: distance loss, divider attenuation, propagation delay, one wall reflection and independent microphone noise. It is not a measured room impulse response or a binaural renderer.

Speaker diarization and audio-scene understanding depend on the connected model and any added task-specific adapters or evaluation code. The simulator supplies acoustic conditions and known source identities; its activity indicators are not inferred speaker labels or evidence of scene understanding.

04 / Two workflows

Observe a conversation.
Keep the interaction.

Inference & evaluation

Connect speech-model workers, set the room and run a bounded conversation. Inspect who spoke, what each agent heard, and whether audio arrived on time. Replay the resulting experiment in the browser without holding a GPU.

Evaluation instructions

Acoustic trajectory collection

Use the same evaluator to retain synchronized source, receiver and model-input audio, per-frame state, acoustic rewards and transport diagnostics. These recordings support analysis and the development of future learning interfaces.

Full speech-model RL is not implemented. Native inference traces do not yet include action tokens, behavior log-probabilities or policy-version barriers. A separate CPU example trains a small speak/listen gate with REINFORCE.

Model-side learning requirements

05 / Get started

Run an experiment.

Install the simulator and connect your model workers using the repository’s setup guide. PersonaPlex and Moshi need separate Python environments because both use the moshi package name.

Record a conversation with running model workers

mudusi evaluate \
  --models configs/local-workers.json \
  --seconds 60 --output runs/conversation

Replay the finished recording on CPU

mudusi serve \
  --replay runs/conversation --port 18766

Open http://localhost:18766/?showcase for the presentation view. Word-timed speech bubbles require the optional offline ASR step. On the shared GPU cluster, launch model inference through the supplied Slurm scripts.

Replay a completed nine-room matrix on CPU

mudusi serve \
  --matrix runs/matrix-four-gpu --port 18767

Open http://localhost:18767/?matrix to compare all nine rooms and select which conversation to hear. See the matrix guide for worker allocation, recording checks, ASR and video export.

Installation & examples Recording & rendering Read the technical report

Technical report

MuDuSi: A Multi-Agent Acoustic Simulator for Full-Duplex Speech Language Models
Kyudan Jung · Jaegul Choo
KAIST AI · Revised October 13, 2026 · Version 1.2

Abstract & citation Full paper (PDF)BibTeX

Help shape the testbed.

Contributions are welcome: speech-model adapters, acoustic scenes, evaluation tasks, trajectory tools and clearer visualizations. Share an experiment, report a limitation or propose an implementation.

Discuss an idea or report an issue Contribution guide

Build the interaction.Then bring it into the room.