AudioMultiChallenge

Overview

Audio MultiChallenge is a multi-turn benchmark that evaluates conversational intelligence in spoken dialogue systems and audio-language models, including both speech-to-speech and audio-input models. Unlike ASR- or TTS-based evaluations, it measures whether models can follow instructions, integrate prior context, remain self-consistent, and handle natural speech corrections across extended dialogues.

The dataset is built entirely from human speech containing disfluencies, interruptions, non-monotonic phrasing, and ambient noise, conditions under which current models frequently fail. Each conversation concludes with a set of atomic rubrics that precisely define the requirements for the model’s final response.

Using a fixed-context evaluation protocol, models are judged solely on their final turn, enabling rigorous measurement of long-horizon reasoning, speech-context integration, and robustness to real acoustic variability. Audio MultiChallenge highlights where speech-to-speech systems still lag behind their text-based counterparts. Read the paper here.

Data Summary

Dataset Composition

• 452 conversations

• 1,712 rubrics

• 47 unique speakers

• 14.99 hours total user audio duration

Audio Characteristics & Conversation Structure

• Realism: natural, conversational human speech

• Sampling Rate: 48 kHz high-fidelity audio capturing breath, tone, and environmental cues

• Turns per conversation: Min: 3, Median: 5, Max: 8

• Each conversation ends on a user turn with rubrics evaluating only the final assistant response.

Axis Distribution

Each conversation is assigned a single primary axis; Audio-Cue-Gated is a subset of Inference Memory.

• Inference Memory: 132 conversations, including 42 Audio-Cue-Gated tasks

• Instruction Retention: 120

• Self-Coherence: 83

• Voice Editing: 117

Dataset Design

Audio MultiChallenge is constructed through a two-stage process designed to surface realistic, reproducible failure modes in audio-language models. The pipeline combines adversarial synthetic generation with human-recorded natural conversation to ensure both coverage and authenticity. Human conversations are constrained to 3–8 turns, ensuring failures arise from reasoning and state tracking rather than unbounded context length.

Adversarial Blueprinting

We begin with a multi-agent system that identifies potential weaknesses in target models. A Planner Agent proposes strategies to break a model along a specific axis (e.g., Inference Memory), while a Tester Agent generates multi-turn probes exploring that strategy. These are converted to speech using TTS and evaluated by an Audio LLM. When a failure occurs, the Planner distills the tactic into a concise, human-friendly blueprint (a strategic guide, not a script) ensuring creativity and variation in the final dataset.

Human-in-the-Loop Collection

Human contributors use these blueprints to record spontaneous speech conversations with an Audio LLM. These recordings capture:

• Natural disfluencies and hesitation

• Non-monotonic speech: mid-sentence corrections, restarts, and barge-ins

• Acoustic realism: accent variation, room noise, breath and timing artifacts

Contributors continue the dialogue until they detect a concrete failure, surfacing errors that synthetic data alone would not expose.

Rubric Writing

After each conversation, contributors create axis-specific, atomic rubrics defining exactly how the final assistant turn should be evaluated. These rubrics:

• Apply only to the last model response

• Are binary and self-contained

• Capture the precise requirement of the failure mode

• Define strict, axis-specific criteria for success (not optional preferences)

The dataset includes 1,712 rubrics across 452 conversations.

Methodology & Core Metrics

LLM-as-a-Judge

Evaluation is performed using o4-mini, selected for strong performance in rubric-based grading. Across 1,712 rubric decisions, the judge achieved Cohen’s κ ≈ 0.87 and Macro F1 ≈ 0.94 indicating high agreement with human annotators. No evaluated model is used as a judge, avoiding self-preference bias.

Core Metrics

• Average Pass Rate (APR): Primary metric. A task passes only if all rubrics are satisfied.

• Average Rubric Score (ARS): Diagnostic metric. Measures partial correctness and failure structure.

Output Modalities

To measure the modality gap, models are evaluated in two modes:

• Text Output: Native text generation.

• Audio Output: Native speech generation. Most models output both audio and text streams in this setup. The text stream produced by the model is evaluated using the same rubrics.

This isolates the performance cost of producing speech rather than text for architectures that support both modes

Key Insights

The Hardest Skills for Today’s Models

• Voice Editing avg APR: 17.99%

• Inference Memory avg APR: 21.55%

Models struggled to apply mid-sentence and prior-turn corrections for Voice Editing and to retrieve details present only in the acoustic signal (e.g. background noise or speaker paralinguistics) rather than the text transcript for our Audio-Cue gated Inference Memory tasks.

Text > Audio for Output Accuracy

Models show a “modality gap”, as they are consistently stronger when generating text rather than audio. For GPT-4o Audio Preview, performance dropped from 25.44% → 23.23% when switching to native speech generation, reflecting the need for stronger S2S post-training not only for speech quality, but for multi-turn coherence and state tracking.

Real Speech ≠ Synthetic Speech

Native S2S models struggle with unscripted human audio. Most models show a relative improvement in scores when evaluated on synthetic TTS instead of real speech, confirming that clean training audio masks real-world failure modes.

Duration Drives Failure

Model performance remains stable across 3-8 conversational turns, indicating that turn count is not the limiting factor. In contrast, performance degrades steadily as the total duration of user audio increases. For the longest conversations (several minutes of cumulative audio), scores drop sharply, with the longest conversations (exceeding 8 minutes of user audio) dropping to roughly 13% on average. These longest-duration results are based on a small number of samples but indicate a clear challenge in long-form audio processing.