Matthew Siegel

<p dir="ltr">Scale has spent years helping build the data infrastructure behind today's language models. Recently, we've been working with teams tackling a similarly difficult problem: teaching AI to understand and generate human speech. While text models could learn from the entire internet, there's no equivalent training corpus for voice&mdash;that data has to be purpose-built. Here's why we think this challenge is worth solving.&nbsp;</p>
<p dir="ltr">Large Language Models have demonstrated that <a href="https://scale.com/blog/text-universal-interface">text is the universal interface</a>, in large part because it is the native language of computing. Code, web Browse, and even chess can be reduced to sequences of text. Since the advent of the Transformer, these models have moved beyond the Turing Test to excel at evaluations once considered the frontier of human ability, such as <a href="https://scale.com/leaderboard/humanitys_last_exam">Humanity&rsquo;s Last Exam</a>.</p>
<p dir="ltr">Yet the physical world we inhabit is just beginning to be transformed by AI. This new era will be guided by agents that interact with their surroundings as we do: through voice and vision. This requires models that can discern the rich, nuanced data layered within human speech, far beyond the words themselves. The problem is that this data doesn't exist in a prepackaged library. While LLMs could be trained on the enormous corpus of the public internet, no such equivalent exists for the millions of hours of labeled, diverse, emotional speech needed to teach an AI to truly listen.</p>
<h2 dir="ltr">Humanity&rsquo;s First Protocol</h2>
<p>The human voice is our original interface. Before the GUI or the printing press, civilization was built on the oral tradition. Central to our prophecies of the future, we have long projected our machines will talk naturally. As Anna Pi&ntilde;ol and Valerie Osband Mahoney of NFX <a href="https://www.nfx.com/post/voice-ai-is-working">write</a>, &ldquo;Every time we imagine a computer in science fiction it has a voice. Jarvis. C-3P0. Samantha from Her. Humanoid or not, our idea of a futuristic machine is one that speaks to us.&rdquo; As an information-dense medium, a single spoken phrase can contain layers of meaning absent from its written counterpart. Sarcasm, misdirection, and humor can all be encoded into the same sequence of words through paralinguistic elements like tone, prosody, and inflection.</p>
<h2 dir="ltr">Mapping the Frontier of Voice</h2>
<p>While the vision for voice AI is clear, the technology is still in its early stages. The primary technical hurdle is to achieve natural, real-time conversation that feels as natural as authentic human speech. The industry has converged on a key threshold for this:<a href="https://research.google/blog/google-duplex-an-ai-system-for-accomplishing-real-world-tasks-over-the-phone/"> sub-300ms latency</a>. Reaching this speed is prompting a shift toward new, natively Speech-to-Speech (S2S) models.</p>
<ul>
<li dir="ltr"><a href="https://qwenlm.github.io/blog/qwen2.5-omni/">Qwen Omni</a> is hacking around S2S nativity with a Speech-to Text-to-Speech model using &ldquo;token streaming&rdquo;, vocalizing every token as it predicted. However, this approach sacrifices some naturalness for speed.</li>
<li dir="ltr">Hume has pioneered Semantic Space Theory, categorizing the layered&nbsp;<a href="https://journals.sagepub.com/doi/abs/10.1177/09637214221150511">emotional tones</a> of utterances. Their Empathic Voice Interface models can detect and express 53 dimensions of emotion with prosody.</li>
<li dir="ltr"><a href="https://x.com/OpenAIDevs/status/1938716933328904240">OpenAI&rsquo;s new Speech-to-Speech model</a> is noteworthy for Function Calling and Internet search informing responses. They co-mingle a "real time conversational model" and "a slower smarter model" for complex tasks.</li>
<li dir="ltr">Eleven Labs is powering real-time, multilingual voice assistants. Most notable for a stocked developer platform for steerable TTS, S2S, Cloning, and Voice Design.</li>
<li dir="ltr">NVIDIA has built a flexible sound machine,&nbsp;<a href="https://blogs.nvidia.com/blog/fugatto-gen-ai-sound-model/">Fugatto</a>, that can seamlessly mash up speech, music, and soundscapes. It lets users shape how voices and tunes morph in real time while inventing entirely new sounds.</li>
<li dir="ltr">Sesame&rsquo;s groundbreaking work employs a multimodal backbone with a smaller audio decoder, simultaneously leveraging semantic and acoustic data with<a href="https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice"> residual vector quantization tokens</a>.</li>
</ul>
<h2 dir="ltr">Starting with the Future</h2>
<p dir="ltr">To understand the data researchers need, we start by asking what products are being built for end users and work backwards. The ambition for voice AI is reshaping both our personal and professional lives.</p>
<h3 dir="ltr">Consumer Applications</h3>
<p dir="ltr">The holy grail is a persistent voice companion with memory and personality, acting as a true digital counterpart. This vision extends to more specific applications:</p>
<ul>
<li dir="ltr">Dynamic Tour Guides that can whisper historical secrets in a quiet museum, not just recite facts.</li>
<li dir="ltr">Truly Interactive Podcasts where you can ask the host to elaborate on a point or skip a segment with a simple command.</li>
<li dir="ltr">Augmented Reality that comes alive through voice, from a simple heads-up display to in-the-wild interactions with agents like Meta's Ray-Bans or Cluely&rsquo;s AI assistant.</li>
<li dir="ltr">Immersive VR and Gaming Worlds where non-player characters have unique personalities and can hold unscripted, natural conversations.</li>
</ul>
<h3 dir="ltr">Enterprise &amp; Business Applications</h3>
<p dir="ltr">In business, reliability, steerability, and observability, are critical. To trust an agent in production requires robust knowledge retrieval and an understanding of specific professional domains.</p>
<ul>
<li dir="ltr">Vertical-Specific Agents that understand the nuances, for example, of case law, medicine, or specific sales needs are necessary. The possibilities are endless from rote tasks like patient intake to inspired tools like context-aware meeting assistants.</li>
</ul>
<p dir="ltr">Many of these future applications are raring to advance far faster, but are hamstrung by the same fundamental challenge: the data bottleneck.</p>
<h2 dir="ltr">The Data Bottleneck</h2>
<p dir="ltr">&nbsp;The breakneck improvement of LLMs was facilitated by a simple foundation: the entire public internet served as a training library for text models. As Dr. Fei-Fei Li has<a href="https://x.com/zeynebnkaya/status/1934761262287872220"> noted</a>, this vast corpus of data fueled rapid progress. But for voice, no such public repository has ever existed. So we built it. Scale&rsquo;s dynamic, high-quality audio data platform is supercharged with:</p>
<ul>
<li dir="ltr">Language, Dialect, and Accent Coverage</li>
<li dir="ltr">Highest &ldquo;Studio&rdquo; Recording Quality</li>
<li dir="ltr">Versatility for Topics, Contexts, and Scripts</li>
<li dir="ltr">Complete Metadata</li>
<li dir="ltr">Trained Voice Actors</li>
<li dir="ltr">Diverse Evaluators and Testers</li>
</ul>
<h2 dir="ltr">A Data Engine for the Full Model Lifecycle</h2>
<p dir="ltr">Building a state-of-the-art voice model requires a structured approach to data, moving from a broad foundation to specialized, high-quality refinement. We provide the data for each phase of this process.</p>
<h3 dir="ltr">Phase 1: Pre-training</h3>
<p dir="ltr">Sutton's 'Bitter Lesson' teaches that massive compute applied to general methods is the winning strategy. Accordingly, the foundation of any great voice model is a massive and diverse dataset. We begin with vast amounts of unscripted conversational audio designed to teach the model the fundamental patterns of real-world human speech. This data captures:</p>
<ul>
<li dir="ltr">Real-world Dynamics: Conversations with 2, 3, 4, or even 5 speakers, complete with the natural interruptions, laughter, and disfluencies that define authentic speech.</li>
<li dir="ltr">Global Representation: Deep coverage across a range of languages, dialects, and accents to ensure the model works for everyone.</li>
</ul>
<h3 dir="ltr">Phase 2: Supervised Finetuning (SFT)</h3>
<p dir="ltr">While pre-training builds the foundation, SFT data defines the upper limit of a model's performance. We produce this data at pristine studio quality, with every hour engineered by seasoned audio professionals through multiple stages of proprietary post-processing.</p>
<blockquote>
<p dir="ltr"><em>&ldquo;Every single contributor is auditioned and then checked week after week to ensure consistent, high-quality audio standards are maintained. Utilizing proprietary workflows... we keep our audio quality at the same high level for all contributors worldwide, regardless of their equipment or recording environment.&rdquo;</em></p>
<p dir="ltr">-Jacob C. Audio Engineer</p>
</blockquote>
<p dir="ltr">Our SFT data is designed to teach specific, high-value skills:</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Topical Expertise: Data from expert conversations and scripted business meetings to teach the model deep knowledge about specific domains like law, medicine, or finance.&nbsp;</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Linguistic Precision: A diverse range of accents, dialects, and even tongue twisters to ensure the model's pronunciation is clear and consistent across any challenge (see Figure 2).</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Technical Transcripts: For training strategies that require them, we provide high-fidelity, word-level transcripts and detailed phonetic data to ensure optimal performance and accuracy.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Emotions and Paralinguistics: Scenarios designed to teach the model how to understand and express nuanced emotions for applications like de-escalating a support call, making a persuasive sale, or roleplaying. This requires capturing the dynamics of real-world, multi-turn conversations, complete with interruptions and overlapping speech (see Figure 1).</p>
</li>
</ul>
<blockquote>
<p dir="ltr"><em>&ldquo;When you train an AI for voice work, you&rsquo;re not just performing&mdash;you&rsquo;re teaching. You&rsquo;re helping the model learn how to interpret emotion, tone, timing, and personality, making it more relatable and emotionally intelligent.&rdquo;</em></p>
<p dir="ltr">- Michelle E., Voice Actor</p>
</blockquote>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=a257c020aab27d37014188f68a364559.png&amp;f=webp&amp;q=75" alt="" width="1198" height="926"></p>
<p dir="ltr"><sup>Figure 1: An example of a full duplex, multi-turn conversation. This level of detail, capturing interruptions and precise timestamps, is crucial for training models on the complex dynamics of real-world human speech.</sup></p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=a3463525ae794b2fc1c2f485acb967b3.png&amp;f=webp&amp;q=75" alt="" width="1198" height="592"></p>
<p dir="ltr"><sup>Figure 2: Detailed phonetic transcription used to fine-tune a model&rsquo;s pronunciation. This ensures the AI can achieve clear, consistent, and human-like pronunciations for any word.</sup></p>
<h3 dir="ltr">Phase 3: Refinement and Alignment with Human Feedback</h3>
<p dir="ltr">Once a foundational model is trained, it must be rigorously tested, refined, and aligned with human preferences to be useful and safe. This process involves a deep partnership between AI and human evaluators across several key areas.</p>
<p dir="ltr">Human Preference Scoring (RLHF)</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">For subjective qualities like &ldquo;naturalness&rdquo; or &ldquo;helpfulness,&rdquo; human judgment is the only ground truth. Our process involves direct conversational evaluation, where human testers rank model responses to create the preference data that powers Reinforcement Learning from Human Feedback (RLHF). This feedback is captured using standard statistical metrics like Mean Opinion Score (MOS) and Likert scales.</p>
</li>
</ul>
<p dir="ltr">Real-World Robustness Testing</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">A model that performs well in a lab can fail in a noisy cafe. We ensure real-world robustness by collecting data &ldquo;in-the-wild&rdquo; across a huge diversity of verified environments (meetings, public transit, etc.) and devices (phones, earphones, wearables). This is made possible by our large, distributed contributor base, which provides the scale necessary for statistically significant evaluation.</p>
</li>
</ul>
<p dir="ltr">Advanced Reward Modeling</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">While simple preference data is powerful, complex tasks require more sophisticated reward signals. While a simple <a href="https://arxiv.org/abs/2111.06353">reward signal can be enough</a> to create surprisingly intelligent agents, the core challenge in a subjective domain like voice is constructing that signal. Building on the success of foundational algorithms like Proximal Policy Optimization (PPO), newer methods like <a href="https://arxiv.org/abs/2402.03300">Group Relative Policy Optimization</a> (GRPO) show a promising path forward. However, their efficacy hinges on the quality of the reward model itself. The key is the craftsmanship in constructing complex, multi-dimensional rubrics that can effectively guide the model, especially for business cases where we can tie rewards to verified outcomes like a resolved support ticket or a completed sale.</p>
</li>
</ul>
<p dir="ltr">Adversarial Testing &amp; Red Teaming</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The safety and alignment requirements for voice AI are far more stringent than for text. We have nove <a href="https://scale.com/blog/rethink-red-teaming">frameworks</a> to protect against adversarial attacks like voice spoofing, phishing, and confidence scams. Our red teams vet models to ensure they don&rsquo;t leak sensitive information or provide harmful advice, ensuring safety at the model, product, and system levels.</p>
</li>
</ul>
<h3 dir="ltr">Phase 4: Extending to Advanced Capabilities</h3>
<p dir="ltr">Agentic Tool Use</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The ultimate goal of a voice agent is to move from conversation to action. We train models to reliably use tools, call APIs, and execute tasks based on voice commands. This is achieved through sophisticated techniques like Process Supervision, where the model is rewarded for following the correct steps in a complex task (not just the final outcome), and Verified Rewards, which ties model success to concrete events like a successfully booked flight or a completed order.</p>
</li>
</ul>
<p dir="ltr">Multimodal Perception (Voice + Vision)</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">As companies like Meta, Snap, and Sesame develop AI-powered glasses, voice is emerging as the primary user interface for interacting with the physical world. True understanding in this context requires multimodality. We merge our industry-leading video and image annotation capabilities with our audio data engine to train models that can see what you see and hear what you hear, enabling truly contextual, real-world assistants.</p>
</li>
</ul>
<p dir="ltr">Expressive &amp; Creative Generation</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The same underlying technology that masters conversational speech can be extended to all forms of generative audio. By training models on tagged music samples, sound effects, and other rich audio, we enable agents to express themselves in a fuller medium, whether by generating<a href="https://suno.com/home"> music</a>, a sound effect, or even singing.</p>
</li>
</ul>
<h2 dir="ltr">The Future is Listening</h2>
<p dir="ltr">The journey from text-based models to truly interactive, voice-driven AI marks a fundamental shift in how we will interact with technology. While the challenges of data complexity are immense, the path forward is no longer theoretical. The benchmarks are being set, the technical frontiers are being explored, and the applications that will define the next decades are being designed today. The future of human-computer interaction will be a conversation.</p>
<h2 dir="ltr">Acknowledgements</h2>
<p dir="ltr">Core Contributors: Gokul Prabhakaran, Rodrigo Belfiore, Matias Devecchi Di Bella, Gorka Ludlow, Michael Chen, Brandon Kyung, George Varelas, Gaurang Goel, Niveditha Patil, Olivia Fu, Ali Khan, Daniel Quigley, Taku Yamada, Alejo Jimenez Chaidez, Utkarsh Tyagi, Advait Gosai &amp; Kilian Butler.</p>
<p>&nbsp;</p>

From text to voice, Scale builds the data powering the next generation of natural, multimodal AI agents.

Matthew Siegel

AI Doesn’t Live in Text Alone