Matthew Siegel

<p dir="ltr">Scale has spent years helping build the data infrastructure behind today's language models. Recently, we've been working with teams tackling a similarly difficult problem: teaching AI to understand and generate human speech. While text models could learn from the entire internet, there's no equivalent training corpus for voice&mdash;that data has to be purpose-built. Here's why we think this challenge is worth solving.&nbsp;</p>
<p dir="ltr">Large Language Models have demonstrated that <a href="https://scale.com/blog/text-universal-interface">text is the universal interface</a>, in large part because it is the native language of computing. Code, web Browse, and even chess can be reduced to sequences of text. Since the advent of the Transformer, these models have moved beyond the Turing Test to excel at evaluations once considered the frontier of human ability, such as <a href="https://scale.com/leaderboard/humanitys_last_exam">Humanity&rsquo;s Last Exam</a>.</p>
<p dir="ltr">Yet the physical world we inhabit is just beginning to be transformed by AI. This new era will be guided by agents that interact with their surroundings as we do: through voice and vision. This requires models that can discern the rich, nuanced data layered within human speech, far beyond the words themselves. The problem is that this data doesn't exist in a prepackaged library. While LLMs could be trained on the enormous corpus of the public internet, no such equivalent exists for the millions of hours of labeled, diverse, emotional speech needed to teach an AI to truly listen.</p>
<h2 dir="ltr">Humanity&rsquo;s First Protocol</h2>
<p>The human voice is our original interface. Before the GUI or the printing press, civilization was built on the oral tradition. Central to our prophecies of the future, we have long projected our machines will talk naturally. As Anna Pi&ntilde;ol and Valerie Osband Mahoney of NFX <a href="https://www.nfx.com/post/voice-ai-is-working">write</a>, &ldquo;Every time we imagine a computer in science fiction it has a voice. Jarvis. C-3P0. Samantha from Her. Humanoid or not, our idea of a futuristic machine is one that speaks to us.&rdquo; As an information-dense medium, a single spoken phrase can contain layers of meaning absent from its written counterpart. Sarcasm, misdirection, and humor can all be encoded into the same sequence of words through paralinguistic elements like tone, prosody, and inflection.</p>
<h2 dir="ltr">Mapping the Frontier of Voice</h2>
<p>While the vision for voice AI is clear, the technology is still in its early stages. The primary technical hurdle is to achieve natural, real-time conversation that feels as natural as authentic human speech. The industry has converged on a key threshold for this:<a href="https://research.google/blog/google-duplex-an-ai-system-for-accomplishing-real-world-tasks-over-the-phone/"> sub-300ms latency</a>. Reaching this speed is prompting a shift toward new, natively Speech-to-Speech (S2S) models.</p>
<ul>
<li dir="ltr"><a href="https://qwenlm.github.io/blog/qwen2.5-omni/">Qwen Omni</a> is hacking around S2S nativity with a Speech-to Text-to-Speech model using &ldquo;token streaming&rdquo;, vocalizing every token as it predicted. However, this approach sacrifices some naturalness for speed.</li>
<li dir="ltr">Hume has pioneered Semantic Space Theory, categorizing the layered&nbsp;<a href="https://journals.sagepub.com/doi/abs/10.1177/09637214221150511">emotional tones</a> of utterances. Their Empathic Voice Interface models can detect and express 53 dimensions of emotion with prosody.</li>
<li dir="ltr"><a href="https://x.com/OpenAIDevs/status/1938716933328904240">OpenAI&rsquo;s new Speech-to-Speech model</a> is noteworthy for Function Calling and Internet search informing responses. They co-mingle a "real time conversational model" and "a slower smarter model" for complex tasks.</li>
<li dir="ltr">Eleven Labs is powering real-time, multilingual voice assistants. Most notable for a stocked developer platform for steerable TTS, S2S, Cloning, and Voice Design.</li>
<li dir="ltr">NVIDIA has built a flexible sound machine,&nbsp;<a href="https://blogs.nvidia.com/blog/fugatto-gen-ai-sound-model/">Fugatto</a>, that can seamlessly mash up speech, music, and soundscapes. It lets users shape how voices and tunes morph in real time while inventing entirely new sounds.</li>
<li dir="ltr">Sesame&rsquo;s groundbreaking work employs a multimodal backbone with a smaller audio decoder, simultaneously leveraging semantic and acoustic data with<a href="https://www.sesame.com/research/crossing_the_uncanny_valley_of_voice"> residual vector quantization tokens</a>.</li>
</ul>
<h2 dir="ltr">Starting with the Future</h2>
<p dir="ltr">To understand the data researchers need, we start by asking what products are being built for end users and work backwards. The ambition for voice AI is reshaping both our personal and professional lives.</p>
<h3 dir="ltr">Consumer Applications</h3>
<p dir="ltr">The holy grail is a persistent voice companion with memory and personality, acting as a true digital counterpart. This vision extends to more specific applications:</p>
<ul>
<li dir="ltr">Dynamic Tour Guides that can whisper historical secrets in a quiet museum, not just recite facts.</li>
<li dir="ltr">Truly Interactive Podcasts where you can ask the host to elaborate on a point or skip a segment with a simple command.</li>
<li dir="ltr">Augmented Reality that comes alive through voice, from a simple heads-up display to in-the-wild interactions with agents like Meta's Ray-Bans or Cluely&rsquo;s AI assistant.</li>
<li dir="ltr">Immersive VR and Gaming Worlds where non-player characters have unique personalities and can hold unscripted, natural conversations.</li>
</ul>
<h3 dir="ltr">Enterprise &amp; Business Applications</h3>
<p dir="ltr">In business, reliability, steerability, and observability, are critical. To trust an agent in production requires robust knowledge retrieval and an understanding of specific professional domains.</p>
<ul>
<li dir="ltr">Vertical-Specific Agents that understand the nuances, for example, of case law, medicine, or specific sales needs are necessary. The possibilities are endless from rote tasks like patient intake to inspired tools like context-aware meeting assistants.</li>
</ul>
<p dir="ltr">Many of these future applications are raring to advance far faster, but are hamstrung by the same fundamental challenge: the data bottleneck.</p>
<h2 dir="ltr">The Data Bottleneck</h2>
<p dir="ltr">&nbsp;The breakneck improvement of LLMs was facilitated by a simple foundation: the entire public internet served as a training library for text models. As Dr. Fei-Fei Li has<a href="https://x.com/zeynebnkaya/status/1934761262287872220"> noted</a>, this vast corpus of data fueled rapid progress. But for voice, no such public repository has ever existed. So we built it. Scale&rsquo;s dynamic, high-quality audio data platform is supercharged with:</p>
<ul>
<li dir="ltr">Language, Dialect, and Accent Coverage</li>
<li dir="ltr">Highest &ldquo;Studio&rdquo; Recording Quality</li>
<li dir="ltr">Versatility for Topics, Contexts, and Scripts</li>
<li dir="ltr">Complete Metadata</li>
<li dir="ltr">Trained Voice Actors</li>
<li dir="ltr">Diverse Evaluators and Testers</li>
</ul>
<h2 dir="ltr">A Data Engine for the Full Model Lifecycle</h2>
<p dir="ltr">Building a state-of-the-art voice model requires a structured approach to data, moving from a broad foundation to specialized, high-quality refinement. We provide the data for each phase of this process.</p>
<h3 dir="ltr">Phase 1: Pre-training</h3>
<p dir="ltr">Sutton's 'Bitter Lesson' teaches that massive compute applied to general methods is the winning strategy. Accordingly, the foundation of any great voice model is a massive and diverse dataset. We begin with vast amounts of unscripted conversational audio designed to teach the model the fundamental patterns of real-world human speech. This data captures:</p>
<ul>
<li dir="ltr">Real-world Dynamics: Conversations with 2, 3, 4, or even 5 speakers, complete with the natural interruptions, laughter, and disfluencies that define authentic speech.</li>
<li dir="ltr">Global Representation: Deep coverage across a range of languages, dialects, and accents to ensure the model works for everyone.</li>
</ul>
<h3 dir="ltr">Phase 2: Supervised Finetuning (SFT)</h3>
<p dir="ltr">While pre-training builds the foundation, SFT data defines the upper limit of a model's performance. We produce this data at pristine studio quality, with every hour engineered by seasoned audio professionals through multiple stages of proprietary post-processing.</p>
<blockquote>
<p dir="ltr"><em>&ldquo;Every single contributor is auditioned and then checked week after week to ensure consistent, high-quality audio standards are maintained. Utilizing proprietary workflows... we keep our audio quality at the same high level for all contributors worldwide, regardless of their equipment or recording environment.&rdquo;</em></p>
<p dir="ltr">-Jacob C. Audio Engineer</p>
</blockquote>
<p dir="ltr">Our SFT data is designed to teach specific, high-value skills:</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Topical Expertise: Data from expert conversations and scripted business meetings to teach the model deep knowledge about specific domains like law, medicine, or finance.&nbsp;</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Linguistic Precision: A diverse range of accents, dialects, and even tongue twisters to ensure the model's pronunciation is clear and consistent across any challenge (see Figure 2).</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Technical Transcripts: For training strategies that require them, we provide high-fidelity, word-level transcripts and detailed phonetic data to ensure optimal performance and accuracy.</p>
</li>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">Emotions and Paralinguistics: Scenarios designed to teach the model how to understand and express nuanced emotions for applications like de-escalating a support call, making a persuasive sale, or roleplaying. This requires capturing the dynamics of real-world, multi-turn conversations, complete with interruptions and overlapping speech (see Figure 1).</p>
</li>
</ul>
<blockquote>
<p dir="ltr"><em>&ldquo;When you train an AI for voice work, you&rsquo;re not just performing&mdash;you&rsquo;re teaching. You&rsquo;re helping the model learn how to interpret emotion, tone, timing, and personality, making it more relatable and emotionally intelligent.&rdquo;</em></p>
<p dir="ltr">- Michelle E., Voice Actor</p>
</blockquote>
<p dir="ltr"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=a257c020aab27d37014188f68a364559.png&amp;f=webp&amp;q=75" alt="" width="1198" height="926"></p>
<p dir="ltr"><sup>Figure 1: An example of a full duplex, multi-turn conversation. This level of detail, capturing interruptions and precise timestamps, is crucial for training models on the complex dynamics of real-world human speech.</sup></p>
<p><img src="https://img.plasmic.app/img-optimizer/v1/img?src=a3463525ae794b2fc1c2f485acb967b3.png&amp;f=webp&amp;q=75" alt="" width="1198" height="592"></p>
<p dir="ltr"><sup>Figure 2: Detailed phonetic transcription used to fine-tune a model&rsquo;s pronunciation. This ensures the AI can achieve clear, consistent, and human-like pronunciations for any word.</sup></p>
<h3 dir="ltr">Phase 3: Refinement and Alignment with Human Feedback</h3>
<p dir="ltr">Once a foundational model is trained, it must be rigorously tested, refined, and aligned with human preferences to be useful and safe. This process involves a deep partnership between AI and human evaluators across several key areas.</p>
<p dir="ltr">Human Preference Scoring (RLHF)</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">For subjective qualities like &ldquo;naturalness&rdquo; or &ldquo;helpfulness,&rdquo; human judgment is the only ground truth. Our process involves direct conversational evaluation, where human testers rank model responses to create the preference data that powers Reinforcement Learning from Human Feedback (RLHF). This feedback is captured using standard statistical metrics like Mean Opinion Score (MOS) and Likert scales.</p>
</li>
</ul>
<p dir="ltr">Real-World Robustness Testing</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">A model that performs well in a lab can fail in a noisy cafe. We ensure real-world robustness by collecting data &ldquo;in-the-wild&rdquo; across a huge diversity of verified environments (meetings, public transit, etc.) and devices (phones, earphones, wearables). This is made possible by our large, distributed contributor base, which provides the scale necessary for statistically significant evaluation.</p>
</li>
</ul>
<p dir="ltr">Advanced Reward Modeling</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">While simple preference data is powerful, complex tasks require more sophisticated reward signals. While a simple <a href="https://arxiv.org/abs/2111.06353">reward signal can be enough</a> to create surprisingly intelligent agents, the core challenge in a subjective domain like voice is constructing that signal. Building on the success of foundational algorithms like Proximal Policy Optimization (PPO), newer methods like <a href="https://arxiv.org/abs/2402.03300">Group Relative Policy Optimization</a> (GRPO) show a promising path forward. However, their efficacy hinges on the quality of the reward model itself. The key is the craftsmanship in constructing complex, multi-dimensional rubrics that can effectively guide the model, especially for business cases where we can tie rewards to verified outcomes like a resolved support ticket or a completed sale.</p>
</li>
</ul>
<p dir="ltr">Adversarial Testing &amp; Red Teaming</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The safety and alignment requirements for voice AI are far more stringent than for text. We have nove <a href="https://scale.com/blog/rethink-red-teaming">frameworks</a> to protect against adversarial attacks like voice spoofing, phishing, and confidence scams. Our red teams vet models to ensure they don&rsquo;t leak sensitive information or provide harmful advice, ensuring safety at the model, product, and system levels.</p>
</li>
</ul>
<h3 dir="ltr">Phase 4: Extending to Advanced Capabilities</h3>
<p dir="ltr">Agentic Tool Use</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The ultimate goal of a voice agent is to move from conversation to action. We train models to reliably use tools, call APIs, and execute tasks based on voice commands. This is achieved through sophisticated techniques like Process Supervision, where the model is rewarded for following the correct steps in a complex task (not just the final outcome), and Verified Rewards, which ties model success to concrete events like a successfully booked flight or a completed order.</p>
</li>
</ul>
<p dir="ltr">Multimodal Perception (Voice + Vision)</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">As companies like Meta, Snap, and Sesame develop AI-powered glasses, voice is emerging as the primary user interface for interacting with the physical world. True understanding in this context requires multimodality. We merge our industry-leading video and image annotation capabilities with our audio data engine to train models that can see what you see and hear what you hear, enabling truly contextual, real-world assistants.</p>
</li>
</ul>
<p dir="ltr">Expressive &amp; Creative Generation</p>
<ul>
<li dir="ltr" aria-level="1">
<p dir="ltr" role="presentation">The same underlying technology that masters conversational speech can be extended to all forms of generative audio. By training models on tagged music samples, sound effects, and other rich audio, we enable agents to express themselves in a fuller medium, whether by generating<a href="https://suno.com/home"> music</a>, a sound effect, or even singing.</p>
</li>
</ul>
<h2 dir="ltr">The Future is Listening</h2>
<p dir="ltr">The journey from text-based models to truly interactive, voice-driven AI marks a fundamental shift in how we will interact with technology. While the challenges of data complexity are immense, the path forward is no longer theoretical. The benchmarks are being set, the technical frontiers are being explored, and the applications that will define the next decades are being designed today. The future of human-computer interaction will be a conversation.</p>
<h2 dir="ltr">Acknowledgements</h2>
<p dir="ltr">Core Contributors: Gokul Prabhakaran, Rodrigo Belfiore, Matias Devecchi Di Bella, Gorka Ludlow, Michael Chen, Brandon Kyung, George Varelas, Gaurang Goel, Niveditha Patil, Olivia Fu, Ali Khan, Daniel Quigley, Taku Yamada, Alejo Jimenez Chaidez, Utkarsh Tyagi, Advait Gosai &amp; Kilian Butler.</p>
<p>&nbsp;</p>

From text to voice, Scale builds the data powering the next generation of natural, multimodal AI agents.

AI Doesn’t Live in Text Alone

<p dir="ltr">It&rsquo;s not everyday an academic study on machine learning garners millions of views on X in just a few days. &ldquo;<a href="https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf">Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity</a>&rdquo; from Model Evaluation &amp; Threat Research (METR) challenges a widely accepted truth that AI makes developers more productive.&nbsp;</p>
<p dir="ltr">In the paper, the authors reveal a finding shocking to many: when experienced developers were given access to state-of-the-art AI tools from early 2025, like Claude 3.7 Sonnet and Cursor Pro, they were 19% slower than their counterparts. Even more surprisingly, those same developers felt as though they had been sped up by 20%. This disconnect wasn't limited to the participants; outside economics and machine learning experts also dramatically overestimated the tools' impact, predicting an average speedup of nearly 40%.</p>
<p dir="ltr">Is it possible that the work just felt faster because it was more enjoyable? Or that the developers were simply taking time to test the limits of new tools?&nbsp;The fuller story exists between the clock and the human experience. The paper offers us an opportunity to examine where we&rsquo;re at with AI coding tools, the kinds of problems we use them to solve, how they are used, and who is able to use them.</p>
<h2 dir="ltr">Reasons for the Slowdown</h2>
<p dir="ltr">The study's results suggest the slowdown isn't a single issue, but stems from several factors. Among them, three major themes of friction stand out:&nbsp;</p>
<ol>
<li dir="ltr" aria-level="1">
<h3 dir="ltr" role="presentation">The Problem of Context</h3>
</li>
</ol>
<p class="whitespace-normal break-words">Think of it as an 80/20 problem: AI can handle the initial 80% of coding tasks reasonably well, but struggles with the final 20% where deep context matters most. This was especially apparent with the expert developers in the study, who averaged five years of experience and 1,500 commits on massive, mature codebases.</p>
<p class="whitespace-normal break-words">Developers reported that AI often behaved like an inexperienced team member struggling with backwards compatibility or proposing edits in the wrong locations. The numbers tell the story: they spent 9% of their time reviewing and cleaning AI outputs, plus another 4% waiting for generations. The problem was worst on tasks requiring institutional memory, because that knowledge represents rich context built from years of human experience.</p>
<p class="whitespace-normal break-words">This means the study tested AI in perhaps its most challenging scenario&nbsp;where human expertise and context are at their peak. The findings may not apply as broadly to junior developers or greenfield projects, though this friction will likely decrease as AI gets better at handling larger contexts.</p>
<ol start="2">
<li dir="ltr" aria-level="1">
<h3 dir="ltr" role="presentation">New Tools = Moving Targets</h3>
</li>
</ol>
<p dir="ltr">Second, effectively managing these tools is a new skill that many are still getting good at. For example, it is a learned skill to know precisely when to abandon a failing dialogue and finish the work manually. Mastering this is difficult because the tools themselves are a moving target, constantly evolving. Imagine a carpenter having to learn how to use a new hammer every few weeks.&nbsp;</p>
<ol start="3">
<li dir="ltr" aria-level="1">
<h3 dir="ltr" role="presentation">Structural Bottleneck</h3>
</li>
</ol>
<p dir="ltr">Finally, the tools themselves impose a structural bottleneck. While a human developer mentally juggles multiple dependencies and plans their next steps while working on a piece of code, interacting with most AI assistants forces them into a rigid, serial workflow: one instruction, one response. This doesn't align with the way experts think.&nbsp;</p>
<p dir="ltr">However, this is already beginning to change, with a potential shift towards a more asynchronous, agent-based workflow. This is a different model than the one tested in the paper, and while it's an interesting evolution, it isn't yet clear if it&rsquo;s more effective at reducing development time, a question future studies may need to address. In this emerging future, we might see a hybrid approach where developers choose their tools based on the complexity of the task using a fully autonomous agent for some things and hands-on AI assistance for others, perhaps even using one to fix the other's work.&nbsp;</p>
<h2 dir="ltr">So, Which Developers are Empowered by AI?</h2>
<p dir="ltr">While the study's focus on expert productivity is valuable, looking beyond this specific research suggests it tells only a partial story. For existing developers, AI's greatest impact may not yet be on finishing complex tasks, but on dramatically speeding up the early stages of a project, allowing them to jump from inspiration to a working prototype much faster. But perhaps its most transformative effect is opening the door to coding for an entirely new population. This is happening in two key ways. First, it lowers the "activation energy" for managers and executives strapped for time, who can now build tools in short bursts. Second, it's empowering a wave of vibe coders including product managers, designers, or other kinds of non-engineers to build their own tools and programs with minimal engineering resources. In both cases, we are seeing entirely new groups of people do things they would never have been able to do before.</p>
<h2 dir="ltr">A Clear-Eyed View of the Future</h2>
<p dir="ltr">On one hand, the study is a valuable, data-driven look at the friction AI can create for experts on complex, context-heavy tasks. On the other, the technology is simultaneously a powerful democratizing force, unlocking entirely new value by empowering new populations of coders. This comes with its own forward-looking questions. A heavy reliance on vibe coding may not engage traditional coding muscles, potentially leading to an atrophy of deeper engineering skills over time.&nbsp;</p>
<p dir="ltr">The paper's authors, however, caution against overgeneralizing. The impact could be entirely different, and perhaps positive, for junior developers or those starting on new, greenfield projects.&nbsp;</p>
<p dir="ltr">With that in mind, perhaps the most telling clue to the future is also the most paradoxical finding in the entire study: despite being measurably slower, 69% of the developers continued using the AI tool after the experiment ended. That single data point proves the ultimate thesis. The value of AI cannot be measured by the clock alone; it&rsquo;s a complex equation of speed, context, enjoyment, and empowerment. Understanding that complexity is the key to navigating the future of work.</p>
<p>&nbsp;</p>

A METR study found AI slowed devs by 19%, but they felt 20% faster. We unpack this paradox of productivity, skill, & AI.

Matthew Siegel

AI Doesn’t Live in Text Alone

Do AI Tools Slow Down Developers? The Answer Isn't Simple.