Can Voice Agents Handle Bilingual Customers? Benchmarking Frontier ASR on Code-Switched Speech

Voice AI systems increasingly power customer service operations worldwide, yet a fundamental challenge remains largely unexamined: their ability to understand customers who naturally switch between languages. A new benchmark study evaluating frontier automatic speech recognition (ASR) systems on code-switched speech—where speakers mix two languages within a single conversation—reveals significant gaps in how these technologies handle real-world multilingual interactions. This research addresses a critical blind spot in AI deployment, particularly for companies serving diverse, bilingual populations across the United States, Europe, Latin America, and Asia.

Code-switching represents authentic linguistic behavior for millions of bilingual speakers globally. When customers mix English with Spanish, Mandarin, Hindi, or other languages, they're not speaking incorrectly; they're communicating naturally. However, most voice agents are trained primarily on monolingual data, creating performance degradation when encountering code-switched speech. This testing framework systematically benchmarks how well the latest ASR models—from major providers pushing frontier capabilities—handle these real-world scenarios.

• Customer Service Equity: Current voice agent limitations may inadvertently create service quality disparities for bilingual customers, raising fairness and accessibility concerns for companies deploying these systems

• Market Opportunity: Demonstrating ASR performance gaps on code-switched speech highlights urgent development priorities for AI companies targeting global markets

• Enterprise Risk: Businesses using voice agents in multilingual regions without understanding these limitations face potential service failures and customer dissatisfaction

• Training Data Requirements: The benchmark likely reveals that existing datasets underrepresent code-switching patterns, pointing toward necessary data collection and annotation efforts

• Competitive Differentiation: Companies improving code-switching performance gain advantages in international markets where bilingualism is prevalent

As voice AI systems become standard infrastructure for customer interactions, understanding their limitations on authentic human speech patterns becomes essential. This research moves beyond theoretical concerns to provide concrete measurements of where frontier models succeed and fail. For the AI industry, the findings underscore that advancing speech technology requires confronting the full complexity of human communication—not simplified, monolingual scenarios. Organizations deploying voice agents in diverse communities must engage with these benchmarks to ensure equitable service delivery.