ITBench-AA: Frontier Models Score Below 50% on the First Benchmark for Agentic Enterprise IT Tasks — by Artificial Analysis and IBM

A new benchmark called ITBench-AA reveals a significant performance gap in enterprise IT operations, with leading artificial intelligence models scoring below 50% on agentic tasks designed to reflect real-world IT environments. The study, conducted by Artificial Analysis and IBM, demonstrates that despite rapid advances in large language model capabilities, current frontier models face substantial challenges when tasked with automating complex enterprise IT workflows.

The benchmark tests how well AI agents can handle authentic IT administration scenarios, including system configuration, troubleshooting, security management, and infrastructure operations. The disappointing results underscore a critical distinction between general AI capability and domain-specific practical application, particularly in enterprise environments where reliability and accuracy are non-negotiable.

• Enterprise AI Adoption Gap: The below-50% performance suggests that organizations cannot yet rely on autonomous AI agents for mission-critical IT operations, slowing enterprise deployment of agentic AI systems

• Training and Architecture Limitations: Current models lack specialized knowledge and reasoning capabilities required for IT domain tasks, indicating that general-purpose training approaches have limitations

• Security and Compliance Concerns: Poor performance on IT tasks raises questions about safety when deploying AI agents in environments handling sensitive data and infrastructure

• Market Opportunity for Specialized Models: The findings point to significant demand for fine-tuned or specialized AI models designed specifically for enterprise IT operations

• Research Direction: The benchmark establishes a clear roadmap for AI developers to focus on practical, enterprise-grade improvements rather than abstract capability measures

ITBench-AA addresses a crucial gap in AI evaluation frameworks. While benchmarks like MMLU and MATH measure general reasoning, enterprise environments demand tested competency in specific operational contexts. This research validates that frontier models, despite impressive performances on popular benchmarks, remain insufficient for autonomous enterprise IT management.

For AI companies, IBM, and enterprise customers, this benchmark signals the need for continued development of specialized agentic systems, better evaluation frameworks, and hybrid approaches combining AI with human oversight. The study emphasizes that the path to truly autonomous enterprise AI requires not just larger models, but fundamentally better solutions for domain-specific task execution.