Keenable AI Open-Sources NEEDLE: A Live Search Benchmark That Rebuilds Its Query Set Every Hour

Keenable AI has introduced NEEDLE, an innovative open-source benchmark designed to address a critical challenge in evaluating web search APIs: preventing evaluation gaming. Traditional benchmarking methods rely on static datasets that search agents can potentially exploit by directly accessing answer keys rather than performing legitimate retrieval tasks. NEEDLE solves this problem through dynamic query regeneration, refreshing its evaluation dataset hourly to maintain test integrity and provide authentic performance metrics.

The core innovation behind NEEDLE lies in its hourly query set regeneration mechanism. Unlike conventional benchmarks with fixed, publicly available datasets, NEEDLE continuously creates new search queries and expected results, making it impossible for agents to pre-download or memorize evaluation answers. This approach ensures that search APIs are genuinely tested on their retrieval capabilities rather than their ability to game static test sets.

The benchmark addresses a fundamental vulnerability in AI evaluation: when gold labels remain publicly accessible, search agents equipped with fetch tools can bypass retrieval entirely by directly downloading the answer key during evaluation. By maintaining a live, constantly updating dataset, NEEDLE ensures that agents must actually perform search and retrieval operations to succeed.

• Enables more authentic assessment of search API performance and reliability
• Prevents artificial benchmark inflation through dataset exploitation
• Establishes a framework for continuous, real-world benchmarking standards
• Supports responsible AI development by creating tamper-resistant evaluation methods
• Provides open-source infrastructure for the entire AI research community

As large language models and AI agents become increasingly sophisticated, benchmarking integrity becomes crucial for genuine progress measurement. NEEDLE represents a significant step toward trustworthy AI evaluation by creating conditions that reflect real-world search scenarios. By open-sourcing this tool, Keenable AI enables the broader research community to adopt more rigorous testing standards, ultimately advancing the development of more reliable and genuinely capable search systems. This approach sets an important precedent for how future AI benchmarks should handle the growing challenge of evaluation robustness.