MiniMax Sparse Attention (MSA): a Two-Branch Block-Sparse Attention Trained on a 109B-Parameter MoE With a 3T-Token Budget

MiniMax has unveiled MiniMax Sparse Attention (MSA), a novel sparse attention mechanism designed to dramatically reduce computational costs while maintaining performance parity with standard attention approaches. Built on the foundation of Grouped Query Attention (GQA), MSA employs a two-branch architecture that intelligently selects which portions of a sequence to attend to, achieving substantial efficiency gains without sacrificing downstream task performance.

MSA introduces a lightweight Index Branch that strategically selects the top-k key-value blocks per query and GQA group, while a Main Branch concentrates computational resources exclusively on those selected blocks. The mechanism was trained on MiniMax's 109-billion-parameter Mixture-of-Experts (MoE) model using a 3-trillion-token budget. The architecture achieves a remarkable 28.4× reduction in per-token attention compute at 1 million token sequences while maintaining performance metrics equivalent to standard GQA implementations across downstream benchmarks.

The advancement carries significant consequences for the language model landscape:

• Computational Efficiency: The 28.4× reduction in attention compute addresses one of the primary bottlenecks in deploying large language models, potentially lowering infrastructure costs substantially
• Scalability Enhancement: Sparse attention mechanisms enable models to process longer sequences more efficiently, expanding practical applications for context-intensive tasks
• Performance Maintenance: MSA's ability to match GQA performance while reducing compute suggests the sparse selection mechanism captures the most relevant contextual information without degradation
• Industry Standardization: This development may accelerate adoption of sparse attention as a standard optimization technique across language model architectures
• Training Efficiency: Demonstration on a 109B-parameter model validates scalability, suggesting applicability across various model sizes

The emergence of MSA represents a critical step toward making advanced language models more practical and economically viable. As organizations increasingly deploy large language models for production workloads, computational efficiency directly translates to reduced operational costs and environmental impact. MiniMax's achievement demonstrates that sparse attention mechanisms can achieve performance parity with dense alternatives, potentially reshaping how future foundation models balance capability with efficiency. This innovation addresses a fundamental challenge in AI scalability—reducing the computational requirements without compromising the capabilities that make modern language models valuable.