Parallax: A Parameterized Local Linear Attention That Keeps Softmax and Adds a Learned Covariance Correction Branch

Parallax represents a significant advancement in attention mechanism design, introducing a parameterized local linear attention system that enhances computational efficiency while maintaining softmax operations. This innovation addresses key limitations in existing local linear attention (LLA) architectures by replacing traditional per-query solvers with learned projectors, achieving measurable improvements in model performance across multiple scales.

Parallax fundamentally restructures how local linear attention operates by substituting the conventional per-query solver with a learned projector mechanism. This architectural modification doubles the arithmetic intensity of the system—a critical metric for computational efficiency—while simultaneously introducing a learned covariance correction branch that refines attention calculations. The approach maintains compatibility with softmax operations, ensuring continuity with established attention mechanisms. Performance testing demonstrates that Parallax delivers improved perplexity metrics at both 0.6 billion and 1.7 billion parameter scales, indicating its effectiveness across different model sizes relevant to practical deployment scenarios.

• Computational Efficiency: Doubling arithmetic intensity reduces computational overhead while maintaining or improving output quality, critical for resource-constrained deployment environments

• Scalability: Demonstrated improvements at multiple parameter scales suggest Parallax's applicability across diverse model architectures and sizes

• Architectural Compatibility: Preserving softmax operations enables easier integration into existing AI infrastructure without requiring complete system redesigns

• Performance-to-Cost Ratio: Enhanced perplexity at lower computational expense strengthens the economic viability of deploying large language models

• Foundation for Future Research: The parameterized approach creates opportunities for further optimization and refinement in attention mechanism design

The development of Parallax addresses persistent challenges in making large language models more efficient without sacrificing performance quality. As AI deployment costs remain a significant barrier to broader adoption, innovations that improve the arithmetic intensity and perplexity of attention mechanisms hold substantial commercial and technical significance. By successfully balancing computational demands with output quality across multiple model scales, Parallax contributes to the ongoing effort to make advanced AI systems more accessible and practical for real-world applications. This advancement signals continued progress in optimizing fundamental AI architecture components.