SubQ 1.1 技术文档：线性缩放稀疏注意力，1200万token下实现98%检索率 [pdf]
SubQ 1.1 发布了一份技术文档，介绍了其线性缩放稀疏注意力机制。该模型在1200万token的超长上下文场景下依然保持98%的高检索率，突破了传统注意力机制在长序列处理上的计算瓶颈。文档详细阐述了SubQ架构如何通过稀疏化实现与序列长度成线性关系的计算复杂度，从而高效处理极长文本而无需牺牲准确性。
相关报道
KV cache compression techniques, including Multi-Query Attention (MQA), Grouped-Query Attention (GQA), Multi-head Latent Attention (MLA), and linear-attention hybrids, have evolved to reduce memory overhead in large language models. These developments have quietly enabled the long context windows required for modern agentic LLM applications by making key-value caching more efficient.