On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models
Apple Machine Learning
State Space Models (SSMs) have emerged as a compelling alternative to
Transformers, enabling sequence modeling with constant memory and linear
compute. Although SSMs exhibit reasonable performance and favorable
computational characteristics, they continue to lag behind Transformers on
tasks that require in-context learning and precise retrieval, slowing their
adoption for large-scale language modeling. In this work, we demonstrate that
both the success and failure of SSMs in these domains can be