LayerNorm Induces Recency Bias in Transformer Decoders

• An official implementation of the paper LayerNorm Induces Recency Bias in Transformer Decoders (ACL 2026 Findings).

Requirements

• $pip install torch matplotlib seaborn numpy

Usage

• concept/simulation.ipynb: Figure 1, 3, 4
• concept/alpha.ipynb: Figure 2, 5
• The outputs of each notebook are saved in the figures directory.

Citation

@misc{kim2026layernormrecencybias,
      title={LayerNorm Induces Recency Bias in Transformer Decoders}, 
      author={Junu Kim and Xiao Liu and Zhenghao Lin and Lei Ji and Yeyun Gong and Edward Choi},
      year={2026},
      eprint={2509.21042},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2509.21042}, 
}