Theoretical Limits of Language Model Alignment
Apple Machine Learning
Language model (LM) alignment improves model outputs to reflect human
preferences while preserving the capabilities of the base model. The most
common alignment approaches are (i) reinforcement learning, which maximizes
the expected reward under a KL-divergence constraint, and (ii) best-of-NNN
alignment, which selects the highest-reward output among NNN independent
samples. Despite their widespread use, the fundamental limits of reward
improvement under a KL budget remain poorly understood. We c