Evaluating the Evaluators: Investigating LLM Judges for Personalized Writing Style Assessment
Apple Machine Learning
Apple researchers investigated the effectiveness of large language model judges in assessing personalized writing styles.
This paper was accepted at the Can We Trust AI Evaluation? Workshop at NeurIPS
2026. We evaluate how good large language model (LLM) judges are for
personalized writing tasks. This is important when a user’s preferred writing
style is known (e.g., “upbeat”) and an LLM judge is used to evaluate whether
generated text adheres to this preference. We examine how the writing task,
the combination of generator and judge LLM, the evaluation objective, and
general commonsense and reasoning ability impac