On the Convergent Validity of Offline Evaluation Designs for Recommender Systems

Sushobhan Parajuli, Samira Vaez Barenji, and Michael D. Ekstrand. 2026. On the Convergent Validity of Offline Evaluation Designs for Recommender Systems. To appear in Proceedings of the 20th ACM Conference on Recommender Systems (RecSys '26), Sep 28–Oct 2, 2026. Acceptance rate: 18%.

Abstract

Offline evaluation on historical interaction logs is the most common evaluation methodology for recommender systems. However, such evaluations depend on sparse, incomplete or biased data which raises concerns about whether commonly used evaluation setups reliably reflect true user preferences. In this work, we study how offline evaluation design choices affect the validity of recommender system comparisons. We evaluate a set of recommendation models across many evaluation configurations that vary key factors including data filtering thresholds, feedback binarization versus graded relevance, candidate set construction, train-test splitting strategies, and evaluation metrics. To assess the validity of these configurations, we measure the correlation between model rankings obtained from conventional train–test splits on sparse interaction data and rankings from evaluations based on dense ground-truth relevance judgments. We use this agreement as an evidence of their validity with respect to true user preferences. Using KuaiRec and extended MovieLens-32M datasets that provide such ground-truth data, we analyze which evaluation setups produce results that better align with ground-truth performance.