Beyond Instance-Dependent Noise: Generalizing Noisy Label Modeling to Multiple Labeler Dependence

Most studies on learning from noisy labels rely on unrealistic models of i.i.d. label noise, such as class-conditional transition matrices. More recent work on instance-dependent noise models are more realistic; however, these models assume a single generative process for label noise across the entire dataset, which does not accurately capture the real-world labeling dynamics observed in modern dataset collection. We propose a new model of label noise, which we call "labeler-dependent noise (LDN)." LDN extends and generalizes instance-dependent noise to multiple labelers, based on the observation that modern datasets are typically constructed via distributed data gathering and labeling methods. Furthermore, inspired by recent research in social learning theory, we introduce a modular learning framework under our proposed labeler-dependent noise model called "labeler-aware learning", in which each labeler is estimated and compared against the others during the learning process. Our experimental results demonstrate how previous state-of-the-art approaches for learning from noisy labels are unable to handle the general model of label noise, while our labeler-aware learning framework remains robust even when large fractions of labelers are spammers.