TechRxiv
release.pdf (383.05 kB)

Learning by Ignoring, with Application to Domain Adaptation

Download (383.05 kB)
preprint
posted on 13.03.2021, 15:29 by Xingchen Zhao, Xuehai He, Pengtao Xie
Learning by ignoring, which identifies less important things and excludes them from the learning process, is broadly practiced in human learning and has shown ubiquitous effectiveness. There has been psychological studies showing that learning to ignore certain things is a powerful tool for helping people focus. In this paper, we explore whether this useful human learning methodology can be borrowed to improve machine learning. We propose a novel machine learning framework referred to as learning by ignoring (LBI). Our framework automatically identifies pretraining data examples that have large domain shift from the target distribution by learning an ignoring variable for each example and excludes them from the pretraining process. We formulate LBI as a three-level optimization framework where three learning stages are involved: pretraining by minimizing the losses weighed by ignoring variables; finetuning; updating the ignoring variables by minimizing the validation loss. A gradient-based algorithm is developed to efficiently solve the three-level optimization problem in LBI. Experiments on various datasets demonstrate the effectiveness of our framework.

History

Email Address of Submitting Author

p1xie@eng.ucsd.edu

Submitting Author's Institution

UC San Diego

Submitting Author's Country

United States of America

Licence

Exports

Licence

Exports