Abstract
Learning by teaching is a broadly used methodology in human learning and
shows great effectiveness in improving learning outcome: a learner
deepens his/her understanding of a topic by teaching this topic to
others. We are interested in investigating whether this powerful
learning technique can be borrowed from humans to improve the learning
abilities of machines. We propose a novel machine learning approach
called learning by teaching (LBT). In our approach, the teacher creates
a pseudo-labeled dataset and uses it to train a student model. Based on
how the student performs on the validation dataset, the teacher
re-learns its model and re-teaches the student until the student
achieves great validation performance. We propose a multi-level
optimization framework to formulate LBT which involves three learning
stages: teacher learns; teacher teaches student; teacher and student
validate themselves. We develop an efficient algorithm to solve the LBT
problem. We apply our approach to neural architecture search on
CIFAR-100, CIFAR-10, and ImageNet, where the results demonstrate the
effectiveness of our method.