MTGPUCB_v3.5 revision Bo.pdf (2.72 MB)
Download fileMulti-Task Gaussian Process Upper Confidence Bound for Hyperparameter Tuning
preprint
posted on 2021-09-27, 16:17 authored by Bo ShenBo Shen, Raghav GnanasambandamRaghav Gnanasambandam, Rongxuan Wang, Zhenyu KongZhenyu KongIn many scientific and engineering applications, Bayesian optimization (BO) is a
powerful tool for hyperparameter tuning of a machine learning model, materials design
and discovery, etc. BO guides the choice of experiments in a sequential way to find
a good combination of design points in as few experiments as possible. It can be formulated as a problem of optimizing a “black-box” function. Different from single-task
Bayesian optimization, Multi-task Bayesian optimization is a general method to efficiently optimize multiple different but correlated “black-box” functions. The previous
works in Multi-task Bayesian optimization algorithm queries a point to be evaluated
for all tasks in each round of search, which is not efficient. For the case where different tasks are correlated, it is not necessary to evaluate all tasks for a given query
point. Therefore, the objective of this work is to develop an algorithm for multi-task
Bayesian optimization with automatic task selection so that only one task evaluation
is needed per query round. Specifically, a new algorithm, namely, multi-task Gaussian
process upper confidence bound (MT-GPUCB), is proposed to achieve this objective.
The MT-GPUCB is a two-step algorithm, where the first step chooses which query
point to evaluate, and the second step automatically selects the most informative task
to evaluate. Under the bandit setting, a theoretical analysis is provided to show that
our proposed MT-GPUCB is no-regret under some mild conditions. Our proposed algorithm is verified experimentally on a range of synthetic functions as well as real-world
problems. The results clearly show the advantages of our query strategy for both design
point and task.
History
Email Address of Submitting Author
boshen@vt.eduORCID of Submitting Author
https://orcid.org/0000-0002-2643-3600Submitting Author's Institution
Virginia TechSubmitting Author's Country
- United States of America