An analysis of covariance parameters in Gaussian Process-based optimization
The need for globally optimizing expensive-to-evaluate functions frequently occurs in many real-world applications. Among the methods developed for solving such problems, the Efficient Global Optimization (EGO) is regarded as one of the state-of-the-art unconstrained continuous optimization algorithms. The surrogate model used in EGO is a Gaussian Process (GP) conditional on data points. The most important control on the efficiency of the EGO algorithm is the Gaussian process covariance function (or kernel), which is taken as a parameterized function. In this article, we theoretically and empirically analyze the effect of the covariance parameters, the so-called ``characteristic length-scale'' and ``nugget'', on the EGO performance. More precisely, we analyze the EGO algorithm with fixed covariance parameters and compare them to the standard setting where they are statistically estimated. The limit behavior of EGO with very small or very large characteristic length-scales is identified. Experiments show that a ``small'' nugget should be preferred to its maximum likelihood estimate. Overall, this study contributes to a better understanding of a key optimization algorithm from both a practical and a theoretical point of view.
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).