Better Fit, Worse Next Move

TL;DR for operators

When each experiment or simulation is expensive, selecting a surrogate because it fits existing observations well can be the wrong criterion. In the paper’s controlled surrogate ablation, the most flexible model achieves the lowest training error but materially worse prediction away from observed points and much more optimistic estimates of candidate value.

MOLRN-BO responds by constraining and sharing nonstationary spectral structure across objectives, correcting future decision posteriors using errors recorded from earlier predictions, and concentrating most acquisition search near observed Pareto solutions while periodically reopening the full domain. Across 12 deterministic bi-objective benchmarks, it ranks best on average for final normalized hypervolume and normalized IGD, although qLogNParEGO remains slightly stronger on anytime rankings.

The broader operating principle is stronger than the method result: evaluate surrogate changes against downstream optimization behavior. One ablation reduces median candidate prediction bias from 0.197 to 0.079 yet slightly worsens final optimization metrics. Better calibration can be desirable without being sufficient evidence of better decisions.

The surrogate is evaluated where the optimizer has not looked yet

Bayesian optimization repeatedly makes a costly choice: use the observations collected so far to decide which experiment, simulation, or design point should be evaluated next.

That creates a mismatch between two evaluation problems. Regression training rewards a model for explaining locations already observed. Optimization depends on predictions at candidate locations that may differ substantially from those observations.

The fixed-design ablation in Zikai Xie’s Adapting Nonstationary Multi-output Gaussian Processes to Bayesian Optimization¹ makes the difference unusually clear. The output-specific model with four spectral components reaches a training RMSE of 0.018. Reducing capacity to two components worsens training RMSE to 0.062.

If training fit were the decisive criterion, the higher-capacity model would win.

It does not win on the quantities that matter for choosing the next evaluation. Global RMSE falls from 1.154 with four components to 0.816 with two. Pareto-region RMSE falls from 1.101 to 0.792. Top-HVI optimism—a measure of how much the surrogate overstates promising hypervolume-improvement candidates—falls from 1.213 to 0.498.

This experiment is an ablation, not the paper’s main benchmark result. Its purpose is to isolate a failure mechanism: greater spectral flexibility can improve fit to observed data while degrading prediction in the regions used for acquisition decisions.

Sharing structure improves more than reducing capacity alone

Lower capacity is only part of the proposed remedy.

With two spectral components held fixed, sharing the spectral structure across objectives reduces global RMSE further, from 0.816 to 0.746, and Pareto-region RMSE from 0.792 to 0.729. Top-HVI optimism drops much more sharply, from 0.498 to 0.087. Training RMSE remains almost unchanged at 0.063.

The proposed covariance therefore separates two roles. Objectives share the same nonstationary spectral components, allowing information about common structure to transfer across outputs, while objective-specific RBF residual kernels retain room for smoother idiosyncratic behavior.

MOLRN-BO adds explicit regularization during Gaussian-process fitting as well. The fitting objective penalizes spectral roughness, boundary-saturating frequencies, mixture imbalance, and other unstable parameter behavior rather than relying on marginal likelihood alone.

For organizations using related objectives—cost and performance, energy and throughput, yield and quality—the business inference is conditional but relevant. A shared multi-output model may provide better decision-region estimates than separate highly flexible models when objectives genuinely contain transferable structure. The benchmark does not establish that such sharing will help when objectives are weakly related or structurally different.

The acquisition policy is part of the model’s reliability

The paper does not assume that improving the surrogate alone solves the decision problem.

MOLRN-BO records prediction errors at acquisition-selected candidates before their outcomes are observed. Those strictly prequential residuals are later used to estimate systematic prediction bias. The acquisition posterior can shift its mean using that history and rescale its covariance when past predictions indicate underdispersion.

The search domain is also constrained. Most acquisition rounds optimize qLogEHVI inside adaptive regions centered on observed Pareto solutions, where the surrogate has more empirical support. Scheduled global rounds still search the full bounded domain. Importantly, the Gaussian process itself continues to use all accumulated observations; only acquisition optimization is localized.

A controlled decision-policy ablation compares full-domain search with pooled bias against local-global search with independent mode-specific bias estimates. The local-global configuration improves final normalized hypervolume on 11 of 12 problems, final normalized IGD on 11, hypervolume-regret AUC on 11, and normalized-IGD AUC on 10.

Because localization and mode-specific bias correction change together, the experiment identifies their joint contribution. It does not establish how much improvement comes from each component separately.

For costly optimization workflows, the design pattern is nevertheless concrete: do not allow an acquisition optimizer to exploit unsupported regions without constraint merely because the surrogate assigns them high value. Restricting routine search to better-supported regions, while scheduling explicit global exploration, can reduce dependence on extrapolative confidence.

Better calibration is not the same as better optimization

One of the paper’s most useful results is a negative one.

Hierarchical pooling improves the intermediate prediction metric substantially. Median problem-level absolute candidate bias falls from 0.197 with independent estimates to 0.079.

Downstream optimization does not improve. Mean final normalized hypervolume declines from 0.590 to 0.582, while normalized IGD rises from 0.201 to 0.207.

That distinction matters for model governance. Calibration, coverage, residual error, and related diagnostics can reveal weaknesses in a surrogate, but an optimization system consumes those predictions through an acquisition rule. Improving an intermediate statistic can alter exploration and exploitation in ways that do not improve the final objective.

Teams deciding whether to deploy a surrogate correction therefore need two evaluation layers: does the correction improve the predictive behavior it targets, and does that improvement produce better sequential decisions?

MOLRN-BO wins mainly on final quality and poor-trial robustness

Across the main comparison of nine algorithms on 12 deterministic bi-objective benchmarks, MOLRN-BO has the best mean problem rank for final normalized hypervolume at 2.25 and final normalized IGD at 2.17.

Its advantage over the original MO-LRN is strong. MOLRN-BO has better problem-level means on 11 of 12 problems for final hypervolume and hypervolume-regret AUC, and on all 12 for final IGD and IGD AUC. All four cross-problem comparisons remain significant after Holm correction.

The comparison with qLogNParEGO is much closer. MOLRN-BO is better on six of 12 problems for final hypervolume and seven of 12 for final IGD. qLogNParEGO has slightly better mean ranks on both anytime AUC measures, and the aggregate differences between the two methods are not statistically significant.

MOLRN-BO’s clearest additional advantage appears in weak runs. It ranks first on all four reported adverse-tail primary measures and has the smallest average trial-to-trial final-IGD standard deviation, 0.0358. The evidence therefore supports stronger final Pareto quality and resilience against poor trials more clearly than universal superiority throughout the optimization trajectory.

The evidence stops before real-world deployment

The evaluation covers small-budget, deterministic, bi-objective synthetic problems. Each run begins with 10 Sobol observations and adds 100 sequential evaluations, one candidate at a time.

Noisy objectives, batch evaluation, many-objective problems, larger budgets, and real operating environments are not tested. Fixed regularization choices are not systematically sensitivity-tested. The study also does not provide a controlled runtime comparison because execution used heterogeneous CPU and GPU hardware.

Those boundaries matter for adoption. In physical experimentation, measurement noise can change calibration requirements. Batch experimentation changes acquisition interactions. More objectives increase Pareto and inference complexity. Computational overhead can matter when candidate evaluation is not overwhelmingly more expensive than model fitting.

The paper therefore supports a decision criterion more confidently than it supports a universal deployment recipe: surrogate quality in Bayesian optimization should be measured where predictions affect future evaluations, and every predictive improvement should ultimately be checked against optimization outcomes.

A model that explains yesterday’s experiments exceptionally well may still be a poor guide to tomorrow’s expensive one.

Cognaptus: Automate the Present, Incubate the Future.