Hi everyone, it may be simpler to read this post I created to see about my exact question, meta.stats.stackexchange.com/questions/2948/…, but I wanted to brain storm ideas with everyone about creating a metric for guiding an optimization algorithm

@RustyStatistician Rust, this is something that I'm interested in as well.

@RustyStatistician A few more recent alternatives to EI are enumerated in A Tutorial on Bayesian Optimization of Expensive Cost Functions

@RustyStatistician upper confidence bound seems to be the most promising among them. Xi penalized EI is purported to have desirable qualities as well, but I think it still is benchmakred below UCB. But the metrics tend to perform similarly, except probability of improvmeent, which is too greedy to be useful. A theme I keep reading over and over again is that the more important decisions are made in the chocie and estimation of the kernel function.

@RustyStatistician So the recommendation is to develop a reasonable choice of kernel for the response surface that you're approximating and integrate out the uncertainty in the hyperparameter estimates.

@RustyStatistician This is my last message for a time because I don't want people to become irate with me for spamming the room, but I'm currently working on implementing a BO/EGO algorithm in R for stochastic functions (i.e. ML hyperparameter tuning), so this is a topic near and dear to my heart at the moment

@user777 Offering thoughtful advice can scarcely be considered "spamming"! Feel free to continue your conversation.

@whuber Ah, well, sometimes it's hard to gauge on the internet whether one is shouting into the Abyss... I'll eagerly await Rust's replies.

@RustyStatistician But the emphasis on good kernel approximations should be intuitively appealing because if our surrogate surface approximation is very poor, then it's hard to make any inferences about which areas are worth exploring. So the best metric in the world, if such a thing exists, is no good if the approximation is bad.

@user777 I agree with evereything you have said so far

and I have seen A Tutorial on Bayesian Optimization of Expensive Cost Function before as well

@RustyStatistician oh, well then, at least we're in agreement.

@user777 What does the BO in BO/EGO stand for? Bayesian optimization?

@RustyStatistician Bayesian Optimization. I used both terms because everyone seems to have their own nomenclature.

@RustyStatistician If you're interested in pursuing the acquistion function question, perhaps consider "portfolio" methods -- several acquistion functions are consulted, and some decision logic is applied to select the next design point

in terms of EI on its own, it uses the current best estimate of the minimum as the yardstick for adjuging fantasy points ... that's sometimes problematic, like when the function values vary widely. I don't know how to fix it, though!