@RustyStatistician An interesting direction to take would be to consider a cooling schedule -based metric, a la simulated annealing, or a better way to parallelize the algorihtm, or a way to build an n-steps-ahead acquistion function. These are all methods that would alleviate different bottlenecks in the existing metrics.

This question: stats.stackexchange.com/questions/194151/best-linear-fit and this question: stats.stackexchange.com/questions/194101/… while made from different users use the exact same model. Before telling off the user is there a way to check which IP they used to post this? If it is the same IP it is probably the same person, otherwise maybe they just take the same class, etc.

Hmm... I think it is the same user because user's econmajorr answer here asks for "negative degrees of freedom". Which seems to be what the unregistered user MichealG asks for... C'est la vie!

@usεr11852 I don't think you should be "telling off' users...maybe explaining the rules to them?

@user777 do you have any paper titles you might recommend on portfolio methods? I tried googling for it and portfolio optimization came up on wikipedia and I think its more or less what you are talking about but I would be interested to read more about it.

@RustyStatistician: I do not plan to excoriate anyone any time soon! :D On the contrary, I want to avoid making even a suggestion that the user is abusing the rules before checking that this is indeed the case. This is why I asked if someone (probably one of the mods?) can check that.

@usεr11852 you don't have a way to identify IPs. But if you flag it, you can let moderators worry about whether its the same person.

@Glen_b: Thanks, will do.

@RustyStatistician Yeah, there's a reference to porfolio criteria in "Taking the Human Out of the Loop" which is another overview article on Bayesian Optimization

@RustyStatistician The authors of the overview article take a very favorable view of portfolio acqusition criteria, but the payoff seems pretty minimal based on the graphs they present. EI is very, very competitive with the portfolio method in their presentation

Can I get a hint of the origin of this ` 2 + mean(1/factorial(ceiling(1/runif(1e5))-2))`? It's from here.

@usεr11852 There are many groups of people using this site who share an IP address. Identifying truly duplicate accounts is sufficiently tricky that this capability was taken away from mods (SE-wide, not just here) four years ago. Thus, we should focus on improving and answering questions (and creating links among dups) rather than wondering who the OPs might be and how they might be interrelated.

@user777 yeah that's one of the hardest things about trying to come up with a better guiding metric than EI, its too darn intuitive! Even its name is intuitive!

@RustyStatistician My suggestions would be to look at how EI behaves in some specific contexts, and especially when it misbehaves. UCB appears to out-perform EI and has some nice additional properties, such as 0 regret!

@RustyStatistician I don't know if anyone has researched this recently, but optimizing the acquisition function is actually very hard. Sometimes EI is only positive in a tiny subspace of the region under optimization, and it can be very hard to find it.

@RustyStatistician My beta implemetnation of BO will occasionaly wander aimlessly because it can't find a positive EI location. Do you know how to fix that?

@user777 I haven't used EI enough to encounter those kind of problems yet. And actually I am moving out of the constrained optimization context (were I was primarily using Constrained EI) and into the unconstrained world.

@AntoniParellada Consider exp(x) = 1+1+1/2!+...+1/n!+... . Split off the initial 1+1=2 and try to look at the remainder as an expectation. One (out of infinitely many) ways is to write 1/(n+2)!=(1/(n+1)-1/(n+2))*1/n!. Take the first factor to be a probability and the second one to be a value. Notice that the sum of the first factors is (1/1-1/2)+(1/2-1/3)+... = 1+(1/2-1/2)+(1/3-1/3)+... = 1, so they really are probabilities.

The problem at this point is to simulate values 1/0!, 1/1!, ..., 1/n!, ... with these given probabilities. That code explicitly inverts the CDF, applies it to independent uniform variates, computes the mean (to estimate the expectation), and finally adds in the initial value of 2.

You could carry this idea much further. For instance, why not split off the initial 1+1+1/2!+...+1/100! and simulate the remainder? You could do this to 158 significant digits with a single realization! This shows the extent to which "simulation" and "numerical calculation" might be considered faces of the same coin.

@whuber: OK. Thanks, I did not know that. I would suspect that this would mostly appear in the case of schools/companies and it would be "relatively easy" to spot (both IP are from the same domain server for example) but I appreciate it could lead to False Positives in many cases.

@whuber, I really don't think there is an answer box available to me for the question

@RustyStatistician I have no way to check--the answer box is still available to me and I can't masquerade as someone else with 10+ reputation (at least not without going to a lot of work to do that!). All I can suggest is trying things like logging off and back on again, or restarting the browser, or trying with another browser.

@whuber can I naively ask you at what part of the screen you see the box to answer?

@RustyStatistician I have to scroll all the way to the bottom of the page, where a "Your Answer" textbox appears just beneath the last answer.

@whuber ok it doesn't appear there for me. I guess I'll message the tech support to see why this is.

@RustyStatistician You might have to resort to that, but please make sure you've tried the obvious and easy solutions first, such as restarting the browser. There very well could be an SE bug here, but you will want to demonstrate that you've made some efforts to investigate and resolve the issue: that ought to help get their attention.

Good luck, then!

@whuber Thank you very much for your explanation.

@user777 has maximizing (or minimizing, however you want to look at it) entropy been considered before?

@RustyStatistician Yes -- "Taking the Human out of the Loop" discusses information-based metrics among the other acquistion functions

@user777 so much for thinking that was going to be novel! haha

@RustyStatistician It may be possible to make a contribution using information metrics, though!

@user777 what I have also been thinking of lately is if there is anyway to take the derivative of the surrogate model, i.e., taking the derivative of the GP which is doable, and then using that derivative information to try to optimize the function.

In my work, we are designing bachelor's degree program in statistics there are co-worker that think only R should be taught but I think statistician must be know about other paradigms like functional programming since that is useful for big data.

@RustyStatistician derivative w.r.t. what?

@user777 well if the initial problem is to minimize $f(x)$ for some $x\in X\subset \mathbb{R}$ (so the simple case where $x$ is a scalar, and we are building our surrogate model such that the assumption is something liek $f(x)\sim GP$ then we could think to calculate work instead with $f'(x)\sim GP$ and use this derivative process to try and minimize the function

it's just a thought, not sure if it would actually work