@user82662 Although I do not wish ever to confuse learning a language with a statistical education, I thought it might be worth mentioning that R supports several paradigms, including functional programming. See the "funprog" help covering "Reduce", "Filter", etc., and note that the *apply functions and many of the vectorized functions fit within this paradigm, too.

@RustyStatistician This sounds like a very different approach than the kinds of problems that I use BO to solve. I don't have the luxury of optimizing analytical functions, so I never know f'(x).

@RustyStatistician I'm just thinking aloud here, but suppose we do have f'(x) ~ GP. We're looking for an optimum, so we're looking for points where f'(x)=0. But there are infinitely many of these points because the (posterior) GP is a distribution over functions. So we're back to looking for where the probability of 0 is large.... but this will tend to focus on regions around the design points, because the posterior derivative usually changes signs at that point (or else is an inflection point)

@user777 this is where I thought this was may be a novel idea

@user777 I may be completely wrong because I am just thinking out loud here as well

but my thought was that you still don't know what the analytical form of f(x) is and you don't actually have derivative observations from f'(x) either

my thought was can we write down the posterior predictive distribution of f'(x)|f(x)?

so can we predict what we think the derivative should look like based on the fact that we have observations of f(x)

clearly you could build a finite difference approximation to the derivative using the posterior predictive of f(x) (I have done this and it works beautifully)

@user777 but I think a more concrete solution, from a probabilistic or statistical point of view, would be better sought without an approximation

finite difference approximations of the GP might be suffiicent. they can be made arbitrarily easily, and we can draw many, many, many of them and have a probaiblity distribution over gradients

*approximations of the derivative usign the GP

I don't know how to go about doing that, myself, but it's an intriguing idea. My concern would be how to make it sufficiently global

@user777 agreed. I have actually coded it up the finite approximation version. However, my main difficulty is applying it to the case the dimension of the input space is greater than 1

because in 1-dimension you can easily check the boundary points and determine where the global minimum exists

but in higher dimensions I don't have an easy way to do that

Well, how does a finite-difference local optimizer do it? Like BFGS?

@RustyStatistician Because it seems like one could do worse than using the GP as a fast approximation to the function and just using a fast local solver with multistart to pick out critical points

@user777 I am not sure I understand your last comment, is it for or against the use of the GP based idea?

No, I'm saying that you can use the GP in conjunction with a fast local optimizer to identify plausible critical points

then assess the probability that those points are minima