a06e

It depends on the sign of \lambda_i: If \lambda_i > 0, then x_i=0 can be a local minimum. If \lambda_i < 0, then some stationary point with x_i=0 is actually a saddle-point

Ok I figured out what's missing. If $\lambda_i < 0$ then $x_i$ cannot be zero at a local minimum.

Ah I think my answer fails because when $\lambda_i<0$, I can't do $\partial\lambda_{i}|x_{I}| \ne \lambda_{i}\partial|x_{I}|$

Please take a look at the answer I have just posted: math.stackexchange.com/a/4910031/10063. I'm not sure if we can get around this somehow and do a faster search over 2^N orthants instead ...

(Too bad LaTeX is not supported in this chat!)

It seems to me one would need to search through all $3^N$ vectors $\mathbf{s}\in\{-1,0,+1\}^{N}$. So not just orthants

Though I'm not sure how to (exhaustively) search for all solutions ...

On second thought I now believe your stationarity condition is actually sufficient for having a local minima.

Yes, stationarity is ok. The problem is with the assumption that "stationarity" implies it is a local minimizer.

Regarding the optimality condition you wrote ... doesn't this assume that the objective function is convex? My objective function fails to be convex at the hyperplanes $x_i=0$ whenever the corresponding $\lambda_i<0$.

Let $x(s)$ be a solution to the inner convex minimization problem (with given $s$). If some coordinates vanish, $x_i(s)=0$, it does not necessarily follow that this $x(s)$ is a local minimum in the original problem. Do you agree? This is the point I'm missing (otherwise I agree with your answer).

Great. Thanks for sticking with this!

I see.

Where is it? I'm curious now too.

Aha!

There's the pesky 1/2 factor. But that's irrelevant if we're focusing on the presence or not of this uAubu term.

Here is the argument I'm using to obtain the variance in the general case, where 'x' is a general multivariate normal.

Do you agree?

I don't see how. I mean, it would just replace A with 2A.

Here's a simple check that agrees with my expression for the variance, without this term.

But I neither find a mistake in mine ....

I don't find a mistake in your calculation.

I still don't agree with the variance though ... if I take my expression in the previous comment, and multiply all the $A$ by 2, you see I never get the $6\mu A\mu b\mu$ term you get.

So now I agree with your expression.

Ah sorry, it's my fault. I was working with the expression $ z = \frac{1}{2}\mathbf{x}^{\top}\mathbb{A}\mathbf{x} + \mathbf{b}^{\top}\mathbf{x} + c$

This is the expression I get for the variance: $\frac{1}{2}tr(\mathbb{A\Sigma A\Sigma}) + \mathbf{u}^{\top}\mathbb{A\Sigma A}\mathbf{u} + 2\mathbf{b}^{\top}\mathbb{\Sigma A}\mathbf{u} + \mathbf{b}^{\top}\Sigma\mathbf{b}$, assuming $A$ is symmetric.

I get a similar expression, but off by some of the factors, and also the second-to last term I have is different.. I will do some numerical verifications to check

Ok, but the variances of the gradients you get are different. Also, what do you mean precisely by "the weighted average" here? Is the the weighted average over a mini-batch?

Ok, I hope to convince you at least that this weighted average is not an unbiased estimate of the gradient. Do you agree on that?

As I said, that gives a biased estimate of the gradient. This is contrast to the unweighted case, where the batch (unweighted) average gives an unbiased estimate of the gradient evaluated on the full data.

I mean precisely this: $\frac{\sum_{i \in \mathcal{B}} w_i f'_i}{\sum_{i \in \mathcal{B}} w_i}$ is not an unbiased estimate of the gradient in the full dataset. If you don't agree, can you try to prove that it is?

The unweighted average is special, because all the weights are equal. Only in that case, the above formula is an unbiased estimate. But if the weights are not all equal, then it is not. You can check the calculation, if you want.

For example, suppose you do $\frac{\sum_{i \in \mathcal{B}} w_i f'_i}{\sum_{i \in \mathcal{B}} w_i}$, where $\mathcal{B}$ denotes the mini-batch. Then this is not an unbiased estimate of the gradient in the full data.

Again, I'm not following precisely what you mean. Can you write explicitly the mini-batch gradient estimate you propose ?

Can you be explicit in how the weighted average is taken? This is the central point of my question.

I don't think this answers my question? It's really a matter of the variance of the gradients estimates and which approach leads to less noisy estimates in a mini-batch.

THanks for the reference though!

It's been a long time since I visited the chat. Very disappointed to see it still doesn't support latex!

@fedja See math.stackexchange.com/a/3020116/10063

@user587192 Absolute integrability is not necessary for integrability

@user587192 Integrate[Exp[I a x^2], {x, -Infinity, Infinity}, Assumptions -> Element[a, Reals]]

@user587192 Yes. I know how to do it in polar coordinates (see my answer). What I need to understand is what I did wrong if I try to do it in Cartesian coordinates and using the Fourier transform of the Dirac delta function, as I attempted to do in my question.

@user587192 Eq. (2.1)

@reuns Yes and yes.

@fedja In fact, $I = n^{n/2-1} \pi^n/2 / \Gamma(n/2)$ can be obtained if one uses polar coordinates (see doi.org/10.1093/qmath/12.1.165). Some step of my derivation leads to a fictitious divergence.

@fedja Why do you say I started with a divergent integral? The first formula I gave for $I$ is (up to some factors) the surface area of a sphere of radius $n$ in $n$ dimensions. It is finite. Only the later formulas diverge. That's why I think I did something wrong. Can you please explain?