I don't think this answers my question? It's really a matter of the variance of the gradients estimates and which approach leads to less noisy estimates in a mini-batch.

THanks for the reference though!

@becko in batch gradient descent you just use weighted average in place of average.

Can you be explicit in how the weighted average is taken? This is the central point of my question.

For example, suppose you do $\frac{\sum_{i \in \mathcal{B}} w_i f'_i}{\sum_{i \in \mathcal{B}} w_i}$, where $\mathcal{B}$ denotes the mini-batch. Then this is not an unbiased estimate of the gradient in the full data.

@becko is an unweighted average over the batch an unbiased estimate for full data? When using batches, each update is given the batch. If you want to update given all data, use all data. Using weights is equivalent to repeating the observations number of times proportional to the weight. So it's like using standard batch gradient descent with varying batch sizes due to the weights if they are not uniform.

Again, I'm not following precisely what you mean. Can you write explicitly the mini-batch gradient estimate you propose ?

@becko is exactly the same as usual gradient descent, but you replace the "$\tfrac{1}{M} \sum_{i=1}^M$" part with weighted average, that's all.

As I said, that gives a biased estimate of the gradient. This is contrast to the unweighted case, where the batch (unweighted) average gives an unbiased estimate of the gradient evaluated on the full data.

@becko I'm not sure what you mean. An unweighted average is just a special case of a weighted average. If you consider sampling according to the weights it is equivalent to using the weights in the loss function.

I mean precisely this: $\frac{\sum_{i \in \mathcal{B}} w_i f'_i}{\sum_{i \in \mathcal{B}} w_i}$ is not an unbiased estimate of the gradient in the full dataset. If you don't agree, can you try to prove that it is?

@becko if weights are proportional to frequencies, the weighted average is the same as you repeated each row the number of times proportional to the weight. So if weights tell you how frequent the value is, it is exactly the same as the standard average. So if you consider sampling proportionally to weights it is the same as using weighted average.

Ok, but the variances of the gradients you get are different. Also, what do you mean precisely by "the weighted average" here? Is the the weighted average over a mini-batch?

@becko there is weighted variance as well, but the variance is not used by the gradient descent algorithm when doing the update. By weighted average, I mean weighted average over batch when doing the update in place of regular average in non-weighted case.

Ok, I hope to convince you at least that this weighted average is not an unbiased estimate of the gradient. Do you agree on that?