I don't see what you mean.

@user1952009 What's unclear?

A typical derivative filter with greater length is $[-\frac{1}{3},-\frac{2}{3},\frac{2}{3},\frac{1}{3}]$.

@user1952009 I'm not sure I see your point. Yes, we use filters with more sample points at times. My second point was just that if you want to only use two points, you don't have to drop second order to do so.

I don't see what you mean with "second order" or with "for smooth $f$, the central difference scheme is second order in $h$". Theoretically, for analyzing a discrete filter replacing $\delta'$ the true derivative filter, we look at its frequency response and compare it to $i \omega$

Ok I just understood, you are looking at the error as $h \to 0$. But then $|f'(x) - \frac{f(x+2h)-f(x)}{2h}| < h^2 \sup_x |f''(x)|$ is very close to $|f'(x) - \frac{f(x+h)-f(x-h)}{2h}|$, it doesn't change the order of the error. And in real life $h$ is fixed, so we look at the total error $\int_a^b |f'(x) - \frac{f(x+h)-f(x-h)}{2h}|^2 dx$ with respect to the frequency. And the derivative filter is $\delta'$ because $(f \ast \delta)' = f' \ast \delta = f \ast \delta'$ (there is some time reversal in the convolution)

@user1952009 Oh, right, I mixed up convolution with just application of a distribution for a moment. And no, you're wrong: the central difference is higher order than the forward difference if $f$ is smooth enough. But yes, in reality $h$ is fixed, but again once $h$ is sufficiently small but finite the central difference scheme is more accurate than the forward or backward schemes.

The central difference with $h$ is almost the same as the forward difference with $2h$ : it doesn't change the order of the error

@user1952009 No, this is just blatantly wrong. $f(x+h)=f(x)+hf'(x)+1/2 h^2 f''(x)+O(h^3)$. $f(x-h)=f(x)-hf'(x)+1/2 h^2 f''(x)+O(h^3)$. Therefore $f(x+h)-f(x-h)=2hf'(x)+O(h^3)$, so the centered difference has an error of $O(h^2)$ for three times continuously differentiable functions. Doing essentially the same calculation reveals that the forward difference has an error of $O(h)$ for twice continuously differentiable functions. So for reasonably small $h$, the centered difference scheme will perform better on the real space side.

??? Come on do the calculation rigorously, both $|f'(x) - \frac{f(x+h)-f(x-h)}{2h}|$ and $|f'(x) - \frac{f(x+2h)-f(x)}{2h}|$ are $< h \sup_y |f''(y)|$. The only difference is that in the 1st case we can bound with $h \sup_{|y-x| < h} |f''(y)|$ while in the second it is $h \sup_{|y-x-h| < h} |f''(y)|$

@user1952009 No, $f(x+2h)=f(x)+f'(x)2h+O(h^2)$. Thus taking the difference and dividing by $2h$ gives you an error of $O(h)$. With the centered difference (again for three times continuously differentiable functions) the error is merely $O(h^3)$, because the second derivative terms cancel one another.

Again the second derivative terms cancel one another in the centered case: $(f(x)+hf'(x)+1/2 h^2 f''(x)+O(h^3))-(f(x)-hf'(x)+1/2 h^2 f''(x)+O(h^3))=2hf'(x)+O(h^3)$. This is exceptionally basic numerical analysis, I'm not sure why I have to explain it 5 times to someone who clearly knows some heavy math.

No ok I got it. I didn't except it changed something assuming it is $C^3$ (and you should see that your answer is really impossible to understand alone)

@user1952009 I said "smooth $f$" to begin with, so that you have all the derivatives you want. I guess I can clarify what "second order" means if you insist, but this really is standard terminology...

I explained : in signal/image processing we look at $h$ fixed, not $h \to 0$. And the forward difference is just the central difference shifted, so all we have to do is take the phase in account. In this context, it wasn't really possible to see what you meant with "second order" and "higher order methods" (the vocabulary of numerical analysis for approximating integrals and differential equations)

@user1952009 The fact that $h$ is fixed does not matter. Once $h$ is small enough, centered difference will perform better (in real space error metric). I see what you mean about one being a shift of the other, so that on the Fourier side one can convert between the two through a phase factor...but this can be viewed as a phase discrepancy, i.e. a flaw in the forward/backward schemes which happens to be easy to correct on the Fourier side. I also disagree that there is enough context here to really be certain of what exactly the OP wants.

@Ian. thanks for the clear explanation. Can you elaborate a bit more on the part where the central difference will have a better (real space) error than the other two? I understand about the $O(h^2)$ and $O(h)$ part. Maybe how $O(h^2)$ can give a better error? If you have a diagram, that will be perfect!

@Ray.R.Chua Is this better?