Hint: use polar coordinates. Heuristically, $\delta(n-|x|^2)$ is like a function accumulated at the sphere of radius $\sqrt{n}$, so the integral should be related to the area of $\sqrt{n}S^{n-1}$.

@user90189 The relation to the sphere area is what suggests that I'm doing something wrong. Obvously the result cannot diverge. But before jumping to polar coordinates (where the algebra gets more complicated) I would like to understand what I did wrong with this approach which should be, in principle, simpler.

I think it is easier to use polar coordinates, the integral is just $\int_{S^{n-1}}\int_0^\infty r^{n-1}\delta(n-r^2)\,drd\sigma$. I'm looking at your calculation, but I don't see the big problem, but the inner integral should be something of the order $1/\sqrt{|\lambda|}$. This is an oscillatory integral, so you should be careful, even more if you try to use Fubini.

Instead of Mathematica, you can try to calculate the inner integral, it is classical and should be in google. Look for Fresnel functions. You can calculate it by residues, using a "pizza"-like path, with the corner at zero.

Did you mean $\int_{\mathbb{R}^n}$ instead of $\int_{-\infty}^\infty$. And $\delta(f(x))= \lim_{a \to 0} \frac{1_{|f(x)| < a}}{2a}$

Well, you started with a divergent integral that needs to be regularized to be made sense of, so it is not a big surprise that you ended up with one. That the singularity at $0$ became even worse and you now have to think hard of how to regularize the resulting monster correctly (@user90189 don't forget the power $n$ to which the Fresnel integral should be raised!) is, of course, bad for the computation but it doesn't necessarily mean that you were doing anything wrong. If you think a bit about $\int e^{i\lambda \|x\|^2}\,dx$, it diverges at infinity quite strongly too.

Dear @fedja, what do you mean by $\int e^{i\lambda|x|^2}\,dx$ diverges at infinity? The function $e^{i\lambda|x|^2}$ has absolute value 1 for real $\lambda$, but do oscillate strongly. And as $\lambda\to\infty$ the integral goes to zero as $1/|\lambda|^{n/2}$. Thank you for your comment.

@user90189 Not so strongly. If you try to integrate over cubes, you are fine, but if you try to integrate over balls, the growth of volume prevails. Why you should prefer one to another in the definition of improper integration is not so clear. Of course, if you make any reasonable sense of it, the scaling is exactly as you said. The problem with divergence the OP has in the end is at $0$, not at $\infty$.

@fedja Why do you say I started with a divergent integral? The first formula I gave for $I$ is (up to some factors) the surface area of a sphere of radius $n$ in $n$ dimensions. It is finite. Only the later formulas diverge. That's why I think I did something wrong. Can you please explain?

@fedja In fact, $I = n^{n/2-1} \pi^n/2 / \Gamma(n/2)$ can be obtained if one uses polar coordinates (see doi.org/10.1093/qmath/12.1.165). Some step of my derivation leads to a fictitious divergence.

Which integral in the linked paper is your integral $I$ corresponding to?

@user587192 Eq. (2.1)

So you want to show that $I=\frac{1}{K}$ where $K$ is defined in (2.6)?

@user587192 Yes. I know how to do it in polar coordinates (see my answer). What I need to understand is what I did wrong if I try to do it in Cartesian coordinates and using the Fourier transform of the Dirac delta function, as I attempted to do in my question.

The function you put into Mathematica is not even absolutely integrable. What command did you use to get that result?

@user587192 Integrate[Exp[I a x^2], {x, -Infinity, Infinity}, Assumptions -> Element[a, Reals]]

@user587192 Absolute integrability is not necessary for integrability

@becko I meant the representation for the $\delta$-function.

@becko can be obtained if one uses polar coordinates (see doi.org/10.1093/qmath/12.1.165). $\int_0^\infty r^{n-1}e^{i\lambda r^2}dr$ diverges at infinity when $n$ is large, doesn't it?

@fedja See math.stackexchange.com/a/3020116/10063