Christian Clason

Happy to have cleared things up :)

And you may be missing that for actually computing a solution (e.g., via Cholesky decomposition or CG method), the positive definiteness of the stiffness matrix is very much necessary.

Yes. You're missing that somebody else did not know that (that the stiffness matrix is positive definite because the bilinear form is elliptic). So I explained it. If you already knew that, that's great; then the answer isn't addressed to you.

Yes. And how do we know that? That's what I explain in my answer (to somebody else's question). I still have no idea what your question is.

And basis functions can never be zero, because then they wouldn't be a basis.

To show that a matrix $A$ is positive definite, we only have to check that $v^TAv >0$ for all non-zero vectors $v$ (since for $v=0$, we obviously have $v^TAv =0$ for any $A$, so that doesn't tell us anything about $A$). We are talking about properties of the stiffness matrix here, not of the (discrete) solution.

Yes. The linked question was specifically about the stiffness matrix, and in my answer I (only) explained how the positive definiteness of the matrix derives from the ellipticity of the bilinear form. (If the bilinear form is not elliptic, the stiffness matrix is not positive definite unless you make careful modifications.)

A norm is always strictly positive for a non-zero vector; that is one of its defining properties.

Can you edit your question to make it clear what you're actually asking (now that the issue of the typo has been cleared up)?

Just as a general comment: if you copy something verbatim from somewhere else, you should always mark it clearly as a quote. It's not so important here, but it's good to get into the habit of good academic practice.

I've fixed that in the linked answer; thank you for pointing it out!

There is indeed a typo: the final claim should be "for all $v_h \in V_h\setminus\{0\}$" (recall the definition of positive definiteness).

@TobiasKildetoft That's why there is a difference between an en-dash (two-person name) and a hyphen (one-person name). Thus, Birch--Swinnerton-Dyer (of conjecture fame).

You're presenting a false dichotomy -- there's the third option of publishing your work by both putting the manuscript on arXiv and submitting it to a journal. (To what extent this is possible, is of course field-dependent; in mathematics, this is usually no problem -- some journals even allow you to specify an arXiv identifier instead of uploading a PDF when submitting.)

Ah, no MathJax in Chat. That's unfortunate.

It's irrelevant. Global optimality means that for all $x$, we have $$f(\bar x)+g(\bar x) \leq f(x) + g(x).$$

OK, let's start from the beginning. Let $f$ be nonconvex, and $g$ differentiable. We also have a global minimizer $\bar x$ of $J:=f+g$. That's the setting, right?

This is probably quicker. Also, I again apologize for nitpicking -- I'm not trying to show that you're wrong.

Yes, it does, because then $-g'(x^{k+1}) \in \partial f(x^{k+1})$. You are completely right that for non-convex functions, the sum rule doesn't hold in general, but in this specific case, it actually does. It bears repeating: Nothing in the definition of the subdifferential requires convexity; it's just that for non-convex functions, the subdifferential will be empty in general and hence useless.

That is not quite true: The set $\partial f$ always exists, but it might be empty unless $f$ is convex. But if $g$ is differentiable, you can directly derive the sum rule for $\partial f + g'$ (actually, in this case, $g'$ is linear and hence (presumably) $g$ is convex), and from this and global optimality, you get the non-emptiness in this case. (If $x^{k+1}$ is a local minimizer, all bets are off.)

Sorry to nitpick, but actually, if $x^{k+1}$ is the global minimizer of $x\mapsto L(x,z^k,y^k)$, then this equation does hold even in the absence of convexity, since then $0\in \partial L(x^{k+1},z^k,y^k)$ as discussed above, and if $g$ is differentiable (as in this case), you don't need the Hahn-Banach machinery (which does require convexity) to derive the sum rule $\partial(f+g)(x^{k+1}) = \partial f(x^{k+1}) + g'(x^{k+1})$, it's enough to work with the definition of directional derivatives (of $g$).

@Lucia There's a crucial difference between this question and the one you linked to: The former specifically and explicitly asks about the mathematics (in particular, topology) behind the physics, and also provides context to argue that it's on-topic (student question, not idle curiosity). Correspondingly, it received an (excellent) mathematical, not physical, answer.

Alright, I'll write an answer to that effect (feel free to edit it) and clean up the comment thread. That was a nice chat, and very helpful, thank you!

Ignoring the whole Bayesian issue (which I'm starting to think is completely irrelevant here), the sensible answer might be: "If in doubt, always make sure your discrepancy and regularization term are discretization-independent, so you can change that without having to fiddle with the parameters again."

It's actually not clear from the question exactly what his problem is (or whether there is in fact a problem); the way it's written, it sounds more like a philosophical point...

Hard to guess where the OP is coming from, so I'd not be so sure. Maybe I can summarize my point (less confrontationally) as "If he's doing proper Bayesian modeling, the scaling should already be correct; otherwise his model is wrong (for example, based on an improper discretization of the likelihood) and he should fix that. If not, all bets are off anyway, and he shouldn't feel bad about scaling the terms to his liking."

Also, there's been some recent work showing that the MAP is actually a proper Bayes estimate, if one considers the right error metric.

And I completely agree with your point about the pdf. If one isn't interested in higher-order moments (or cannot at least use the Bayesian interpretation to model their parameters as well -- e.g., using a hyper-prior), the Bayesian framework is really pointless, and one shouldn't feel restricted by it.

(Not saying you're wrong, of course; just explaining where I come from. This has been a useful lesson in how backgrounds and expectations differ.)

OK, I think I see the issue here. I don't really care about the whole theory of Bayesian inference in statistics, only about Bayesian inverse problems (in mathematics). In this context (and when only considering MAP estimators), the only difference to classical regularization (and hence what really matters) is the modeling step.

About that link: These are statisticians, for whom "Bayesian" is a philosophical position (as opposed to "frequentist") -- not dissimilar between (say) realists and formalists in mathematics. (Maybe constructivists would be a better alternative, since those actually disagree on practical issues.)

Or even better (and less contentious): Bayesian inference includes the modeling step.

I think my point can be summed up as "if there's no statistical basis, then it's not Bayesian" :)

Also, there's a difference between fixing a class of priors -- say, normal / L^2-based -- and also fixing the statistics -- say, the variance. For example, you can do the first, and then use statistical modeling for the variance (using a hyper-prior). That would also be within the framework of Bayesian inference.

Since the internet is notoriously bad at conveying tone, let me make clear that a) I don't think we fundamentally disagree; I just want to try to hone the point I'm trying to make and b) as someone whose research field is inverse problems, I've seen way too many works on "Bayesian" inverse problems that weren't, so this is a bit of a pet peeve.

Sure, and today Tikhonov regularization includes arbitrary norms and even distance functionals (what used to be called "generalized Tikhonov regularization in the eighties and nineties).

Yes, but what are you basing your pick on? Is it -- exaggerating -- trial and error, or is there some statistical basis? Only the latter is proper Bayesian inference, the former is just a fancy new name for what people have always done.

Also, I can be very blunt here: My point is that Bayesian inference is what gives you the choice of the prior. If you pick the prior yourself, you're not doing Bayesian inference, you're just applying new buzzwords to classical (generalized) Tikhonov regularization.

Let's hash this out here, and I'll summarize the conclusion as an answer.

@Kirill: If the choice of prior is free, then it's not really Bayesian, and he can do whatever he wants :) If it isn't free, then the values of $\alpha$ and $\sigma$ are fixed by the modeling, and if it is done correctly, should already include the correct scaling. So there's nothing to be done. (This is getting too long already; I'll stop here, and if needed consolidate the comments into an answer.)

See siltanen-research.net/LassasSaksmanSiltanen.pdf for some discussion on this (in a slightly different context), as well as the works by Georg Stadler.

Moral: Set up your likelihood and prior in a way that they are discretization invariant. If it arises from the discretization of a PDE, use the function space norms (e.g. $L^2(\Omega)$) and discretize those (depending on your discretization, you'll end up with something like $(m-m_p)^TM_h(m-m_p)$, where $M_h$ is a mass matrix, which takes care of the scaling). Otherwise you can use weighted $l^2$ norms (simply divide the sum of squares by the number of terms). This will take care of the scaling without changing the interpretation of the parameters. (The values might still change.)

@Kirill If you are only concerned about the numerical computation of the MAP estimate, you are right -- the value of $\alpha$ has no inherent meaning apart from a weighting. If you consider the whole Bayesian setup, this is different -- the whole point of the Bayesian formulation is to give you a fixed scaling (which of course depends on the modeling of likelihood and prior). Any re-scaling afterwards breaks this interpretation, making the Bayesian setup moot.

@Kirill The regularization parameter is never arbitrary. One of the advantages often given for the Bayesian approach (which, if you are only interested in the MAP estimate, is completely equivalent to standard Tikhonov regularization) is that there's a clear, objective, statistical interpretation of this parameter. (Of course, for the purpose of minimization, only the ratio $\sigma/\alpha$ matters.)

If you can't decide, then any eigenvector is as good for you as any other, and you can just take Matlab's and don't worry about it :)

No, there are infinitely many: if $x$ is an eigenvector corresponding to an eigenvalue $\lambda$, then so is $c x$ for any $c\in \mathbb{R}$.

That depends on the constraints; in the question, you are just writing that they are opposite. How would you know which of the two (yours and Matlab's) are the "right" ones?

In your first approach, you are constructing a specific set of eigenvectors. On the other hand, Matlab doesn't know anything about the ellipses. So you just get what you asked for: a set of eigenvectors. If you need a specific eigenvector, you need to somehow include these constraints.

They're not really a new basis; just a rescaling of the same basis; they span exactly the same subspace (which is all eigenvectors care about).

Both are correct eigenvectors. If you want to have (say) $u_{11}\geq 0$, just multiply $u_1$ by $\mathsf{sign}(u_{11})$.