What about personal certificates / Key pairs?

Well, as I say, you have to decide on a common Trusted (Third) Party from which you can reliably get a public key. If you can have this (whether it's an official CA, or your own) you can make the PKI works. Otherwise, you will have problem with MitM attacks.

The problem remains. If the server trusts only some PKI, then that becomes a limiting factor for user adoption. I should have explained that this is for an "open to the public" situation, not in-house corporate where the users and their personal keys can be enforced to a specific PKI service.

This is not a limiting factor. In the case of HTTPS, a website trusts a single CA, while client browsers trust multiple CAs (as you can see if you take time to explore the list of root certificate on your machine). As a server/website, you just have to choose to trust a CA that is already in that list of client-trusted-CAs. Of course if you are running your own CA, you have to make the clients accept it as trusted.

Either there is something huge that I am not aware of (Eg users can with zero or near zero interaction obtain a personal certificate from a CA), or you don't understand my issue.

A user connecting to an TLS enabled website does not need anything (except the root CA certificate embedded in the browser) to validate the authenticity of the website's certificate. Then the client can start a secured communication with the website. Once you have a secure channel, you are free to exchange anything you want (a user key or whatever you need to).

But the problem is that the server doesn't know who is talking to it at that point.

Indeed. To do that you need either a client certificate from the CA (user has to do that himself), or you have to find a way to authenticate a user and give him a key/certificate you generated yourself for him. You can't magically authenticate a random client.

Even if I "authenticate a user" I have no way of knowing whether that user is actually signed in into my web site (connected to my server directly) or is in fact connected to a malicious MITM who forwarded the authentication data "on behalf of the user". So I can generate a key pair for him, or better: provide some way for him to generate the key pair so that only the pubic key need to be transited, but since the server accepts ANY RANDOM CONNECTION, the server won't know whether that key has been compromised or not.

Even if I do make the user sign up for and get a certificate from a "Trusted CA" the "sufficiently motivated" MITM could still intercept that key, register another key for which they will have access to the secret/private half, and continue to intercept all communications, modify the requests and responses, and even impostor the end user.

Well, the idea of using a "Trusted" Third Party is to have this TTP issue certificate only to the real person. If I cannot provide an identification as Mr X, the TTP will not give me a certificate stating I am Mr X. If the CA operates according to this standard, there is no more MitM possible. The owner of the certificate has the only copy of the private key, only he can decrypt message coded with his certificate's public key.

Most CAs accept that, for individual/personal certificates, you are who you say you are. The only thing the cert proves is that the Email address supplied to the CA was in fact used by who-ever operated the system. And that assumes that email was not compromised. CAs make an effort to ensure that a real human is behind the request for a certificate, but CAPCHA is not infallible. During the sign-up process the motivated MITM could substitute the email address from the real user with the one used to obtain the certificate.

You have to find a TTP that does what you need. And with a "motivated MITM" you can basically devise any scenario you want (one that would compromise the CA for example). Security is a trade-off.

Good day. It occurs to me that there may be a simpler way for a MitM to obtain the user's keys, and that would be to convince the user to hand them over. In any case, I have all but decided that since much of this is dependant on the user being authenticated to start, I may just use that channel for authenticating requests.

Verifying request integrity by having the user sign the content does not appear to add to the security beyond what is offered by TLS except that it makes it harder for a MitM if the session is compromised somewhere AFTER the registration process.

The thing is, for client authentication: you either need to see the man yourself and hand him over some keys (or get his keys if it's a public key) or you need to find someone else that does

TLS has the advantage that the client can authenticate to whom he's talking too without too much effort.

If said client register on your site, eventually provides some public key or certificate, you can thereafter authenticate this particular guy later on.

TLS has the disadvantage that the server doesn't have any idea whether the client did authenticate the server or not.

@Johan well, since the client uses the server's public key from the certificate, he has implicitly accepted that this is the server he wants to talk to. You cannot force the user to actually do things :)

How many non security consious users actually check all relevant part of the connection/URL/browser indicators?

The client accepted the public key of whatever he is connected to. The user may or may not check whether that is in fact the server which they are supposed to be connected to.

@Johan Normally, when a problem arise, the browser do issue a warning. Well, most people will dismiss warnings, but you cannot prevent people being careless.

@Johan I agree, you cannot make someone realise they are on gogle.com instead of google.com

But again, how would you do that. There cannot be a security officer behind every computer to police users

The only relevant thing to do is awareness.

I'm still trying to just improve the situation. One idea that I don't yet know whether it will come to any fruition is to have a native app on a mobile device. This app can have access to a user's real IdP details, without any man-in-the-middle because the app is on the same device where the IdP details are located.

Besides the risk of a compromised mobile device, what else do I need to consider?

Will that be an application in some "store"? (like google play?) or do you control the devices?

Cause, native apps still uses the internet. You can program a script that does the same thing

The system I'm working on will have a "published" app, but that is for the end users. This relates to a "management app" which I can possibly not publish. We could require this app to be installed by an agent.

But I don't want to make that a limitation just yet.

A native app can be set to sign with a hard coded key, encrypt the traffic, generate personal keys for the user, securely deliver that key to the right server (encrypted and signed).

Another similar-ish solution may be a Chrome-store app... I don't know enough about Chrome Apps to know whether they are vulnerable to the same risks as a web site.

Yeah, they also can be reverse-engineered to find the key and break into the security. But as I stated earlier, security is a trade-off

At the end, you have to see what are the real risks and see what you are wanting to invest to be protected against them

Yes, I don't know the answer

heading back home. be back in 30 mins~, I'll read any other comments you have in there when I'll be home.