Ja1024

@MiniRagnarok: This is exactly your problem: You've become obsessed with this one aspect instead of seeing the bigger picture of information gathering. Trying to find the original intent is one possible approach. One among many. It can be successful and yield a deep understanding of the code. And it can fail. Then you move on. The idea that I somehow forced the OP to find the intent of the code only exists in your head. I've been a developer myself, I know that sometimes nobody remembers what the intent was, or the persons who do aren't reachable. Then you move on. Which you should too.

@MiniRagnarok: I think you still don’t understand that I’ve made multiple suggestions to gather any useful information. I’ve suggested trying to talk the original developer to understand why they’ve chosen this implementation – they didn’t want to talk, fine. Then I’ve suggested getting any info about what the whole point of this project is. That doesn’t mean formal requirements. It can be some vague idea some superior has in their head. This worked out. If it hadn’t, I would have suggested something else. This is very basic information gathering which any developer can and should do.

@MiniRagnarok: When I write an answer, I care about the concrete problem at hand, not the issues of other projects somebody else may have encountered at some point in their career. Yes, there absolutely are projects so dysfunctional that the developers have to reverse-engineer the code to get useful information. Thankfully, there are also projects where you can talk to developers, get documentation and figure out the requirements. The only way to find this out is to ask. It’s a bit of work and not necessarily pleasant, but if you do get the information you’re looking for, it definitely helps.

@MiniRagnarok: Did you miss that the OP was able to get useful information, came to a reasonable conclusion, marked my reply as an answer and thanked everybody for their advice? The only person who is still strangely unhappy with the replies or comments is you. What exactly do you have a problem with? That I told the OP to do some research before deleting code? The whole point of asking a question on this site is to make an informed decision, and I’ve made multiple suggestions to help with that. I'm not quite sure what your goal is.

@MiniRagnarok: [...] the quickest way to figure out a) what the project is all about and b) how the implementation works. In the OP’s case, the prior developer seems uselesss in every aspect, so I agree that asking them won’t help at all. I’m fine with that. This is why I told the OP to instead look for somebody else who knows the requirements. The OP did that, explained this is an internal project they got directly from their boss and gave some background. Great! I believe my comments have been fully addressed by the OP, and I hope the extra information was beneficial to everybody. That's all

@MiniRagnarok: No offense, but if it’s your experience that the work of developers is completely detached from the project requirements, or that the requirements fundamentally change during a project, then you must have worked on some pretty dysfunctional projects. Sure, both can happen. Sometimes developers have no idea what they’re doing, and sometimes customers (or whoever initiated the project) have no clue what they want. But if a company is in good shape and has competent (and friendly) developers, asking them why they made specific implementation choices is often [...]

@MiniRagnarok: In any case: When working on a project with unclear requirements, I would definitely not assume that nobody cares about the project and that each developer can do whatever they feel like (like randomly removing code). Unclear requirements don’t mean no requirements.

@MiniRagnarok: When companies pay a developer to work on a project, they rarely do that just for fun. They want something from the project. Sometimes the requirements are clearly written down, sometimes they’re informal (especially in the case of internal projects). I’ve encountered all of this. My point is: Unless the company is entirely dysfunctional and pays its employees to do nothing, somebody knows what the project is for. The OP already found that person (their boss), so I’m not sure what you’re arguing for or against.

@MiniRagnarok: Who cares about the prior developer? I said the OP needs to understand the requirements and the implementation before throwing code out. Do you disagree? Talking to prior developers is one option, but if the devs are uncooperative, that's also fine. However, there should be somebody who knows the requirements – in the case of the OP's, this is an internal project by their employer, so they do have somebody to ask. If you're working on a project where nobody knows why the project even exists, there's something very wrong.

@MFSCraft: Are you working on the code as part of your job? Is this a project for some customer? As there's apparently money involved, I'm pretty sure this isn't just a hobby project. Then there should be a contract with requirements which say what the code is supposed to do. And there should be somebody you can ask about this – like your boss. If the requirement is “The traffic must be encrypted in transit” and nothing else, then suggest using TLS and getting rid of the useless keys. If there are other requirements, you need to find out what those are.

@MFSCraft: If even unauthenticated clients can get the keys, or if anybody can register and then get the keys, they're indeed useless. Still, if there's any way you can talk to the original developers or obtain design documents, figure out what the original goal was and why this implementation was chosen. Was this about traffic encryption? Or encryption at rest? Maybe something else entirely?

@ThoriumBR: Work experience in general IT or IT security can help assess risks (although a decade isn’t all that much), but it doesn’t make you infallible.

@ThoriumBR: It’s ridiculous to assume that everybody who argues against you or downvoted your answer is just inexperienced or stupid, while you have some exclusive insights into IT security. Multiple people have explained exactly what the practical risks are. Instead of addressing this, you just keep repeating that they’re irrelevant, or you point to other risks (as if that nullifies the problem of sharing a password). Have you considered that maybe your current answer really is poor?

@Joshua: If you don't have a local JVM for testing, run this code online. If you get an exception complaining about incompatible encodings (or "types", as you call it), I'm thoroughly impressed.

@Joshua: You're confused. There's just one string type in Java, and it's called String. You can create strings from raw byte sequences and an encoding argument. However, this encoding information isn't retained, so when Java concatenates two strings, it has no idea what the initial encodings were. It cannot throw an error (and it most definitely doesn't throw a type error). Just -- try -- it. There are languages like Ruby which keep the original encoding and will trigger an error for incompatible encodings, but that doesn't apply to Java (and an encoding error isn't a type error).

@Joshua: You can concatenate UTF-8 and UTF-16 in Java. Just try it. I'm pretty sure that Java doesn't even retain any information about the encoding of a string, in which case the check you describe is impossible.

@Joshua: Consider the following case: The database connection and all queries are UTF-8-encoded. However, the home-made SQL escape/filter function incorrectly treats the user input as UTF-16 data. If an attacker provides the input 0x2027, this is perfectly valid in both UTF-8 and UTF-16. However, the same byte sequence is interpreted in two different ways: In UTF-8, it’s a space followed by a single quote (which can be used for injections). In UTF-16, it’s a harmless hyphenation point which has no special meaning in SQL.

@Joshua: There are multiple Unicode encodings, and, no they aren't all compatible with each other in how they represent ASCII characters.

@Joshua: You’re missing the point. The GBK example is just the tip of the iceberg. There are other encodings which are also incompatible in terms of how they represent a single quote (like UTF-8 and UTF-16). And telling people to just use one encoding (like UTF-8) consistently is a bit like saying “just don’t make a mistake”. Developers do screw up encodings. The actual solution to the problem is to either use prepared statement (which doesnt depend on the right encoding )or the escape functions provided by the database library (which takes the encoding into account).

@Jasen: Do you understand that there are more character encodings than UTF-8? Just look at the list I linked to. And, no, 0x27 isn't a single quote in every encoding.

@Jasen: In case my point wasn't clear: Yes, PostgreSQL by itself will absolutely handle encodings correctly. But once you start inventing your own filters outside of PostgreSQL (as the OP originally considered), then you're completely on your own.

@Jasen: There are many more character encodings available in PostgreSQL, including problematic ones like GBK. In an ideal world (from a security perspective), everybody would consistently use the same character encoding like UTF-8. But in reality, encoding mismatches do happen. In many cases, it only leads to harmless bugs (gibberish characters etc.). But there's no guarantee, and I don't think the OP should try it out.

@user13964273: There are a lot more character encodings than ASCII, and, no, they aren't all compatible with ASCII.

@AkiT: You would have to escape the entire input, not just individual characters. But the preferred method is to use prepared statements with, for example, the JdbcTemplate#query(String sql, PreparedStatementSetter pss, ...) methods. Yes, I understand it's slightly inconvenient to build the query template dynamically and pass the parameters in a loop. But security should be more important than developer convenience.

@AkiT: As some of the commenters have already said, you should explain which database system and programming language you use. Then I'm pretty sure we can suggest a proper solution.

@AkiT: If the restriction only exists in the frontend, then an attacker can simply bypass it. You need to address this in the backend by either using prepared statements (which is by far the most secure option) or the encoding-aware escape functions provided by the database library. Trying to prevent users from entering single quotes is nonsensical. And you still have the problem that you might screw up the character encoding.

@oleg_zh: I'm glad you finally realize that the probability of every(!) password x in a uniform distribution is p(x) = 1/52^10. Now all you have to do is insert this exact value into your own definition of entropy. Don't give us a lecture, just insert the value and calculate the result: For the summands, you get p(x) log p(x) = 1/52^10 log 1/52^10 = -1/52^10 * 10 log 52. Agreed? Now calculate the sum: There are 52^10 possible passwords, so you get 52^10 * (-1/52^10) * 10 log 52 = -10 log 52. The total result is H(X) = -(-10 log 52) = 10 log 52 ~ 57 bits. That's it.

@oleg_zh: But if they were sampled from the gigantic set of Unicode characters, the probability is obviously much, much smaller (something like 1/150,000²). Since the probability cannot be 1/3² and 1/150,000² at the same time, that should tell you it depends on the distribution. In the case of passwords, the distribution must be assumed by the attacker based on available information. So, yes, it does matter what the attacker knows about the password.

@oleg_zh: Taking an individual password and then claiming probabilities for individual characters is complete nonsense, because (1) probabilities are assigned to the whole password and (2) you have no clue what the probabilities are when you don’t know how the password was generated. What’s the probability of the password 'äü'? If you’ve come up with a concrete number, you’re already wrong, because the result depends entirely on the probability distribution. If the characters were independently sampled from the small set of German umlauts äöü, then we’re talking about a probability of 1/3².

@oleg_zh: This is basic math and has nothing to do with entropy yet. Now insert p(x) = 1/52^10 into your own formula and calculate the result. Try it. You don’t need an online calculator for that, just simplify the terms. If you’ve been successful, you’ll end up with H(X) = 10 ld(52) bits, which is roughly 57 bits. Does this look familiar? Hint: It’s the result I’ve been telling you the whole time.

@oleg_zh: You clearly have no idea how probabilities, probability distributions and entropy work. When you consider 10-character passwords, then the X in the Shannon formula is a distribution which assigns probabilities to each 10-character password. Not individual characters. There's no such thing as “p('s')”, and whatever value you've assigned to that is pure fantasy. You can calculate the probability of entire passwords, e.g., p('myPassword). The value of this does depend on the distribution assumed by the attacker. In a uniform distribution, the probability is 1/52^10.

@oleg_zh: Of course you can help the attacker and give them more information about your password (for whatever reason). For example, you may tell them the fifth and sixth character is actually the same. This leads to a reduced entropy of (l – 1) ld(n). Or you tell them the password is constructed of two words from a specific dictionary – this again leads to a different entropy result. What you don’t seem to understand is that the entropy calculations discussed here refer to a password pattern, not an individual password. If the exact password is known, the entropy is zero bits.

@oleg_zh: You’re hopelessly confused. An attacker doesn’t know the password in advance – that’s the whole point. They don’t know if and where characters are repeated, or whether a passphrase was constructed by simply repeating a short string. The attacker can only make assumptions about how the password/passphrase is constructed and then calculate the entropy based on those assumptions. For example, they may assume that there are l independently sampled characters from an alphabet of size n. This leads to an entropy of l ld(n). Calculate it yourself or look it up. It’s really simple.

@oleg_zh: I'm not mixing up anything. I'm using the standard definition of Shannon entropy which is the formula you wrote down yourself. This exact formula can be simplified to ld(n) given a discrete uniform distribution with n possible values. You actually seemed to agree with me in your first comment where you correctly stated the entropy of a single character in the [a-zA-Z] range as ld(52). You can easily extend this to words of length l using the same formula, which results in l * ld(n).

@oleg_zh: That's the definition, but in the case of independently chosen characters from an alphabet of size n, you can simplify this to H(X) = -n * (1/n) * log(1/n) = log(n). For a password of length l, the total entropy is l * log(n).

@oleg_zh: You said the attackers only knows the password has 10 characters from the alphabet a-zA-Z. This leads to an entropy of 57 bits. If the attacker has more information (like duplicate characters), then we're talking about a different password pattern with a different entropy.

Your entropy calculations are off. Given 52 possible choices (a-z, A-Z), the entropy per character is ld(52) ~ 5.7 bits. For 10 characters, the entropy is 57 bits.

@Falco: Since the cache is limited, the attacker could fill it with irrelevant data to reduce (or even eliminate) the chance of running into a cached real account. At the same time, they can login multiple times with completely arbitrary names to make sure the dummy account is in the cache. The combination of both should allow them to reliably distinguish between (non-cached) real accounts and the (cached) dummy account.

@BoppreH: And if you think the username doesn‘t lead to the e-mail address: Are you sure you haven‘t used any of the e-mail addresses that can be found with a simple Google search using your name? Did you really create a completely random address just for this site?

@BoppreH: I know that. I‘m talking about the public username, display name or whatever you want to call it. My point is: You‘ve chosen this name yourself. You‘ve apparently decided to reuse it on several different platforms, which reveals information about you (including multiple e-mail addresses). This isn‘t a privacy or security issue of the Stack Exchange site. It‘s your own decision. So your whole argument that websites shouldn‘t allow users to choose an arbitrary username (because they might make bad decisions) is defeated by your own choices. You clearly did not use a random name here.

@Falco: You're forgetting about the query cache (and many other factors). If there's a fixed dummy account which is used whenever there's no real user account, then the database system will cache this data and not do any database lookups at all. So you haven't fixed the timing issue. Of course you can now create multiple dummy accounts and select one randomly. But I hope you realize this is a never-ending cat-and-mouse game.

The whole username hiding smells of security through obscurity. Instead of taking care of the password as the actual secret, you’re trying to hide information like usernames, hoping this will somehow make attacks harder.

@marcelm: Did you even read the answer? The idea that you can just hide all usernames is highly dubious. Complex applications leak timing information all over the place. People also leak information, either purposely by linking to websites they’re registered at, or unwillingly by reusing names or choosing obvious ones. Do you really believe that attackers who routinely find passwords (i.e., the one thing everybody agrees should be secret) will have any trouble figuring out usernames? By the way, your own username is visible for everybody here. Are you going to claim a security incident now?

@BoppreH: I’m generally not convinced that you can force users to make good decisions. Passwords are an excellent example: For decades, people have tried to “forbid” weak passwords with all kinds of rules and policies, but weak passwords are still all over the place. What seems much more promising is to encourage the use of password managers or passwordless authentication methods like WebAuthn. So instead of restricting usernames, I’d (1) stress that the name is public (2) let users choose whatever name they want (3) point out that password managers make it easy to securely store credentials.

@BoppreH: You have a strange understanding of privacy if you think website owners should decide what users can and cannot do with their own data. Like I said, you’ve chosen to reuse your name on many different sites – who is to say this is a bad decision? Would you really like the Stack Exchange administrators to intervene and tell you to change your name for the sake of privacy?

@BoppreH: Third, it‘s naive to assume that password reuse isn‘t a problem (or is less of a problem) as long as the username stays hidden – if that‘s even possible. Once a password is leaked, attackers can try it on any account and any service they want. Sure, there‘s some extra work if the attacker cannot simply match a leaked username to a username within the target service, but that won‘t prevent them from looking for reused passwords across all accounts. It‘s not like they will give up just because a user is called “foo” in one service and “bar” in another.

@BoppreH: Secondly, if somebody willingly puts sensitive information into their public username, that‘s not a privacy problem – it‘s their personal choice. Just like it was your choice to reuse your Stack Exchange name on GitHub, your personal website and whatnot. Are you saying you should not have been allowed to do that? Letting users pick their own name (as opposed to using the e-mail address, for example) means they can decide how much or how little they reveal. If they want to use a completely random name, they can do that.

@BoppreH: Your objections don‘t make much sense. First, if you agree with me that it‘s very difficult (or even completely impractical) to hide the usernames, then there‘s no way around the fact that usernames can be found out, no matter if we like it or not. So what are you suggesting? To actually use long random numbers as usernames? Good luck trying that. For many people, even just remembering their own passwords is a major challenge – not everybody uses a password manager. If you additionally force them to deal with some random username, this can quickly turn into a usability nightmare.

@kaya3: The implementation is described right in the question, not somewhere in the comments. Did you even read the question?

@kaya3: The OP has already explained how the comparison is actually implemented: by encrypting the old passwords. I don't care how it could have been done when you're willing to spend hours of computing time or how it was done back in the days or how Facebook did it 10 years ago. I'm interested in facts, not speculations. If you want to do a brainstorming session on how a similarity check could be implemented, this is the wrong question, possibly even the wrong site.