Anyone who has studied Unicode and is honest will admit that UTF-16 was kind of a mistake. It was born from the early assumption that 16 bits would be enough for all of Unicode. Then a hack was invented to extend it to 21 bits using "surrogate pairs". This led to lots of software not working c...
I guess I'm thinking of some very forward-thinking designs from long ago, such as the Unix timekeeping system, which fully supported time zones from the very beginning, unlike every other timekeeping system, even ones invented decades later.
Even UTF-8 doesn't guarantee 1 byte == 1 character, an assumption long-held by most programmers. A lot of code was written assuming direct random access and modification of characters in a string was permitted. Even if the Unicode Consortium had landed on UTF-8 at the outset, it still required all text-handling code to be made UTF-8 clean, which many coders balked at.
While this looks really like a fiasco, I would mention: I've never heard from this 21-bit extension of the UTF-16. The idea looks heavily surreal to me. UTF-16, having practically all the characters, looks to me a pretty nice standard, for supporting random access to files.
@peterh-ReinstateMonica UTF-16 (including the 21-bit extension) or UCS-2 (no 21-bit extension)? UCS-2 does not have practically all the characters. There are 143,859 characters which means over half of them can't fit in 16 bits. And UTF-16 isn't any better at random access than UTF-8 is. (This also means in about 2100 UTF-16 will be obsolete and we'll be using extended UTF-8)
@user253751 Thanks, it is good to know. First I would think that it is meaningless to me (requiring at most utf-8/16 characters to me), but then I think to the USA developers having regularly problems with anything different than ascii-7, thinking likely the same from our chars. So it is good to know :-)
Re: Much later, UTF-8 was invented [...] UTF-8 was invented quite early, during 1992, and presented in 1993 according to Wikipedia. UTF-16 came later, again according to Wikipedia, for Unicode 2.0 in 1996. UTF-8 is the best solution for pretty much all applications, since there is no special treatment of higher-valued characters and no byte-order-mark needed. As a bounus, it's backward-compatible with 7-bit ASCII.: of course there is special treatment of higher-valued bytes, it's just used much more often than surrogates, and more immediately obvious when it' broken...
...being backwards compatible with ASCII is not a bonus, it is the essential feature of UTF-8, which started its life as UTF-FSS, filesystem-safe, with path separators and string terminators being the same in ASCII and UTF-FSS.
When Java came along with its 16-bit characters, people freaked out, especially C programmers. RAM was pretty tight in the days of Java 1.0. 16-bit characters were seen as outrageously wasteful by many. There was even at least one third party attempt at writing a “better” String class which stuck to 8-bit characters. If characters had been 32 bits in size, I think the language would have had a much harder time being adopted than it was already having. As for UTF-8, processing power was also much weaker; UTF-8 encoding/decoding might have been seen as too CPU heavy.
@VGR: Given that Java doesn't allow programmers to directly access the characters within strings, I would think that performance could have been improved by making String an inheritable class with a private constructor, and then having subclasses for byte-backed strings, halfword-backed strings, concatenated strings, or other formats.
@MooingDuck: The vast majority of strings used by most programs won't contain any code points outside the range 0..255. Even if all strings that might contain code points outside the range 0..255 are encoded using two bytes per code point (rather than using UTF-8), performance for most purposes would be far superior to the performance obtained by loading and storing twice as much data as necessary.
We have byte-oriented character sets because we have byte-oriented computers. We have byte-oriented computers because we have byte-oriented character sets.
@another-dave: Multiples of 4 bits were probably useful for processing decimal digits. A 36-bit size was useful for storing six characters or nine digits in each word. The 8-bit character size was the smallest convenient multiple of the four bits needed to hold a decimal digit.
Can anybody please explain why UTF-16 is 'much more complex than UTF-8'? From what I've read, UTF-16 has either 1-word characters or 2-word characters, and that's all, not counting BOM. UTF-8 could have multibyte characters, what seems to be more complex indeed.
@lvd for starters, UTF-16 has 2 different byte orders, requiring a byte order mark, which may or may not be present. UTF-8 does not have this issue. But the main problem with UTF-16 is that hardly any developers understand it completely. Many of them think it's always a 16-byte character. And that assumption is not immediately challenged, because code that makes that assumption will kind of work. That's a recipe for disaster.
@lvd: IDK about complex, but it is more error prone. When support for UTF-8 is broken, it's obvious with the first character outside of ASCII. When support for UTF-16 is broken, ...
Discussion on question by JoelFan: Co…
Discussion on question by JoelFan: Co…