« first day (1303 days earlier)      last day (3916 days later) » 

15:00
Oh, thanks. Just a min.
The value you hope to see is "AL32UTF8"
@Mr.ShinyandNew安宇 If the data is in Western European, or if it is a web page even in an Asian script, UTF-8 wins.
Table does not exist.
If (and only if) the data is in purely Asian script, not with XML/HTML/SGML markup, then UTF-16 wins.
If you are a programmer, once you get it inside, you probably want it converted to UTF-32/UCS-4 for convenience’ sake. But sometimes not.
15:01
@tchrist What do you mean? We are storing our data in UTF-8, but the max-length in the db is expressed in bytes. So we need some way to tell the user how much more they can type.
@Mr.ShinyandNew安宇 You can’t know until they type it.
@tchrist exactly.
@KitFox SELECT * FROM NLS_DATABASE_PARAMETERS
Worst case is bytes = 4 * UTF-8 chars, which is virtually impossible to ever actually occur.
NLS_NCHAR_CHARACTERSET AL16UTF16
NLS_CHARACTERSET WE8MSWIN1252
Well.
Wait.
15:04
1252
Those are different.
isn't on the code page and the other the encoding
so it's fine
oh
You can map 1252 to UTF16, but you can only go back the other way if you started with 1252, because UTF16 has byzintillions more characters than 1252 has.
I guess not
@KitFox That's not UTF-8.
15:05
That’s sad.
But UTF-16 is also variable width, one notes.
However, all 1252 codepoints map to exactly 2 bytes in UTF16.
Well, legal ones.
There are illegal 1252 code points.
The map is not full.
I wonder what kind of Indians we're talking here?
UTF16 does not per se specify such.
And I’d hate to think of BOMming every field.
so, @Kit, you are using a non-unicode character encoding for VARCHAR2. It is one-byte-per-char
We ate MS Win 1252. I think they just missed out an h
but there are empty regions devoted to characters unique to charsets
What do you mean? The holes in the 1252 map?
wait, i confused myself. ignore me today, i'm not all here
15:08
@Mr.ShinyandNew安宇 No, it's not.
Also if means if you put UTF-8 data in there, strange things will happen. For one thing, you won't be able to query the data properly. For another thing, you will run into strange encoding issues if you ever have two different clients connected to the DB and one follows the rules and one doesn't.
cries
These are the cp1252 holes:
cp1252       81  ⇒  U+FFFD  < � >  \N{REPLACEMENT CHARACTER}
cp1252       8D  ⇒  U+FFFD  < � >  \N{REPLACEMENT CHARACTER}
cp1252       8F  ⇒  U+FFFD  < � >  \N{REPLACEMENT CHARACTER}
cp1252       90  ⇒  U+FFFD  < � >  \N{REPLACEMENT CHARACTER}
cp1252       9D  ⇒  U+FFFD  < � >  \N{REPLACEMENT CHARACTER}
It's not my database.
It's a horrible piece of work.
Those 5 code points are illegal in cp1252, and thus have no Unicode mapping.
I don’t know why they exist.
I just know that they do.
15:10
There. I've said it. I didn't want to bad-mouth my developer—my friend—but this work is crap.
Anyway, it's nothing I have to do anything about. I'll just advocate for client-side control of the string lengths and be done with it.
@KitFox It will be fine as long as you don't try to put in any characters that aren't in Windows-1252 code page.
@Mr.ShinyandNew安宇 And what are the chances of that, really.
@MattЭллен Not counting the 5 holes, it has 27 code points that do not map to Latin-1.
Whereas MacRoman has 48.
It's annoying that she just moved it to a new server, but we didn't update it at all.
@KitFox That depends on the person entering the data.
I have had people where one uses a Mac to enter stuff with and another uses Windows, and both encodings show up.
15:14
We do have Mac and PC users entering data.
And for some reason, everyone here thinks doing stuff manually is really important.
And that visitors to the site want to search through pages and pages of text to find the thing they are looking for.
And also that web visitors really want to read paragraphs of text.
@Mr.ShinyandNew安宇 I’ve been out of the web world too long to remember whether there is some elegant solution to this. I am used to specifying the page’s charset on output, but I don’t know what to do about specifying or at least learning the input charset they use to fire back in the POST data.
"Click here for a pdf of this report." Me: Um. You know you could reduce that to an icon.
Most stuff I use now seems to accept UTF-8 input, but I don’t actually know “why”.
15:17
Inbred Strain;
Additional information on Inbred Strains.
Why do we do stuff like this?
@tchrist The browser sends the character encoding in the request. You can force a certain encoding in a form attribute. On the server side, you need to handle the possibility of wrong data.
In 11 pt font, on a page where you are absolutely required to scroll down.
@Mr.ShinyandNew安宇 So the charset is part of the HTTP POST response, but may be wrong?
@tchrist No, I mean it may be something other than what you want.
You can't really control what a browser will send you.
But can you get it to tell you what it sent you?
15:19
But usually browsers default to iso-8859-1 or to whatever the page's encoding is.
I am complaining here.
Now, this is a different matter.
pouts
@tchrist It sends the encoding with the request, otherwise you have to assume it's iso-8859-1
The web standard says that if you do not specify an encoding, it is ISO-8859-1.
jinx
I just don’t remember ever coding to detect the post encoding and decode/convert as appropriate.
15:21
But I could make a site that serves iso-8859-2 pages and creates forms with iso-8859-2 forms and some browser sends me a UTF-8 POST. I can't stop that from happening. I should attempt to do something intelligent with the request. like reject it, or translate it
That’s what I was wondering whether happened.
When they send you the UTF8 POST, does the HTTP header admit that, or not? I bet not.
@tchrist if it doesn't, then it's an iso-8859-1 post.
With (potentially) corrupt data.
I see.
Ok, so what is happening “here”?
I'm using curly quotes. What is SE receiving?
macbook# byte2uni -e MacRoman | grep QUOT | grep DOUB
MacRoman     C7  ⇒  U+00AB  < « >  \N{LEFT-POINTING DOUBLE ANGLE QUOTATION MARK}
MacRoman     C8  ⇒  U+00BB  < » >  \N{RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK}
MacRoman     D2  ⇒  U+201C  < “ >  \N{LEFT DOUBLE QUOTATION MARK}
MacRoman     D3  ⇒  U+201D  < ” >  \N{RIGHT DOUBLE QUOTATION MARK}
MacRoman     E3  ⇒  U+201E  < „ >  \N{DOUBLE LOW-9 QUOTATION MARK}
macbook# byte2uni -e CP1252 | grep QUOT | grep DOUB
cp1252       84  ⇒  U+201E  < „ >  \N{DOUBLE LOW-9 QUOTATION MARK}
15:25
Content-Type: application/json; charset=utf-8
That's what my browser POSTed to SE when I typed that horse
Ahah.
So I am typing the Mac escape that makes a “ and a ”, but it sends not 0xD2 and 0xD3 but U+201C and U+201D encoded as UTF-8?
Wouldn’t that solve @Kit’s dual-input-system issue, too?
The only valid character set for "HTML" is Unicode. But on transfer over HTTP, or storage to disk, etc, any encoding is possible.
@tchrist Not really.
How come?
Not enough control?
15:28
In your browser, the HTML and JS are running in a unicode world. But to transfer the data to the server, some encoding must be specified. The default is (oddly) iso-8859-1.
@Mr.ShinyandNew安宇 I’m not sure I understand that statement.
Hmm, room title should be changed to "Programming Languages and Usage" :P
I know when they did the SIP protocol, they explicitly used the HTTP protocol as a base but assumed UTF-8 not ISO-8859-1 by default. I think XML may default to UTF-8, also. A default of ISO-8859-1 is really not a good idea.
The browser takes the form data and does its best to encode that data in the requested encoding. So if you have Chinese text in a text field, but the form is set up to send iso-8859-1, who knows what it will send? I've seen it send html escapes: &nnnnn; style
But even so, our data entry people kept sending us CP1252 and MacRoman in the SIP data.
It’s usually caused by when you do customer data entry for businesses.
You have no control over the upstream form data entry folks.
15:31
The server gets the HTTP request and has to parse it using the encoding specified (or the default). Then it's up to the server to deal with those bytes/chars. I don't know what server Kit's using; various platforms do it better or worse. In Java, it gets automatically parsed and then stored in memory using UCS-2, which is at least some kind of unicode.
But then when you want to store it in the database, you need a database connection and those are all vendor-specific. You pass a string in and the combination of your application platform and your DB library and the DB itself conspire to store the data, as bytes, on a disk somewhere.
By using "unicode" throughout, you can ensure that everything works.
But Kit's VARCHAR2 columns are Windows codepage 1252.
So like let’s just hypothetically consider a UK customer named Prêt-à-Manger. Someone entered that customer name in the database, and that name is sent for SIP transactions. But whether they entered it in CP-1252 or MacRoman or UTF-8 is like playing Unicode Roulette.
So I'm not sure what will happen if she tries to store, say, Chinese in there.
@Mr.ShinyandNew安宇 That would be a problem.
With long text, you can virtually always figure out the encoding. But not for short snippets.
I mean, programmatically/deterministically.
@tchrist I'm not familiar with SIP. but with regular web sites, the browser takes the inputted characters from the OS and enters them into the DOM and does whatever tricks it needs to to handle all of unicode, because "HTML" is unicode.
I don't remember what server it was moved to. It was on a Sun system, moved to a Linux system, I think.
15:34
Then it transfers the data to the server using some encoding.
If you have some application that builds on top of HTTP and demands a particular encoding, it is probably a bug in the client software if it sends a different encoding.
If they enter on a Mac, and it just sticks them in as is without converting to 1252, you will have trouble.
Three of the five "illegal" 1252 code points turn out to be not to rare in MacRoman:
macbook# byte2uni -e cp1252 | grep REPL cp1252 81 ⇒ U+FFFD < � > \N{REPLACEMENT CHARACTER}
cp1252 8D ⇒ U+FFFD < � > \N{REPLACEMENT CHARACTER}
cp1252 8F ⇒ U+FFFD < � > \N{REPLACEMENT CHARACTER}
cp1252 90 ⇒ U+FFFD < � > \N{REPLACEMENT CHARACTER}
cp1252 9D ⇒ U+FFFD < � > \N{REPLACEMENT CHARACTER}
macbook# byte2uni -e MacRoman | egrep -w '8[1DF]|9[0D]'
We wouldn't use those, probably.
The worst ones are when they differ, though.
@tchrist For websites, the OS's native encoding is never an issue, because the website can control the page's encoding. I don't know about SIP clients. Maybe you have some that are broken.
@Mr.ShinyandNew安宇 It’s the low-level telecom wholesaler operators that are the problem.
15:38
The best thing for Kit to do is to convert the entire stack from 1252 to UTF-8.
morning :)
> These mutant mice may be used as a Cre-reporter strain; to test the tissue/cellular expression pattern of cre transgenic mice.
See what I have to deal with?
@Mr.ShinyandNew安宇 I have no control over that.
@tchrist according to RFC 3261 you can have non-UTF-8 requests in SIP, if they provide the character set in a header.
Eek!!!
I missed that!
I coded it to assume UTF-8.
@KitFox Without using unicode everywhere, you can't ensure that any characters not in 1252 will work. The results are basically undefined.
15:44
The problem is, by the time it gets to us, some feeds did not have the full SIP headers, they had been condensed.
@tchrist We now come to the conclusing of Mr. Shiny and New's SIP knowledge. If you enjoyed this broadcast, please join our pledge drive and become a member.
@Mr.ShinyandNew安宇 I still have no control over it.
@KitFox I'm just pointing it out. It is a major limitation and there is only one solution. But until you fully unicodeize everything, some illegal stuff may appear to work in some cases and data might get stored in an illegal way. Then when you convert later, you'll find that some of your data is actually garbage. And there will be no way to tell the difference.
I have good news and bad news.
@Mr.ShinyandNew安宇 Well, it's really the least of our problems.
@tchrist What is it?
15:48
@KitFox yay! hidden data-corruption is the least of your problems! ;)
The good news is that CP1252 and MacRoman both represent the same 231/256 possible code points.
The bad news is that 124 of them are in different positions.
We have so much duplication and crappy code and rickety data structures and poor practices...
@tchrist Ugh.
@KitFox Just fix things up a bit at a time.
Well, that would be nice except it's not up to me.
I just have to design around it.
For example, the EN DASH is the most common non-ASCII character used in English-language papers submitted to PubMed and to Elsevier.
But watch:
cp1252 96  ⇒  U+2013 <–> \N{EN DASH}                     MacRoman 96  ⇒  U+00F1 <ñ> \N{LATIN SMALL LETTER N WITH TILDE}
cp1252 D0  ⇔  U+00D0 <Ð> \N{LATIN CAPITAL LETTER ETH}    MacRoman D0  ⇒  U+2013 <–> \N{EN DASH}
15:51
Which is also funny because I'm not a designer either.
It's 96 in cp1252 and D0 in MacRoman.
That's why you'll go insane if you don't upconvert everything to Unicode.
In fact, MacRoman and cp1252 share only 5 different 8-bit code points in common.
It's not so bad, as long as any particular bit of text has a known, correct, tag identifying its encoding.
@tchrist Because why wouldn't they.
@Mr.ShinyandNew安宇 True.
cp1252 A2  ⇔  U+00A2 <¢> \N{CENT SIGN}            MacRoman A2  ⇔  U+00A2 <¢> \N{CENT SIGN}
cp1252 A3  ⇔  U+00A3 <£> \N{POUND SIGN}           MacRoman A3  ⇔  U+00A3 <£> \N{POUND SIGN}
cp1252 A9  ⇔  U+00A9 <©> \N{COPYRIGHT SIGN}       MacRoman A9  ⇔  U+00A9 <©> \N{COPYRIGHT SIGN}
cp1252 B1  ⇔  U+00B1 <±> \N{PLUS-MINUS SIGN}      MacRoman B1  ⇔  U+00B1 <±> \N{PLUS-MINUS SIGN}
cp1252 B5  ⇔  U+00B5 <µ> \N{MICRO SIGN}           MacRoman B5  ⇔  U+00B5 <µ> \N{MICRO SIGN}
Those are the overlappers.
So in text with all 7-bit ASCII plus only those 8-bit code points, it is indeterminable -- and immaterial -- which of the two encodings it was in.
15:54
I recently had a great issue with encoding, and transferring text across the Java/Ruby boundary (using jruby).
Ick.
Ruby is troublesome about that.
Java can do it right, if you’re not uncareful.
The beauty of it is: Ruby doesn't use just one encoding internally (I learned).
That is not beauty.
It is sin.
And I have a talk about it.
But Java DOES use just one encoding. So WTF doesn't jRuby just use Unicode?
I mean, it should be completely transparent that every fucking string coming from Java and passed into any jruby anything anywhere is unicode.
It cannot be anything else.
Because it was written by Japanese people who wanted to be able to have rare non-Unicode characters.
It is a super asspain.
I know I know I know.
15:57
And. And And. And the fucking HTML parser I was using was defaulting to US_ASCII encoding.
@tchrist That's Ruby's excuse. jRuby is a Java Ruby. It has no excuse for not "Just Working"(tm)
I have to run, but here for @Kit's benefit is the full 8-bit mapping for 8-bit data between CP1252 and MacRoman. It should tell her why not unifying on Unicode will hurt hurt hurt.
cp1252 80  ⇒  U+20AC <€> \N{EURO SIGN}                              MacRoman 80  ⇒  U+00C4 <Ä> \N{LATIN CAPITAL LETTER A WITH DIAERESIS}
cp1252 81  ⇒  U+FFFD <�> \N{REPLACEMENT CHARACTER}                  MacRoman 81  ⇒  U+00C5 <Å> \N{LATIN CAPITAL LETTER A WITH RING ABOVE}
cp1252 82  ⇒  U+201A <‚> \N{SINGLE LOW-9 QUOTATION MARK}            MacRoman 82  ⇒  U+00C7 <Ç> \N{LATIN CAPITAL LETTER C WITH CEDILLA}
cp1252 83  ⇒  U+0192 <ƒ> \N{LATIN SMALL LETTER F WITH HOOK}         MacRoman 83  ⇒  U+00C9 <É> \N{LATIN CAPITAL LETTER E WITH ACUTE}
@cornbreadninja麵包忍者 you have ?1
@Mr.ShinyandNew安宇 Oh, I missed it was jRuby, not Ruby. That’s sick!
15:59
@tchrist yeah. Oh, and the problem only exists if Java's default code page is UTF-8.
If Java's default is cp-1252, it works fine.
W. T. F.
To be honest, the MacRoman set is the more useful of the two.
But they should all be normalized out of their legacy encodings into Unicode.
@tchrist It's not up to me!
I'm heading out for a while. ttyl
@KitFox I’m so sorry. They should be storing stuff in UTF-8.
Me too, bye.
Bye!
16:06
Thanks! @Mr.Shiny
All I did was copy the link. Hmm.
Great, I've killed the chat. goes back to watching Hulu
posted on June 06, 2014 by sgdi

I’m not gonna sell you a sonnet That you can display on your bonnet I’ve got fifty verses To write on to hearses I’ve a soliloquy if you want it

I think it's more that everyone left, not that you killed chat.
time for me to go!
16:23
@MattЭллен No!
Oh! Hi.
16:33
@KitFox hello.
Is it me you're looking for?
^_^
Are you somewhere feeling lonely, or is someone loving you?
16:48
Both, kind of.
Imaginary lover?
You're mine, anytime.
Aww.
She's an easy lover.
I'm trying not to be too disappointed that my friend hasn't read my story yet.
Also, I'm trying not to freak out about how much stuff I have to do right now.
Oh! I was going to ask you if you finished your Nose piece yet!
I haven't started.
I mean, I had notes from before I thought I would do it for this series.
but nothing new
Are you going to write something?
No, 'cause he wanted it like, today.
16:54
I know you can't for the series, but maybe write something anyway?
@cornbreadninja麵包忍者 Well, just prior to the day he would publish it.
I started something in that vein a few weeks ago.
Oh, but you're new, so he might have wanted more lead time.
@AwalGarg Command !!norris' does not exist.
@<> "!!norris"
@KitFox can you tell @KitSox to go to sandy?
What's the room number?
16:57
1
@cornbreadninja麵包忍者 Registered; need 0 more to execute
!!summon 1
Did she go? I was distracted.
She did.
@AwalGarg oh, duh.
16:58
And she is behaving weirdly there...
@Kit may I read your story?
@cornbreadninja麵包忍者 Which one?
@KitFox the one your friend hasn't.
But yes, to all of them.
Oh, it's Spider. I think you've read it already, haven't you?
I have. I rather liked it.
17:13
Me too. It's one of my favorites.
I wish the damn laundry would finish so I could put it away and leave for a while.
I thought she'd like it too, but she's also really busy.
So she probably hasn't had time.
But also, it's not like she said she would read it. I just asked her to.
17:32
I think my new maxi shrunk enough that I won't have to hem it!
raises eyebrow
!!define maxi
@KitFox maxi having a hemline at ankle length
I am listening to The One Lilium asmr videos, getting my tingle on and feeling very, very lesbian today.
I mean, I can't help it. She totally wants me.
17:39
raises eyebrow
I get my tingle on with Elvis Costello via earbuds.
Oh. I have a dress like that. To me, maxi always meant menstrual pad.
I like the pattern on that.
It's my first.
I should have said maxi dress.
Yay!
It's good to know that's in fashion. I like that style.
Is she better than most?
@KitFox they're on sale now for $25.
Oh, you mean ear tingle.
17:43
I do mostly get ear tingles, although I realized last week sometime that I get a kind of eye tingle too.
And sometimes tingles make me sneeze.
I thought you meant tingles in your southern hemisphere.
Oh.
Oh.
Like when I see the Southern Cross for the first time.
Elvis Costello, huh? makes note
My god.
Might not be your thing, I dunno.
Um...what's a good length for a blurb?
Three lines of text at 12pt.
ish.
Lorem ipsum dolor sit amet et cetera?
17:47
How many characters do you suppose that is?
@cornbreadninja麵包忍者 Cute
Is that @Mr.Shiny's kid?
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc at ullamcorper tellus, vel iaculis ligula. Aenean massa magna, sodales vel eleifend id, aliquam ac mauris. Donec dolor risus, aliquet sed faucibus at, egestas convallis ipsum. Sed in dapibus enim. Curabitur convallis, tortor in varius pretium, massa erat cursus urna, sed malesuada purus libero pellentesque felis. Donec tellus quam, porttitor vel mi ornare, aliquet consectetur lorem.
@KitFox I hope so.
@cornbreadninja麵包忍者 Stop that!!
May 27 at 1:01, by Cerberus
@ThatBrazilianGuy Speak proper Latin in this room please!
17:50
It's OK.
Hey, I was just trying to help Kit.
Just don't let it rehappen.
Thanks.
250 characters.
Sheesh!
Ima go to the grocery store.
There.
Some of these are good.
Mostly I like his accent.

« first day (1303 days earlier)      last day (3916 days later) »