English Language & Usage: Multi-Layered Discourse Room

Meysam

8:01 PM

Hello

Is this correct? "Let's get over with it"

Cerberus

> Let's get it over with.

That's the normal way to say it.

Meysam

It's like saying: "Let's get it along with" instead of "Let's get along with it"

ninja is it you?

is it just me, or is it really you?

tchrist

@Cerberus Um, that doesn’t mean what you think it does.

Not your fault, though.

Cerberus

@Meysam Umm I guess; but the point is that "let's get it over with" is an idiomatic expression. I don't think you could use a rule to predict or explain it. Or perhaps you could, but I can't think of any.

@tchrist Oh, no?

David Wallace

Tom, can you please explain to all of us why Unicode isn't a way of formatting files? It would be so useful to everyone here.

tchrist

8:07 PM

David, please be nice and stop trolling me.

cornbread ninja

And if you feel like it, tell us how to pull a variable from another .ksh using the dot operator.

tchrist

@cornbreadninja If you mean dot as the source-that-input-file operator, you will get it, but not the way you want it.

Cerberus

@tchrist I open an .ahk file in Notepad++. I tell NP++ to convert the file into UTF-8. Then I save the file.

tchrist

@Cerberus Yes, that’s the right way to say it.

Cerberus

It wouldn't seem unreasonable to say that the file is now formatted in unicode.

tchrist

8:09 PM

Just saying "formatted in Unicode" doesn’t say what its format is.

Cerberus

Formatted, encoded—to a layman, these are very similar words.

"That what"?

David Wallace

Ah, but "Unicode" and "Unicode Transfer Format 8" are not.

tchrist

lame man

David Wallace

Bye all.

tchrist

Unicode != UTF-8 != UTF-16

He isn’t going to be nice, and he isn’t going to stop trolling me.

JSBձոգչ

8:11 PM

Unicode == UTF-32?

tchrist

Asshole.

No, Unicode != UTF-anything.

It’s confusing the concepts.

JSBձոգչ

oh, so you mean that Unicode == UTF-64

Mr. Shiny and New 安宇

What tchrist is trying to say is that "unicode" is not a character encoding of a file.

JSBձոգչ

(i understand what tchrist is trying to say)

cornbread ninja

@tchrist hmm.

Mr. Shiny and New 安宇

8:12 PM

@JSBձոգչ Yes but not everyone else does

Cerberus

Whatever it is, there is something different about the file and it has to do with unicode or no unicode, so it seemed reasonable for me to say it was formatted in some unicodish way.

I did not use "format" in any technical sense, of course, because I wouldn't know how to.

Mr. Shiny and New 安宇

@Cerberus Yes and no. It is reasonable if you do not care to be precise about it.

tchrist

You should just say saved as UTF-8.

It is not possible to save as Unicode.

Cerberus

@Mr.ShinyandNew安宇 I did not care very much at the time, no...should I?

tchrist

Probably.

Mr. Shiny and New 安宇

8:14 PM

@Cerberus Depends. If tchrist is listening, then yes.

Cerberus

Any other reason?

tchrist

A program needs to know what format a file is in.

And so do people.

Mr. Shiny and New 安宇

@Cerberus There are times when the specific choice of encoding might be important, and you might need to specify which encoding you are using that supports, or attempts to support, Unicode.

tchrist

Random stargames?

Cerberus

@Mr.ShinyandNew安宇 Well, in this case, that was obviously not important, or I would have said UTF8 or something.

Mr. Shiny and New 安宇

8:16 PM

@tchrist I'm sure most people who are worried about learning AHK from Cerb understand perfectly well what he means when he says "formatted as unicode". There aren't many interpretations that make sense.

Cerberus

I used code and format as similar concepts.

tchrist

@Cerberus But code and data are different, unless you have a pronounced lisp.

Cerberus

I wouldn't know any official/technical definition of format anyway, except for hard drives.

@tchrist Depends on content.

tchrist

whoosh

Cerberus

Well, I still don't fully comprehend the issue, but OK.

Mr. Shiny and New 安宇

8:19 PM

Unicode

Unicode is a computing industry standard for the consistent encoding, representation and handling of text expressed in most of the world's writing systems. Developed in conjunction with the Universal Character Set standard and published in book form as The Unicode Standard, the latest version of Unicode consists of a repertoire of more than 110,000 characters covering 100 scripts, a set of code charts for visual reference, an encoding methodology and set of standard character encodings, an enumeration of character properties such as upper and lower case, a set of reference data computer ...

Unicode is a set of characters.

tchrist

The Lisp programming language intentionally conflates program data with program code. Or can.

Mr. Shiny and New 安宇

A file is a bunch of bytes.

tchrist

Hence, a pronounced lisp.

Mr. Shiny and New 安宇

An encoding is a mapping of characters to bytes.

Cerberus

But can't "in Unicode" be short for "in a format/code using Unicode or being compatible therewith, as opposed to ANSI"?

Mr. Shiny and New 安宇

8:20 PM

UTF-8 is a popular encoding which can represent Unicode.

tchrist

No.

ANSI is a non-word.

It is just fucked-in-the-head levels of stupid and wrong. Sorry.

Cerberus

Then pick a word.

Mr. Shiny and New 安宇

@Cerberus ANSI, in your case, is (probably) referring to a set of encodings.

tchrist

I don’t know what you mean.

So I cannot pick a word.

Mr. Shiny and New 安宇

And there are several popular encodings for Unicode.

Cerberus

8:21 PM

@Mr.ShinyandNew安宇 No doubt. All I know is that I commonly read about Unicode v. ANSI, whatever they are.

tchrist

No, there is one popular encoding. :)

Cerberus

@tchrist Something not using Unicode or not compatible therewith.

tchrist

@Cerberus You have been poisoned by $Bill.

The popular encoding is UTF-8.

Cerberus

No doubt.

Mr. Shiny and New 安宇

@tchrist If there is only one popular encoding then it is safe to just call it "unicode".

tchrist

8:22 PM

The unpopular encoding is one or another kind of UTF-16.

All others are marginal.

Cerberus

The premise is that the distinction between UTF-8 and UTF-16 and such is not important.

MετάEd

Saying "saved in Unicode" for "saved in UTF-8" is like saying "saved in IBM" for "saved in EBCDIC".

Mr. Shiny and New 安宇

@Cerberus well, it is important, if your program cannot read the input data because it expects one encoding and is fed another.

MετάEd

On the other hand, "saved in EBCDIC" is problematic. But it serves to illustrate.

Cerberus

@MετάEd This doesn't help...

tchrist

8:23 PM

Microsoft has defined Unicode to be UTF-16LE. It is a big fucking problem.

Because it is wrong.

Mr. Shiny and New 安宇

@Cerberus It's like saying "saved in Microsoft" for "saved in MS Word format"

Cerberus

@Mr.ShinyandNew安宇 I declare that it is not important. You have to accept that.

MετάEd

@Cerberus Okay. "Saved in Unicode" for "saved in UTF-8" is like "said in language" for "said in Dutch".

tchrist

And they used to define it to be UCS-2, which introduces a whole nother class of idiocy.

Cerberus

@Mr.ShinyandNew安宇 Well, that's fine, if the obvious choice is between Word and LibreOffice only.

Mr. Shiny and New 安宇

8:24 PM

@Cerberus No, that's just it. There is no obvious choice. Well, there are some more likely choices. But not definitively.

Cerberus

@MετάEd That's fine if there are only two options: in Dutch and in <not language>.

@Mr.ShinyandNew安宇 Even if there were no obvious choice, it is irrelevant here, because the addressee will know what format to pick.

Or so I assumed.

Mr. Shiny and New 安宇

Anyway, the point is, any programmer will know that "saved in unicode" means "saved in an encoding that supports unicode [that my target program understands]" and be able to figure out the rest as needed. A lay-person may not know the second half and may need to be told which specific encoding they need.

@Cerberus Which is why earlier I said it was often only important to be so precise when tchrist is listening.

Cerberus

@Mr.ShinyandNew安宇 And that's exactly what I meant. I felt it wasn't necessary to take laymen into account when addressing Ed.

@Mr.ShinyandNew安宇 Right, then we are agreed.

Mitch

@cornbreadninja depends on the math. what's the question?

Mr. Shiny and New 安宇

@Mitch that conversation went to the jquery room

I can haz jQuery?

Programming chat

Cerberus

8:28 PM

Being imprecise is not something I would see as a problem, especially if I am not sure what the precise option is.

tchrist

Character Set. A collection of elements used to represent textual information.
Character Repertoire. The collection of characters included in a character set.
Charset. (See coded character set.)
Coded Character Set. A character set in which each character is assigned a numeric code point. Frequently abbreviated as character set, charset, or code set; the acrony
m CCS is also used.
Character Encoding Form. Mapping from a character set definition to the actual code units used to represent the data.

Mr. Shiny and New 安宇

@Cerberus ha! I am totally holding you to that.

Cerberus

@Mr.ShinyandNew安宇 That doesn't mean I wasn't being imprecise.

I was.

Mr. Shiny and New 安宇

@Cerberus No, but the next time you whip out some bit of pedantry, I am going to quote you.

Cerberus

And not unwittingly so.

Mr. Shiny and New 安宇

8:30 PM

because this whole conversation is one pedant calling another pedant not pedantic enough.

Mitch

@Mr.ShinyandNew安宇 I don't. They're all unicode related. 'this file is "in" unicode' could mean a bunch of things, but it narrows it down quite a bit away from 'this file is "in" ascii' so it has some informational content.

Cerberus

@Mr.ShinyandNew安宇 Then I will have to argue why precision does matter in that particular case.

@Mr.ShinyandNew安宇 I suppose.

tchrist

Q: What is a UTF?
A: A Unicode transformation format (UTF) is an algorithmic mapping from every Unicode code point (except surrogate code points) to a unique byte sequence. The ISO/IEC 10646 standard uses the term “UCS transformation format” for UTF; the two terms are merely synonyms for the same concept.

Cerberus

@Mitch Yes. You should know that the addressee already knew this was about a plain text file.

Mr. Shiny and New 安宇

@Mitch You don't what? know what tchrist means?

tchrist

8:31 PM

Each UTF is reversible, thus every UTF supports lossless round tripping: mapping from any Unicode coded character sequence S to a sequence of bytes and back will produce S again. To ensure round tripping, a UTF mapping must also map all code points that are not valid Unicode characters to unique byte sequences. These invalid code points are the 66 noncharacters (including FFFE and FFFF), as well as unpaired surrogates.

Mitch

@Mr.ShinyandNew安宇 oh. I'm slow. and anyway I wouldn't know enough about jQuery to say.

Cerberus

But what I have learned today is that UTF-8 is the "normal" format for unicode stuff or dingamabobs, correct?

Mr. Shiny and New 安宇

@Mitch nah, jquery is just the name of the room

the question was akshully about korn

Cerberus

And what is the "normal" version: LE or BE?

Mitch

@Mr.ShinyandNew安宇 right. or at least it's a precision which is below my perception.

Mr. Shiny and New 安宇

8:32 PM

@Cerberus It's a common format

Mitch

@Mr.ShinyandNew安宇 even worse (or even further a lack of knowing) I'd 'man ksh' and that would be all I know.

tchrist

For the formal definition of UTFs see Section 3.9, Unicode Encoding Forms in the Unicode Standard. For more information on encoding forms see UTR #17: Unicode Character Encoding Model.

Mr. Shiny and New 安宇

@Cerberus UTF-8 doesn't have a BE/LE variation.

Cerberus

@Mr.ShinyandNew安宇 Without specification, that's what people would normally usemean, right?

@Mr.ShinyandNew安宇 Oh, OK. Good.

tchrist

Q: Which of the UTFs do I need to support?
A: UTF-8 is most common on the web. UTF-16 is used by Java and Windows. UTF-8 and UTF-32 are used by Linux and various Unix systems. The conversions between all of them are algorithmically based, fast and lossless. This makes it easy to support data input or output in multiple formats, while using a particular UTF for internal storage or processing.

Mr. Shiny and New 安宇

8:34 PM

@Cerberus No, it's hard to say what people would normally use. it depends on the software. Probably that's what they would use.

Cerberus

Right.

Mr. Shiny and New 安宇

@tchrist UTF-16 is used internally by Java. Java doesn't imply anything about external encodings, i.e. encodings that "I need to support"

tchrist

Q: Is the UTF-8 encoding scheme the same irrespective of whether the underlying processor is little endian or big endian?
A: Yes. Since UTF-8 is interpreted as a sequence of bytes, there is no endian problem as there is for encoding forms that use 16-bit or 32-bit code units.

Cerberus

Well, all this unicode stuff will always remain a bit of a mystery to me. It's too complicated, and a couple of tries usually get me the right result.

tchrist

@Mr.ShinyandNew安宇 I am quoting the Unicode FAQ from the UTC. Don’t whine at me. They do know their language.

Mitch

8:36 PM

@Mr.ShinyandNew安宇 that is, (I think that this is @Cerberus's position too) when I hear or use 'uincode' all that it evokes is something like 'kinda like ascii but allows lots more characters like Chinese and Cherokee'. that's all.

Cerberus

@Mitch Hmm I didn't take you for such a layman as myself.

tchrist

No, that would be ISO-10646.

Unicode is not simply more characters.

Mitch

@Cerberus right. if the program doesn't work with 'Unicode' specified' then tell me what it should be, or let me try 'uincode-8' or whatever until it works.

tchrist

Unicode is more characters with a huge whole lot of very fancy rules that go along with those characters, for how to use and not use them.

Unicode is not just more characters.

Mr. Shiny and New 安宇

@tchrist well, it's somewhat irrelevant to worry about java's in-memory representation of characters. Only java developers need to worry about it. actually the default format for reading/writing text files in java is the platform default, which is not any unicode encoding on many systems.

tchrist

8:37 PM

And adding more characters does not suffice to make a program "able to handle Unicode".

Cerberus

@Mitch Haha, exactly what I do.

tchrist

Because it takes rules.

Mitch

@Cerberus with respect to unicode, yes, I am farily unknowledgeable.

tchrist

@Mr.ShinyandNew安宇 Which is one of the most horrible things in the world.

Mr. Shiny and New 安宇

@tchrist what, the default thing? yeah, it's frickin' annoying.

There's lots to hate in java. I use it every day, I know.

Cerberus

8:38 PM

@tchrist But my program has more than 28 characters it uses. So it must be a unicode program.

tchrist

I have not the words to describe how fucked up it is, nor any longer the hair for having once tried.

Mitch

@Mr.ShinyandNew安宇 right. that freaks me out that the java compiler (or is it the jvm) needs to know about unicode. it just seems wrong. but there, that's my lack of understanding.

Cerberus

I looooove Unicode programs!

tchrist

loovers

Mr. Shiny and New 安宇

I have no idea why they didn't make it use unicode by default for its various text-io features. Or better yet, NO default, make the programmer aware of the problem.

Cerberus

8:39 PM

They're like so cool?

Mitch

@tchrist what, you can get longer hair for trying real hard?

Cerberus

Haha.

tchrist

I would rather quit life as a programmer than get buggered by the spiky ramrod from Hell yclept Java. It is the Cobol of our times.

Mr. Shiny and New 安宇

@Mitch wait, what? The JVM doesn't really know unicode. It just uses 2 bytes for its "char" type.

Mitch

@Cerberus "I'm a lover, not a fighter"

tchrist

8:40 PM

@Mr.ShinyandNew安宇 Yet another disaster.

Mr. Shiny and New 安宇

@tchrist now now, it's not so bad.

Cerberus

@Mitch What's that?

Mr. Shiny and New 安宇

@tchrist what should it do?

tchrist

@Mr.ShinyandNew安宇 Have a char type that can hold a character, of course.

Mitch

@tchrist wow. Another metaphor with an uncomfortable image.

MετάEd

8:41 PM

@Cerberus I looked at putting in a utility function. That's getting confusing because I should not call Input more than once. I think what I'm going to need is a state machine or a complex if .. then structure.

Mitch

But luckily -I -can- unread that.

Mr. Shiny and New 安宇

@tchrist And how many bytes should it be?

tchrist

@Mr.ShinyandNew安宇 Wrong question.

@Mitch searches for a certain jpg

Mr. Shiny and New 安宇

@tchrist Oh wise Tom, master of all things unicode, please edit my question into shape then opine upon me!

tchrist

The right answer is that a "char" datatype can hold any valid codepoint.

Mr. Shiny and New 安宇

8:43 PM

@tchrist And when they invented Java, that was true.

tchrist

I once cordially disliked Java, but this was a cordial dislike borne largely out of ignorance. I am no longer ignorant, and I am no longer cordial.

MετάEd

@Cerberus And I can certainly figure out how to do those. But I was curious how to proceed using a function.

tchrist

THAT IS NO FUCKING EXCUSE FOR THE CATASTROPHE!!!!

Fix it.

Just

Fix

It

.

Wake up. It isn’t 1990 anymore, Jdauvmaasses.

Mr. Shiny and New 安宇

Please. It's hardly as bad as you say. The problem is that fixing it requires changes to the specification of the JVM, and to existing programs. And Java has been more conservative than, say, .Net, and is backwards compatible. So they can't change it.

Cerberus

@MετάEd No, you just need to 1.) put all possible input strings as the "matchlist" parameter to the Input command, as one long parameter, but separated by commas;

tchrist

8:46 PM

I have a zero-tolerance policy against technical stupidity that borders on evil and wrong.

Which this is.

Cerberus

2. You need to Put lots of the following afterwards:

tchrist

If you cannot fix it, discard it.

Period.

Because it is too fucked up to live.

I worked in Java for two years. I do know whereof I speak.

Mr. Shiny and New 安宇

My eyes are rolling so hard right now.

s/speak/greatly exaggerate/

tchrist

It is just fucking insanely wrong.

There, that’s a week’s worth of profanity. That is how strongly I feel about the disaster that ruins lives.

Cerberus

@MετάEd StringReplace, OutputVar, InputVar, SearchText, ReplaceText

If Errorlevel = 0

Return

tchrist

8:49 PM

►

Mr. Shiny and New 安宇

Anyway, I'm outta here. @cerb: you could read this for some background on character encodings.

Cerberus

@MετάEd What is a state machine? I actually don't think a function will change much, because you should be able to put all input strings in one Input command. You could replace the three "Replace" lines with a function: that would save two lines apiece.

tchrist

Rob is much nicer than I am in public.

His private words are not that far removed, of course.

And he is right.

Cerberus

@Mr.ShinyandNew安宇 Maybe I will...bye!

Mitch

@Mr.ShinyandNew安宇 well, I hate to repeat this, but I don't know jack, but something like that is what I've heard, for possible a large range for 'like that'.

Cerberus

8:53 PM

@MετάEd Lastly, not all strings may behave as desired, because 1.) some characters may need escaping, and 2.) a shorter string cannot be contained within a longer string. I mean, those things can be done, but then you would need to use Regexreplace and make sure you include and exclude the right things.

tchrist

@Cerberus That is not an uncommon restriction.

Mr. Shiny and New 安宇

@Mitch read the spolsky article I linked for Cerb. it's a good primer.

tchrist

I’m not really that fond of it. It needs updating.

Cerberus

@tchrist I know.

> Toast 2, origin: 1690–1700; figurative use of toast 1 (noun); the name of a lady so honored was said to give flavor to the drink comparable to that given by spiced toast

> Toast 1: sliced bread that has been browned by dry heat.

Oh, oops.

Wrong room.

Mitch

@Cerberus :)

so embarassing.

@Cerberus "that lady is toast"

Cerberus

8:59 PM

cowers in shame

@Mitch Haha.

That means her defences are no match for your assault, doesn't it?

And you will burn her with your love-flame.

I can haz jQuery?

Transcript for

English Language & Usage: Multi-Layer…