Wolfram Mathematica

« first day (2140 days earlier) ← previous day next day → last day (2345 days later) »

halirutan

5:31 AM

@VitaliyKaurov Hey Vitaliy. Could you ask your colleagues at WRI about the missing talk of TWJ about the new compiler?

7 hours later…

Kuba

12:12 PM

There is a nice question about hashing/strings/encoding etc, which I hastily answered but now the answer is deleted and I'd appreciate an informative answer :)

https://mathematica.stackexchange.com/q/160589/5478

The issue that prompt me to delete my answer was a realization that I had a wrong idea of how to generate list of bytes:

Import[Export["temp.txt", FromCharacterCode[Range[0, 2^16 - 1]]], "Binary"]

without exporting and importing file.

@Szabolcs @halirutan and others, any help?

Vitaliy Kaurov

@halirutan Sure, I'll see what I can do.

Kuba

I also can't find a good reference as why "Unicode" is listed in encoding formats.

Vitaliy Kaurov

12:40 PM

@halirutan response posted.

halirutan

@VitaliyKaurov Hehe :)

Then let there be patience.

Szabolcs

12:56 PM

@halirutan I hope that means that it'll be in 11.3!! (But I'm not too hopeful. It's a big feature that I didn't expect til 12.0.)

@Kuba I believe that is for converting from Unicode code points. UTF-8 would encode a single code point with 1-4 bytes. Thus α can be FromCharacterCode[{945}, "Unicode"] or FromCharacterCode[{206, 177}, "UTF-8"]

Kuba

@Szabolcs I more or less am aware of encoding, strings, text etc but the point above is to get what Import[ line does without using intermediate file. The problem with FromCharacterCode is it already returns code points not bytes.

@Szabolcs ah, sorry, you were referring to the second question

Szabolcs

@Kuba Yes. With Hash, I'm confused ...

Kuba

@Szabolcs so my guess is that what is done during export is what From/ToCharacterCode docs describes, that certain ranges are treated specially. Then encoding is used (which? $CharacterEncoding), end that is it.

Though I'm not sure

Szabolcs

@Kuba Are you on Windows? What's your $CharacterEncoding? I'm on Mac where it's always UTF8

What do you get for Hash["Erős Pál"]? I get 4787872487826373006

Kuba

WindowsANSI so, right, that is not it probably.

@Szabolcs same hash, fortunately

Hash[FromCharacterCode[Range[0, 2^16 - 1]], "SHA"]

800243062733252430858382968136877388101024049505

@Szabolcs the one above is different I think, do you confirm?

Szabolcs

1:16 PM

@Kuba You mean if I get something different? It's the same.

@Kuba I am unable to figure out how to get the same thing with Hash and FileHash for a string when there are non-ASCII characters.

Kuba

string = FromCharacterCode[Range[2^16] - 1];
Export["test.txt", string, "String"];
FileHash["test.txt", "SHA"]
Hash[string, "SHA"]

@Szabolcs But "String" clearly does something behind my back so can't elaborate

Szabolcs

@Kuba "String" will do things such as convert ő to o''

Kuba

@Szabolcs as a result of decoding for >255, or did I get it wrong? Also Hash[Import["test.txt", "String"], "SHA"] returns the same

Szabolcs

@Kuba Aren't there some invalid unicode code points in that range? I am not sure if those are only invalid in UTF-16 encoding or are simply invalid code points.

Sorry, I don't know :-(

Kuba

@Szabolcs no worries, thanks for feedback, at least I know it is not something basic now. And about invalid code points, I have no idea.

I think I will update linked question with all those doubts

Would also be nice if Hash was overloaded for ByteArray

or RawArray but this I'm not experienced with