last day (14 days later) » 

1:51 AM
3
A: Zipfile character encoding challenge

Y Treehugger CymruI have just tested this for myself, using the link you provided, and downloaded a file entitled Thank You For You Project.zip, which I had to unzip using passcode 864 with 7-zip, and encountered no problems at all. Did I have the right file? ...and before someone decides to p*ss all over my repu...

 
I tried 7z l archive.zip, 7za l archive.zip, and 7z x archive.zip, and I still get the garbled filename Figure_<U+0082>ป<U+0082>ฬ‘ผ.fx using 7-Zip [64] 16.02.
 
I could send you the un-archived files, if you want? - I'm guessing that I will not be able to send them via superuser.com without countless thousands of reputation or something; but private message an email address instead and I will sort it out now for you. I notice, too, that you are using the command-line version of 7-zip, so this may be a factor, too, as the command-line and GUI versions may be functionally different.
Further thought: I have just been looking at the commands for 7-zip (command-line) and cannot see any special functionality for encoding; so I am wondering if, possibly, the archive is working for me as a result of additional encoding pages in my operating system (Windows 7, x64)? ...so, with this in mind, have you tried adding further encoding pages / languages to your system, and then unzipping the archive after a reboot? The link given by JakeGould is also very informative.
 
Thanks, but I have the files. Finding out the real filename did take a lot of work, partly because what looked like four non-printable bytes was actually six (this isn't dirt on your screen! <82><BB><82>̑<BC>), but now I have it. This question is about whether unzip is working correctly.
 
No problem, but I am still inclined to think that there may be an issue with the encoding pages for your system: do the file names display correctly when transferred to a different machine or system? Also, maybe this will answer some of your questions: unicode.org/charts
 
Are you sure the specific file's name was decoded correctly? It reads 'Figure_その他.fx'? There's definitely a lot I don't know about encoding, but the mystery here is that it seems like there's two filenames in the zip file (using two different encodings), and unzip is using the bad one.
 
1:51 AM
The original archive was "Thank You For You Project.zip", which unzipped to directory "Thank You For You Project". Inside that directory are two sub-directories: "DT Ext Loreley Meiko" and "DT Ext Orbit Miku". I have had a quick look at the coding on the .zip file and the directory names are correct. I also remember using tools with similar functionality to "gucharmap", but none of them worked after Windows 98... =[
 
Most of the filenames are in ascii. Do you have the file 'DT Ext Loreley Meiko/Normalmap Effect/Skin/Figure_その他.fx' or does it use other characters? (I deleted my comment about character map because it seemed offtopic)
 
Is that how it is supposed to read?! =] - No, I have something 'orrible that turns into blocks when I try to post it in this comment box; but even the characters that post correctly do not look like your characters. - Give me a couple of minutes whilst I PM you an image.
 
Ok, so 7zip doesn't do better. Check the screenshot; I'm guessing it looks like the part that looks like [...]w.fx. I do wonder if I should post this on (or move to?) another Stackexchange site. This question is a bit more in-depth than most ones about decompressing. The linked question has nothing new, except some incorrect advice about what unzip -a does. I also think not being able to comment if you don't have reputation is silly, so I upvoted your answer even though it doesn't answer the question.
 
No problem, - if you do not feel that I can help you then fair enough (no offense taken). The image should be viewable at: [2a00:b900:10a4:1::a]/figure.png . The IPv6 address needs to be in brackets when entered into a browser, but I am not certain whether the /figure.png does or not (currently having some fun getting my website working properly).
The name displays correctly, though, only not in ASCII. - The characters are familiar, but I am certain that they are not Katakana.
 
Thanks for the effort. Something is wrong with my IPv6 connectivity and it appears beyond my ability to troubleshoot it. This is where a character map comes in handy though! I highlighted the filenames in the screenshot in the question; the second one contains Thai script, and I'm guessing it's similar to what you see.
 
1:51 AM
No problem, - got it in one: Thai script, - I knew those characters were familiar. Right, in unicode the filename reads: 0E1A 0E2C ' 0E1C . fx - unicode.org/charts/PDF/U0E00.pdf . BTW: that apostrophe is like the start curly single quote (as opposed to the end one).
 
Still ignoring the message about chat: usually if characters display uniquely instead of question marks, but are still wrong, it's because a program tried to convert them from some encoding to Unicode, which is what Linux uses and might be what some forms of Windows use. If the bits remain unconverted and a program tries to read them as if they were Unicode, that's when errors and question marks appear. In this case, I don't know what the wrong conversion steps were; helpful site doesn't help in this case.
So I'm just ignoring the problem of how the Thai script happened; I'm more concerned about the part of the zip file that reads 'Figure_����.fx'.
 
No, not ignoring it: I have now moved over to chat =] ...
Have you looked inside the .fx file?
It is not corrupted (or, at least, mine is not), and the script would appear to be a mix of English and Katakana. - Could you not infer the file name from the script?
 
Yes, the contents are decoded correctly in every case. (Which means if it's a text file saved as Shift-JIS, then it can appear garbled, but I figured out that I can get my text editor to open them correctly.)
Like I said in the question, the correct filename is 'Figure_その他.fx'; but since it isn't in that text file, it took some work to find it
 
OK, so the issue is still one of you want some way in which to extract the files with correct naming?
I am not sure what to suggest, there, because that is dependent on the archivers and the operating system fully supporting whatever encoding is being used in the archive, so, outside of the good advice being given in some of the posts that have been listed, I do not see what more can be done to achieve that.
 
Not really relevant, but I 1) searched for strings from the text file, 2) eventually found ux.getuploader.com/BeamMan2/download/10, 3) did some searches including for "float SpecularPow = 1" , which found the right filename but I assumed it was probably wrong because I still thought it was 4 bytes long
4) wrote the relevant part of the zip file to a new blank file, then used a hex editor to isolate it so iconv wouldn't give errors while also doing 5) figure out why the "DT Ext Loreley Meiko2.emd" file with question marks couldn't be pasted from clipboard, then convert it to get the same filename as from #4
No, I want to see if it's a bug that unzip can't extract it, or possibly the lack of a feature
I was thinking of contacting whoever maintains unzip for Linux, but I wasn't sure if there was another answer like "just use 7zip"
 
2:02 AM
Fair enough, and good on you, - I hope that you can come up with something that works. - You will probably still receive an answer directing you to use something else, no matter what, but that's not to stop you forking unzip into your own version, though...
 
I guess that's sort of right though. If the part of the zip file that lists the 'correct' name (in non-UTF8 format) is actually read by an unzipping program, then extracting files with that name would be a goal
 
I guess, then, that you would be using unicode, seeing as that would keep the naming consistent right across the board and in all languages...
 
yes
I don't know if Japanese versions of Windows use Unicode, just that most or all .zip files that come from Japan use Shift-jis for filenames
 
...so ideally you would want to create a zip / unzip fork (if no interest from the maintainer) that would enforce unicode conversion during the zipping ...and then absolutely no problems unzipping. It should be simple enough to map the Shift-jis to the relevant unicode, though.
 
So I don't know if they're actually saved to disk as Shift-JIS, or if Windows is automatically converting them to Shift-JIS for all programs using the 'locale' setting, because programs were built to assume this would continue to work. But Linux uses UTF8
Keeping it as Shift-JIS would be fine, sort of; the problem is that the 'wrong' filename isn't in Shift-JIS at all
 
2:11 AM
I remember Katakana is one of those awkward character sets, where trying a one-fits-all approach does not work very well...
 
It was valuable to get your input that it doesn't work better on Windows. I still wonder how it was named on the computer of the person who created the zip file
 
An actual name ...or slang, perhaps?
...or just an abbreviated form / shorthand for something?
 
Slang would still be with the same character set. The mystery is that the 'right' filename is there in the zip file, represented by the question marks, but unzip doesn't use it and apparently neither does 7zip
 
Pattern matching that failed, perhaps?
 
I would hope pattern matching isn't used for that! I linked Wikipedia for en.wikipedia.org/wiki/Zip_(file_format)#Local_file_header though I don't know if that's what the data structure is. Probably that or "Central directory file header"
 
2:19 AM
Not sure myself, but it is late where I am and I need to get some sleep. I do hope that you manage to solve this, and would be glad to help further if I can. - It does not seem to be possible to PM on this site, so just comment on your original post again if you need to.
 
Yes, thank you
The last time I had a complex question, it was answered after about a day
Good night
 
G'day for now.
 

last day (14 days later) »