/dev/chat

8:20 AM

@maulinglawns I think his post was intended as "philosophical". He wasn't expressing dissatisfaction or disaffection. Maybe he's a little burned out. But I know how he feels. There comes a time when you think - there's got to be more to sitting in front of a computer all day long.

Unless you are doing something really interesting with it, which isn't the case most of the time.

@maulinglawns With which part, exactly? :-)

@maulinglawns Ah. Ok.

He doesn't use the words "burned out", though.

That's from reading between the lines.

@maulinglawns Hmm. Sometimes I find coding relaxing too (or I used to).

I even used to like writing C++ for a while. Then I came to my senses.

It's kind of like that passage in the Phantom Tollbooth. When the demon of something or other sets Milo and his companions a completely pointless task to do.

Some kind person has the text (presumably illegally) online at butterfluff1066.tripod.com/toll/phantomtollbooth.htm

Ch 17:

""Quite correct!" he shrieked triumphantly. "I am the Terrible Trivium, demon of petty tasks and worthless jobs, ogre of wasted effort, and monster of habit.""

That sums up writing C++ in a nutshell.

@maulinglawns Placating a C++ compiler is kind of like living in a fascist dictatorship. One can get used to it.

@maulinglawns Humans are ok, in carefully selected and small doses.

Michael Felt

11:19 AM

Hello all - yesterday I asked a question that was characterized as too broad. Can happen - learning. However, as I am new - "Unix and Linux" and Unix and Linux - Meta" are these the same SE forums (my head is spinning somewhat with the large spread of different topics - to the point of being lost - next will be the frustration of not being able to comment over and over again).

So, to not waste a question "Are 'Unix and Linux' and 'Unix and Linux Meta' the different names for the same site? - :) Certainly not about *nix - so 'off topic' again

terdon

@MichaelFelt Hey, hi! Glad you came in here.

So, no, meta is where we discuss how the site works. For example, the question you just asked would have been on topic on meta.

If you have a bug report, or an issue about something that happened on the site, or an idea about how to improve the site or somehow change it, or a complaint, or a question about how the this place works and why it's so confusing: that belongs on meta.

Actual questions about Unix and Linux belong on the main site.

Meta: meta.unix.stackexchange.com

Main: Unix & Linux

Now, your question was closed probably because it, well, rambled a bit. We tend to like to keep things streamlined here. We keep chit-chat to a minimum, even to the point of removing greetings and salutations.

Everything in a question should be actually relevant to understanding the question. If it isn't, it's considered noise.

We only have two types of posts: questions and answers. No discussion.

A question needs to be about a real, specific, technical problem/query that is related to the site's subject matter (the Unix and Unix-like operating systems).

Posting a self answered question is great! It is very much encouraged.

However, it needs to be done in such a way as to fit the rules of the site. Specifically, it needs to be asked as though you are an actual user experiencing the problem. So, instead of "I've often wondered about this and have done these various things in the past, so I thought I'd just post a nice how-to about installing MINIX" you could ask "How can I install MINIX"?

This way, it will be easier to find if someone else has the same issue, it fits with the format of the site and you can post your how-to as an answer.

And please feel very free to come badger me (I'm a moderator here; you can tell by the ♦ after my name where I appear on the site and by the fact that my name appears in blue here, in chat) and ask me about anything you don't get. You can also ask such things on meta, as explained above.

Welcome aboard, @MichaelFelt!

Michael Felt

11:36 AM

thanks for the greeting. And yes, You are probably quite correct about the rambling part. I tend to give background because I believe in teaching to fish, rather than give a fish.

terdon

And there is often a place for that in the answer. Far less so in the question.

I was thinking of editing it but I'm afraid I know too little about MINIX to know what is actually useful and relevant and what can be safely removed.

Michael Felt

the motivation for "my question" was someone asking about a compiler flag to support large_files. The answer to that is not simple. I was working on a framework aka question to naswer a larger question: HOW to port OSS to AIX.

terdon

OK. Great!

If that can actually be made into an answerable question, please do post it. It sounds like it'd be very useful.

Michael Felt

That was about AIX, not MINIX - although I was one of the students who 'proofed' minix, and worked on the compiler kit

terdon

Can you actually provide an answer to as broad a question as "how can I port OSS to AIX"? Wouldn't that depend on the specific software in question?

I assume you mean open source software, by the way, not the oss sound daemon.

Michael Felt

11:41 AM

What I need to cut is "my dispair" with mixed run-time-environments. And yes, OSS software - should I use FOSS instead for non-ambuigity?

sorry: ambiguity

as far as "port OpenSourceSoftware" to AIX - there is an 'opinion' based part - and 'facts' to base, rather choose one side or the other. So, the one liner question becomes something like: "Howto port OpenSource using mkinstallp?" except Howto is more a tutorial format - so again - "How can I use mkinstallp to package software on AIX?"

Is that a question that would fit well?

i.e., forgo the discussion about using RPM as a packager manager/packager

bbl - thanks for the feedback!

ojs

I for one would say that "How can I use mkinstallp to package software on AIX" would be more on topic here. It has a much more precise question then your original one.

Perhaps it would be better to ask how to use mkinstallp to make a package for a specific OSS software, just pick anything that fancies you. Then the answer could give a specific solution and also a general solution.

Michael Felt

before I go shopping -

Junaga

and for all you kids out there, this was the 1990s internet, when ‘the internet’ was mostly email, usenet and FTP, and you accessed it over a dial-up modem that cost 2p a minute and got yelled at by your parents when you stayed online all night downloading all six megabytes of the Quake shareware release…

holy shit

Michael Felt

@Junaga - that was the 1980's internet and uucp, when 20MByte was enough to store everything from USENET for two weeks !

Junaga

what is usenet anyways?

terdon

11:54 AM

@MichaelFelt No, not at all. That's what I thought, I was just checking.

ojs

:-)

terdon

@MichaelFelt Yes, that sounds fine.

Michael Felt

Usenet

Usenet is a worldwide distributed discussion system available on computers. It was developed from the general-purpose UUCP dial-up network architecture. Tom Truscott and Jim Ellis conceived the idea in 1979, and it was established in 1980. Users read and post messages (called articles or posts, and collectively termed news) to one or more categories, known as newsgroups. Usenet resembles a bulletin board system (BBS) in many respects and is the precursor to Internet forums that are widely used today. Usenet can be superficially regarded as a hybrid between email and web forums. Discussions are...

terdon

And what @FaheemMitha said.

Michael Felt

mkinstallp - by itself, imho, is a horrid program. I have spent time to write scripts to automate the hard parts.

terdon

11:56 AM

@Junaga Oh yes. I remember that. I also remember painstakingly building a collection of 1.4G of music (that was huge at the time) over a period of a year and a half, downloaded over a 14.4K modem and then deleting it all when installing a new Linux and formatting the wrong drive.

Good times. . .

Michael Felt

So, my personal issue with asking and answering the mkinstallp question is that I would be promoting my scripts - too much. And I fear that may violate other guidelines/expected behavior

off to shop - bbl

ojs

@Junaga guess your too young also then to recognize the sound of a dial up modem :-)

Junaga

@terdon damn that's sad, I am glad I still have all my music x)(even back from when I used windows, Heay it's not my fault alright. If you are 10 years old and someone sits you in front of a desktop. You don't ask what os this is)

ojs

youtube.com/watch?v=gsNaR6FRuO0 just so you know how we had it in our youths :-)

Junaga

@ojs I think I know what you mean it's this weird peeping that . That sounds like a dieing bug

It’s not 1996 any more. It’s not the year of the Linux desktop. It’s never gonna be the year of the Linux desktop. It’s never even going to be the year of the desktop ever again.

Now it’s just what I do…because it’s what I do.

I feel sad and sorry.

user334283

12:04 PM

I'm having problems with installing backports in kali-linux

Junaga

>kali-linux
oh boy

terdon

@ojs Oh man. . . I used to love that sound. It was the sound of a door opening and a new world spreading out before me.

ojs

@user334283 please post a question then to the main site, and remember to formulate it as well as possible and include what commands you have tried and what errors you get from them.

Faheem Mitha

@terdon You're chatty today. Slow day at work? :-)

terdon

Not really, actually :) But Michael came in with a valid site question so I was doing mah mod business.

Junaga

12:10 PM

@terdon where do you actually work If I may ask?

terdon

@Junaga saphetor.com

Junaga

>making better diagnoses and selecting optimal therapies.
for what? cancer?

terdon

Among other things. Basically, medical doctors don't know squat about genetics and can't use these new tools we now have like fully sequenced genomes. That's where we come in. We do the analysis and provide the doctor with information they can understand.

Braiam

Anyone on firefox can visit this? blogs.unity3d.com/en/2015/08/26/…

Junaga

works

>Unity Comes to Linux
another step into making Linux an os for gaming I love it x)
But I don't think I've ever played a unity games for longer then 15mins (trying it out)

Braiam

12:16 PM

so, it crashes firefox...

installing -dbgsym's again, le sigh

Junaga

>tools we now have
Oh I know about watson how much a thing is that actually?
Can it replace my doctor with just an app?

:33685354

terdon

@Junaga No! Good grief no! Nor will it ever.

We're simply talking about genetic disorders (most health issues have little or nothing to do with your genome) and how to diagnose them.

Basically, the doctor will come to us and say "Look, this patient of mine has something wrong with them, I asked the lab to sequence their genome and they gave me 40GB of text data. What the hell am I supposed to do with that?"

And we analyze that data and extract the regions of the patient's genome where there are differences from the "standard" genome, see whether any of those are known to be linked to disease and present the doctor with a web-based GUI whether they can filter these "variants" to find the one(s) most likely to be causing their patient's illness.

More or less .

Junaga

Oh, Yeah that is a problem. If he can't read the data it's like there was no test made at all.

>they gave me 40GB of text data
how does such a text look like? :D
do you maybe have an example link, I am very curious

Faheem Mitha

@terdon Do you do any validation of the results?

terdon

FASTQ format

FASTQ format is a text-based format for storing both a biological sequence (usually nucleotide sequence) and its corresponding quality scores. Both the sequence letter and quality score are each encoded with a single ASCII character for brevity. It was originally developed at the Wellcome Trust Sanger Institute to bundle a FASTA sequence and its quality data, but has recently become the de facto standard for storing the output of high-throughput sequencing instruments such as the Illumina Genome Analyzer. == Format == A FASTQ file normally uses four lines per sequence. Line 1 begins with a '@'...

@FaheemMitha Sigh. Not really, unfortunately. We can't. We validate our methods, of course, using known data but when it comes to actual real-life analyses there's no way to validate since there's no way of knowing the truth set. We're generating the truth set.

No way around this though.

Faheem Mitha

12:29 PM

@terdon It's probably possible to do some kind of statistical validation with large enough data sets. If I understand what you do correctly. Checking for genes that are indicators of disease? Something like that?

terdon

@FaheemMitha Not really, no. We take a real genome, yours for example, and try to locate regions where it differs from the "standard". The only way to validate would be to then go and re-sequence your genome using what's known as the Sanger method. And sure, that's been done (as I said, we validate our methods) but it can't be done for every analysis we perform. That defeats the purpose of running such analyses.

Next generation sequencing methods, as they're called, which are what we use can sequence a whole genome in hours as opposed to months (Sanger) and ~1000$ or less as opposed to several dozen thousand.

That's precisely why they're used. Even though they're not quite as 100% reliable as Sanger sequencing.

Faheem Mitha

I meant, if you have a large enough data set, then you could check if disease profiles vary in agreement with known disease causing genes. That sort of thing.

Of course, you'd need quite a large collection to do anything. That's usually where these things fall down.

But I don't know if I'm expressing myself clearly.

terdon

@FaheemMitha Not really, no. We're not looking for genes (there's no such thing as a disease causing gene, anyway) we're looking for variants. Places where your sequence (your gene, perhaps) is different from the norm.

But yes, the general statistical validations have been done. We do the best we can do: we use high quality, manually curated data sets and run our tools on them and then see how well we do. Sensitivity/Specificity etc.

Faheem Mitha

@terdon well, disease markers, if you prefer. I know there are things like that.

I'm terrible at the biology lingo.

terdon

Yes, but those markers are precisely what we're looking for. So we can't use their presence to evaluate a method designed to detect their presence :)

Faheem Mitha

12:37 PM

I guess I came to it too late.

Junaga

>we're looking for variants. Places where your sequence (your gene, perhaps) is different from the norm.
and how would you fix/kill the differing ones?

terdon

@Junaga 1) We don't, that's the doctor's job. 2) Doctor's can't either. They use that information to find out what disease the patient has and then prescribe the right medication.

We don't make drugs that can fix DNA errors yet though. Not really. There are some gene therapies available but they're still in their infancy.

Usually it's more like "Oh, so you have a variant that makes you incapable of producing X? Ok, take a pill of X every day to supplement your body's insufficient production"

Faheem Mitha

@terdon I don't see how that's even possible.

terdon

Not easily, no

Junaga

>take a pill of X every day
Hm, that doesn't really sound like 21th century style.

terdon

12:44 PM

Sorry, work stuff, but look up gene therapy, there should be a wikipedia article about it

Junaga

good luck :D

Faheem Mitha

Interesting. I'm surprised that's even an option conceptually.

"even conceptually an option", perhaps.

What about new cells?

terdon

Oh conceptually, there's all sorts of cool stuff. We're actually not too far from being able to generate synthetic viruses that excise and replace a specific region of your DNA, thereby removing the mutation. Not too far. Not too close either, but I wouldn't rule out seeing it in my lifetime.

Faheem Mitha

Aren't there places in the body where new cells are generated dynamically on a "as needed" basis?

E.g. white blood cells - immune system stuff. Also RBC.

terdon

Sure. With very few exceptions all cells replicate and then die, new ones replacing them.

However, if you change the genome of the parent cell, the daughter cell will inherit the changes.

Faheem Mitha

12:50 PM

@terdon Oh. I thought there were more levels of indirection involved. But my knowledge of biology is extremely sketchy.

I've tried to learn it. It didn't take. Sometimes I find stuff hard to learn, for no particular reason. Perhaps my brain has ossified.

Junaga

Well, I know absolutely nothing about biology x)
does someone maybe have a read recommendation?

or I guess wikipedia is fine.

terdon

@FaheemMitha That's how I always felt about math :)

That's why us bio types collaborate with you math people.

@Junaga Start with wikipedia, yes. Ask me if you need more direction (I'm actually a biologist, all evidence to the contrary notwithstanding) or check out Biology.

Junaga

wow is there a stack site for everything?

Does DNA have the equivalent of IF-statements, WHILE loops, or function calls? How about GOTO?

Michael Felt

what are the 32.1k and 201k numbers below your names - increased reputation?

Junaga

damn that's interesting

terdon

12:59 PM

@MichaelFelt Reputation score. In chat, that's total reputation across all Stack Exchange sites you have an account on.

Which
is
why
I
have
so
much

You get a +100 rep bonus when you have >200 on one site and register on another, so you don't have to deal with the new user hurdles all over again.

Michael Felt

glad they get combined somewhere! Just suppose you had to that by yourself ;P

ah - which explains my dilemma. Never stayed with asking questions on AskUbuntu, and I have no clue anymore why I registered on 'the other one'. Probably pressed a wrong button, or a cross-posting 'took' me there.

I hope to feel much more "at home" here - having been a UNIX 'user' since early 79.

terdon

Sounds like you will be :)

Faheem Mitha

@terdon I've wondered why some things are easy to learn and other things are hard. I don't have any answers. I doubt anyone else does either.

Michael Felt

still struggeling with 'the right question' though. Answers are easy - imho. Good questions are hard.

Faheem Mitha

Hmm, some activity on this channel. Unusually.

Michael Felt

1:07 PM

@Faheem - of course you have answers. A key issue/cause of miscommunication is assuming the listner/reader is capable of making the correction. I must assume you mean: I do not have any answers about why some things are easy to learn and others are hard. If you literally had no answers, you would not be here - my best guess!

terdon

@MichaelFelt Very true. Questions are often harder than answers.

Michael Felt

Back when he UNIX manual was just two books - and I was re-reading chapter 7 and thought - oh my goodness - the answer has been here all along. Point being, I knew enough to recognize the answer. (But I ramble...) - please notice my big smile! as I say this.

terdon

@MichaelFelt And I see you're a pedant. Yes, you will indeed fit right in.

Michael Felt

pedant - nice short word - for how I explained the difference between myself and a perfectionist.

A perfectionist looks and finds missing dots above the letter i, and un-crossed t's (when we were still writing on paper).

I am the **** that spots the missing pixel in the dot above the i

it got some laughs when done 'live'.

Faheem Mitha

@MichaelFelt Yes, it's calling selectively leaving out context in the interests of brevity. It's what us humans do.

When we're not writing mathematical proofs, anyway.

Michael Felt

1:13 PM

are you implying I am not human?

Faheem Mitha

@MichaelFelt Nope. I have no information one way or the other.

Michael Felt

Part of my work is finding and resolving 'miscommunication' between humans and systems. Systems do not, generally, place context. Generally, 'instructions' are taken and executed literally.

Faheem Mitha

Yes, computers are very literal-minded. Bless their little silicon hearts.

I can be literal-minded too. I used to do math.

Michael Felt

And, so it is my 'bad habit' or 'strong characteristic' depending on the discussion. I accept that I cannot help myself.

Faheem Mitha

In fact, I was described as "maddeningly literal-minded" by someone in the Frying Pan. Who wasn't a math person, obviously.

terdon

1:16 PM

@MichaelFelt You are not alone:

6

Q: The letter i is not displayed properly.

I've been meaning to post about this for a while. On my machine (LMDE, firefox, various versions) the letter i on AU is not displayed properly: You will need to zoom in a bit to see it clearly, but the i looks like this:

Faheem Mitha

As I recall, I was just trying to get stuff clear.

Michael Felt

So, I guess, read expect, we will get along well.

terdon

@FaheemMitha Heh, that actually made me chuckle :)

Faheem Mitha

@terdon Thanks.

So, @MichaelFelt, visiting? I see you do AIX.

Michael Felt

(just a smile here, sorry if I do not laugh or rofl much - being an old dog it hurts my sides)

Faheem Mitha

1:18 PM

We aren't the busiest place over here.

Michael Felt

"We" is ambigous

Faheem Mitha

Occasionally there are spurts of activity.

@MichaelFelt On the internet nobody can see you laugh.

Or hear you, for that matter.

@MichaelFelt Intentionally so.

Michael Felt

:rofl: not in the literal sense

terdon

Here's a question since we're all chatting. I'll probably end up posting it on the main site as well, but is there a simple way of checking that a gzipped file ends in a newline? I am thinking of files that are several GBs in size so I am trying to avoid decompressing them . Ideally, I'd like to do tail -c 1 file | grep -E '^'$'\n''$' || "echo bad file" but that won't work for compressed files

Michael Felt

my "best guess" is that you must uncompress to see the actual data.

terdon

1:20 PM

I'm thinking I'll have to insert a gzip header and do something like printf '%s%s' $(command that prints header) $(tail -c1 file.gz)

Michael Felt

gzip -dc will at least unpack via a pipe

terdon

Yeah. But that can still take ages for huge files.

Michael Felt

gzip -dc file.gz | tail -c1

terdon

Yeah, that's what I'm doing now (but with zcat, same idea though).

Michael Felt

the point is - is is compressed. Only after uncompressed is there clear text to test on.

terdon

1:22 PM

Yes. And there's no way of knowing if the last bytes of the compressed file actually correspond to the last bytes of the decompressed file, right?

Don't compression algorithms move stuff around?

Michael Felt

I do not know the algorithum well enough to know if that is included in meta info about the file. I would expect not.

Faheem Mitha

@terdon I'm not sure that's possible. But why do you need to know in advance?

terdon

@FaheemMitha We had an issue at work today where a user (luckily one of our testers) uploaded an input file which ended up being truncated. We don't know if the upload failed or if the compression of the file failed (we compress after the upload finishes) or if the copying from one file server to another failed. Either way, we're trying to think of quick methods to check for obvious inconsistencies such as the file not ending ina newline.

When your files can be in excess of 60 gigabytes in size, you don't just decompress them for fun.

Michael Felt

actions differ, but generally a fixed size is read and compressed to a smaller size. Some algortithms may also vary the size of what gets read (e.g., start at 4k blocks, but later work with 16k blocks, or change size dynamically when it sees it is all one character - "ask the math guys"

terdon

Well, I'm asking one guy with a PhD in math and another who seems to be some sort of hardcore UNIX expert/sysadmin. Seems to me like I'm in the right place :)

Faheem Mitha

1:26 PM

You really want an expert in compression.

Michael Felt

then - the question should be - what are the possibilities of alg. gz, bz2, xz, etc. to determine in an archive ended normally.

terdon

Dammit! I knew I'd been here before. And I was right, it isn't really possible. You can't be sure of what the last bytes are.

2

Q: How can I decompress and print the last few lines of a compressed text file?

I have 6 gzipped text files, each of which is ~17G when compressed. I need to see the last few lines (decompressed) of each file to check whether a particular problem is there. The obvious approach is very slow: for i in *; do zcat "$i" | tail -n3; done I was thinking I could do something clev...

shell command-line compression

Faheem Mitha

I think it's probably possible to check for a sane gzip file format, without actually trying to uncompress it. What was the problem. Was the file not zipped correctly? Or was only part of the gzipped file uploaded?

Michael Felt

what you might be able to be sure of is whether the entire file is there - IF the compression includes file size. Let me experiment a bit

Faheem Mitha

@MichaelFelt Yes, that's what I was getting at.

terdon

1:28 PM

@FaheemMitha It was zipped, but the file was truncated. 99% sure the original file was truncated and the compression worked fine.

Michael Felt

a) the file could have been zipped correctly, but not (all) uploaded.

terdon

Yes

Faheem Mitha

For example, gzip almost certainly include some sort of preamble. But you'll need to check the specs.

terdon

@MichaelFelt That's probably what we're looking at here.

Michael Felt

happens/happened with some frequescy that a file did not ftp completely, e.g., the last block.

terdon

1:29 PM

@FaheemMitha It does, that's what I meant by the header, but it's no good. You can't decompress only a subset of a gzipped file.

Gilles said so, it must be true.

Faheem Mitha

Sounds like a reasonable question for the site. What can I do to check if a file has been gzipped correctly without actually unzipping it.

Michael Felt

better - that I have the complete file, not whether it was compressed correctly.

Faheem Mitha

@terdon No, I mean run some sanity checking on the gzipped file. I expect that's possible. Gzip itself probably does that.

terdon

@FaheemMitha Yeah, but my issue is more likely to be that the file was gzipped fine, but was crap to begin with. Classic crap in crap out.

Faheem Mitha

@terdon Ah, well, I don't see what to do about that.

terdon

1:30 PM

And that's basically what the question I posted above was asking.

Michael Felt

pedantic me ... :P

terdon

My question. And for pretty much the same situation. Damn.

Michael Felt

for starters: are the files you are concerned about ending in an odd-byte number.

Faheem Mitha

Interesting answer by Radovan. I didn't know there were special gzip methods.

Michael Felt

those I would check last, or, the ones I would check first would be exact multiples of 1024 in size (or maybe 4096)

Faheem Mitha

1:32 PM

Presumably there's some reason why this dictzip isn't standard.

terdon

@MichaelFelt Ah. That might be a good idea there.

No, in this case. But still, I guess that should catch 50% of such errors.

Michael Felt

your work is looking for patterns - is it not.

terdon

Heh, indeed.

Michael Felt

I would hope, after virification, that "odd" sizes would be a DD for file is okay

terdon

Odd as opposed to even or just weird, as in not multiples of 1024?

Faheem Mitha

1:34 PM

I'm unclear what the actual issue is. You've got a compressed file. But you want to know what the original file looks like, without uncompressing it?

terdon

@FaheemMitha i) A user uploads a file; ii) we compress it; iii) we analyse it iv) analysis fails because the file is somehow bad.

Faheem Mitha

And do you have the original file handy for comparison?

Michael Felt

my initial "skip" choice would a) odd; b) mod % 1024 != 0

and, actually only ! %1024 ( as odd would also always be ! %1024

Faheem Mitha

@terdon is the compression automatic? I'm unclear how the compression fits into this.

terdon

The problem could be in any of i or ii and I want to check whether the original file was OK without decompressing it. yes, we can check before compressing, but then we'd also have to check after compressing in case ii went wrong. So, ideally, I'd like a quick and dirty way of detecting what's easy to detect.

@FaheemMitha Not after compressing which si done to save disk space.

Faheem Mitha

1:36 PM

@terdon oh

terdon

@MichaelFelt I'll give that a go, thanks.

Faheem Mitha

I'd do a check before compressing, then. The compression part is under your control. Why not assume that went right?

Michael Felt

-t, --test
Test the integrity of compressed files. This option is
equivalent to --decompress --stdout except that the
decompressed data is discarded instead of being written
to standard output.

that is from the man page for xz, but my hope is that gzip has the same

a simple test:

root@x064:[/data/prj/gnu/dist]gzip -t nano-2.4.2.tar.gz
root@x064:[/data/prj/gnu/dist]echo $?
0
root@x064:[/data/prj/gnu/dist]dd if=nano-2.4.2.tar.gz of=NANO-2.4.2.tar.gz
3708+1 records in
3708+1 records out
1898633 bytes (1.9 MB, 1.8 MiB) copied, 0.0730941 s, 26.0 MB/s
root@x064:[/data/prj/gnu/dist]dd if=nano-2.4.2.tar.gz of=NANO-2.4.2.tar.gz count=3708
3708+0 records in
3708+0 records out
1898496 bytes (1.9 MB, 1.8 MiB) copied, 0.0527734 s, 36.0 MB/s
root@x064:[/data/prj/gnu/dist]gzip -t NANO-2.4.2.tar.gz

terdon

@FaheemMitha Yes, that's what I'll do. I'll still have to check after compressing though since that can still fail. For example, I think we copy files over between servers at some point so that will be after compression and can also go wrong.

@MichaelFelt No, that won't help. The integrity will be fine if the original file was just truncated.

Michael Felt

nods...

Faheem Mitha

1:42 PM

@terdon You could do md5sum checks before and after.

terdon

Yeah, thought of that, but these are files uploaded by users. I'd have no way of knowing what the correct md5sum would be.

Michael Felt

gzip: NANO-2.4.2.tar.gz: unexpected end of file
root@x064:[/data/prj/gnu/dist]dd if=/etc/motd of=/dev/null
1+1 records in
1+1 records out
880 bytes copied, 0.000247654 s, 3.6 MB/s

Faheem Mitha

Copying the compressed file, though. And check for exit codes. I expect the gzip algorithm gives an error code if it errors out.

@terdon No, I meant of the compressed file. I thought you meant it might get corrupt if were to copy it around.

terdon

@FaheemMitha Ah, yes. True.

Michael Felt

the +1 could be an idicator that a transfer completed - completely. +0 indicates an ending at a "storage" end-point. Not invalid, but a potential

Faheem Mitha

1:44 PM

@terdon You could also return the md5sum of the uploaded file to the user, and ask them to check their file locally. Or have them type in the md5sum when uploading. The latter would probably be more effective. As in, don't allow the upload if that field is blank.

Well, dinner time.

Michael Felt

Do users upload complete (uncompressed) files, or compressed files?

terdon

@FaheemMitha yeah, but no. That assumes a level of technical savvy that our users don't have.

They are mostly medical doctors who thing Excel is the pinnacle of statistical analysis.

@MichaelFelt Compressed, usually.

But not always. Not in this case, for example.

Michael Felt

ok, so you have no way of knowing if the file they comopressed is okay or not.

gzip -t xxx.gz tells if the .gz file received is okay as far as gzip is concerned.

that was my example above. I did a dd as a 'transfer' emulation, and deliberately left off the closing block. gzip reported an error.

root@x064:[/data/prj/gnu/dist]dd if=nano-2.4.2.tar.gz of=NANO-2.4.2.tar.gz count=3708
3708+0 records in
3708+0 records out
1898496 bytes (1.9 MB, 1.8 MiB) copied, 0.0527734 s, 36.0 MB/s
root@x064:[/data/prj/gnu/dist]gzip -t NANO-2.4.2.tar.gz

gzip: NANO-2.4.2.tar.gz: unexpected end of file

terdon

Yep. That's easy enough to catch. It's when they (or we) have compressed an already truncated file that we have problems.

Michael Felt

in my experience the "real-time" involved with decompressing is the time to write the output. -t writes to /dev/null, and speeds things a great deal.

ok. looking further...

Michael Felt

1:59 PM

so, if I am willing to accept an input file as potential incorrect (truncated) when the original size is a multiple of 1024 I could use:

#!/bin/sh
for archive in $*; do
print checking archive: $archive
gzip -l $archive | grep -v compressed | while read comp uncomp ratio file
do
check=`echo "$uncomp % 1024" | bc`
[[ $check -eq 0 ]] && print "verify file: $file in gzip file:$archive"
done
done

This is not going to catch everything - because some files end natuarally on 124 boundary, e.g., a tar file:

root@x064:[/data/prj/gnu/dist]sh /tmp/checker.ksh nano*gz
checking archive: nano-2.4.2.tar.gz
verify file: nano-2.4.2.tar in gzip file:nano-2.4.2.tar.gz
checking archive: nano-2.5.1.tar.gz
verify file: nano-2.5.1.tar in gzip file:nano-2.5.1.tar.gz

but: a normal text file gets skipped:

root@x064:[/data/prj/gnu/dist]sh /tmp/checker.ksh /tmp/*.gz
checking archive: /tmp/motd.gz

root@x064:[/data/prj/gnu/dist]gzip -l /tmp/motd.gz
compressed uncompr. ratio uncompressed_name
173 880 82.9% /tmp/motd

Is, I hope, easy enough to test on a known truncated file.

terdon

2:15 PM

Nice! Thanks, I'll try it out.

ojs

@terdon how do the users upload the files to you? wondering if the uploading mechanism can somehow check the file before uploading.

terdon

Yeah, we're looking into that.

This is an nginx server, it should be able to do something along those lines.

Junaga

Guys does maybe someone know this? are physical sectors always 256 byte large?
is this some sort of de facto standard? And if yes how spread is it? are there any actual storage mediums that don't use it? Like IBM special stuff?

or should I ask a question? but again, this isn't unix related xD

Michael Felt

physical sectors? I think much larger.

AIX has a term PP for physical partition, and the default size is 4MB, but on most systems the actuall size of the PP is GByte these days.

ojs

Disk sector

In computer disk storage, a sector is a subdivision of a track on a magnetic disk or optical disc. Each sector stores a fixed amount of user-accessible data, traditionally 512 bytes for hard disk drives (HDDs) and 2048 bytes for CD-ROMs and DVD-ROMs. Newer HDDs use 4096-byte (4 KiB) sectors, which are known as the Advanced Format (AF). The sector is the minimum storage unit of a hard drive. Most disk partitioning schemes are designed to have files occupy a multiple of sectors regardless of the file's actual size. Files that do not fill a whole sector will have the remainder of their last sector...

Michael Felt

2:22 PM

Most hard-disks, originally had a physical partitioin size of 512 bytes, and most today use 4096 bytes

Junaga

@ojs thank you^^ after I googled I only found this one en.wikipedia.org/wiki/Cylinder-head-sector

Junaga

2:59 PM

just read this, https://www.gnu.org/software/grub/manual/grub.html#Features

and, holy shit. the name grand universal boot-loader is more then a joke, it's actually deserved and fitting.

terdon

3:11 PM

@MichaelFelt Doesn't work, I'm afraid. And, come to think of it, I see no reason why it should. Why do you expect file sizes to be multiples of 1024? I may be missing something here, but I can see why that would make sense for disk usage, but not for actual byte size of a file.

Michael Felt

1024 as a possible "truncated" symptom.

previous experience when text files were transferred by ftp, and 'something went wrong' - the data received was a multiple of 1024.

ojs

1024 bytes then?

Michael Felt

Having a real example would help greatly with creating a "scanner". As there is no certainity without unpacking and examing the extracted output - the hope/goal is to look for indicators that give a good choice to look at.

yes, 1024 bytes (ftp likes to organize in 1024 bytes, while the underlying layer may concatenate that into larger sizes)

at least, ftp going back far enough. maybe today the default is 4k.

so, back to XXX as a mod value. The goal is to find a value that is predictive, without having massive false negatives.

You mention your users - how are they submitting these files? Is it browser assisted?

if there is an 'assistent' used, e.g., a browser - you might be able to add some javascript, or whatever, that can do a simple check for you of the uncompressed input, e.g., the \n\r, or \n ending you are thinking may be predictive enough.

Faheem Mitha

@MichaelFelt Actually, it might be possible to get some client side code to calculate some information about the file. I've heard rumors about something called Javascript.

ojs

@Fah

@FaheemMitha you should check snopes about those rumours ;-)

Faheem Mitha

3:24 PM

@ojs Nah. It's probably a collective mass hallucination.

ojs

@FaheemMitha thats my impression at least ;-)

Faheem Mitha

lol

terdon

@MichaelFelt Yeah, that's probably how we'll go.

Faheem Mitha

3:42 PM

Are any of you guys listed on SE Careers.

Faheem Mitha

3:58 PM

Sorry, that was a question.

terdon

For whom?

Faheem Mitha

@terdon Anyone. And that should have read "you guys".

terdon

Ah! That makes more sense :) No, I'm not.

Faheem Mitha

@terdon ok

I see you edited my post. I suppose that is one of your superpowers.

You could make it a question mark at the end too.

user334283

4:16 PM

How long this chatroom will be active before getting closed? Can I post here tomorrow at this same time?

terdon

@user334283 Yes, this will stay here indefinitely.

I don't expect it to ever close. Not unless Unix & Linux itself closes.

Faheem Mitha

@user334283 It will probably shut down if WWIII happens.

user334283

I've to use mobile site to connect with the community..wish if there was an application for SE chat

terdon

@user334283 There is, actually. But the new mobile chat UI is pretty good. I find it much better than the application (which isn't very good).

I think the app was (is?) called chatSEy or something.

Yep:

29

Q: ChatSEy - An Android App for SE Chat

App removed Now that we have a mobile chat that doesn't suck, I've taken this down. Screenshot About Completely overhauled styles, including default and dark themes Reply to, star and flag messages Slide-out Sidebar for easy menu/star list access Tweaks to make typing easier (including...

app android chat

No longer active, apparently.

shirish

7:09 PM

can somebody figure out why the script doesn't work unix.stackexchange.com/questions/325395/…

Junaga

from the Debian/Gnu installation guide:
If you are running another Unix-like system, you could use it to install Debian GNU/Linux without using the debian-installer described in the rest of this manual. This kind of install may be useful for users with otherwise unsupported hardware or on hosts which can't afford downtime. If you are interested in this technique, skip to the Section D.3, “Installing Debian GNU/Linux from a Unix/Linux System”. This installation method is only recommended for advanced users when no other installation method is available.

shirish

@Junaga why are you sharing the debian/gnu installation guide ?

Junaga

why is it recommended to only be used if the other options are inaccessible?
Is it just to keep new users away form doing it this way? Because it is hard and complicated?
Or are there any actual compatibility problems?

@shirish I can't type that fast^^ sorry:p

I think it's the first case because even for the poor man's installation there is this sort of, warning:
2.4.4. Hard Disk
Booting the installation system directly from a hard disk is another option for many architectures. This will require some other operating system to load the installer onto the hard disk. This method is only recommended for special cases when no other installation method is available.

shirish

that complicates things quite a bit, using d-i (debian-installer) from usb stick is easy and clean.

Also if the hdd's are the same brand, size etc. you could easily end up over-writing the second hdd disk itself.

Junaga

7:25 PM

actually I planned to do it on just another partition x)

shirish

the best way would be to use a netinstall usb stick and be done with it. This way the disk reads and knows all other partitions and there is no confusion.

also net-install disks are cheap (something like 50-100 MB or something) , so you could have a basic skeleton in place pretty shortly.

Junaga

yes but with a netinstall

don't I have to wait for the download of the entire thing?

shirish

this would be useful if you are thinking of getting the debian stretch alpha which has just been released.

Junaga

oh yes I am retarded I just set up my own host form another pc

lul

I still know nothing about networking, I don't even consider all the options that networking provides.
I hate that reading takes so long

and that I need sleep

shirish

That really depends, you could just have the basics installed and then install the components which you want as and when required.

the whole apt-get install foo applies here.

Junaga

7:32 PM

What exactly is the difference between apt and apt-get

I used both like 50 times already couldn't figure it out

Faheem Mitha

@Junaga because it's messy? I assume. I've never tried.

@Junaga aptis the new simplified command. Has features of both apt-cache and apt-get. I'm sure someone has asked this here already.

shirish

I actually prefer apt and aptitude and I find they are much better than apt-get. The only reason I use apt-get at times to see what packages are suggested. I haven't been able to incorporate 'suggests' in apt or aptitude.

this is when installing or updating packages, which happens quite a bit.

Faheem Mitha

See debian-handbook.info/browse/stable/sect.apt-get.html and askubuntu.com/questions/445384/…

Junaga

Honestly to me it appears less messy xD
I mean that installer Is system software that needs to do a lot of things. I needs to able to create files, directories do decompress things, these are different task probably there are running different thread/processes for only this.
I have all of this tools already on my running system/ why leave them out? and use that magic tool?
I just need that config that says like, what keyboard layout I want.

Faheem Mitha

@Junaga Use whatever you like, of course. But you asked.

Junaga

7:39 PM

Should I ask a question about?

how could I even ask such a question..

literally this?
why is it recommended to only be used if the other options are inaccessible?
Is it just to keep new users away form doing it this way? Because it is hard and complicated?
Or are there any actual compatibility problems?

Faheem Mitha

@Junaga I suggest you try it and see.

The estimable Russ Allbery: Some thoughts on the US elections. Not exactly on topic.

Transcript for