/dev/chat

Faheem Mitha

01:54

@Basj what's on your mind?

Hi @Anthon. What's happening?

Anthon

Not much, cannot sleep, so I prowl the nets for things to edit or review

Faheem Mitha

02:17

@Anthon That sucks. I know about the not sleeping thing.

Anthon

02:35

@FaheemMitha I norally would read in bed, but my gf has the early shift, needs all the sleep she can get and while reading I might wake her up.

Done any interesting pythoning recently?

Seth

@Anthon You should come over to AU if you can't find anything here, we've always got things to edit and review :P

Anthon

02:50

@Seth Thanks for the invite. I'm not sure why I spent so little time there. It might be the subconscious affront the orange theme poses for my latent monarchistic tendencies.

Faheem Mitha

@Anthon I see.

@Anthon Not really. I did fix a depreciation warning in biopython a few days ago. They changed a function to handle large data better, I think. That was kind of a pain. And I'm seeing a SQLAlchemy warning in the same place I don't understand. Haven't fixed that one yet.

Seth

@Anthon o_O

;p

Faheem Mitha

@Anthon I think you should stay here. :-)

Hi @Seth. Stop trying to lure him away. :-)

@Anthon is your gf a nurse or a doctor?

Seth

@FaheemMitha Hi! :P

Faheem Mitha

@Anthon I don't get the orange reference.

@Seth What's happening?

Seth

02:58

Watching some penetration testing videos, playing some chess.

And talking to you ;P

And you?

Faheem Mitha

@Seth Penetration testing videos?

@Seth Proofreading this freaking letter.

Seth

@FaheemMitha ya

@FaheemMitha :/

I hate writing..

Faheem Mitha

8.30 am here. I'm trying to sync with the rest of humanity for the moment.

Seth

Which reminds me.. I have some to do D:

Faheem Mitha

@Seth Well, here, it's more the content that actual writing that is the problem. Some nasty legal things.

Also, a url is getting cut off. Have to fix that.

LaTeX stuff.

Braiam

03:02

so, pbuilder would get removed if the FTBFS isn't fixed for one of the dependencies...

Faheem Mitha

I VTC this one as a programming question:

0

Q: How to fix Python ValueError:bad marshal data?

Running flexget Python script in Ubuntu, I get an error: $ flexget series forget "Orange is the new black" s03e01 Traceback (most recent call last): File "/usr/local/bin/flexget", line 7, in <module> from flexget import main File "/usr/local/lib/python2.7/dist-packages/flexget/__init__.py", line...

Braiam

@FaheemMitha do you think that SO guys would answer that?

Faheem Mitha

@Braiam No idea. Do you think it is unsuitable for SO?

Braiam

@FaheemMitha is definitively not about coding something... is more akin to "here's my core dump, find and fix the problem" which nobody likes

Faheem Mitha

@Braiam Hmm, fair enough. Should I change my close vote? And if so, to what? It is definitely off topic here.

Anthon

03:25

@FaheemMitha no nurses/doctors don't carry a gun

House of Orange-Nassau

The House of Orange-Nassau (in Dutch: Huis van Oranje-Nassau, pronounced [ˈɦœy̯s fɑn oːˈrɑɲə ˈnɑsʌu̯]), a branch of the European House of Nassau, has played a central role in the politics and government of the Netherlands — and at times in Europe — especially since William I of Orange (also known as "William the Silent" and "Father of the Fatherland") organized the Dutch revolt against Spanish rule, which after the Eighty Years' War led to an independent Dutch state. Several members of the house served during this war and after as governor or stadtholder (Dutch stadhouder) during the Dutch Republic...

Anthon

04:03

@Seth the other problem is that my rep on AU is so low I cannot review anything and only suggest edits

Faheem Mitha

@Anthon Ah, a European political joke. Went over my head, I'm afraid.

@Anthon You could probably bring up your rep pretty fast. Not that I recommend it.

Anthon

04:26

@FaheemMitha Somehow I assume the whole world associates dutchmen with orange colored soccer-hooligans (not that I watch soccer or any other sports)

Faheem Mitha

@Anthon You do the whole world way too much credit.

In assuming they know much about anything.

Seth

@Anthon That's easily fixable, just ask @terdon! :D

He now has more rep than I do >.>

Which obviously means I have more of a life, right? Right? ;p

Braiam

@Seth just answer regex questions on SO... you will feel dirty, but you will have more rep than terdon anyways

Anthon

@seth I am sure Terdon has a life as well.

Getting to the top of a review queue on AU would be a challenge....

Faheem Mitha

04:58

What is a suitable tinyurl thingy? One that won't be blocked by sites?

is bitly better than tinyurl?

Anthon

05:12

@FaheemMitha Which sites are blocking those? SO?

Faheem Mitha

@Anthon That's hypothetical blocking.

As in, I wouldn't want the url to be blocked.

Braiam

13:01

@FaheemMitha check this meta.stackoverflow.com/a/295085/792066

mikeserv

14:37

@StéphaneChazelas - yup: gnu.org/software/gzip/manual/html_node/Invoking-gzip.html

The gzip format represents the input size modulo 2^32, so the uncompressed size and compression ratio are listed incorrectly for uncompressed files 4 GiB and larger. To work around this problem, you can use the following command to discover a large uncompressed file's true size: zcat file.gz | wc -c

I wonder how far off it is?

terdon

@Seth Probably. Sadly. :P

Stéphane Chazelas

15:10

@mikeserv yes, gzip was written in 1992. Newer compression formats (bzip2/xz) don't have that low a limitation and some (like xz) can allow random access.

mikeserv

@StéphaneChazelas Yeah. It's a shame - that was pretty easy that way. I can't figure on how else to do it otherwise. I can't even tell what the other answers do, how are they splitting the output files reliably? If the top half of the file has a more regular data set than the bottom half, then the top half would be far smaller, I think.

I tried faking it - just piping the gz header at zcat - like the first 10 bytes (and several other amounts) and then sending the next Mb to >/dev/null, and then catting the rest at it, but it just kept telling me input violates the format.

Faheem Mitha

@Braiam Thanks, but it doesn't seem terribly relevant. And I wasn't planning to use url shorteners on this site, anyway.

casey

16:23

@Anthon I certainly and unequivocally associate orange with dutch. I also have first hand experience with the football folks, more with the nice than the hooligans though (I encountered them in force in Bern, Switzerland during Euro 2008)

Anthon

17:42

@casey You met nice dutchman there? That a new definition for "Das Wunder von Bern" :-D

Stephen Kitt

20:51

@mikeserv What are you trying to do?

3

A: Fastest way of working out uncompressed size of large GZIPPED file

I believe the fastest way is to modify gzip so that testing in verbose mode outputs the number of bytes decompressed; on my system, with a 7761108684-byte file, I get % time gzip -tv test.gz test.gz: OK (7761108684 bytes) gzip -tv test.gz 44.19s user 0.79s system 100% cpu 44.919 total % ti...

mikeserv

I actually saw that earlier today - but I forgot to upvote it then. That is pretty cool. It's just that, well, that's what I was trying to avoid doing. You should offer that as anwer though.

0

A: How to efficiently split up a large text file wihout splitting multiline records?

I don't think you can do this - not reliably, and not the way you ask. The thing is, the archive's compression ratio will probably not be evenly distributed from head to tail - the compression algorithm will apply better to some parts than others. That's just how it works. And so you can't factor...

@StephenKitt ^ that was it.

The post history will show that I initially divided gzip -l's output by 4 before Stephane let me know it wouldn't work - I hadn't tried it on a 4gb file.

Stephen Kitt

@mikeserv Ah right, I see...

mikeserv

So while the 4 line thing is cake - and it needn't even be blocked at a specific byte boundary to start with either (as dd can conv=block anyway) the 25% of stream thing just can't be done, I don't think. And really, I don't think that guy should, anyway.

Stephen Kitt

No, I don't think it's possible either, not with gzip anyway.

mikeserv

21:10

Yeah, that makes two of us.

Stephen Kitt

It's not clear from the question though whether the input is supposed to be processed in compressed form or not; the Python answer just works on the uncompressed file.

Stéphane Chazelas

21:39

@mikeserv That would be possible on a filesystem that support cp --reflink and fallocate -c with 1-byte granularity.

Don't know of such a filesystem though

mikeserv

@StéphaneChazelas - which ones are those?

@StephenKitt - the Python answer just works on the uncompressed file? I couldn't even tell what those other answers did.

@StephenKitt - where does it put it?

Stéphane Chazelas

btrfs can do cp --reflink, but not fallocate -c, ext4 can do fallocate -c but only with block granularity and can't do reflink yet. Don't know about things like nilfs2. One could always have a fuse filesystem on top of the full uncompressed file to fake the 4 files though. Though that'd be cheating.

mikeserv

Yeah. fuse is always cheating.

But on a btrfs filesystem you would just mark the file for gz compression and forget about it.

@StéphaneChazelas - is it ext4 that does the -c thing - is dig holes or something? I was look at that man page the other day.

I actually did man $(pacman -Qil util-linux | sed -n 's|.*/usr/bin/$.[^ ]*$|\1|') and just tabbed through all of them.

mikeserv

22:03

@StéphaneChazelas - dang, man. You're cryptic sometimes. How could you do it with fallocate or --reflink? You'd still have to uncompress it, wouldn't you?

Stéphane Chazelas

@mikeserv Yes, I just mean make 4 reflinked copies of the uncompressed file (and record the line number/size... upon uncompressing) and collapse the unwanted parts in each of those files.

lwn.net/Articles/587819 for fallocate -c

mikeserv

@StéphaneChazelas - yeah, that's genius. collapse - that was it.

@StéphaneChazelas - what's i_size - is that for inode?

Transcript for