/dev/chat

Faheem Mitha

00:09

@derobert hi. unfortunately the switch failed. i got what looked rather like a kernel traceback

@derobert they do keep a couple of copies

minor comment - prefer update-initramfs -u -k all

i think when i ran update-initramfs -u it just updated the first kernel, which i don't actually use

however, that one didn't work either. 3.13. i use the wheezy default, 3.02

i got delayed because my drives all get fscked and then X failed to come up with different kernels

hmm, no i guess the X log does get overwritten. bummer

Faheem Mitha

00:26

@derobert if you want to add an answer, go ahead. maybe it will work for someone.

does anyone know how to configure debian so the x logs don't get overwritten on reboot or whatever?

Braiam

02:55

only for sysadmins

mikeserv

04:53

@derobert probably multilib was a stretch, but look at this: discussions.citrix.com/topic/…

On virtualization-supported hardware, nvidia actively blocks 64-bit memory addressing because, at least as far as I can tell, it cannot actively shadow its ROM. If it were mapped into 64-bit space its instructions could be read in real-time - which is a lot different than pulling a BIOS image off of a card.

At least since 3.11

The nvidia driver readme has a bit to say about valgrind along the same lines as well.

Anyway, that's their prerogative I suppose, but it sucked 6 months ago when I went to their forum after x mysteriously quit working following a (later than usual) kernel upgrade and all I found were months' old posts describing my same issue and not a single reply from any nvidia rep.

Faheem Mitha

09:21

Ok, one more time - Does anyone know how to configure debian so the x logs don't get overwritten on reboot or whatever? In many cases Debian logs keep backups of older logs but that does not seem to happen here. Does anyone know why, and if/how one can configure Debian to keep older copies of the x log around?

Oh, and this seems like a reasonable question for the main site, unless there it has already been asked. I didn't find anything.

mikeserv

13:14

@Graeme I have useradd -m

terdon

15:34

We don't seem to have a good Q&A on how exactly are shell scripts executed. Things like the difference between executing binary files and scripts and the difference between scripts with and without a shebang line. This question could be a good opportunity to write a one:

0

Q: How Does Linux deal with shell scripts?

For this question, let's consider a bash shell script, though this question must be applicable to all types of shell script. When someone executes a shell script, does Linux load all the script at once (into memory maybe) or does it read script commands one by one (line by line)? In other words...

Any of the heavyweights want to give it a go?

Faheem Mitha

16:00

I doubt bash would read the whole thing in in one go in general. suppose it was 10,000 lines?

terdon

16:11

@FaheemMitha Yeah, that's the thing. I tried and it doesn't. Seems it will read in blocks but also has some strange tricks.

Faheem Mitha

@terdon well, it must have some algorithm it uses.

i'm surprised it is so complicated

if it is not documented, i suppose looking at the source code is an option. but asking on the bash mailing list is also an option

the devs would know what it does.

Gilles

@terdon I've written about that before, at least the part about with vs without shebang

Faheem Mitha

python for example, afaik, reads the whole thing in one go, no matter how large

Gilles

@FaheemMitha so do most shells

terdon

@Gilles Ah, I was hoping you had. I didn't manage to find it though.

Faheem Mitha

16:13

you can tell, because it complains about parsing errors, no matter where they are

@Gilles i see. bash is an exception, then?

terdon

@Gilles Stephane has posted an answer showing that bash reads blocks

10

A: What happens if you edit a script during execution?

bash goes a long way to make sure it reads commands just before executing them. For instance in: cmd1 cmd2 The shell will read the script by blocks, so likely read both commands, interpret the first one and then seek back to the end of cmd1 in the script and read the script again to read cmd2...

Faheem Mitha

my impression is that shells don't typically parse the script for syntax correctness before they get going.

Gilles

@FaheemMitha IIRC, it is, among the usual shells

Faheem Mitha

@Gilles I see. Interesting.

Wonder why.

i think even R reads the whole thing in first.

terdon

So if I have a 1GB csh script for example, I will see a 1GB spike in RAM usage when I try to execute it?

Faheem Mitha

16:16

unlike Python, it doesn't do the byte-compile thing first, though.

@terdon possibly. try it and let us know. :-) at least 1 GB, I'd have thought. either that, or you'll crash the interpreter.

python for example, is amazingly inefficient in how it stores stuff

terdon

@FaheemMitha I tried it with a bash script (without a shebang line) and there was no noticeable change in RAM

Faheem Mitha

@terdon how big was the script?

terdon

1.6G or so

Faheem Mitha

does the shebang make a difference?

terdon

I think so but I'm really not sure

Faheem Mitha

16:18

@terdon that's pretty big. must be reading in pieces.

@terdon ok

Gilles

ok, actually I was wrong, all the usual shells (bash, dash, ksh93, mksh, zsh) start executing the script before reading it fully

Faheem Mitha

@Gilles ok

i'm surprised that syntactically possible. suppose you have a really big function for example? doesn't it need to read to the end? or more generally, any syntax block

Gilles

@FaheemMitha then it needs to read the whole function (see Stephane's answer posted above)

Faheem Mitha

@Gilles Ok. so, i guess you could force a memory spike then.

write a function that does nothing but read thousands of text lines, and returns nothing

slm

@terdon We covered some of this before in another Q I cannot find now that I showed the command reading chunks of data in from a file in blocks.

Gilles

16:26

@slm terdon posted a link to that just above

slm

16:40

@Gilles @terdon - this is somewhat related, though it's how grep reads in a block at a time from a file. I would assume that interpreters are doing the same thing.

1

A: How much text does grep/egrep consume each time?

Most of the tools do not actually read a single line in from a file at a time, rather they use a buffer in memory to store chunks of lines. The tools operate a line at a time on the data in this buffer. NOTE: By "line" I mean split by a \n, in grep's case, or whatever character is denoted as th...

mikeserv

@terdon this is the best discussion on the general subject I've seen about. unix.stackexchange.com/questions/119929/…

terdon

@mikeserv ah, yes, I'd seen that one but forgotten. Thanks.

slm

@terdon @mikeserv sometimes we should not answer these questions, we're just encouraging madness with some the answering 8-)

mikeserv

Those answers are the only reason I'm here.

I answered first that . and sh would both show different $0s or something.

Then toxalot came back and said no. So I tested it and dug into /process and posted like a tiny bit. Then I opened an edit on it and starting writing this crazy loop interpreter.

Graeme had answered with the sh -c ' ' argv0. So I was making -i invoke -c with different argv0s so you could customize $0.

I forgot about it after getting distracted and found it in an open tab later. Checked the page. "Woah."

slm

16:56

@mikeserv @terdon - I worry for Stephane that these answers are causing him to have some sort of out of body experiences every time he produces one of the them 8-)

mikeserv

I was writing that little toy and Stephane just friggin execed printf. It was cool. I dumped the edit tab.

terdon

heh :)

mikeserv

Anyway, I learned a lot from that post so I figured I'd remind you guys...

Combine it with that loop one - the one with rewriting the script midexecute? Think I'm gonna break my computer...

That's how you get your 1gb memspike, @terdon.

@slm something tells me sc has got a handle on it. Or maybe he's put of control...

slm

Yeah the happy faces are me joking 8-)

mikeserv

17:14

I know. I'm capable of that too... I just suck at emoticons.

slm

8-)

Faheem Mitha

17:59

re my question about keep older versions of the X log file around on Debian, shall I post this on the main site?

mikeserv

18:15

It doesn't already keep xorg.log.{0,1....}?

Faheem Mitha

18:37

@mikeserv it doesn't seem to

it does for other logs, not for this one.

mikeserv

Weird.

And I guess its supposed to be Xorg.0.log

Faheem Mitha

@mikeserv correct, /var/log/Xorg.0.log

mikeserv

Yeah. Probably you can adjust this in .xinitrc

Faheem Mitha

i may have somehow disabled it on my system, but I am not aware of having done so, and this is a relatively new install - I did a fresh install for wheezy. before that, this system was last installed feb 2007. my recollection before that is that the x log was not kept around either

@mikeserv adjust what?

mikeserv

The .log path.

I doubt it will already be set, but you need to define a command line operand for log location when it's called.

I'm trying to grep /etc for it and I get 0 results for a predefined log path. Maybe it's compiled in?

Faheem Mitha

18:49

@mikeserv my system currently doesn't have a .xinitrc

mikeserv

What starts X?

Faheem Mitha

dlocate xinitrc also turns up nothing

@mikeserv dunno

mikeserv

Is it in .profile?

Faheem Mitha

@mikeserv is what?

mikeserv

Whatever starts X.

You don't have xinitrc in /etc/X11?

Faheem Mitha

18:51

@mikeserv no

mikeserv

You don't necessarily need one in $HOME...

Well, then you don't use xinit...

Faheem Mitha

@mikeserv OK. I see there are files in /etc/X11/Xsession.d/ that look like they might do something

mikeserv

But SOMETHING starts X. You get no clue out of the contents of /etc/X11?

Yeah. That.

Faheem Mitha

i'm tempted just to ask the question

mikeserv

That's what you want. That will set options to X at invocation. So you need to add one for log location. I'll check cmdline params. Yeah, do, sure. Probably there are 100 ways to set a log path. That will do it though.

strugee

18:55

Debian installs a DM by default, wouldn't it be that?

Faheem Mitha

@mikeserv One for log stuff in /etc/X11/Xsession.d?

@strugee Wouldn't what be what?

I thought logrotate might be relevant, but I don't see any specific logic there.

strugee

@FaheemMitha what's starting X

sorry I have no idea what we're trying to figure out so if that's completely irrelevant, my bad

mikeserv

On point, I'd say.

Faheem Mitha

@strugee X only keeps one copy of the X server log file on debian. i want it to keep more copies. that's all

strugee

oh. them why we're you discussing what starts X?

Faheem Mitha

19:00

@strugee i'm using kde, so maybe kdm. not sure what that would control how many log files are kept, however.

@strugee dunno.

odd, a self answer on how to download Debian source code.

strugee

@FaheemMitha sudo service kdm status

@FaheemMitha hah :)

Faheem Mitha

@strugee what does that do?

i'm wary about typing random commands

strugee

@FaheemMitha it should show if kdm is running. I think.

if that's not what it does it won't do any harm

sorry I'm only used to systemd systems

Faheem Mitha

I get "[ ok ] kdm is running."

i'm running KDE so perhaps not surprising

strugee

there you go then

Braiam

19:05

anyone thinks this is an answer? unix.stackexchange.com/a/115239/41104

strugee

19:19

I'm with you. looks like a comment

mikeserv

Apparently xorg accepts configuring the log file path only at compile time. Logrotate - or whatever your logfile service is - would be the program that handles changing the filenames.

So you've gotta figure out why it doesn't do it for X.

Faheem Mitha

@mikeserv well, maybe it is just a matter of adding the logic, as i comment in my question

mikeserv

A short term solution could be ( tail -f x.log >someother.log & ) &

Faheem Mitha

Eg. for apt, there is

/var/log/apt/history.log {
rotate 12
monthly
compress
missingok
notifempty
}

from /etc/logrotate.d/apt

So maybe I just need to create a similar file. Dunno.

mikeserv

That'll run forever as long as there's an x.log and a writable someother.log path

Maybe. Me either.

I know the above will get you a log to check on next reboot.

Faheem Mitha

19:25

@mikeserv Ok

mikeserv

Or at least, I think it will. Should work like nohup, basically.

As for logrotate - I never mess with that stuff. I always do silly hacks like above. Probably I should learn how it should be done properly eventually...

@Braiam comment

It even opens with "just [a] note..."

derobert

19:42

I forget what does the X log rotation... I know I found it at one point.

Trying to find it again quickly

Braiam

@derobert when you do post a question about it :P

Faheem Mitha

hey, @derobert

derobert

Hey

derobert

20:04

hw/xfree86/common/xf86Helper.c
1228:#define LOGOLDSUFFIX ".old"
1238:    xf86LogFile = LogInit(xf86LogFile, LOGOLDSUFFIX);
1245:#undef LOGOLDSUFFIX

Faheem Mitha

I just realised that the two other X log files there are actually copies, so it is in fact keeping copies. i.e. Xorg.0.log.old, Xorg.0.log.old2

the thing is, i use old for a suffix myself, and old2 as well, so I thought it was me doing it. looks like that is not the case

derobert

hmmm, I've never seen an .old2 file... That one is probably you

Faheem Mitha

plus debian doesn't use old in other places, so why is it doing so here?

derobert

Xorg is using .old

Faheem Mitha

@derobert oh

so it keeps one copy, then?

derobert

20:08

yep

well, it moves the old one

Faheem Mitha

@derobert ah, and if I want more copies?

and does logrotate do anything here, or not applicable?

derobert

logrotate is not involved... it's handled directly by X

Faheem Mitha

@derobert ah. and is it configurable or not?

derobert

at compile time

Faheem Mitha

@derobert eek

derobert

20:09

i.e., if you want to recompile xorg

Faheem Mitha

@derobert i don't, no. So the answer is, you can't do it? Bummer

derobert

you could maybe hack around it in a display manager config, by moving the log file before starting X

Faheem Mitha

@derobert yuck

@derobert you could answer that nvidia question. my attempt failed, but the answer might still be useful for someone.

derobert

I could, or I could play Stick of Truth...

Faheem Mitha

@derobert what is Stick of Truth?

I tried to stick the X log for my failed attempt into my question, but I ran up against a character limit...

apparently something called south park.

should i post the X log somewhere else and link to it?

derobert

20:31

@FaheemMitha amazon.com/South-Park-The-Stick-Truth-Playstation/dp/B006IOAHPK/…

@FaheemMitha wow, the X log was that long?!

I guess you could put it on pastebin or similar, so someone can find the relevant section

Faheem Mitha

@derobert Yes, doing that now

Ok, done now.

So, @derobert Stick of Truth or Nvidia?

There is a song title in there somewhere.

This guy deserves an upvote for his epic battle.

2

Q: Debian wheezy: cannot create "Hello World" module (and NVIDIA, and VirtualBox)

(EDIT: This is now solved (see the accepted answer). The problem lied with GREP_OPTIONS='--color=always'. Changing the value to --color=never resolved the problem.) First off, the details. BEFORE: kernel: 3.2.0-2-amd64, nvidia driver: 295.59 AFTER: kernel: 3.2.0-3-amd64, nvidia driver: 302.17...

debian kernel drivers nvidia kernel-drivers

Almost as epic as the Stick of Truth.

derobert

@FaheemMitha both are proprietary. One is a lot more fun, though.

Faheem Mitha

@derobert you need to have Free Software Epic Quests. Else RMS will disapprove.

Braiam

@FaheemMitha what the heck has the title to do with the body of the question?

Faheem Mitha

@Braiam Ask dimitko

weird problem - he had a color option set on grep.

Braiam

20:41

the solution is more bizzare

Faheem Mitha

that broke the kernel build, apparently

go figure

mikeserv

21:01

It's the bytecode that comes with the colors - the escape sequences. Grep and sed are both used extensively by gnu compilers. If they're pumping out their results with terminal escapes there's gonna be a problem.

Faheem Mitha

@mikeserv i would have thought this problem would come up quite often, then.

or is colorizing grep not so common, then?

mikeserv

It used to. Colorizing grep --always isn't. Also, most compilations you do will be automated with fakeroot and chroot which clean their env and drop your shell aliases.

Faheem Mitha

@mikeserv so what went wrong here?

fakeroot yes, chroot no. i never use chroot for compilations

mikeserv

Did he replace grep with a shellbscript or something - I didn't look hard at it - just your comment.

Faheem Mitha

@mikeserv No, I didn't really look at it either.

He said he just had

export GREP_OPTIONS='--color=always'

changing to

export GREP_OPTIONS='--color=never'

fixed the problem

mikeserv

21:10

See his $PATH? ruby's at the head - priority. There are Ruby and perl scripts out there that parse the output from common apps and colorize them for you. Maybe that was part of it. They break compilers.

That's just a guess though.

I can't think of any other reason to put ruby scripts ahead of /bin though.

Its gotta be something like that - the ruby scripts directory is even called .../ruby.../bin

If he had just cleansed his $PATH he probably would have achieved the same results.

You should try chroot for compilations. For instance, with a typical compile chroot, you would --bind mount or link in only the bare minimum to do the job - so in this case, the ruby stuff would have been ignored cause it wouldn't have been there.

Graeme

23:13

@mikeserv did some investigating...

0

A: What does adduser do that useradd doesn't?

First off, the respective man page snippets highlight the differences between the two commands and give some indication of what is going on. For adduser: adduser and addgroup add users and groups to the system according to command line options and configuration information in /etc/adduser.c...

slm

@terdon - I made a example that shows that the interpreter is reading the file in in 8K blocks if you're interested.

4

A: How Does Linux deal with shell scripts?

If you use strace you can see how a shell script is executed when it's run. Example Say I have this shell script. $ cat hello_ul.bash #!/bin/bash echo "Hello Unix & Linux!" Running it using strace: $ strace -s 2000 -o strace.log ./hello_ul.bash Hello Unix & Linux! $ Taking a look insid...

terdon

@slm Nice, that's the kind of thing I was hoping someone would answer. Dammit, I have to stop upvoting your answers just cause they're correct. I should wait until you make gems out of them so I can reward you :)

slm

I think you know me by now, crap like this bugs me and I have to figure it out 8-). But I like to see it with an example.

terdon

23:29

I had written a similar test but got lost in the boonies of shell buffers and didn't feel up to answering.

I'm playing with this one now:

2

Q: Merge fields in a file

I have a file with 7 columns, a GFF file having chromosomal regions.I want to collapse the rows where REGION ="exon" to only one row in the file.The row has to be collapsed on the basis of regions being overlapping with each other. REGION START END SCORE STRAND FRAME ATTRIBUTE exon 26...

bash shell sed awk bioinformatics

It's off topic really, it is a much more complex problem than simple text parsing and needs a full-fledged script but I want to answer anyway :P

AAAAAAA Calling the Perl Monks. What the hell is going on here?

#!/usr/bin/perl
@foo=(1,2,3);
for (my $i=0; $i<=$#foo; $i++) {
    chomp($foo[$i+1]);
    print "$i : $#foo\n";
}

slm

Here's something you don't see every day.

35

Q: Can I configure my Linux system for more aggressive file system caching?

I am neither concerned about RAM usage (as I've got enough) nor about losing data in case of an accidental shut-down (as my power is backed, the system is reliable and the data are not critical). But I do a lot of file processing and could use some performance boost. That's why I'd like to set t...

linux filesystems performance fstab sysctl

Notice the deleted A? 21 UV's + a 300+ bounty, the OP deleted it.

terdon

@slm yeah, wow. Some self control there

Do you have any idea what't going on in that perl snippet I just posted? I'm sure it's staring me in the face and I can't see it but why in the world is chomp($foo[$i+1]) apparently adding things to @foo?

slm

@terdon Ah it's a Ron Q. I swear he outsources his scripting to us suckers here 8-)

terdon

Does he? He only has 4 Qs so far. I think he just does not know how to write scripts. But he's gonna have to learn if he's going to work in bioinformatics.

slm

We've got a couple from him before.

terdon

23:41

@slm can you confirm the behavior of that perl I just posted? Is it specific to my version or does that cause an infinite loop for you too?

slm

I was trying it out now.

loops for me too

terdon

Am I going crazy or does that make no sense at all?

slm

I think the chomp is doing something, changing the control from $#foo to 3 and it stops.

for (my $i=0; $i <= 3; $i++) {

terdon

Same thing on perl v5.8.8

slm

I'm on 5.14

terdon

23:44

@slm yeah, me too, I just tested on another machine to be sure

It looks like chomp is changing the value of $#list which seems very strange

slm

that was just a guess, I would assume that the control value has already been evaluated and set by the time chomp rolls around

terdon

Odd, I get the same thing for scalar(@foo) instead of $#foo

slm

Ah I think I see it

terdon

Huh, no, it's actually adding whitespace (not sure what kind yet, but I'm guessing $/ since it's chomp()) to the array!

slm

I think when you do foo[$i+1] you're adding elements to the array

terdon

23:47

@slm apparently but why?

print "$foo[$i+1]" won't

slm

When I capped it at $i <= 3
$ ./p.pl
2
0 : 2
1 : 2
2 : 3
3 : 4

terdon

It's adding spaces. Just checked with od -c

That's even more confusing

slm

#!/usr/bin/perl
@foo=(1,2,3);
$len = $#foo;
for (my $i=0; $i <= $len; $i++) {
    chomp($foo[$i+1]);
    print "$i : $#foo\n";
}

$ ./p.pl
0 : 2
1 : 2
2 : 3

See if I latch the position of the n-th element of foo and use that it works as you expect, it's manipulating the @foo on you when you access it. I think this is the autovivification in Perl that you're tripping on

Autovivification

Autovivification is a distinguishing feature of the Perl programming language involving the dynamic creation of data structures. Autovivification is the automatic creation of a variable reference when an undefined value is dereferenced. In other words, Perl autovivification allows a programmer to refer to a structured variable, and arbitrary sub-elements of that structured variable, without expressly declaring the existence of the variable and its complete structure beforehand. In contrast, other programming languages either: 1) require a programmer to expressly declare an entire variabl...

terdon

Let me just read that article see if it makes sense to me

Sounds like a zombie flick.

slm

Hey got you for a change 8-)

2

A: How to find out all the files in other machines using bash shell script?

There are 2 approaches that I can think of off the top of my head. The first would involve using rsync the second would be to use a combination of ssh & diff. NOTE: Both of these approaches would compare a directory on machineA that has all the files with a subset of these files on machine's B ...

I might have the wrong term, but basically it's when Perl will autocreate a data structure purely on accessing it

terdon

23:55

@slm for a change? When did I get you?

Anyway, you nailed it, sure enough, this works as expected:

#!/usr/bin/perl
@foo=(1,2,3);
for (my $i=0; $i<=$#foo; $i++) {
    chomp($foo[$i+1]) if exists($foo[$i+1]);
    print "@foo\n";
}

slm

I thought you had gotten me on a couple accepts + there was a reversal in there

terdon

@slm Oh maybe, but only cause I don't know enough to go into as much detail as you do and you might have scared the OP :)

slm

lol

I'm fighting not laughing out loud on that comment thanks 8-)

terdon

My favorite case of that was my answer here:

10

Q: How does mounting on the GUI work "under the hood"

UPDATE Please correct me if I'm wrong: For working on my computer, with a GNU/Linux Distribution named Debian, I know two ways to enter a command, start an application, open a file, etc.: a Command Line Interface where I enter text a Graphical User Interface [a.k.a GUI]: an interface which pr...

mount automounting fuse

By far the least interesting/informative but that's what the OP wanted. You guys gave too much detail :)

But seriously, thanks for finding that bug, it was driving me crazy.

slm

Any time, Perl's my favorite language

Transcript for