/dev/chat

Gnouc

1:37 AM

@Gilles: Are you there?

Gilles

@Gnouc yes but not for long

Gnouc

OK, so let next time for more disscussion. I will edit my answer about pathname and filename to make it clearer

mikeserv

4:03 AM

@FaheemMitha - 5 fold? That is a considerable optimization!

Tim

5:11 AM

hi @mike. what does / mean?

mikeserv

Well, written like that, it means the root directory of filesystem.

But, more generally, it is the pathname separator.

Tim

/ in ls -d */

mikeserv

Filenames can contain really any character but null and /, because / is what delineates them.

So your shell has a thing called pathname expansion builtin.

Tim

ls -d * output directories without / at their ends

mikeserv

The *globstar means any/or/all

Right - I'll get there.

When you pair the *globstar and the pathname separator you're saying, shell I want an argument list amounting to any or all directories contained within this one

sh -cx 'ls */'
+ ls one/ three/ two/
one/:
three/:
two/:

@Tim - see that + bit?

That's actually the shell's evaluation of my command.

That's what actually happens.

So doing ls */ is the same in this case as if I had typed ls one/ two/ three/ because the shell expands that out before ever calling on ls at all.

You don't get the / at the ends because that's actually a quality of default settings - which are ignored when you specify -d because that option is specifically for treating all files as a directory.

Tim

5:21 AM

$ sh -cx 'ls */'
+ ls */
ls: cannot access */: No such file or directory

mikeserv

It's a little confusing, but the short answer is, specify -p also.

You must be in a directory with no child subdirectories.

As I said any or all. So, if it evaluates to none, it is not expanded, and the string is handed literally to the invoked process as an argument.

And that, as a case in point, is what is special about the / character.

The same thing would happen if there were subdirectories and you disabled pathname expansion, like with set -f.

Tim

A side question: does "null" in "Filenames can contain really any character but null and /" mean empty?

mikeserv

Typically when people talk about null they refer to the ASCII 0 byte. It's a long sordid story, but it goes back to punchcards. If you were processing data on punchcards, you could make filler space by just not punching and holes - that's a null byte

it is \000

There's also the concept of a null string - which is an empty expansion. so printf %s ''

printf gets an argument there, but it is empty - the null string.

So I can do: printf '%s this still gets printed' ''

But a null byte, nowadays, is not a blank punchcard, it's an actual byte. Anyway, it is still forbidden in filenames. Only it and the pathname separator / share that distinction.

Tim

5:36 AM

"You don't get the / at the ends because that's actually a quality of default settings" what does the default setting says?

show directories without / at their ends?

mikeserv

A lot of things, but by default ls distinguishes by filetype.

Ok... welll... this goes deeper because it's about dentries.

So files are not the names you cal them.

files are inode numbers.

Hang on...

touch file
ls -i
397399 file

A directory is special type of file - it's a file's file - i guess.

It stores information about files.

When you move a file from directory to directory you directly affect the file - you affect which directory stores information about it.

The filename is stored in the directory as a quality of the file's directory entry - its dentry.

The filename refers to an inode - which is how the kernel refers to a file.

A hardlink is another dentry for the same inode - just different listings for the same file. It is not a quality of the file, but a quality of the directory listing it.

ls does dentries.

That's what it does - it reads the directory and reports on its listings.

It doesn't do too much with the files within unless you tell it to - with -d.

That's when it treats each file as a directory and reads it.

ls -dlp ./*
drwxr-xr-x 1 mikeserv mikeserv 0 Jul 28 23:43 ./dir/
-rw-r--r-- 1 mikeserv mikeserv 0 Jul 28 23:38 ./file

Well, mine gives me a slash following dir with or without -p, but I probably have that configured in $LS_COLORS or something.

Oh no - it's probably a shell alias.

yeah - that was it.

alias ls
ls='ls -CF --color=auto'

/usr/bin/ls -dl ./*
drwxr-xr-x 1 mikeserv mikeserv 0 Jul 28 23:43 ./dir
-rw-r--r-- 1 mikeserv mikeserv 0 Jul 28 23:38 ./file
/usr/bin/ls -dlp ./*
drwxr-xr-x 1 mikeserv mikeserv 0 Jul 28 23:43 ./dir/
-rw-r--r-- 1 mikeserv mikeserv 0 Jul 28 23:38 ./file

Tim

5:58 AM

(1) "which are ignored when you specify -d because that option is specifically for treating all files as a directory." Do you mean treat a dir as a file instead?

(2) with -p, dirs are output with / at the ends. without -p, no / at the ends of dirs.

mikeserv

No, vice versa. Like I said, ls does dentries - so when it does a listing it typically only has to stat the containing directory. When it treats each of its arguments as a directory in its own right, it stats each one.

-d is a lot more like find.

@Tim - this is the base spec for ls if you're interested: pubs.opengroup.org/onlinepubs/009695399/utilities/ls.html

Tim

(3) I am not sure about differences between ls -p * and ls */.

mikeserv

Well the -p is an argument to ls. It says ls please append the pathname separator to dir types.

The */ glob is an instruction to your shell which calls ls.

It says, shell, before calling ls, expand this glob out to represent every directory within the current one, and then hand that list to ls as arguments

Because ls read directories - not the files described by their dentries, it doesn't amount to much of a difference at all.

However, if you did ls -dp * and ls -p */ you might see the difference.

Tim

when shell expands */, it expands subdirs only and doesn't include non-dir files under the current dir, and put / at the end of each subdir name?

when shell expands *, it expands subdirs and files under the current dir, and doesn't put / at the end of each subdir name?

mikeserv

when the shell expands */ it expands to a list of all directories (not beginning with a .) in the current directory.

It does the same for all files in the current directory with just * - it does not limit the list to subdirs.

Tim

6:11 AM

Both */ and * are part of bash globbing?

mikeserv

Those globs are portably specified by POSIX - they should do the same in any shell. But yeah, nothing to do with ls.

Tim

** and *@ are not bash globs?

mikeserv

I believe those are - they're extensions on the spec.

In any case, you can the arguments yourself. Like: printf '%s\n' *

Tim

/ in */ doesn't mean the literal /, but @ in *@ mean the literal @.

mikeserv

Um, I honestly don't know about that @.

If it's not some bash thing, then it should expand to any file in the current directory ending with the character @

But ls uses it in the ls -F listings to represent... is it a link? or some file type - I forget which.

I think soft links.

But no, it doesn't mean the literal / because there aren't any filenames which contain a /.

Unless, of course, */ expands to nothing.

Tim

6:18 AM

Are dentries covered in OS textbooks, or in Linux/Unix books?

Are */ and * covered in bash books, but not in Linux/Unix books or OS textbooks?

mikeserv

Yes, definitely. They're how any programmer has to translate a filename to an inode number and back again. Which is needed when handling files with relation to the kernel.

I wouldn't know why the */ thing would be covered in one and not the other.

Tim

where is */ (and *) covered?

I'd like to learn more about globbing

mikeserv

Here's a the POSIX spec on pathname expansion: pubs.opengroup.org/onlinepubs/009695399/utilities/…

Tim

In the link, there is no */

mikeserv

@Tim - are you the guy that was asking about softlinking to directories?

Tim

6:25 AM

I don't remmeber

mikeserv

Well, you have to link through it. But still - it's there. It's number three:

Specified patterns shall be matched against existing filenames and pathnames, as appropriate. Each component that contains a pattern character shall require read permission in the directory containing that component. Any component, except the last, that does not contain a pattern character shall require search permission. For example, given the pattern:

/foo/bar/x*/bam

search permission is needed for directories / and foo, search and read permissions are needed for directory bar, and search permission is needed for each x* directory.

If the pattern matches any existing filenames or pathnames, the pattern shall be replaced with those filenames and pathnames, sorted according to the collating sequence in effect in the current locale. If the pattern contains an invalid bracket expression or does not match any existing filenames or pathnames, the pattern string shall be left unchanged.

There's also the pattern matching notation thing there.

Still, the spec can be a little difficult to swallow sometimes.

I mean, it's a bit like a stereo manual.

You can also just do man sh or man bash to learn more. That's probably a good place to start.

Tim

Does */ in echo ${pathname##*/} mean the same as */ in ls */?

mikeserv

This is better:

pubs.opengroup.org/onlinepubs/009695399/utilities/…

No.

Your ${pathname##*/} is a parameter expansion. It means strip from the head of this $pathname variable any and all characters up to and including the last / occuring within its value

so like:

var=aabbccdd

echo ${var##*c}
dd

But the weak form only matches as little as possible:

echo ${var#*c}
cdd

Tim

Does / in echo ${pathname##*/} mean the literal /?

mikeserv

Yeah.

But there are a couple of layers of expansion there.

Tim

6:36 AM

I don't know why / in */ doesn't mean literal /, but in echo ${pathname##*/} it does?

mikeserv

Of course, this can't happen because you strip them all, but if you had a literal / stored within a variable and expanded it can expand again to a pathname - because variable expansion happens first

In that case it's literal at first, and then it isn't.

Tim

Are these covered in bash books?

mikeserv

Because $pathname is its own expansion. The variable named pathname evaluates to a value. When you expand it with...

umm... any good ones will.

Tim

what good ones?

mikeserv

You already have the link to pattern matching thing though.

I don't know.

But $pathname is a string - it's not a file object. Any operation the shell performs on it will treat it as such.

Tim

6:39 AM

thanks for explaining. i really have a lot to catch up

mikeserv

So you store a string in the variable, then expand that string, and then it can expand to the rest.

Yeah, maybe. It mostly just comes with practice. And man.

Use man.

Tim

man doesn't tell me to understand the two examples with */

I want to sleep now.

mikeserv

good night

Tim

are you at night?

mikeserv

Uh, yeah. Are you not? You sleep dayside - vamp style?

Tim

6:42 AM

I am. it is too late

globs?

are those special characters ##*/ and %/* globs too?

mikeserv

They're special in that context, yeah.

The shell treats a lot of different types contextually.

They're not globs.

Well, the ## %% are not globs.

The *c is.

Kind of.

Tim

echo ${pathname##*/}
echo ${pathname%/*}

mikeserv

The ## %% are instructions from which end to strip and how much.

# means head and % means tail

Tim

what topics I can read to understand them?

mikeserv

Two of either means as much as possible and a single one means as little as.

Tim

6:47 AM

they are not globbing, and what are they?

mikeserv

This is the spec on parameter expansion: pubs.opengroup.org/onlinepubs/009695399/utilities/…

Tim

they are not regex either

mikeserv

They are not regex - they are shell patterns. It's described in the pattern matching notation link I gave you already.

globs use the same to glob

Tim

Will check that out in the day. thanks

mikeserv

Good luck, man.

Tim

6:50 AM

tnk.

Faheem Mitha

@mikeserv Yes, 50 min to 10 min. But only because I'm a clod.

mikeserv

@FaheemMitha - are we talking 50 mins years ago - or 50 mins executed on today's machine?

Faheem Mitha

@mikeserv Todays machine.

50 -> 10 min from a code rewriting.

mikeserv

Gee whiz - what did you do to so drastically improve it?

Faheem Mitha

Approximately, of course.

@mikeserv I was doing insanely stupid things. I stopped doing them. Do you want details?

mikeserv

6:57 AM

Well, maybe. I mean, was there an overarching method that you altered? It was more than a function decoration or something, I assume.

Faheem Mitha

Well, off to lunch, back in a bit.

mikeserv

Ok.

mikeserv

7:15 AM

@Braiam - just found this:

meta.unix.stackexchange.com/questions/3021/…

I upvoted it as soon as I read the stole, borrowed bit.

Faheem Mitha

7:41 AM

@mikeserv It's not a secret, just mildly embarrassing.

I didn't think I was that dumb.

Braiam

@mikeserv mm? That has been there for a week or so

mikeserv

@Braiam - Well, its age notwithstanding, I just now found it.

@FaheemMitha - Oh. Well what was the fundamental difference? There must have been some big standout thing, right? Something?

Faheem Mitha

@mikeserv Certainly. Short version: I was creating a bunch of files, and loading it into a db in parallel. The only problem is that I was creating and loading each file in pieces and loading each pieces of each file into the db directly. So, the runtime was dominated by the loading process.

This is sort of textbook moron. I really have to wonder what I was thinking. Ironically, the slower version is also significantly more complicated. Usually the optimized version is more complicated.

mikeserv

And the optimization came from loading them with parallel?

Faheem Mitha

@mikeserv The optimization came from (a) not breaking up each file into pieces. That doesn't make much sense to start with, and (b) waiting to the end to load the files into the db.

mikeserv

7:51 AM

Oh!

So you were pumping a bunch of parallel outputs into a single pipe, kinda?

Faheem Mitha

Actually, I'm looping over the files to load them. I'm wondering if concatenating them beforehand or piping them in to COPY would be faster.

mikeserv

You should stream when you can, maybe, as long as you can still distinguish them if necessary...

Faheem Mitha

@mikeserv Well, dbs can handle parallel loads. That's not the problem. The problem is that doing repeated loads thousands of times is incredibly expensive. For one thing, one has to drop and load constraints, which of course gets more expensive as the db gets bigger.

@mikeserv Well, I'll try it with a pipe. There is a link somewhere. One sec.

mikeserv

Is that maybe because it relinks at each load?

Faheem Mitha

2

A: Copy multiple CSV files into postgres

In linux pipe the output of the files listing to psql. Make copy use the standard input: cat /path_to/ys*.csv | psql -c 'COPY product(title, department) from stdin CSV HEADER' Look for the equivalent in other OSs

user44517

7:55 AM

anyone playing hadoop here?

i have an error ERROR: Can't get master address from ZooKeeper; znode data == null

Faheem Mitha

@mikeserv No, I think partly it is that dropping and recreating the constraints is expensive. I'm not sure.

I'm not sure what you mean by "relinks at each load", but I don't really know anything about db implementation.

mikeserv

@user44517 - what's hadoop? Is that literally a game? Or a play on words?

user44517

@_@

ok then

Faheem Mitha

I actually asked on dba.sx and they were like - well, yeah...

hadoop the db?

Apache Hadoop

Apache Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware. Hadoop is an Apache top-level project being built and used by a global community of contributors and users. It is licensed under the Apache License 2.0. The Apache Hadoop framework is composed of the following modules: Hadoop Common – contains libraries and utilities needed by other Hadoop modules. Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster...

mikeserv

hadoop.apache.org

@user44517 Not I.

user44517

7:58 AM

that thing is powerful but sucks for noob

mikeserv

@FaheemMitha I know only what they taught me in my cheapass online college course Intro to Database Management

Faheem Mitha

@mikeserv This is one of the reasons that working with other people is a good idea. The most useful function of a collaborator is to say - what the !^&&** are you doing, you incredible idiot?

mikeserv

And, really, less than that.

Faheem Mitha

@mikeserv Well, it is possible to learn things in a much larger and better university called the Internet.

mikeserv

But the tables have to be linked somehow - each item is keyed.

Faheem Mitha

8:00 AM

That's mostly where I've learnt things. I've never take a formal course in anything with the word computer in the title.

@mikeserv True. That's one kind of constraint. But there are others.

Usually the linking is done using foreign keys.

mikeserv

@FaheemMitha Of this I am, thankfully, already aware. But it never hurts to be reminded.

Faheem Mitha

One thing that strikes me listening to dbs chatter is how complicated the whole thing is, considering we are basically just talking about a bunch of tables. Wrt relational dbs, of course.

@mikeserv Formal universities are surprisingly dysfunctional places. The major advantage is the contacts you make. Also, they sometimes have good libraries, and journal access.

mikeserv

8:44 AM

@FaheemMitha - aren't they usually stocked with 20-somethings and/or nutty professors? I wouldn't call it shocking.

Faheem Mitha

8:57 AM

@mikeserv What, you mean that they are dysfunctional?

mikeserv

Yes.

Faheem Mitha

@mikeserv I don't see what is so dysfunctional about 20-somethings, though.

mikeserv

Have you ever been one?

The professors though - no argument there?

Faheem Mitha

@mikeserv Gosh, no. I think I skipped that decade. :-)

@mikeserv No argument. They're often as mad as hatters.

mikeserv

you're cracking me up, fm

Faheem Mitha

9:04 AM

Not all of them, of course. But tenured professors, at least, can be a pretty screwy bunch. The British variety of academic is particularly weird.

I thought the teens were supposed to be the off-kilter age.

Hormanal crazies etc. etc. As depicted in numerous American high-school films. Though there may be an element of exaggeration.

E.g. "Can't Hardly Wait".

mikeserv

Yeah, maybe, but I spent a lot of my 20s off kilter.

Faheem Mitha

@mikeserv Ok. Me too, I guess.

A US teenager once remarked to me that they don't make Hollywood films about good students. Which I suppose is true. It might not be particularly exciting watching people study.

mikeserv

@FaheemMitha - you were a good student?

Faheem Mitha

@mikeserv As a child, no. I was miserable and hated my school. I did like maths, though. I got better as an adult.

mikeserv

I liked maths too.

Faheem Mitha

9:17 AM

I think schools are actually designed to make you hate learning. I'm not kidding.

mikeserv

And the literature stuff.

Faheem Mitha

@mikeserv literature stuff?

mikeserv

I dunno... Mrs. Rahaman had two prominently encouraging features.

Well, here, it's some variant on English, usually.

Faheem Mitha

@mikeserv Who was Mrs. Rahaman?

You mean reading Shakespeare or Dickens, something like that?

mikeserv

My eighth grade algebra teacher.

Yes.

Faheem Mitha

9:20 AM

I think it is natural to like learning, but schools are good at beating it out of you. Of course, that was India.

The whole testing / artificial competition thing is particularly unhealthy. Good for teaching deference to authority, though. Which is what the people who run things like.

mikeserv

Well, I didn't fare too well in school the first couple tries anyway, but I'll never forget Mrs. R.

Faheem Mitha

@mikeserv Well, it's nice you had one teacher you had positive feelings about. I had an art teacher I got on quite well with, though she was quite eccentric. Though maybe part of the reason I liked her was because she was eccentric.

mikeserv

positive feelings is right

Faheem Mitha

"two prominently encouraging features"?

mikeserv

yes

Faheem Mitha

9:23 AM

Which were?

mikeserv

alluded to

Faheem Mitha

@mikeserv I must have missed that.

mikeserv

they don't do innuendo in india?

Faheem Mitha

@mikeserv Oh, I see.

:-) I didn't realise you were talking dirty. So, you didn't appreciate her teaching, then?

mikeserv

I appreciated it very much!

Every breathy sigh of it!

Faheem Mitha

9:27 AM

@mikeserv :-)

Faheem Mitha

9:43 AM

Well, good to know I'm not the only one who keeps weird hours.

Gnouc

10:53 AM

@mikeserv: Are you there?

mikeserv

yeah

Gnouc

Do you have a better ideal for my solution in this: unix.stackexchange.com/questions/147149/…

Like your comment, how can I avoid the two consecutive line which does not start with alpha?

mikeserv

@Gnouc - You're already pretty well setup to gather it on a branch test anyway, since you've got the label. So... hang on, maybe I can come up with something if I stare at it....

@Gnouc - maybe that's not even the best way, maybe mine's too complicated... This could be a very good candidate for a paired address - a range address.

sed '/[[:alpha:]]/,/[^[:alpha:]]/{/[[:alpha:]]/{N;s/\n/<tab>/;};}'

No. That does the opposite of what it should... damn...

ok, the branch thing...

@Gnouc:

sed ':a
/[^[:alpha:]]/{N;/\n[^[:alpha:]]/!ba;s/\(.*\)\n/\1\t/}
' <<\DATA
NAME_A
NAME_B
NAME_C
12,1
NAME_D
21,2
DATA
NAME_A
NAME_B
NAME_C  12,1
NAME_D  21,2

No.I

That's wrong - I don't know why it worked - I guess just because of the second address.

Anyway - the first one needs to not complement

/^[[:alpha:]]/{$!N

Anyway, for a line that starts with a alphabetic character, it pulls in the next if it's the last line, then if it can address a newline followed by a non alpha it selects everything in pattern space up to the last newline there and does a substitute on it - changing the last new line to a tab

But not it branches back and tries again.

Gnouc

11:18 AM

Oh, good job!

I never good at thinking sed-way

mikeserv

I like it cause it's so simple. There are no variables, no config options.

Everything that happens next is a direct result of what happened before.

Gnouc

Yeah, I can use it, but still thinking like perl-way or awk-way

Trying my best to change my mind :)

mikeserv

Whatever works best for you, man.

that whole right tool for the job thing is a bunch of baloney if you ask me.

they've all been around several decades at least - there's plenty of overlap.

Faheem Mitha

11:40 AM

@mikeserv either you skipped the sleeping bit entirely, or you are a really early riser.

mikeserv

I skipped it. I was just thinking about repaying that debt to myself though, to be honest.

goldilocks

3:28 PM

Anyone ever notice that even though the votes on a migrated question go with it, if one of those is yours, you can still upvote it again on the site it was migrated to?

Tim

not yet

terdon

@goldilocks Votes do not go with it. In theory anyway.

Tim

WOnder which one is better: gnu screen, or tmux

which one should I use?

terdon

Nope, @goldilocks, take that back, yes they are. Sounds like a bug there.

Gnouc

@Tim: try both and pick the one make you comfortable

Tim

3:31 PM

I used screen a while ago, and forgot how to use it.

nowadays, I heard of people yelling tmux to me on the internet.

very few mention screen

Does that mean tmux surpass screen? google.com/trends/…

Gnouc

I personally use tmux

Tim

I personally have a spot for screen, simply because I used it before

derobert

Personally, I use screen. Not for any good reason; just because I've always used screen...

Tim

good to hear that someone still using scren

bad because "not for any good reason"

derobert

shrug, it does what I need it to do, I've already got it configured, ...

Patrick

3:44 PM

@VolkerSiegel You've got 3 different people telling you to use SIGTERM. If you're such an expert on signals, why did you even ask the question? — Patrick 30 secs ago

Must not kill. Must not kill. Must not kill.

6

casey

@Tim the people shouting tmux are the same ones who shout htop and ncmpcpp to complete their holy grail desktop of text mode tiling windows

well, not all of them of course, just the loudest ones

Tim

Casey, I have trouble understand what you write. do you mean they are right that tmux is better than screen?

casey

@Tim I do not mean that

I mean they believe it to be best for them and are quite vocal that it must therefore be best for everyone else

if you want an example of that behavior with other software just throw emacs and vim into a sentence together.

I don't often use either screen or tmux, but when I need something like that I just need a detached remote session, and screen fits the bill better than tmux since I don't need tmux's ability to multiplex

Tim

what is multiplex? can screen do that?

Transcript for