« first day (1389 days earlier)   

1:37 AM
@Gilles: Are you there?
 
@Gnouc yes but not for long
 
OK, so let next time for more disscussion. I will edit my answer about pathname and filename to make it clearer
 
 
2 hours later…
4:03 AM
@FaheemMitha - 5 fold? That is a considerable optimization!
 
 
1 hour later…
Tim
5:11 AM
hi @mike. what does / mean?
 
Well, written like that, it means the root directory of filesystem.
But, more generally, it is the pathname separator.
 
Tim
/ in ls -d */
 
Filenames can contain really any character but null and /, because / is what delineates them.
So your shell has a thing called pathname expansion builtin.
 
Tim
ls -d * output directories without / at their ends
 
The *globstar means any/or/all
Right - I'll get there.
When you pair the *globstar and the pathname separator you're saying, shell I want an argument list amounting to any or all directories contained within this one
sh -cx 'ls */'
+ ls one/ three/ two/
one/:
three/:
two/:
@Tim - see that + bit?
That's actually the shell's evaluation of my command.
That's what actually happens.
So doing ls */ is the same in this case as if I had typed ls one/ two/ three/ because the shell expands that out before ever calling on ls at all.
You don't get the / at the ends because that's actually a quality of default settings - which are ignored when you specify -d because that option is specifically for treating all files as a directory.
 
Tim
5:21 AM
$ sh -cx 'ls */'
+ ls */
ls: cannot access */: No such file or directory
 
It's a little confusing, but the short answer is, specify -p also.
You must be in a directory with no child subdirectories.
As I said any or all. So, if it evaluates to none, it is not expanded, and the string is handed literally to the invoked process as an argument.
And that, as a case in point, is what is special about the / character.
The same thing would happen if there were subdirectories and you disabled pathname expansion, like with set -f.
 
Tim
A side question: does "null" in "Filenames can contain really any character but null and /" mean empty?
 
Typically when people talk about null they refer to the ASCII 0 byte. It's a long sordid story, but it goes back to punchcards. If you were processing data on punchcards, you could make filler space by just not punching and holes - that's a null byte
it is \000
There's also the concept of a null string - which is an empty expansion. so printf %s ''
printf gets an argument there, but it is empty - the null string.
So I can do: printf '%s this still gets printed' ''
But a null byte, nowadays, is not a blank punchcard, it's an actual byte. Anyway, it is still forbidden in filenames. Only it and the pathname separator / share that distinction.
 
Tim
5:36 AM
"You don't get the / at the ends because that's actually a quality of default settings" what does the default setting says?
show directories without / at their ends?
 
A lot of things, but by default ls distinguishes by filetype.
Ok... welll... this goes deeper because it's about dentries.
So files are not the names you cal them.
files are inode numbers.
Hang on...
touch file
ls -i
397399 file
A directory is special type of file - it's a file's file - i guess.
It stores information about files.
When you move a file from directory to directory you directly affect the file - you affect which directory stores information about it.
The filename is stored in the directory as a quality of the file's directory entry - its dentry.
The filename refers to an inode - which is how the kernel refers to a file.
A hardlink is another dentry for the same inode - just different listings for the same file. It is not a quality of the file, but a quality of the directory listing it.
ls does dentries.
That's what it does - it reads the directory and reports on its listings.
It doesn't do too much with the files within unless you tell it to - with -d.
That's when it treats each file as a directory and reads it.
ls -dlp ./*
drwxr-xr-x 1 mikeserv mikeserv 0 Jul 28 23:43 ./dir/
-rw-r--r-- 1 mikeserv mikeserv 0 Jul 28 23:38 ./file
Well, mine gives me a slash following dir with or without -p, but I probably have that configured in $LS_COLORS or something.
Oh no - it's probably a shell alias.
yeah - that was it.
alias ls
ls='ls -CF --color=auto'
/usr/bin/ls -dl ./*
drwxr-xr-x 1 mikeserv mikeserv 0 Jul 28 23:43 ./dir
-rw-r--r-- 1 mikeserv mikeserv 0 Jul 28 23:38 ./file
/usr/bin/ls -dlp ./*
drwxr-xr-x 1 mikeserv mikeserv 0 Jul 28 23:43 ./dir/
-rw-r--r-- 1 mikeserv mikeserv 0 Jul 28 23:38 ./file
 
Tim
5:58 AM
(1) "which are ignored when you specify -d because that option is specifically for treating all files as a directory." Do you mean treat a dir as a file instead?
(2) with -p, dirs are output with / at the ends. without -p, no / at the ends of dirs.
 
No, vice versa. Like I said, ls does dentries - so when it does a listing it typically only has to stat the containing directory. When it treats each of its arguments as a directory in its own right, it stats each one.
-d is a lot more like find.
@Tim - this is the base spec for ls if you're interested: pubs.opengroup.org/onlinepubs/009695399/utilities/ls.html
 
Tim
(3) I am not sure about differences between ls -p * and ls */.
 
Well the -p is an argument to ls. It says ls please append the pathname separator to dir types.
The */ glob is an instruction to your shell which calls ls.
It says, shell, before calling ls, expand this glob out to represent every directory within the current one, and then hand that list to ls as arguments
Because ls read directories - not the files described by their dentries, it doesn't amount to much of a difference at all.
However, if you did ls -dp * and ls -p */ you might see the difference.
 
Tim
when shell expands */, it expands subdirs only and doesn't include non-dir files under the current dir, and put / at the end of each subdir name?
when shell expands *, it expands subdirs and files under the current dir, and doesn't put / at the end of each subdir name?
 
when the shell expands */ it expands to a list of all directories (not beginning with a .) in the current directory.
It does the same for all files in the current directory with just * - it does not limit the list to subdirs.
 
Tim
6:11 AM
Both */ and * are part of bash globbing?
 
Those globs are portably specified by POSIX - they should do the same in any shell. But yeah, nothing to do with ls.
 
Tim
** and *@ are not bash globs?
 
I believe those are - they're extensions on the spec.
In any case, you can the arguments yourself. Like: printf '%s\n' *
 
Tim
/ in */ doesn't mean the literal /, but @ in *@ mean the literal @.
 
Um, I honestly don't know about that @.
If it's not some bash thing, then it should expand to any file in the current directory ending with the character @
But ls uses it in the ls -F listings to represent... is it a link? or some file type - I forget which.
I think soft links.
But no, it doesn't mean the literal / because there aren't any filenames which contain a /.
Unless, of course, */ expands to nothing.
 
Tim
6:18 AM
Are dentries covered in OS textbooks, or in Linux/Unix books?
Are */ and * covered in bash books, but not in Linux/Unix books or OS textbooks?
 
Yes, definitely. They're how any programmer has to translate a filename to an inode number and back again. Which is needed when handling files with relation to the kernel.
I wouldn't know why the */ thing would be covered in one and not the other.
 
Tim
where is */ (and *) covered?
I'd like to learn more about globbing
 
Here's a the POSIX spec on pathname expansion: pubs.opengroup.org/onlinepubs/009695399/utilities/…
 
Tim
In the link, there is no */
 
@Tim - are you the guy that was asking about softlinking to directories?
 
Tim
6:25 AM
I don't remmeber
 
Well, you have to link through it. But still - it's there. It's number three:
Specified patterns shall be matched against existing filenames and pathnames, as appropriate. Each component that contains a pattern character shall require read permission in the directory containing that component. Any component, except the last, that does not contain a pattern character shall require search permission. For example, given the pattern:
/foo/bar/x*/bam
search permission is needed for directories / and foo, search and read permissions are needed for directory bar, and search permission is needed for each x* directory.
If the pattern matches any existing filenames or pathnames, the pattern shall be replaced with those filenames and pathnames, sorted according to the collating sequence in effect in the current locale. If the pattern contains an invalid bracket expression or does not match any existing filenames or pathnames, the pattern string shall be left unchanged.
There's also the pattern matching notation thing there.
Still, the spec can be a little difficult to swallow sometimes.
I mean, it's a bit like a stereo manual.
You can also just do man sh or man bash to learn more. That's probably a good place to start.
 
Tim
Does */ in echo ${pathname##*/} mean the same as */ in ls */?
 
This is better:
No.
Your ${pathname##*/} is a parameter expansion. It means strip from the head of this $pathname variable any and all characters up to and including the last / occuring within its value
so like:
var=aabbccdd
echo ${var##*c}
dd
But the weak form only matches as little as possible:
echo ${var#*c}
cdd
 
Tim
Does / in echo ${pathname##*/} mean the literal /?
 
Yeah.
But there are a couple of layers of expansion there.
 
Tim
6:36 AM
I don't know why / in */ doesn't mean literal /, but in echo ${pathname##*/} it does?
 
Of course, this can't happen because you strip them all, but if you had a literal / stored within a variable and expanded it can expand again to a pathname - because variable expansion happens first
In that case it's literal at first, and then it isn't.
 
Tim
Are these covered in bash books?
 
Because $pathname is its own expansion. The variable named pathname evaluates to a value. When you expand it with...
umm... any good ones will.
 
Tim
what good ones?
 
You already have the link to pattern matching thing though.
I don't know.
But $pathname is a string - it's not a file object. Any operation the shell performs on it will treat it as such.
 
Tim
6:39 AM
thanks for explaining. i really have a lot to catch up
 
So you store a string in the variable, then expand that string, and then it can expand to the rest.
Yeah, maybe. It mostly just comes with practice. And man.
Use man.
 
Tim
man doesn't tell me to understand the two examples with */
I want to sleep now.
 
good night
 
Tim
are you at night?
 
Uh, yeah. Are you not? You sleep dayside - vamp style?
 
Tim
6:42 AM
I am. it is too late
globs?
are those special characters ##*/ and %/* globs too?
 
They're special in that context, yeah.
The shell treats a lot of different types contextually.
They're not globs.
Well, the ## %% are not globs.
The *c is.
Kind of.
 
Tim
echo ${pathname##*/}
echo ${pathname%/*}
 
The ## %% are instructions from which end to strip and how much.
# means head and % means tail
 
Tim
what topics I can read to understand them?
 
Two of either means as much as possible and a single one means as little as.
 
Tim
6:47 AM
they are not globbing, and what are they?
 
This is the spec on parameter expansion: pubs.opengroup.org/onlinepubs/009695399/utilities/…
 
Tim
they are not regex either
 
They are not regex - they are shell patterns. It's described in the pattern matching notation link I gave you already.
globs use the same to glob
 
Tim
Will check that out in the day. thanks
 
Good luck, man.
 
Tim
6:50 AM
tnk.
 
@mikeserv Yes, 50 min to 10 min. But only because I'm a clod.
 
@FaheemMitha - are we talking 50 mins years ago - or 50 mins executed on today's machine?
 
@mikeserv Todays machine.
50 -> 10 min from a code rewriting.
 
Gee whiz - what did you do to so drastically improve it?
 
Approximately, of course.
@mikeserv I was doing insanely stupid things. I stopped doing them. Do you want details?
 
6:57 AM
Well, maybe. I mean, was there an overarching method that you altered? It was more than a function decoration or something, I assume.
 
Well, off to lunch, back in a bit.
 
Ok.
 
7:15 AM
@Braiam - just found this:
I upvoted it as soon as I read the stole, borrowed bit.
 
7:41 AM
@mikeserv It's not a secret, just mildly embarrassing.
I didn't think I was that dumb.
 
@mikeserv mm? That has been there for a week or so
 
@Braiam - Well, its age notwithstanding, I just now found it.
@FaheemMitha - Oh. Well what was the fundamental difference? There must have been some big standout thing, right? Something?
 
@mikeserv Certainly. Short version: I was creating a bunch of files, and loading it into a db in parallel. The only problem is that I was creating and loading each file in pieces and loading each pieces of each file into the db directly. So, the runtime was dominated by the loading process.
This is sort of textbook moron. I really have to wonder what I was thinking. Ironically, the slower version is also significantly more complicated. Usually the optimized version is more complicated.
 
And the optimization came from loading them with parallel?
 
@mikeserv The optimization came from (a) not breaking up each file into pieces. That doesn't make much sense to start with, and (b) waiting to the end to load the files into the db.
 
7:51 AM
Oh!
So you were pumping a bunch of parallel outputs into a single pipe, kinda?
 
Actually, I'm looping over the files to load them. I'm wondering if concatenating them beforehand or piping them in to COPY would be faster.
 
You should stream when you can, maybe, as long as you can still distinguish them if necessary...
 
@mikeserv Well, dbs can handle parallel loads. That's not the problem. The problem is that doing repeated loads thousands of times is incredibly expensive. For one thing, one has to drop and load constraints, which of course gets more expensive as the db gets bigger.
@mikeserv Well, I'll try it with a pipe. There is a link somewhere. One sec.
 
Is that maybe because it relinks at each load?
 
2
A: Copy multiple CSV files into postgres

Clodoaldo NetoIn linux pipe the output of the files listing to psql. Make copy use the standard input: cat /path_to/ys*.csv | psql -c 'COPY product(title, department) from stdin CSV HEADER' Look for the equivalent in other OSs

 
7:55 AM
anyone playing hadoop here?
i have an error ERROR: Can't get master address from ZooKeeper; znode data == null
 
@mikeserv No, I think partly it is that dropping and recreating the constraints is expensive. I'm not sure.
I'm not sure what you mean by "relinks at each load", but I don't really know anything about db implementation.
 
@user44517 - what's hadoop? Is that literally a game? Or a play on words?
 
@_@
ok then
 
I actually asked on dba.sx and they were like - well, yeah...
hadoop the db?
Apache Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware. Hadoop is an Apache top-level project being built and used by a global community of contributors and users. It is licensed under the Apache License 2.0. The Apache Hadoop framework is composed of the following modules: Hadoop Common – contains libraries and utilities needed by other Hadoop modules. Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster...
 
@user44517 Not I.
 
7:58 AM
that thing is powerful but sucks for noob
 
@FaheemMitha I know only what they taught me in my cheapass online college course Intro to Database Management
 
@mikeserv This is one of the reasons that working with other people is a good idea. The most useful function of a collaborator is to say - what the !^&&** are you doing, you incredible idiot?
 
And, really, less than that.
 
@mikeserv Well, it is possible to learn things in a much larger and better university called the Internet.
 
But the tables have to be linked somehow - each item is keyed.
 
8:00 AM
That's mostly where I've learnt things. I've never take a formal course in anything with the word computer in the title.
@mikeserv True. That's one kind of constraint. But there are others.
Usually the linking is done using foreign keys.
 
@FaheemMitha Of this I am, thankfully, already aware. But it never hurts to be reminded.
 
One thing that strikes me listening to dbs chatter is how complicated the whole thing is, considering we are basically just talking about a bunch of tables. Wrt relational dbs, of course.
@mikeserv Formal universities are surprisingly dysfunctional places. The major advantage is the contacts you make. Also, they sometimes have good libraries, and journal access.
 
8:44 AM
@FaheemMitha - aren't they usually stocked with 20-somethings and/or nutty professors? I wouldn't call it shocking.
 
8:57 AM
@mikeserv What, you mean that they are dysfunctional?
 
Yes.
 
@mikeserv I don't see what is so dysfunctional about 20-somethings, though.
 
Have you ever been one?
The professors though - no argument there?
 
@mikeserv Gosh, no. I think I skipped that decade. :-)
@mikeserv No argument. They're often as mad as hatters.
 
you're cracking me up, fm
 
9:04 AM
Not all of them, of course. But tenured professors, at least, can be a pretty screwy bunch. The British variety of academic is particularly weird.
I thought the teens were supposed to be the off-kilter age.
Hormanal crazies etc. etc. As depicted in numerous American high-school films. Though there may be an element of exaggeration.
E.g. "Can't Hardly Wait".
 
Yeah, maybe, but I spent a lot of my 20s off kilter.
 
@mikeserv Ok. Me too, I guess.
A US teenager once remarked to me that they don't make Hollywood films about good students. Which I suppose is true. It might not be particularly exciting watching people study.
 
@FaheemMitha - you were a good student?
 
@mikeserv As a child, no. I was miserable and hated my school. I did like maths, though. I got better as an adult.
 
I liked maths too.
 
9:17 AM
I think schools are actually designed to make you hate learning. I'm not kidding.
 
And the literature stuff.
 
@mikeserv literature stuff?
 
I dunno... Mrs. Rahaman had two prominently encouraging features.
Well, here, it's some variant on English, usually.
 
@mikeserv Who was Mrs. Rahaman?
You mean reading Shakespeare or Dickens, something like that?
 
My eighth grade algebra teacher.
Yes.
 
9:20 AM
I think it is natural to like learning, but schools are good at beating it out of you. Of course, that was India.
The whole testing / artificial competition thing is particularly unhealthy. Good for teaching deference to authority, though. Which is what the people who run things like.
 
Well, I didn't fare too well in school the first couple tries anyway, but I'll never forget Mrs. R.
 
@mikeserv Well, it's nice you had one teacher you had positive feelings about. I had an art teacher I got on quite well with, though she was quite eccentric. Though maybe part of the reason I liked her was because she was eccentric.
 
positive feelings is right
 
"two prominently encouraging features"?
 
yes
 
9:23 AM
Which were?
 
alluded to
 
@mikeserv I must have missed that.
 
they don't do innuendo in india?
 
@mikeserv Oh, I see.
:-) I didn't realise you were talking dirty. So, you didn't appreciate her teaching, then?
 
I appreciated it very much!
Every breathy sigh of it!
 
9:27 AM
@mikeserv :-)
 
9:43 AM
Well, good to know I'm not the only one who keeps weird hours.
 
 
1 hour later…
10:53 AM
@mikeserv: Are you there?
 
yeah
 
Do you have a better ideal for my solution in this: unix.stackexchange.com/questions/147149/…
Like your comment, how can I avoid the two consecutive line which does not start with alpha?
 
@Gnouc - You're already pretty well setup to gather it on a branch test anyway, since you've got the label. So... hang on, maybe I can come up with something if I stare at it....
@Gnouc - maybe that's not even the best way, maybe mine's too complicated... This could be a very good candidate for a paired address - a range address.
sed '/[[:alpha:]]/,/[^[:alpha:]]/{/[[:alpha:]]/{N;s/\n/<tab>/;};}'
No. That does the opposite of what it should... damn...
ok, the branch thing...
@Gnouc:
sed ':a
/[^[:alpha:]]/{N;/\n[^[:alpha:]]/!ba;s/\(.*\)\n/\1\t/}
' <<\DATA
NAME_A
NAME_B
NAME_C
12,1
NAME_D
21,2
DATA
NAME_A
NAME_B
NAME_C  12,1
NAME_D  21,2
No.I
That's wrong - I don't know why it worked - I guess just because of the second address.
Anyway - the first one needs to not complement
/^[[:alpha:]]/{$!N
Anyway, for a line that starts with a alphabetic character, it pulls in the next if it's the last line, then if it can address a newline followed by a non alpha it selects everything in pattern space up to the last newline there and does a substitute on it - changing the last new line to a tab
But not it branches back and tries again.
 
11:18 AM
Oh, good job!
I never good at thinking sed-way
 
I like it cause it's so simple. There are no variables, no config options.
Everything that happens next is a direct result of what happened before.
 
Yeah, I can use it, but still thinking like perl-way or awk-way
Trying my best to change my mind :)
 
Whatever works best for you, man.
that whole right tool for the job thing is a bunch of baloney if you ask me.
they've all been around several decades at least - there's plenty of overlap.
 
11:40 AM
@mikeserv either you skipped the sleeping bit entirely, or you are a really early riser.
 
I skipped it. I was just thinking about repaying that debt to myself though, to be honest.
 

« first day (1389 days earlier)