« first day (3348 days earlier)      last day (40 days later) » 

Tim
12:08 PM
Good morning!
Today I watch a video from a guru about deploying something. Very enlightening! @StephenKitt
 
@Tim glad you enjoyed it!
 
o/ ..ooO( Good morning! )
hmmm, guess ASCII-art isn't one of my skills
 
@JeffSchaller maybe you had to lose your ASCII-fu to get your Google-fu back
 
mmmm, possibly! The one google search I did this morning was successful ...
and I won't say how many times I had to type o before it came out lower-case
 
Tim
For anyone interested in databases and distributed systems, I found Design Data Intensive Applications has good coverage. Not sure if anyone has heard of it or read it.
 
12:21 PM
"The Big Ideas Behind Reliable, Scalable, and Maintainable Systems"?
 
Tim
yes
Another book "Database Internals" seems a good one too. I haven't read it
 
I hadn't heard of it, so thank you!
 
Tim
I have the feeling that they are written for practitioners but also have academic traits
 
haha! A coworker used to have a sticker with "There is no cloud — it's just someone else's computer." (from youtube.com/watch?v=oRZoeDRACrY -- "End-to-end encryption: Behind the scenes" by Martin Kleppmann, Diana Vasile)
 
1:22 PM
@Kusalananda since the OP is specifically asking about parallel, it might make more sense to use parallel as the main solution and offer xargs as the alternative here.
Especially since parallel doesn't need any loop or printf and can just do parallel -P 16 sha256sum ::: *
Ah no, the OP wants each one to be in its own file.
Still possible, but slightly more complex. Never mind.
parallel -P 16 sha256sum {} ">"{}.sha ::: *
Parallel really is ridiculously powerful.
 
1:39 PM
@terdon Yes, I opted for the simpler xargs syntax and then to say "just replace xargs with parallel".
parallel is really not needed here.
I believe they reference it since they may not know that xargs can do the same thing. And it's got "parallel" in its name.
 
2:01 PM
@Kusalananda Well, it makes everything easier.
Not strictly needed, no, but it lets you avoid the shell loop and gives you more fine grained control.
All you need is parallel -P 16 sha256sum {} ">"{}.sha ::: *
I don't really see that as simpler than
printf '%s\0' * | xargs -0 -n 1 -P 16 sh -c 'sha256sum "$1" > "$1".sha' sh
 
The quoted ">" is odd, and I'm assuming that you'd need more specialised quoting if you want to do other things like piping?
 
@Kusalananda piping probably not, I'm not sure. But yes, the ">" is odd. But I find the extra sh` for xargs very odd too. Unless you're very familiar with this sort of thing, you'll have no idea what it's doing or why it's needed. But, de gustibus...
 
My thing is more general as it uses sh -c to launch a generic in-line script in a generally well understood fashion.
 
@Kusalananda Heh, understood by some :) I had to read that thing several times to understand it.
But then I never use xargs.
What does the -n 1 do?
 
Well, it's the same semantics as when you launch sh -c from find.
The -n 1 could probably be replace by -L 1. "Give one argument".
 
2:05 PM
It is, yes. But I very rarely do that. But sure, I know what the "sh" is for, I'm just saying it won't be obvious to the non-expert.
 
In fact, I'll change that.
 
What's the difference?
 
Something to do with lines.
 
-L max-lines
Use at most max-lines nonblank input lines per command line.
Trailing blanks cause an input line to be logically continued on
the next input line. Implies -x.

-n max-args, --max-args=max-args
Use at most max-args arguments per command line. Fewer than
max-args arguments will be used if the size (see the -s option)
is exceeded, unless the -x option is given, in which case xargs
will exit.
I guess -L does make more sense, yes.
 
I'm just not sure how it plays with -0.
(no actual lines)
Need to test
 
2:07 PM
No idea. I doubt I have even run xargs more than a dozen times in my entire life.
 
There does not seem to be a big difference.
I'm using it every now and again. Not often though.
And now that I'm on zsh, I'm using zargs instead (a loadable function).
 
I really need to switch to zsh. All the cool kids are using it, dammit.
 
Well, it does render both xargs and find obsolete, so to speak. Apart from the speed issues that we discussed the other day.
I mainly use it for the glob modifiers though.
And the default failglob (equivalent) and **.
Oh, and the non-splitting/globbing of unquoted expansions.
 
All I know is that each time I've posted some convoluted bash thing to answer a question, Stéphane comes along with his "In zsh, just say the magic word and it works" answer.
 
What really brought me over was some question about getting the N most recently modified files from a directory hierarchy. That's a short glob modifier in zsh.
dir/**/*(.NDom[1,5])
 
2:12 PM
yes, that's one of the ones I've seen
 
2:27 PM
@Kusalananda came here to say this, too
(and good afternoon, you two!)
 
@JeffSchaller Hi there! You mean that zsh rocks, or that I should focus that answer on GNU parallel rather than xargs?
 
@Kusalananda zsh rocks for N most/least recent file solutions
feels like a very "shell"-y thing to do
 
No need to parse pathnames.
I mean, worry about newlines and nuls etc.
 
it can still end up looking a little bit like line noise, but useful is useful
 
 
4 hours later…
6:16 PM
@Jesse_b I'm sure someone has figured out how to attach enough rockets and/or jet engines to accomplish that.
(And maybe they even lived to tell the tale, at least after a several month hospital stay)
@Kusalananda I wonder if it has better ways (other than the obvious answer, switch to Perl) of doing something like this...
groups=('-mtime -45' '! -mtime -45')
for group in "${groups[@]}"; do
    find -maxdepth 1 -type d -not -name '.' \( $group \) -print0 \
        | grep -xzvFf <(find -maxdepth 2 -type f \( -name 'WAITING' -or -name 'WAITING.*' \) | cut -d/ -f1-2) \
        | xargs -0 du $du_args | sed -e 's!\./!!' \
        | sort -nr \
        | egrep -v '^[01]\s'
    echo
done
(And yes, that gives a list of directories, with their sizes in front, excluding directories that have a WAITING or WAITING.* file in them, and excluding directories which are tiny)
 
7:18 PM
@derobert Some of that: du -hs -- ./**/*(/e['set -- $REPLY/WAITING.*(N); [[ $# -eq 0 ]] && [[ ! -e $REPLY/WAITING ]]'])
 
Wow, shell code inside an expansion to filter the expansion, that does sound powerful.
 
It is, but I'm very likely not using it correctly. There's probably a snappier way to do it.
 
 
1 hour later…
8:35 PM
Sometimes you try to be positive, but the other person's just not in a positive mood :-). unix.stackexchange.com/questions/556196/…
 
9:15 PM
Can anyone point me to a brief description of how pattern matching algorithms work? Like regexes, but more generally. I'm currently trying to deal with Lua patterns.
I'm not clear how these things arrive at a match.
 
Well, here's one interesting take on how (not to) implement regexes: swtch.com/~rsc/regexp/regexp1.html
 
@FaheemMitha Work, as in algorithms to implement them? Or more at a user level?
 
@FaheemMitha This is much too broad of a question to have a brief answer
 
"They start at the start, see what they can match there, then go back and start again at the next letter if they don't match until the end"
or should I just say "they start and the start, see what they match and go from there"
 
(Some) pattern-matching algorithms operate on things that don't have starts
 
9:22 PM
well, regexes are used for strings, so if we're in that context, then there's probably a start somewhere
 
For regular expressions specifically, there is a visualisation tool I built here, which runs the NFA/DFA constructions
 
cool
I can't remember my NFA's that well, but it seems to generate some unnecessary-looking states for some expressions. Ones where ε is the only way out, like with the re a(xx) (vs. axx) Is it just me?
 
It doesn't condense them, yeah, it's purely syntax-driven
 
yeh
 
If you put +reverse into the regular expression box you get a "Reverse" button that generates the FA that match the reverse string, so you can double-click that to get the minimal automata
 
9:35 PM
:)
 
9:49 PM
@derobert User level. As background, I should say, for those who aren't aware, that my normal reaction to having to deal with regular expressions and their pattern matching cousins is to scream and run. But in the case of Lua, which doesn't have much by way of libraries, one had to handle things with patterns that other languages would offer libraries for.
In this particular situation, I'm trying to extract the basename of a file, given its full path. Python, for example, has a simple library for such things.
But to return to my question, Lua patterns, at least, don't appear exactly deterministic.
Based on the sketchy documetation I have at any rate. For example, how are conflcting requirements resolved?
 
string.match(path, "[^/]+$")?
 
Yes, in fact "([^/]+)$" is what I have to get rid of the forward slashes. But I also wanted to get rid of everything after the final ., including the dot, assuming there is one.
 
@sourcejedi I'm trying to be friendly to them. They want to reopen the question because they spent time on it. That is IMHO not a valid reason to reopen a dupe. Also, the text of the question is opinionated throughout ("not in a positive mood" as you say), which is another reason I'm not going to reopen it.
 
Lua patterns are just not-very-good regular expressions, and regular expressions are deterministic in a "does this match" sense, but they can match multiple different subsequences and the particular library might choose a different one to give you first/at all
I don't have lua in front of me now to try out ([^/]+)([.][^./]+)?$
 
@MichaelHomer Yes, one issue is that if there are multiple matches, how are they handled?
 
10:00 PM
I have a feeling you can't do (xy)? in them
 
I'm using "(.-)%.?[^%.]*$" for getting everything before the last .
 
Most practical systems are greedy (longest match) by default
 
This works on the cases I've tried, at least. But I'm still not very happy about my poor understanding of the logic involved.
 
For complex string processing in Lua I'd probably go to LPEG pretty quickly from the looks of this documentation
 
How does [.] differ from plain .?
@MichaelHomer Oh, I didn't think of that. Good point.
 
10:02 PM
I wanted it not to be a wildcard and I don't remember how to escape it
So I made a character class that just holds .
 
@MichaelHomer You prefix it with a %.
Is LPEG generally considered better than the out of the box Lua patterns?
And is the documentation better?
One thing I was wondering. Suppose the pattern is greedy at both ends? Which end wins?
 
PEG are a fundamentally different complete parsing system and it feels like overkill for this problem, but I'm not sure you can do it in one step with the regular expressions
 
This looks relatively complete and through, at least by Lua's standards - lua-users.org/wiki/LpegTutorial
@MichaelHomer Yes, it probably is overkill. The natural way to this would be to use a library, IMO. Not screw around with matching.
 
Usually the leftmost item consumes as much as it can first, and if the whole thing fails then backtracking happens right-to-left
 
@MichaelHomer Is that standard behavior across systems?
 
10:07 PM
i.e. (a*)(a*) will tend to result in a full group 1 and empty group 2
Not standard, no, but it's the most natural implementation
 
@MichaelHomer I see. By backtracking from right-to-left, you mean it backs up one character and tries again?
@MichaelHomer Do you think LPEG could do it in one step?
 
I mean it goes back to the last place it consumed something it didn't have to, and doesn't consume it this time
 
@MichaelHomer So it retains some state between tries?
 
PEGs can parse whole context-free grammars in one (external) step
@FaheemMitha It's usually recursive
 
@MichaelHomer I don't know what that sentence means, but ok.
 
10:12 PM
@FaheemMitha Performance can be quite bad, consequently, when a lot of backtracking is required
But that probably doesn't matter for your case, so if you're getting the right answers out I'd just call it a day and move on
 
@MichaelHomer If there is a lot of parsing, yes, I would imagine so.
@MichaelHomer Yes, I think I'll do that.
 
If you can get to PCRE somehow ([^/]+)(\.[^/.]*)?$ will give you what you want in captured group 1
I don't think a single match can do it with the built-in Lua ones
 
@MichaelHomer Yes, I did it in two steps.
There are PCRE libraries for Lua. But I'm not sure how reliable/well-maintained they are.
The Lua ecosystem is on the sketchy side. I think I've now used sketchy thrice.
is PCRE preferable to (L)PEG? Or are they just different?
 
PEGs are essentially a way of representing a whole context-free grammar (your friend Chomsky!) and then you try to apply the whole grammar against an input to get back a yes/no and a syntax tree if it worked. Internally they're very complicated but from the outside, if you're already thinking in terms of a grammar, they just work magic
They are just different
PCRE is regular expressions plus a bunch of non-regular extensions and syntactic shorthands that make them much more pleasant to write
 
It turns out Debian has the Lua Rex libraries. E.g. lua-rex-pcre. How out of date, I don't know. Some of its Lua libraries are years out of date. I don't know why.
The maintainer of Luarocks is MIA, for example.
Perhaps they just don't have the manpower.
 
10:20 PM
The maintainer of Luarocks is definitely not MIA, I know him
 
@MichaelHomer The Debian maintainer.
Sorry for the imprecise language.
 
Ah. Debian is a problem like that, yeah
 
Last Rex update, Tue, 06 Sep 2016 in Debian.
 
I sort of suspect that installing Luarocks separately from APT would work better in any case
 
@MichaelHomer Possibly, but a local installation of something that is supposed to manage local installed makes me uncomfortable.
I've generally had good experiences with Debian packages, out of date though they may often be.
 
10:25 PM
The interaction between distribution package managers and language-specific package managers is a bit of a mess everywhere, not just Debian
 
@MichaelHomer Thank you for the education. Have you worked on these things, or studied them in a class or something?
 
LPEG does look like it has more muscle than the Lua patterns. I'll put a pin in it and check it out further when the next pattern matching event comes up.
 
Hisham is the LuaRocks person
 
@MichaelHomer I see.
 
10:30 PM
The super-short version of the paper is that it's very complicated and all the answers have major problems, but here are some names to describe those anyway
 
@MichaelHomer Oh, I see you're a co-author.
 
Always cite your own work
 
Sounds like a smart move.
 
@Kusalananda I don't expect there's anything you could have said they would have taken as friendly :-). My last comment was hard-disengage and just see if I could make them feel a bit better. 'cos having your question closed sounds bad at first, but (from my POV) having your question closed as duplicate with valid answers is actually positive.
 
@sourcejedi Well, I asked them to open a new question on Meta about it, and so they did.
 
10:36 PM
@Kusalananda oh, didn't see that.
 
@sourcejedi I'll answer it if nobody else does (but that may not be a good ideas since I comment on their original question), but not tonight.
 
@Kusalananda reasonable :-)
@Kusalananda I assume "all links get redirected" was a complaint about the post notice. I didn't think the site does a weird instant redirect to the duplicated-against question, in any situation.
 
@sourcejedi I believe it does, for low-rep users (I don't know the actual criteria).
 
@Kusalananda so if I try a private browsing window...?
 
@sourcejedi Yes, that will insta-redirect, but it may be because you're not logged in?
 
10:42 PM
@Kusalananda confirmed. Thanks!
I can see why that feels sucky then.
 
11:04 PM
I'm surprised at the idea Google does anything useful with "Duplicate questions that use different words to describe the same problem", if we serve Google a redirect to the dupe target. I would have guessed Google would ignore the original, either assuming it's some sort "we;re redirecting you now" message, or as some weird SEO trick.
 
11:47 PM
@MichaelHomer The page doesn't seem to recognize the negated bracket expression [^a].
 

« first day (3348 days earlier)      last day (40 days later) »