/dev/chat

11:45 AM

@Tim In terms of the Unix utility cat, evaluating sequences would probably equate to reading them from disk. Doing this in parallel would be possible, but parallel reading quickly becomes I/O bound if you only have a single disk. The cat utility would have to implement large enough buffers to read all input into memory, and then proceed to start writing the sequences (data) as they are read to completion, in order.

Closest I can come up with is car on lists

This would stop the utility from working on arbitrary sized data and would, for many inputs, quickly saturate disk I/O.

If this is not actually regarding the cat utility, then I'm not quite sure how I would understood that.

For the cat utility, no benefit would be had in parallel reading, as the writing is by necessity sequential.

Tim

12:15 PM

@Kusalananda Thanks. That is from real world perspective, and is also what I hope to hear

The book however was perhaps written in a way ignoring the details. The two input expressions can be obtained by evaluating other expressions, such as seq expressions.

car =/= cat

unless

cat car! And then we're only one step away from KITT car

@JeffSchaller WHO SUMMONED ME

I WILL NOW INVOKE EDITING RE-EDUCATION

Seriously unix.stackexchange.com/review/suggested-edits/316509 aargh

It was a coin flip between you or Michael Homer :)

"What is it, Michael?"

@JeffSchaller LOL

We’re unmasked

@StephenKitt you're actually Chris Blasius from Olmsted Falls, Ohio

(maker of a KITT car replica)

12:32 PM

@JeffSchaller well, we’ve got the same shoes

and I've never seen you and Chris Blasius in the same room, so....

@JeffSchaller then again, no one has ever seen you and me in the same room

(dramatic music intensifies)

try not to come down too hard on Łukasz; I would have improved that edit, but it's going in the right direction. Unless you intend to go after the approvers?

Hold on, we elected one of Stephen's sock puppets as mod? Or is Jeff's sock puppet the user with the fourth most reputation on the site?

@JeffSchaller I’m trying not to come down too hard on him. But I’m getting the impression he’s not paying much attention to the reject reasons given to him, or the comments left on some of his approved edits. (I’ve given up on approvers, that’s playing whack-a-mole.)

@Kusalananda a few of my sock puppets got together to elect another of my sock puppets as mod

12:37 PM

psst. Stephen -- we need a quick distraction, Kusalananda's on the trail. Hey how about that new DVORAK keyboard that just came out??

@JeffSchaller ah yes, the one that plays your favourite music as you type, but only if you get the cadence right!

heard the arrow keys play the Knight Rider theme

and every time you hit backspace it starts from the beginning

@JeffSchaller What?! Where?!

/me runs off to file a patent application

12:39 PM

@Kusalananda tvtropes.org/pmwiki/pmwiki.php/Series/KnightRider

Stephen, I sent him a TV Tropes link - that should buy us a few hours

@JeffSchaller well played

what were we plotting before he chimed in?

@StephenKitt guiding editors that were angled slightly off the mark

You were going to ghost-write something that I would post on the site for them to see

We could honeypot him with a badly written question that we keep rolling back... and that we "correct" badly. That'll keep him busy.

@JeffSchaller right, sounds like a plan

It’s great that he cares about editing the site, it would be better if the site offered good ways to help improve his editing skills!

and point him to the tour page ("Next badge: Informed")

Member for 4 years, 1 month!

12:44 PM

But grammar only really need fixing if understanding the question is difficult because of it. Or when editing the question for other reasons.

It's possible they just haven't looked at the rejected edit messages; if I recall, you have to seek it out pretty specifically

I sympathize because that edit is one I would have made, except for the extraneous code highlighting

@JeffSchaller especially the .service mix-up at the end (which could be an honest mistake of course)

@StephenKitt The "all the grammar is wrong" bit got stuck 'on'

@JeffSchaller yes, which is a recurring problem

I'm searching our Meta for any guidance on not-formatting nouns (or what should be code-formatted)

meta.stackexchange.com/questions/155904/… is the best I have so far

Bonus for Stephen: "Slapping trouts into the face of the editor would also be a good idea"

A: Highlighting technical words?

12:57 PM

12

No, you should not "highlight" any of them. The backtick operator is meant for code highlighting and those words are not code. However, if something in your text is actually code or code-like (a filename might qualify for example) using this formatting is fine. On a side-note: Edits adding such ...

@JeffSchaller I’ve depleted my stock of trout though (and yes, it started off at 0)

A: editing guidelines for Unix SE

Give a man a fish, and he corrects a misguided editor. Teach a man to fish, and he might stop visiting U&L

2

Obviously we can come up with our own policies, but this has been discussed on the main meta quite a bit, so that's probably a good place to start; What is the etiquette for modifying posts? is particularly helpful. There's also a blog entry on it

1:14 PM

>This user< probably needs to recreate his home directory after letting the directory entry grow to 360+ MB. Could somebody help him? (I don't have time and head-space at the moment)

that sounds like a repost

nope, I'm just hallucinating again

:-)

wait! I'm not! They even linked to it! unix.stackexchange.com/questions/551939/…

oh the frayed ends of sanity, so early in the day

This is not a repost. It's a followup.

I thought "4 million small files" sounded familiar

yes, because I commented on it

1:19 PM

Yeah. the 1st is about deleting them. The 2nd is about discovering that performance now is very bad for that directory.

I meant my comment as a suggestion for a duplicate

going offline for a bit -- internet modem's being replaced shortly

@JeffSchaller Ah. Not directly related to the followup then.

Thanks @StephenKitt

@Kusalananda You’re welcome! I’ve tried to make it more general than just the home directory scenario.

I suppose rsync -aH could also be used, that's what I would do. I very seldom use cp for recursive copying... I don't trust it (but I don't really know why).

@Kusalananda heh, it could indeed; for some reason rsync is firmly lodged in the “remote file transfer” section of my brain

1:28 PM

There's no low level ext-tool that can rewrite the directory entry?

A sort of "prune dir" operation...

@Kusalananda I’m not aware of any. Directory shrinking is still officially on the extX developers’ todo list, AFAIK, but I don’t think much progress is being made.

ilkkachu

3:12 PM

fsck.ext4 has -D to "Optimize directories in filesystem. This option causes e2fsck to try to optimize all directories, either by reindexing them if the filesystem supports directory indexing, or by sorting and compressing directories for smaller directories, or for filesystems using traditional linear directories."

and fwiw, cp -a dir newdir seems like a rather bad default idea for defragmenting, since the tree can be arbitrarily large. linking would seem like a much better idea.

@ilkkachu right, I should really go for cp -al directly

ilkkachu

yeah, that would seem like a good idea.

@ilkkachu done.

3:45 PM

What's this with Python script wanting to write to tty device files lately? I think I've seen about three or more questions popping up related to this.

7:17 PM

@FaheemMitha I think you were asking me about btrfs the other day. Realized I'm using it for my systemd-nspawn containers as well, so, e.g., my Nextcloud instance is on btrfs

@derobert Yes, I was. I hear it offers some form of version control.

btrfs? It has snapshots

Why do so many people try to use text processing tools for tasks that are more suitable for databases? (Example: unix.stackexchange.com/q/552713/135943)

@Wildcard If all you have is a hammer...

@derobert Not the same as version control, then.

the curse of powerful tools?

7:21 PM

@FaheemMitha but they don't have hammers, either. They're asking for a hammer to help them pound in a screw. And hammer-wielders happily oblige.

@derobert have you considered using Mercurial for your filesystem? It has version control ;)

@JeffSchaller As much as you joke about that, not all people are sane... pypi.org/project/hgfs

@derobert lol I should have guessed

(There are of course equivalents for git. More than one, it'd appear.)

@Wildcard In fairness, the poster didn't specify any specific tool.

7:24 PM

And remember, it was a built-in feature of Subversion!

He might be just fine with SQLite, for example.

@FaheemMitha the tags are "text-processing" and "awk" and the asker put them on.

@FaheemMitha she, judging by username.

@Wildcard Yes, I just noticed. I take that back.

(Subversion could be configured to take ordinary WebDAV writes, so you could use any WebDAV client and treat it like a filesystem. Like Windows...)

@Wildcard Well, if you just have to do that one filter, it'll take a lot more time to load it into a database than just doing the filter on a text file.

@derobert True, but a database is much more scalable when it comes to that sort of thing. And its cousins, like R data frames, and Python Pandas are also better.

7:27 PM

@derobert only if you don't count the time spent writing and debugging your awk script. Take a look at the extensive discussion on the Awk-based answers.

@Wildcard well, that depends on how long loading all that data into PostgreSQL takes. Especially if you don't already have a Pg instance tuned for fast loading. Loading e.g., a few hundred million rows to Pg can take quite a while.

@derobert SQLite, not PG.

@FaheemMitha I wrote my answer with PG because I have it up and ready and it's easy for me.

Not to mention it'd be much easier in Python or Perl, instead of awk :-)

I'm not sure how well SQLite scales to large number of rows, but it's supposedly quite fast. Of course PG is undustrial strength and can basically do anything.

@Wildcard Well, SQLite is easier to deploy.

Given a choice, I'd go with that. For demonstration purposes, anyway.

7:31 PM

@FaheemMitha yeah, I agree. I just wasn't going to spend the time. I don't know sqlite syntax/commands so well.

And the looseness of their type system bugs me, so I never got into it fully.

R probably wouldn't do too well with several hundred million rows. Not sure about Python. Probably not great either. Neither of them manage memory particularly well. R is probably worse.

@Wildcard Fair enough.

@Wildcard Yes, SQLite is a bit of a mixed bag. I think they're trying to be "flexible".

Anyway, for gene sequencing/bioinformatics, I highly doubt sqlite would ever be appropriate. The datasets are just too large.

Personally, I like my dbs fussy.

@Wildcard Possibly not. Though tuning PG for large datasets also isn't trivial.

> Think of SQLite not as a replacement for Oracle but as a replacement for fopen()

@Wildcard fopen?

7:35 PM

@FaheemMitha I was quoting sqlite.org/about.html but you can just run man fopen

@Wildcard I don't see the relevance.

@FaheemMitha There is no reason (with that format) to have more than two rows in memory at once.

@FaheemMitha they're not really trying to be a database per se. They're trying to encourage application developers to store their internal data in a structured fashion rather than in random bespoke binary formats.

I don't actually know how SQLite copes with large data sets, because I've never tried. Has anyone?

My largest SQLite datasets are in the 100MB-range, so not large

7:37 PM

Hence, they're hoping when an application developer reaches for fopen() to store some data, they would instead reach for sqlite.

@FaheemMitha But yes, I like my DBs fussy also. :)

@derobert Out of curiosity, what do you use it for?

@Wildcard The largest is an index of a video backup (large btrfs filesystem, where there are a ton of dated snapshots, each one basically just being an rsync of the primary storage). That is around 85MB (used to be larger, until I zapped more redundancy from it). Next largest is from an S3QL filesystem, around 50MB. My Podist database is 37MB.

All this stuff is open source... so gitlab.com/derobert/random-toys/tree/master/backup/… is the backup indexer and gitlab.com/derobert/Podist is Podist

S3QL is written by someone else; github.com/s3ql/s3ql

Q: Do we need to git a new tag for Git Bash questions?

8:24 PM

@JeffSchaller "git a new tag"?

0

I'm seeing some questions lately pertaining to "git bash" or "gitbash" and had to go look to see if this was something different from windows-subsystem-for-linux, and it seems that it is different. I learned more about it from an Atlassian Git bash tutorial and the Git for Windows page. It's summ...

discussion tags

What manner of sed is the "git bash sed"?

Asking for a friend...

No, sorry, I'm genuinely interested. I saw it mentioned in a question a few hours ago. I suppose it's GNU sed?

@Kusalananda "git" being a country/slang way of saying "get"; maybe that's too much of a pun/stretch

Oh :-) I thought it was a typo!

I couldn't come up with a quick pun on "bash" so went with "git".. But now I have to see if I am hallucinating again

I disappeared for a while there. My internet line went out. I switched to the other one, but it was out too. Fortunately, it came up after a bit.

@JeffSchaller whew, I found external reinforcement: Entry 2 of merriam-webster.com/dictionary/git

8:31 PM

@derobert Tell that to R. It's memory management used to be terrible. It would just dump everything into memory.

I think I might be addicted to the Net. I got a bit panicky there.

and @Kusalananda I'm 99% unaware of gitbash, so I do not know what sed it has

Pop!OS is another theme I'm seeing lately that I'm curious whether it needs its own tag, but I'm gunshy of the whole Linux vs Distribution thing that I know I get wrong 50% of the time

Obviously sed is one of those "common bash utilities"

no one calls sed from zsh or tcsh

/s

9:17 PM

@Wildcard Nice work on the PG answer. Do you use PG regularly?

9:53 PM

@Jesse_b We're talking about it. I'm not very optimistic.

forum.codidact.org

2

10:16 PM

@FaheemMitha I do indeed.

@Gilles'SO-stopbeingevil' Why are you not optimistic?

And if you don't mind me asking, where did you get that Discourse site running? And was it hard to get it running?

@Gilles'SO-stopbeingevil' just my two cents - I think a large part of what drove SE's popularity is the gamification. If you try to do away with the rep and only work like a Wiki (as I read in one of your suggestions), I think you drastically cut down your potential users.

@Wildcard Agreed.

@FaheemMitha I can't answer for Gilles, but I can say why I'm not optimistic. It's too many cooks, each with their own idea of what they should make. "Not SE" isn't a good recipe when one person wants soup and another wants fish fillets.

@Wildcard I thought he might just be concerned about lack of manpower.

10:59 PM

@Wildcard [citation needed]

Sure, some people are here for the gamification. But that doesn't automatically mean that gamification does more good than harm. And that doesn't automatically mean that there has to be as much gamification as SE.

@Wildcard Right. There's a sizable group who want to build SE under different management, which 1. repeats SE's mistakes and 2. has nothing to offer so I don't think it'll sell. And there are people who want something different, but none of us want the same thing.

@Wildcard I want to avoid having rep, but I do want to preserve the concept of post ownership.

11:20 PM

@Gilles'SO-stopbeingevil' gamification sells. :)

And as for trying to build something different—the gathering you have is people who are dissatisfied with SE management, primarily.

They may also have quibbles with various design factors in SE, but all of them were using SE extensively at least up to the point of management blunders.

So SE featureset and design is your common denominator of agreement.

@Wildcard It sells to some people. It repulses others.

@Gilles'SO-stopbeingevil' yes, but that's true of absolutely any design decision.

If you want to attain a critical mass of questions and answers, it has to be popular enough to work.

Wikipedia doesn't have much gamification and it works.