@terdon Huge gold medal for Greece today!

ed tag?

@AndrasDeak It's the standard tag.

> ed (pronounced as distinct letters, /ˌiːˈdiː/)[1]

mind blown

Do I now have to look up sed, awk and the rest?

@AndrasDeak For their pronunciation? Well, "sed" is "stream-ed", and "awk" is an acronym of "Aho, Weinberger, Kernighan".

But one is pronounced "said" and the other "awk" as in "awkward" or I need to have a sit down.

> The acronym is pronounced the same as the bird auk, which is on the cover of The AWK Programming Language.[7]

phew

let me gee ahr ee pee sed's article to see if the pronunciation is mentioned there

@terdon ("said")... which is partly why I pronounce "ed" as "ed", not "e-d".

@Kusalananda You and everyone else, I would have thought until a couple of minutes ago.

@terdon Well, I know about the Bell Labs custom to say "e-d" from before, but I just ignore it.

let me just edit the wikipedia page...

@AndrasDeak Too much documentation is better than too little. At worst, one has the option to ignore it. Whereas if there is too little, one has an option between staring at the screen and trying to guess what is going on, reading the source code, or screaming for help.

Yes, it wasn't a complaint. Just a fun fact.

Is this output for free -m normal? Because this is just before my machine freezes up, presumably for lack of memory.

                   total          used        free        shared    buff/cache   available
Mem:          15941       14949      130         692         860             48
Swap:             0           0             0

no, that's not normal

48 MB of free memory, 860 MB buffered

               total        used        free      shared  buff/cache   available
Mem:           15884        9700         535        1186        5648        4669
Swap:          17165        1584       15581

you probably added whitespace on the first line

that ^ is my output when a sysstem monitor says I have 71% memory use

I'm especially surprised that your "available" is less than your "free", I thought that was never the case

@AndrasDeak Probably. So you just pasted in the file as is? Let me try again.

yes, and pressed ctrl+k before sending

Is there a way of copying a file in here by typing in the file path?

no

              total        used        free      shared  buff/cache   available
Mem:          15941       14949         130         692         860          48
Swap:             0           0           0

I'm sure there's a sandbox room here which you can use to try things

Better?

yes

Anyway, around this point my system stops responding. Sometimes I manage to reboot before that happens.

Like I did just now. But saved this output before I did.

next time you can look at top and sort by memory to see if there's anything hogging it

if you start top and press shift-m, the rows get ordered by memory

    PID USER        PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2021483 adeak       20   0 6054768   1.1g 425180 S   5.6   7.2   1026:02 firefox-bin
2021703 adeak       20   0 4261536   1.1g 158504 S   1.7   7.1 202:17.42 Web Content
  28405 adeak       20   0 3374936 698196  50752 S   1.3   4.3 132:11.39 teams
2021730 adeak       20   0 3602152 691120 122584 S   1.0   4.2  95:02.14 Web Content
   4663 adeak       20   0 3482268 634132  85392 S   0.0   3.9  66:01.27 thunderbird

all those Web Content rows are also firefox, my main memory hog

I restarted discord and it starts up with 25 GB virtual memory... I bet that's a mistake as well

My feed reader, liferea, has 82 GB :/ Now I know why I never look at these...

As I recall, it's normal for the Linux kernel to grab all the memory and sit on it. Normally very little shows up as free.

But my computer has not been behaving normally memory-wise for a while now. And I don't have another Linux-based machine to use as a reference point for normal behavior.

@AndrasDeak As I recall, top isn't ideal for printing, though I also seem to recall there was an alternative that was.

There's htop, which is more colorful. But I'm not sure if it is otherwise better.

There’s a related question with no good answer:

But as Andras says, available memory should be greater than free

@StephenKitt Perhaps the poster had some memory issues too.

In my case, it seems odd that both shared and buff/cache are so low.

Normally shared is quite large. And I would have thought the buffer would be fairly large too.

Any thoughts about whether this is memory, processor or motherboard? Those seem to be the main suspects. I can't imagine where else the problem could be.

If you’re running out of memory, buff/cache will be low, especially since you don’t have any swap.

Any thoughts about whether this is memory, processor or motherboard? Those seem to be the main suspects. I can't imagine where else the problem could be.

@StephenKitt OK. But surely I shouldn't run out of memory in 2 days?

What is it with people working with structured document formats?! It's "but I don't have xmlstarlet installed!" and "I don't want to install any dependencies" and "can't you do it with sed instead?"... It's as if they are allergic to wanting to work with the tools actually designed to handle their data correctly.

In contrast, I have hundreds of open tabs in my phone, which has 6GB. And so far that hasn't skipped a beat.

@FaheemMitha that depends on what you’re running...

@StephenKitt Two browsers, Firefox and Chromium. Mostly just regular browsing, with some video watching. It's possible the video processes could be quite leaky, especially Hotstar, which seems to have been designed by someone on drugs. But would they be this bad?

Hotstar is the local Disney+ provider, as Disney continues in its mission to buy all the media on Planet Earth.

Bits of other things too, but nothing particularly memory intensive. A little bit of programming. TeX, which doesn't have persistent processes, and couldn't possibly cause any issues.

That’s why Andras asked for your top output

@StephenKitt I'm happy to provide it, but again, top isn't particularly well suited to writing to file.

I just tried it. It's a mess. Lots of ESCs.

top -b -n 1 -o %MEM

@Kusalananda Many of them probably don't have admin rights so can't install something new.

@StephenKitt Thank you for the tip. I got as far as the -b.

And many more would be reluctant to learn a whole new tool for something they assume (however incorrectly) can be done by naive text processing utilities.

This is my current output. I'll try to resave when I'm close to freezing again, which will be a couple of days. gist.github.com/a8a02ea1fd8c72582a133136952fb78a

@AndrasDeak ^^

Just rebooted.

@terdon Do you tell them exactly what to type, or so they have to figure things out for themselves?

?

@FaheemMitha IME answers involving xmlstarlet, jq etc. give the exact command to use

@StephenKitt OK

@terdon Sorry, was I unclear?

Yes. :) But apparently Stephen got it. All answers that are useful give the exact command. Or almost.

@terdon OK. Well, that doesn't sound like much work, then.

@terdon I would say that an arcane sed solution is harder to learn than a straight-forward (specialized-tool CLI)

(harder to learn and harder to remember all the edge cases where it breaks)

@terdon I see I should have addressed @Kusalananda instead. Since he's the person who made the comment. Which is probably why you were puzzled. Sorry for the noise.

Though I must say I'm not an XML fan. It just looks so ugly.

@JeffSchaller Well yes, most sed solutions are arcane. I don't really use sed for anything complicated, that's why $god created Perl!

On cue:

(for some value of XML, apparently)

Same OP as the previous one, that's a follow-up question.

I need to borrow ilkkachu's sewing needle for the typos in the title /o\

@terdon can be done... by other people :/

@terdon There are virtually no tool that needs admin right to install in a home directory.

@terdon Then they should not be working with that type of data.

@AndrasDeak Kind of an interesting article, though India specific. thewire.in/politics/benjamin-zachariah-fascism-sangh-parivar

@FaheemMitha Luckily, you very seldom have to read it, and it's fairly easy to translate it to other structured document formats.

@Kusalananda I didn't claim my antipathy was rational.

But humans do have aesthetic preferences, even when it comes to computers.

At least, this human does.

@Kusalananda True, but that requires considerably more work, especially if the tool has dependencies which you then also need to configure and compile from source. That can get very, very complicated and isn't for the faint-hearted.

@Kusalananda If we lived in a world where people didn't do what they weren't supposed to do, things would be very, very different :)

But I do get the urge not to use a dedicated parser. Yes, it's fragile, but 9 times out of 10 you can do what you need perfectly well with simple tools. You just need to be sure of your input and double check your output.

That’s what can end up frustrating in some cases here: the Q author only wants a solution for their specific problem (including input), whereas A writers are often expected to produce a general answer.

So yes, it’s better to use xmlstarlet for XML, jq for JSON, etc., but sometimes a quick search-and-replace using sed or even a text editor followed by a diff is faster than figuring out how to do it properly.

I'd be OK with "sed" type answers if there were caveats listed with the answer.

I'm actually thinking it'd be useful for the site to have one or more canonical Q/A's on processing structured data. It could explain all these caveats once & for all.

Unfortunately, doing things properly is often very time consuming, and also requires much more effort than doing it badly.

That's true as regards computers, and for other things too.

The canonical Q/A could explain the caveats with the line-based approach as well as demonstrate how to install the proper tool either system-wide or as a user.

@mas it’s not outlandish at least; I have 11,523 files there

I'm convinced that something about my scrapyd software is eating all my inodes but I have no idea what and all these one liners to find the biggest inode dirs seem to not be giving me the results.

Maybe there's some service that's creating tons of files in separate directories.

It uses chromium headless to scrape some sites, maybe chromium's cache is blowing up somehow?

I mean, there's over 6.2 million inodes used on this system, I don't get how the biggest inode directory is only 17k (man3).

@terdon Do these people all work on systems with no package manager?

Man, that's still just a drop in the bucket.

@terdon Sorry, but it sounds as if you're giving up. If a thing can be done in a way that is 1) convinient, 2) correct, and 3) safe, then it would strange not to want to do it in that way.

@Kusalananda How would a package manager help? We're talking about someone without admin privileges who would need to install something locally to their home dir. That requires also installing all of the thing's dependencies, also locally and also without admin privileges. So the package manager won't help.

@Kusalananda Of course. But learning a whole new tool is never convenient, so your argument would fail at point 1. Convenient is using the tool I know. I am not arguing that it wouldn't be better to always use jq or something. However, I can honestly say I have never used it once outside answers I've posted here. Granted, I don't parse XHTML files on a daily basis, but I need to often enough and when I do, I will always write a little perl script for it.

@FaheemMitha Huh? How would a short xmlstarlet expression (or jq expression or whatever) take longer to write than a multiline awk program that you have to fix every once in a while to stop it breaking on each edge case?

That is far more convenient and fast for me than figuring out how to use jq, and since I know the dangers and limitations, it's never caused me any problem at all. Sure, a dedicated parser is The Right Way® but sometimes the right way isn't necessarily What Works For Me©.

@terdon We're literally giving them the option to not learn the tool if they don't feel like it. They get a command that works.

Is it just me or is everyone acting really strange today? Like "it's ok to parse XML with awk and sed"!

@Kusalananda Oh, you mean people who get an answer and then don't want to use it?

@terdon Yes.

That's different sure. That would only make sense if you cannot install new tools. But that's a very common situation as explained above.

I can also understand someone who doesn't want to clutter their system with some tool they will only use once, this one time they need to parse this one XML file.

@terdon They can always install new tools. Our answers should not be different because they are lazy.

If they can't, then their sysadmins can.

@Kusalananda No, they can't. If you don't have admin rights, installing new tools is hard. And beyond the ken of most users.

And if they need to jump through beaurocratic hoops to get a tool installed, then it is likely not worth it.

Sysadmin are people too, and can be contacted and talked to.

@terdon It does not change the fact that there is a right way to do it, and conequently also some very bad ways of doing it.

@Kusalananda Spoken like someone who has not spent decades working at large, public institutes ;) As I'm sure you know, it isn't always that simple.

@Kusalananda OK.

Nobody is arguing that dedicated parsers are not the right way. They are. But sometimes, the right way is more trouble than it's worth.

@terdon :-) I'v worked for state intsitutes in Sweden, New Zealand, the EU and in Britain since the 90s :-)

@Kusalananda I know. That was my point :)

Those places are the easy ones to work with though.

Yeah, it's even worse in the corporate world.

Agreed

In any case... I'll continue giving the correct answers. It's up to the user if they feel like using them.

As you should!

... and I'll throw the random tantrum every once in a while too.

:-)

Everything I know about jq (namely, that it exists, is named jq and can parse JSON) I've learned from your answers here.

So please keep posting them!

Alright, I think I found the problem, five million inodes in /root/.cache/Python-Eggs. What. The. Heck.

@terdon without thoroughly reading the discussion: a key difference is that you can write your own code, and you're willing to reinvent your own wheels. The typical asker gets a specialized package that does what they need, but then when they need the third column instead of the second, they are lost and have to ask another question. Which ties in with @StephenKitt's remark that answers are more general: in the hope of teaching the asker how to fish.

@mas nice

@AndrasDeak yep

Thanks, guys, for letting me use the room as a rubber duck. :-)

@AndrasDeak do you know why I'd be getting so many files in Python-Eggs? Bug with scrapyd or something?

@mas Always the most effective debugging method!

@mas I'm not familiar with that path, so no.

@terdon I actually bought a duck and I never use it. I like to open a stack overflow question and because I know how much people hate malformed questions I usually end up answering my own just in better formulating the question.

"python eggs" sounds like something that should be on the python path, and would have to do with installations

from .cache I assume they are temporary files... not sure why they aren't created in /tmp though

@mas that's the magic of MCVEs

I'm pretty sure scrapyd-deploy packages everything as eggs and since I'm running the scraper as a service I think it's defaulting to root.

scrapyd-deploy doesn't seem maintained anymore, I ended up forking it to fix a bug that's been sitting in the PRs for like 2 years.

Another possibility is some logging, perhaps... but this doesn't sound normal in any way. Any chance that your own code does something wrong?

It must, I haven't touched it in like 3 months, but I remember finding a huge memory leak because I kept opening new instances of Chromium webdriver without closing them.

I nearly killed a raspberry pi with that.

But I have no idea why it would be packaging eggs except when I explicitly deploy.

I'm going to keep checking that directory to see when new files are created, that might give me some clue.

Maybe I can try a deploy and see if a buttload of files are created on every deploy too (maybe like 3 deploys results in millions of files?)

That sounds really unlikely since a deployment takes like 10 seconds at the extreme end.

But there were so many files I couldn't just rm -rf *. Damn thing said too many arguments!

Not bad

But the directory is fried by then anyway

Looking at the files would probably help, yes

@Kusalananda I was speaking generally.

@Kusalananda Of course, if doing it the right way isn't hard, that's obviously the way to go.

@Kusalananda I don't know, I mean, maybe we should parse XML with ed? :)

@AndrasDeak Not an hour later and there are 40 new files in egg

Every time crontab curls scrapyd's schedule.json route it creates a new copy of the entire project

It has to be a bug in scrapyd.

In the mean time I wrote a cronjob to just literally delete all the files in /root/.cache/Python-Eggs/ every hour.

@mas Oh yes, I know that feeling well!

@mas For something like this one would normally have a cron job which expired suitably old files. I think Debian has stuff like that for logs and such.

@mas perhaps this means that your workflow goes against the grain of the project?

after all "deployment" is not something that I would expect to happen every five minutes

@AndrasDeak obligatory xkcd.com/1172

yup

lol, I don't think I'm doing anything outside of the docs!

I'm not doing deployments, just scheduling jobs, apparently that creates eggs. So weird.

yeah

I've been wanting to set up a VPS for this thing but I was initially worried about the storage, I've built a 170mb database in just 3 months, but it turns out that would still take like 25 years to fill up a 25gb digital ocean droplet.

Sometimes storage space just boggles my mind. In 3 months I completely filled out all my inodes but it would take me 25 years to fill up a measly 25gigabytes

Or else I did my math really really wrong

No, 170mb over 66 days is 2.5mb/day which is about a gig a year.

I suppose it's rude to treat a community chat like an SMS thread with correction*

I guess it depends on the locals. I'm a lot pickier than the regulars, so don't take me too seriously.

If one doesn't set a password for the postgres user in a new installation and opens the postgres server to external connections, no one can access that without sshing into the machine, right?

can't you try this over LAN?

I guess you might not anticipate certain attack directions

It's on a vps, it seems like psql postgres://postgres@localhost does not work

Is it insane to wonder why ln -sf isn't able to just determine the target and the link from context? Like, if the file doesn't exist for one of them, that's the link.

I guess then you can't overwrite a dead link

@mas The ln utility does not check whether the target of a symbolic link exists or not. You can create dead links to your heart's content. I could envisage situations where it would help to be able to create dead links, like if you want to create (absolute) symbolic links inside a chroot without having to actually chroot into it first.

So I guess I'll just have to keep looking up what order to but the target/link every time I use ln for the rest of my life :-(

that's like one of those paintings that someone paints upside-down, and it only makes sense when it's complete and the painter flips the canvas the right side up

@mas I always do that, and I need three reads each time to deduce the order. For some reason I find it very hard to parse the manpage.

somehow TARGET LINK_NAME don't strike me as obvious names for the two

If I think about it I understand that LINK_NAME -> TARGET, but I'd rather not have to think about it

Yeah, I think I also think about it backwards, and I always have to look up to make sure I have it the right direction

this is neat unix.stackexchange.com/a/77576/416002 (a list of 2-letter commands)