@derobert hey thanks for the reply! Yeah but there might be better approach, I am searching since 1 hr. Other devices also pass absolute coordinates how do drivers work in that case.

In my actual data file, only about 4% of the lines contain a colon at all.

@MathematicalOrchid But do dupes always have one?

For every line that ends :FOO, there's always a dupe on the line before.

There shouldn't be any other dupes.

OK, there's something weird going on with the /:

$ printf 'Foo/\nFoo/Bar\n'
Foo/
Foo/Bar
$ printf 'Foo/\nFoo/Bar\n' | uniq -f1
Foo/

Ah! OK, it looks like uniq takes any non-alphanumeric string as a delimiter.

Dang.

That's not what the documentation says though:

Fields are sequences of non-space non-tab characters that are
 separated from each other by at least one space or tab.

Ouch! That's not fun...

Yup.

Let me see if I can think of a workaround.

Maybe this is just too complex for a shell script... (?)

It looked simple, but it seems it isn't.

Oh, it's a simple awk or perl one liner anyway.

I don't grok awk. Maybe someday...

You want to comment your answer to help the next poor soul?

I'll fix it up as soon as I get a working solution.

OK.

@MathematicalOrchid how big are these files?

$ grep -hFxv "$(grep ':FOO' file | cut -d: -f1)" file
red.7
green.2:FOO
blue.6
yellow.9:FOO
/Foo
/Foo/Bar/
/Foo/Bar/Baz

It needs two passes of grep (and also probably needs GNU grep)

Hey @steeldriver, nice sed solution. it really would benefit from an explanation though, sed labels and commands look like line noise to the uninitiated.

What's the context?

@terdon Ok. Thank you.

@StephenKitt What would a version just to exclude non-English locales look like?

The first part would presumably be:

# Drop locales except English
path-exclude=/usr/share/locale/*
path-include=/usr/share/locale/en/*
path-include=/usr/share/locale/locale.alias

And the second part:

# Drop translated manpages except English
path-exclude=/usr/share/man/*
path-include=/usr/share/man/man[1-9]/*
path-include=/usr/share/man/en*/*

Is that right?

@FaheemMitha wow, you working on embedded to care that much about <1G space?

If anyone else is interested, this is a reference to unix.stackexchange.com/a/281549/4671

@derobert No, but space adds up. Why not save it?

Especially if it is a one-time thing.

@FaheemMitha Configuration is not free (time to learn the syntax, write the config, merge your changes on upgrade, etc.) Neither is running non-defaults (more likely to hit bugs).

@derobert I agree that running non-defaults is more likely to hit bugs.

But Stephen provided the syntax. Do you see a problem with it? Or do you think localepurge is safer?

Looks fine... though somewhat pointless, unless you have a tiny disk.

/me notes his entire /usr is 18G, fairly small—and that's on a pretty bloated install.

Parenthetically, what's the most efficient way of finding out the space used in a directory? I use du -hs, but it isn't exactly fast.

I don't think you're going to find anything faster.

Unix filesystems track the size of each file... they don't track it per-directory. So you have to stat each file, and total them up.

@derobert Hmm. All the filesystems?

@FaheemMitha Maybe there is an exception. But there isn't AFAIK any API to get that information out of the kernel, so even if there were a filesystem that did...

Many can track quotas. So you could get space per-user quickly.

Or per group.

@derobert I see.

I get du -hs -> 59G. But that's wrong, because /usr/local/src is separate but included. I need to exclude that. It is possible to tell du to not cross filesystems?

Yes, -x.

du -hsx -> 15G

yes, and 3TB is ~\$100, \$200 if mirrored.

(bloody mathjax)

so, about a dollar. For the whole install.

@FaheemMitha That looks correct, yes.

@StephenKitt Ok. Thanks.

:29492834 Sorry, I was trying to fix up my message — Enter sends the message...

@StephenKitt No problem.

@derobert I'm not sure if I can get those prices here.

@FaheemMitha Right, I guess your install probably costs ₹300 (Google says before)

and my $1 figure was mirrored. Because who doesn't mirror? :-P

@derobert real users use raid0 and then ask a question on Unix.SE when they lose their data ;-)

@StephenKitt LOL, but only after trying to fix it themselves with mdadm --create a few times.

@derobert preferably with a patched mdadm...

@derobert I don't follow.

@FaheemMitha Can't be more than 4.5x as expensive, can it?

@derobert That's true, it can't.

Assuming you mean 4 or 5 times.

Even Indians can't jack up more than that.

I used 4½ times, because that gave a nice round number :-P

Actually most electronic/computer stuff here is within 2x of the US values. Sometimes less.

@derobert What did you use for rupee/dollar conversion?

Google

google.com/…

@derobert ok

Rs 300 goes further here than in the US. There are plenty of people in this city who live on that daily. Or less.

I wonder if one could use a db to solve unix.stackexchange.com/q/281513/4671

@FaheemMitha Well, they should definitely avoid apt install kde-full, then....

@derobert That's unlikely to happen. Because they don't have computers.

In India, at least, being able to connect to the net requires a certain minimum income, I 'd say.

You also need a place to put your computer.

unix.stackexchange.com/questions/281554/… ... does my answer here look sane?

@derobert I'm behind the times on PC booting restrictions; grub's able to handle booting a partition that's >9 Gb from the top?

@JeffSchaller yes... for quite a while. Early 2000s, maybe?

@derobert that'd be the timeframe that I stopped booting PC's! Thanks :) Seems like a reasonable answer to me, otherwise!

@derobert Looks reasonable. I generally recommend LVM to people too.

@JeffSchaller Anything with LBA mode could do more than 9GB. LBA32 gets you to 2TB.

LILO did LBA32... I forget when that was added.

@derobert yeah, looks fine. If you like, you could also suggest a simpler solution: create a new partition and then move /usr and /var to it, replacing them with links in /. Those are usually the biggest dirs in / and that would allow the OP to sort out their space problems without too much pain.

@terdon You could add that answer.

Yeah, I was thinking of doing so but thought that yours was close enough. If you don't, I'll post one in a bit.

I'm guessing all the space is in /usr. At least, that's where it winds up on mine.

Well, except /var/cache and /var/tmp... which of course you clean up if you're out of space.

@derobert Yeah, but since those can grow, I'd move both over.

@terdon Right, but the problem is if its all in /usr ... then OP will just be out of space on /usr.

Point. I hadn't really checked how much space they had available.

(plus, separate-/usr is barely supported if at all in a lot of distros)

@derobert even with manual partitioning? or do you mean that the guided partitioning won't offer a separate /usr?

@derobert Why? Things in /usr needed before non-/ partitions are mounted?

@terdon yep. Often /usr/lib or /usr/bin.

It's far too easy to accidentally have your boot scripts try to use a library or binary out of /usr

Fair enough. Not to mention that many distros these days have /bin -> usr/bin

Some distros have given up and now demand /usr be mounted from initramfs. My guess is the rest will do that soon, too.

guess the days of read-only or centrally-mounted /usr are long-gone!

Well, might still be here if you mount it from initramfs.

good point

Is anyone here a fan of Ken MacLeod, by the way?

You can wind up needing a lot of stuff for separate /usr. You can, for example, need a full network stack (including up to NetworkManager) and GSSAPI authentication stack to mount an NFS share. That's a lot to make sure it doesn't need to access anything in /usr/lib.

@terdon had to google him to make sure, but - nope, haven't heard of him

Haven't heard of him, either.

Oh well. He's very good, certainly worth a read. Especially his early, political SF. Anyway, he answered a question on ELU. I love it when that happens!

Does this make sense to anyone?

@terdon It reads like the OP needs a good book on kernels.

Or a course.

Since that's also true of me, I didn't feel up to passing judgement on it. Dunno if it makes no sense because of my ignorance or the OP's.

"The Rapture of the Nerds" is a funny (and true) way of discribing that Singularity nonsense.