/dev/chat

12:42 AM

Replacement disk arrived and ready. Unfortunately, btrfs dev remove /dev/foo /path does not seam to feature a "ummm, yeah, read from the other mirror if at all possible" mode.

derobert

12:56 AM

Oh dear, it's worse!

sdi (5TB) and sdj (3TB) are the old devices. sdj is failing, and is being dropped. sdk (5TB) is the new device. And yet, when it finally got some I/O off (and not just waiting for sdj to time out again)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.00     4.33    0.03  216.47     0.53 13637.80   125.99    17.24   79.65  180.00   79.63   0.62  13.38
sdj               0.07     5.08  204.35  177.87 12812.73 11216.40   125.74   187.63 3628.98 1941.08 5568.18   2.61  99.93
sdk               0.00     0.00    0.00    0.02     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

#@!#)*((@)!#(08!@()#

Oh. Looks like there is a different command for that. Ummm. Wonder how I abort the remove.

Hopefully control-C does it :-/

derobert

3:10 AM

So, aborted the remove (after control-C didn't work) and Google results suggested 'errr... reboot?' by having device-mapper fail all requests (as there is actually a device-mapper encryption layer in there, so could have dmsetup do that). Then I could unmount the filesystem.

Remount took a good half hour or so, mostly because it waited through a bunch of timeouts from the failing drive (each timeout = 360s)

Right now, waiting for the new drive to drop, which hopefully will finish in 30 minutes or so. Then can try replace.

I'd love to decrease the 360s timeout, but only RedHat knows where it comes from, and they're not telling.

@Braiam let's see if I can more effectively destroy storage than you :-P

Anyway, back to reading Men at Arms.

derobert

3:28 AM

Replace started... let's hope this actually works.

derobert

4:05 AM

Well, for about 30s, it goes like this:

Device:         rrqm/s   wrqm/s     r/s     w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sdi               0.10     0.00 2662.70    0.00   166.40     0.00   127.98   124.42   46.73   46.73    0.00   0.37  98.60
sdj               0.20     0.00    6.10    0.00     0.10     0.00    33.05     0.55   89.51   89.51    0.00   2.43   1.48
sdk               0.00   787.30    0.00  541.30     0.00   164.29   621.59    40.19   73.52    0.00   73.52   1.82  98.36

... then it decides to write to sdj, which takes a good 6 minutes to time out. Or actually, two writes I think, so 12.

Guess I'll cancel it, stack a dm_snapshot over top of sdj to take all the writes, then try again. Bloody annoying.

derobert

5:07 AM

Yeah! Finally running with dm-snapshot inserted, and it's now done more work in three minutes than it had over the last five hours.

derobert

5:40 AM

0

A: I have a disk failing in a RAID1 btrfs such that writes are iffy, but reads mostly work. How to replace it?

This has turned out to be a royal PITA. First, it's important to note that btrfs now has a proper replace command, which is very much better than add new, remove failing. First, start by partitioning the new disk and setting up dm-crypt on it. Go ahead and unlock it. If your disk wasn't having ...

Wrote up what I did as an answer.

Braiam

1:23 PM

@derobert well, I only lost a single file from my repo, meanwhile your entire storage is compromised :P unix.stackexchange.com/a/312060/41104

s/ is / was /

busukxuan

2:05 PM

hey guys, does anyone use OpenSUSE here?

Vijay

3:09 PM

Hi...guys...i have problem with swap memory...can someone help me...

I have system with 64GB RAM and 32GB SWAP

I am running 3000 docker container....

and the server runs on Ubuntu 15.10

the RAM usage is 32GB but SWAP is slowly growing everyday....by few MBs.....after few days i left wtih no free SWAP memory....

even when i am have 32GB RAM left free...

derobert

@Vijay you should be able to see which process is using swap (with top if nothing else...) If you can find out if it's one process, that might solve the issue, or if it turns out to be a bunch, give enough detail to ask a question on the site.

Not personally familiar with it, but you could just have a container hitting its memory limit and thus starting to swap out. I think cgroups let you limit a container's swap usage too (but not sure if Docker exposes that knob)

Vijay

3:43 PM

top - 17:43:42 up 13 days, 5:49, 1 user, load average: 3.40, 3.49, 3.37
Tasks: 3182 total, 1 running, 3181 sleeping, 0 stopped, 0 zombie
%Cpu(s): 11.1 us, 1.7 sy, 0.0 ni, 76.2 id, 10.9 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 65788456 total, 64653728 used, 1134728 free, 464660 buffers
KiB Swap: 33554428 total, 6047288 used, 27507140 free. 31575272 cached Mem

This is top current result @derobert

but i use htop most of the time...

@derobert ....but here in htop it say around 32GB RAM and 6GB SWAP....and the SWAP memory will just increase over period of time...

I have summed all process swap usage from top output....it came to 1.2GB but no information why it using 5.8GB in swap....can you plz help me out in this...

derobert

@Vijay I personally haven't used Docker—but if you post on a question on the site, there are plenty of other people who have.

I'd hope Docker gives some tools to look at container memory usage, but if they don't and/or it turns out to be some obscure Docker problem—well, really need someone who knows Docker to help you.

Vijay

@derobert thanks alot for your reply.....i do that

derobert

BTW, the site is a lot less busy on the weekends. A lot of folks seem to use it from (and probably for) work.

Vijay

3:59 PM

oh...ok...thanks...again.... :)

Braiam

4:16 PM

@derobert how is the rebuild going?

derobert

6:10 PM

@Braiam it finished without any problems, I updated my answer

Faheem Mitha

6:51 PM

@derobert Are you now using btrfs? We may have talked about this before - I don't remember.

derobert

7:50 PM

@FaheemMitha I've been using btrfs for a bit now. The first backup on those external drives is from 2014-07-05

I think my TV PC has been on btrfs for longer.

Faheem Mitha

@derobert Ok. So working for you then? Is it an option in the Debian installer yet?

derobert

Actually, yeah, its predecessor was on btrfs too...

@FaheemMitha Yeah, I think it's in d-i. Probably since at least Wheezy, actually.

Faheem Mitha

@derobert oh. Would you recommend it over the default, currently ext4?

derobert

@FaheemMitha Depends if you want the new features (e.g., compression, deduplication, checksums) more than you want ext4's very proven stability.

I've now got several servers on btrfs...

The biggest thing to be aware of is that btrfs raid1 is fine. btrfs raid5/6, though, are not and shouldn't be used

Faheem Mitha

@derobert The builtin RAID? But how are people supposed to know that?

derobert

7:57 PM

@FaheemMitha well, they're supposed to read the docs before using it. E.g., btrfs.wiki.kernel.org/index.php/RAID56 has up top "The parity RAID code has multiple serious data-loss bugs in it. It should not be used for anything other than testing purposes."

Faheem Mitha

@derobert I see.But not everyone reads the docs. Is it better than mdadm for RAID 1 then?

derobert

@FaheemMitha It's quite different. btrfs RAID is done on a per-file basis. So you could have some files be RAID1, and others RAID0. Not that there is an API for that yet. But also it handles drives of different sizes—mdadm doesn't really (other than just using the smallest)

Faheem Mitha

@derobert So apples and oranges?

derobert

Sort of. Accomplishes the same goal, but then again I suppose apples and oranges are both fruit.

Faheem Mitha

Ok. Well, maybe I'll experiment at some point. How is recovery from a crash with btrfs?

derobert

8:04 PM

It's supposed to be crash-safe (just like ext4).

Faheem Mitha

Ok.

derobert

I haven't had any issues with filesystems mounting again after a crash, power outage, etc.

But I wouldn't trust it with my only copy of something important (of course, I wouldn't trust a disk with my only copy of something important, period).

In theory, it should survive some things better than ext4/mdraid. E.g., btrfs stores checksums—so if a disk corrupts something, it can pull it from the other copy. ext4 doesn't have checksums, but even if it did, it can't ask mdraid for the other copy.

Also, btrfs RAID1 only supports 2 copies. mdraid let's you have more.

Faheem Mitha

8:52 PM

@derobert That sounds useful in theory.

@derobert RAID 1 is just mirroring, right? It is useful to have more than 2 mirrors?

And doesn't it affect performance adversely?

derobert

10:11 PM

@FaheemMitha sure, if you have three mirrors, you can survive any two disks failing

and you're still redundant after one disk failed (during the rebuild)

it hurts writes a little, depending

Transcript for