A short backstory: something happened on my Synology NAS, apparently a buggy mailserver program that came with an update stopped the device from seeing the 2x3TB drives I have. After a lot of searching and a trying a lot of different solutions I eventually got the drives to show up again but the...
Let's start with two simple things, first, ls -l /dev/md3. Second, grep /mnt /proc/mounts. And a third thing, a little less simple, do those mount lines add any kernel messages (dmesg)?
(And, also, for anyone else who hits a similar problem: mkfs is seldom, nay, almost never the right way to recover from a raid failure.)
Yes, the /mnt directory exists. @derobert: first gives me brw-rw---- 1 root root 9, 3 Jan 18 2006 /dev/md3, second is nothing and third, there's no messages in dmesg pertaining to the failed mount.
@AndyE What is in /proc/mounts? Trying to figure out where the busy message came from (makes it sound like its already mounted somewhere...)
@AndyE Also, it appears your md3 uses 1.2 superblocks. And that your md0 and md1 do not (I'm guessing they use 0.90, but --detail would say for sure). Are you sure you recreated the array with the right options?
@derobert: no, I don't think I did recreate the array with the right options. I was following the advice of someone on the Synology forums who replied to someone with a similar issue, can't seem to find it now, though. I've added the /proc/mounts result to the post. Note that the /dev/vg* stuff is how the NAS mounts storage usually.
I realise it was a bad idea to do things this way, but I was losing a lot of time sending emails back and forth to Synology's tech support who weren't very helpful. I figured if I formatted one drive I could run data recovery on the other, which I successfully did, but I have around 80k unsorted files which I'm not looking forward to sorting. If I could get the drive mounted I could just copy the data to the formatted drive manually with the directory tree intact.
So, it looks like you have LVM running on that as well. So you may actually have a LVM PV on that RAID set. What do you get from lvs, pvs, and vgscan? Oh, and also, what about file -s /dev/sdb5?
@derobert: I'm sorry, it's my fault, I've been trying to mount the wrong disk! sdb is already mounted, as indicated by lvs. However, I'm still unable to mount /dev/sda1 with mount -t ext4 /dev/sda1 /mnt. The error message is "mount: mounting /dev/sda1 on /mnt failed: Invalid argument".
@AndyE that's not going to work if its actually a raid disk... Though please check dmesg to see if there is an error there.
It sounds like you probably have disk -> mdraid raid1 -> lvm pv -> lvm lv -> ext4
If you're lucky, all that stuff turns out to be mostly transparent, and you just have disk -> ext4 at a weird offset
In which case, you just need to find where the filesystem starts... look for the ext4 magic number 0xEF53 at offset 0x38 in each sector (or even 4K block).
I think there is a program called "testdisk" that can do this for you...
Disk /dev/sda - 3000 GB / 2794 GiB - CHS 364801 255 63
Partition Start End Size in sectors
D MS Data 256 4980607 4980352 [1.41.12-1958]
D Linux Raid 256 4980735 4980480 [md0]
D Linux Swap 4980736 9174895 4194160
D Linux Raid 4980736 9175039 4194304 [md1]
>P Linux Raid 9453280 5860516583 5851063304 [DiskStation:2]
it looks like a custom Synology thing, they have their copyright on it
fdisk gives me
fdisk: device has more than 2^32 sectors, can't use all of them
Disk /dev/sdb: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 267350 2147483647+ ee EFI GPT
$ (parted /dev/sda p ; parted /dev/sdb p)
Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 4840MB 3001GB 2996GB boot
Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 131kB 2550MB 2550MB ext4 raid
it didn't like the --read-only option so I removed it, and got this:
$ mdadm --assemble -v /dev/md5 /dev/sda1
mdadm: looking for devices for /dev/md5
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has no superblock - assembly aborted
hmmm, maybe its from the testdisk running, not sure. Anyway, you can try that dd line on sda1... if it doesn't work (you don't get the magic number) add in a +2048 and see if that gets the magic number
And, BTW, its more expensive, but the normal first thing to do when trying data recovery like this is "make a copy of the disks, and work on the copy".
Ok, I plopped in my summary answer. Feel free to edit it.
I'm sure I don't have to tell you of the importance of backups, either :-P
Also, FYI, when you ran mdadm --create (assuming you actually did so), it wrote to the disk, there is a chance that scribbled on one of your files. Also, whatever failure caused the initial problem may have left corruption. So, there is a chance that one or more of the files you're copying off may be corrupt.
No idea what type of files you're storing, but beware that corruption potential. If you have trusted copies of important files, you'll want to compare them to check.
It's mostly dev stuff, almost all of it was backed up (some of it is backups of what I have on my laptop) except for about a weeks worth of work, but there were also some rips of my daughter's dance recital DVD that I didn't have backed up. Once again, thank you :-)
Discussion between derobert and Andy E
Discussion between derobert and Andy E