last day (15 days later) » 

1:35 PM
Q: Unable to mount raid on my NAS, trying to rescue the data, how should I proceed?

Andy EA short backstory: something happened on my Synology NAS, apparently a buggy mailserver program that came with an update stopped the device from seeing the 2x3TB drives I have. After a lot of searching and a trying a lot of different solutions I eventually got the drives to show up again but the...

Let's start with two simple things, first, ls -l /dev/md3. Second, grep /mnt /proc/mounts. And a third thing, a little less simple, do those mount lines add any kernel messages (dmesg)?
(And, also, for anyone else who hits a similar problem: mkfs is seldom, nay, almost never the right way to recover from a raid failure.)
Yes, the /mnt directory exists. @derobert: first gives me brw-rw---- 1 root root 9, 3 Jan 18 2006 /dev/md3, second is nothing and third, there's no messages in dmesg pertaining to the failed mount.
@AndyE What is in /proc/mounts? Trying to figure out where the busy message came from (makes it sound like its already mounted somewhere...)
@AndyE Also, it appears your md3 uses 1.2 superblocks. And that your md0 and md1 do not (I'm guessing they use 0.90, but --detail would say for sure). Are you sure you recreated the array with the right options?
@derobert: no, I don't think I did recreate the array with the right options. I was following the advice of someone on the Synology forums who replied to someone with a similar issue, can't seem to find it now, though. I've added the /proc/mounts result to the post. Note that the /dev/vg* stuff is how the NAS mounts storage usually.
I realise it was a bad idea to do things this way, but I was losing a lot of time sending emails back and forth to Synology's tech support who weren't very helpful. I figured if I formatted one drive I could run data recovery on the other, which I successfully did, but I have around 80k unsorted files which I'm not looking forward to sorting. If I could get the drive mounted I could just copy the data to the formatted drive manually with the directory tree intact.
So, it looks like you have LVM running on that as well. So you may actually have a LVM PV on that RAID set. What do you get from lvs, pvs, and vgscan? Oh, and also, what about file -s /dev/sdb5?
1:35 PM
@derobert: I'm sorry, it's my fault, I've been trying to mount the wrong disk! sdb is already mounted, as indicated by lvs. However, I'm still unable to mount /dev/sda1 with mount -t ext4 /dev/sda1 /mnt. The error message is "mount: mounting /dev/sda1 on /mnt failed: Invalid argument".
@AndyE that's not going to work if its actually a raid disk... Though please check dmesg to see if there is an error there.
It sounds like you probably have disk -> mdraid raid1 -> lvm pv -> lvm lv -> ext4
If you're lucky, all that stuff turns out to be mostly transparent, and you just have disk -> ext4 at a weird offset
In which case, you just need to find where the filesystem starts... look for the ext4 magic number 0xEF53 at offset 0x38 in each sector (or even 4K block).
I think there is a program called "testdisk" that can do this for you...
2:20 PM
Yeah, I have testdisk, I'll run it now
Hey, thanks for your help so far. I still feel a bit sheepish about trying to mount the wrong disk
photorec is what I used to recover the files before
2:33 PM
Hopefully testdisk finds your ext4 partition
if it does, mount it read only, and attempt to recover your data from it...
this is what the testdisk result looks like:
Disk /dev/sda - 3000 GB / 2794 GiB - CHS 364801 255 63
     Partition               Start        End    Size in sectors
 D MS Data                      256    4980607    4980352 [1.41.12-1958]
 D Linux Raid                   256    4980735    4980480 [md0]
 D Linux Swap               4980736    9174895    4194160
 D Linux Raid               4980736    9175039    4194304 [md1]
>P Linux Raid               9453280 5860516583 5851063304 [DiskStation:2]
that's after it did the scan?
yeah. Not really sure what to do here, do I try and change the type to ext4?
hmmm, it means it didn't find it, or your version doesn't support ext4
can you have it do a deeper search on that last linux raid?
running the deeper search now. This is probably going to take a while, so no rush :-)
2:44 PM
2 hours later…
4:27 PM
3% :-)
1 hour later…
5:46 PM
oh dear
well, you could try the other way while its scanning, did pvscan find anything on that array (when you had it started?)
  PV /dev/md3   VG vg1000   lvm2 [2.72 TB / 0    free]
  Total: 1 [742.00 GB] / in use: 1 [742.00 GB] / in no VG: 0 [0   ]
just that
indicating the formatted drive only
Oooh, you created 1.2 superblocks on the device when you re-made it, right?
I don't know, that was for the other drive, I think.
use mdadm -E on it to find out
It was all done automatically using the Synology web interface.
5:56 PM
I mean, when you attempted to recover by using mdadm --create
oh, yeah that's right
$ mdadm -E /dev/sda1
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 01dd80d9:5b5dfe30:7cbc82a0:15cbc91d
           Name : DiskStation:2  (local to host DiskStation)
  Creation Time : Thu Nov 15 17:48:12 2012
     Raid Level : raid1
   Raid Devices : 1

 Avail Dev Size : 5851063680 (2790.00 GiB 2995.74 GB)
     Array Size : 5851063296 (2790.00 GiB 2995.74 GB)
  Used Dev Size : 5851063296 (2790.00 GiB 2995.74 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
wait, is sda1 the right partition?
there's only /dev/sda and /dev/sda1, although testdisk seems to have found more
$ cat /proc/partitions
major minor  #blocks  name

   8        0 2930266584 sda
   8        1 2925531652 sda1
   8       16 2930266584 sdb
   8       17    2490240 sdb1
   8       18    2097152 sdb2
   8       21 2925532864 sdb5
  31        0        512 mtdblock0
  31        1       2048 mtdblock1
  31        2       1280 mtdblock2
  31        3         64 mtdblock3
  31        4        128 mtdblock4
  31        5         64 mtdblock5
  31        6       4096 mtdblock6
   9        0    2490176 md0
what does mdadm -E /dev/sda1 -e 0.90 give?
mdadm: No super block found on /dev/sda1 (Expected magic a92b4efc, got 00000000)
5:59 PM
darn :-(
I guess that's bad... I'm sorry I'm pretty clueless about all this superblock stuff
well, I was hoping that the original superblock somehow got spared
did you repartition this disk as well?
I used the Synology tool to remove the volume before I formatted the other disc, which I thought afterwards may have been a mistake
I think that may have done something to the partition table, not sure.
that may have, or it may have removed the LVM logical volume
hmmm, do you happen to have an /etc/lvm/backup/ directory on that NAS?
or /etc/lvm/archive ?
6:03 PM
if you have archive, that'd be awesome. Especially if you have one from before the failure.
$ ls /etc/lvm/archive/
us ls -l to check the dates... Also, those are plain text files, you can read them with less (or whatever)
there's two from feb 13 2011 lol
those only change when you add/remove volumes
so that might be new enough
Well, I lol'ed because that is from before I bought the thing. Although thinking about it, the date probably wasn't set before the volume was created.
6:07 PM
hah, that could be.
Look at them, and see if you can find information about your volume
            device = "/dev/md2" # Hint only
 25             status = ["ALLOCATABLE"]
 26             dev_size = 5851063296   # 2.72461 Terabytes
 27             pe_start = 1152
 28             pe_count = 714240   # 2.72461 Terabytes
there's not much to it other than that
there isn't a logical_volumes { ... } section?
Hmmm, come to think of it, I bet it recreated it on your blank disk exactly the same way as it created it originally
What does mdadm -D /dev/md3 say?
        Version : 1.2
  Creation Time : Fri Nov 16 11:10:39 2012
     Raid Level : raid1
     Array Size : 2925531648 (2790.00 GiB 2995.74 GB)
  Used Dev Size : 2925531648 (2790.00 GiB 2995.74 GB)
   Raid Devices : 1
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Fri Nov 30 18:12:10 2012
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : DiskStation:2  (local to host DiskStation)
Interesting, so it does use 1.2 superblocks. That's good, I suppose, as that's what you used
what about sfdisk -d /dev/sdb (and for /dev/sda as well)
it says Please specify -z and -N
6:16 PM
odd, my sfdisk doesn't even have a -z...
what about fdisk -l /dev/sdb... does that work?
it looks like a custom Synology thing, they have their copyright on it
fdisk gives me
fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdb: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1               1      267350  2147483647+ ee EFI GPT
ah, its using gpt
how about mdadm -E /dev/sdb5 ... let's see how different that is than whats on sda1 at this point
$ mdadm -E /dev/sdb5
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6fa3143c:344c9ebe:3d472d13:b7c7c298
           Name : DiskStation:2  (local to host DiskStation)
  Creation Time : Fri Nov 16 11:10:39 2012
     Raid Level : raid1
   Raid Devices : 1

 Avail Dev Size : 5851063680 (2790.00 GiB 2995.74 GB)
     Array Size : 5851063296 (2790.00 GiB 2995.74 GB)
  Used Dev Size : 5851063296 (2790.00 GiB 2995.74 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
Interesting, they're the exact same size, even though the partitions are different
you did run mdadm --create on sda1 at some point, right?
yeah, I believe so. would I lose that after a reboot though?
6:26 PM
nope, mdadm --create writes to the disk
ah I see
do you have parted on there? To try and print the two partition tables?
or gparted
(parted /dev/sda p ; parted /dev/sdb p)
$ (parted /dev/sda p ; parted /dev/sdb p)
Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      4840MB  3001GB  2996GB                     boot

Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name  Flags
 1      131kB   2550MB  2550MB  ext4                  raid
That is an interesting partitioning setup they have there...
Ok, trying to figure out how to get the actual start sector (of your newly formatted volume) out of device mapper
ok... I think I figured it out
the command is dmsetup table
root@Zia:/etc/lvm/backup# dmsetup table Zia-root
0 195305472 linear 9:127 11715584
that is saying that my logical volume Zia-root (which would be volume group Zia, lv root) starts at 11715584
6:45 PM
$ dmsetup table
vg1000-lv: 0 5851054080 linear 9:3 1152
It's Synology Hybrid Raid, their own custom implementation I think
so, now, I just need to figure out how to convert 1152 to a block number... and then we can use losetup and mount the filesystem
cool, you're a real life saver
Which RAID device is that currently mapped to (which /dev/mdX number)?
dd if=/dev/md127 bs=512 skip=$((1152+2)) count=1 | xxd
use the right number instead of 127
and then look at this line:
0000030: 124a aa50 1400 1900 53ef 0100 0100 0000 .J.P....S.......
and let's hope that you see the 53ef
to be clear, the raid device that /dev/sda1 (the one we're trying to recover data from) is mapped to
hmm, it's not mapped to an mdX
maybe I didn't use mdadm --create on it. Although I really thought I did.
7:01 PM
wait, if you didn't this may be real easy
mdadm --assemble -v --read-only /dev/md5 /dev/sda1
I just made up md5, just make sure it isn't already in use
it didn't like the --read-only option so I removed it, and got this:
$ mdadm --assemble -v /dev/md5 /dev/sda1
mdadm: looking for devices for /dev/md5
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has no superblock - assembly aborted
check /proc/mdstat ... see if its already in an array
$ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid1 sdb5[0]
      2925531648 blocks super 1.2 [1/1] [U]

md1 : active raid1 sdb2[0]
      2097088 blocks [2/1] [U_]

md0 : active raid1 sdb1[0]
      2490176 blocks [2/1] [U_]

unused devices: <none>
hmmm, maybe its from the testdisk running, not sure. Anyway, you can try that dd line on sda1... if it doesn't work (you don't get the magic number) add in a +2048 and see if that gets the magic number
got it with the +2048
0000030: 1419 a550 1a00 1f00 53ef 0100 0100 0000  ...P....S.......
7:08 PM
ok, that's good news, you've found the ext4 filesystem
so, run losetup -f to find a free loop device
$ losetup -f
then losetup /dev/loop0 -o $((512*(1152+2048))) -r /dev/sda1
that should set up a read-only (-r) mapping
then you could try, for example, e2fsck -n /dev/loop0
or be optimistic and mount -text4 /dev/loop0 /mnt (or wherever)
well, add an -o ro in there, though it should be default due to the r/o loop device
7:12 PM
it worked?
it worked, thank you so much
And next time, ask before attempting data recovery... :-P
I'm going to post an answer attempting to summarize.
lol I will
thanks again, I really appreciate the whole thing.
And, BTW, its more expensive, but the normal first thing to do when trying data recovery like this is "make a copy of the disks, and work on the copy".
yeah, I just didn't have another 3TB drive (nor could I afford one right now) to copy to
I'll be more careful in the future ;-)
7:26 PM
Ok, I plopped in my summary answer. Feel free to edit it.
I'm sure I don't have to tell you of the importance of backups, either :-P
Also, FYI, when you ran mdadm --create (assuming you actually did so), it wrote to the disk, there is a chance that scribbled on one of your files. Also, whatever failure caused the initial problem may have left corruption. So, there is a chance that one or more of the files you're copying off may be corrupt.
No idea what type of files you're storing, but beware that corruption potential. If you have trusted copies of important files, you'll want to compare them to check.
Anyway, I'm off.
8:21 PM
It's mostly dev stuff, almost all of it was backed up (some of it is backups of what I have on my laptop) except for about a weeks worth of work, but there were also some rips of my daughter's dance recital DVD that I didn't have backed up. Once again, thank you :-)

last day (15 days later) »