Let's start with two simple things, first, ls -l /dev/md3. Second, grep /mnt /proc/mounts. And a third thing, a little less simple, do those mount lines add any kernel messages (dmesg)?

(And, also, for anyone else who hits a similar problem: mkfs is seldom, nay, almost never the right way to recover from a raid failure.)

Yes, the /mnt directory exists. @derobert: first gives me brw-rw---- 1 root root 9, 3 Jan 18 2006 /dev/md3, second is nothing and third, there's no messages in dmesg pertaining to the failed mount.

@AndyE What is in /proc/mounts? Trying to figure out where the busy message came from (makes it sound like its already mounted somewhere...)

@AndyE Also, it appears your md3 uses 1.2 superblocks. And that your md0 and md1 do not (I'm guessing they use 0.90, but --detail would say for sure). Are you sure you recreated the array with the right options?

@derobert: no, I don't think I did recreate the array with the right options. I was following the advice of someone on the Synology forums who replied to someone with a similar issue, can't seem to find it now, though. I've added the /proc/mounts result to the post. Note that the /dev/vg* stuff is how the NAS mounts storage usually.

I realise it was a bad idea to do things this way, but I was losing a lot of time sending emails back and forth to Synology's tech support who weren't very helpful. I figured if I formatted one drive I could run data recovery on the other, which I successfully did, but I have around 80k unsorted files which I'm not looking forward to sorting. If I could get the drive mounted I could just copy the data to the formatted drive manually with the directory tree intact.

So, it looks like you have LVM running on that as well. So you may actually have a LVM PV on that RAID set. What do you get from lvs, pvs, and vgscan? Oh, and also, what about file -s /dev/sdb5?

@derobert: I'm sorry, it's my fault, I've been trying to mount the wrong disk! sdb is already mounted, as indicated by lvs. However, I'm still unable to mount /dev/sda1 with mount -t ext4 /dev/sda1 /mnt. The error message is "mount: mounting /dev/sda1 on /mnt failed: Invalid argument".

@AndyE that's not going to work if its actually a raid disk... Though please check dmesg to see if there is an error there.

It sounds like you probably have disk -> mdraid raid1 -> lvm pv -> lvm lv -> ext4

If you're lucky, all that stuff turns out to be mostly transparent, and you just have disk -> ext4 at a weird offset

In which case, you just need to find where the filesystem starts... look for the ext4 magic number 0xEF53 at offset 0x38 in each sector (or even 4K block).

I think there is a program called "testdisk" that can do this for you...

Yeah, I have testdisk, I'll run it now

Hey, thanks for your help so far. I still feel a bit sheepish about trying to mount the wrong disk

photorec is what I used to recover the files before

Hopefully testdisk finds your ext4 partition

if it does, mount it read only, and attempt to recover your data from it...

this is what the testdisk result looks like:

Disk /dev/sda - 3000 GB / 2794 GiB - CHS 364801 255 63
     Partition               Start        End    Size in sectors
 D MS Data                      256    4980607    4980352 [1.41.12-1958]
 D Linux Raid                   256    4980735    4980480 [md0]
 D Linux Swap               4980736    9174895    4194160
 D Linux Raid               4980736    9175039    4194304 [md1]
>P Linux Raid               9453280 5860516583 5851063304 [DiskStation:2]

that's after it did the scan?

yeah. Not really sure what to do here, do I try and change the type to ext4?

hmmm, it means it didn't find it, or your version doesn't support ext4

can you have it do a deeper search on that last linux raid?

brb

running the deeper search now. This is probably going to take a while, so no rush :-)

yep

3% :-)

oh dear

well, you could try the other way while its scanning, did pvscan find anything on that array (when you had it started?)

  PV /dev/md3   VG vg1000   lvm2 [2.72 TB / 0    free]
  Total: 1 [742.00 GB] / in use: 1 [742.00 GB] / in no VG: 0 [0   ]

just that

indicating the formatted drive only

Oooh, you created 1.2 superblocks on the device when you re-made it, right?

I don't know, that was for the other drive, I think.

use mdadm -E on it to find out

It was all done automatically using the Synology web interface.

I mean, when you attempted to recover by using mdadm --create

oh, yeah that's right

$ mdadm -E /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 01dd80d9:5b5dfe30:7cbc82a0:15cbc91d
           Name : DiskStation:2  (local to host DiskStation)
  Creation Time : Thu Nov 15 17:48:12 2012
     Raid Level : raid1
   Raid Devices : 1

 Avail Dev Size : 5851063680 (2790.00 GiB 2995.74 GB)
     Array Size : 5851063296 (2790.00 GiB 2995.74 GB)
  Used Dev Size : 5851063296 (2790.00 GiB 2995.74 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors

wait, is sda1 the right partition?

there's only /dev/sda and /dev/sda1, although testdisk seems to have found more

$ cat /proc/partitions
major minor  #blocks  name

   8        0 2930266584 sda
   8        1 2925531652 sda1
   8       16 2930266584 sdb
   8       17    2490240 sdb1
   8       18    2097152 sdb2
   8       21 2925532864 sdb5
  31        0        512 mtdblock0
  31        1       2048 mtdblock1
  31        2       1280 mtdblock2
  31        3         64 mtdblock3
  31        4        128 mtdblock4
  31        5         64 mtdblock5
  31        6       4096 mtdblock6
   9        0    2490176 md0

what does mdadm -E /dev/sda1 -e 0.90 give?

mdadm: No super block found on /dev/sda1 (Expected magic a92b4efc, got 00000000)

darn :-(

I guess that's bad... I'm sorry I'm pretty clueless about all this superblock stuff

well, I was hoping that the original superblock somehow got spared

did you repartition this disk as well?

I used the Synology tool to remove the volume before I formatted the other disc, which I thought afterwards may have been a mistake

I think that may have done something to the partition table, not sure.

that may have, or it may have removed the LVM logical volume

hmmm, do you happen to have an /etc/lvm/backup/ directory on that NAS?

or /etc/lvm/archive ?

yes

if you have archive, that'd be awesome. Especially if you have one from before the failure.

$ ls /etc/lvm/archive/
vg1000_00000.vg  vg1000_00001.vg  vg1000_00002.vg  vg1000_00003.vg  vg1000_00004.vg  vg1000_00005.vg

us ls -l to check the dates... Also, those are plain text files, you can read them with less (or whatever)

there's two from feb 13 2011 lol

those only change when you add/remove volumes

so that might be new enough

Well, I lol'ed because that is from before I bought the thing. Although thinking about it, the date probably wasn't set before the volume was created.

hah, that could be.

Look at them, and see if you can find information about your volume

            device = "/dev/md2" # Hint only
 24
 25             status = ["ALLOCATABLE"]
 26             dev_size = 5851063296   # 2.72461 Terabytes
 27             pe_start = 1152
 28             pe_count = 714240   # 2.72461 Terabytes

there's not much to it other than that

there isn't a logical_volumes { ... } section?

Hmmm, come to think of it, I bet it recreated it on your blank disk exactly the same way as it created it originally

What does mdadm -D /dev/md3 say?

physical_volumes

/dev/md3:
        Version : 1.2
  Creation Time : Fri Nov 16 11:10:39 2012
     Raid Level : raid1
     Array Size : 2925531648 (2790.00 GiB 2995.74 GB)
  Used Dev Size : 2925531648 (2790.00 GiB 2995.74 GB)
   Raid Devices : 1
  Total Devices : 1
    Persistence : Superblock is persistent

    Update Time : Fri Nov 30 18:12:10 2012
          State : clean
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           Name : DiskStation:2  (local to host DiskStation)

Interesting, so it does use 1.2 superblocks. That's good, I suppose, as that's what you used

what about sfdisk -d /dev/sdb (and for /dev/sda as well)

it says Please specify -z and -N

odd, my sfdisk doesn't even have a -z...

what about fdisk -l /dev/sdb... does that work?

it looks like a custom Synology thing, they have their copyright on it

fdisk gives me

fdisk: device has more than 2^32 sectors, can't use all of them

Disk /dev/sdb: 2199.0 GB, 2199023255040 bytes
255 heads, 63 sectors/track, 267349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks  Id System
/dev/sdb1               1      267350  2147483647+ ee EFI GPT

ah, its using gpt

how about mdadm -E /dev/sdb5 ... let's see how different that is than whats on sda1 at this point

$ mdadm -E /dev/sdb5
/dev/sdb5:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 6fa3143c:344c9ebe:3d472d13:b7c7c298
           Name : DiskStation:2  (local to host DiskStation)
  Creation Time : Fri Nov 16 11:10:39 2012
     Raid Level : raid1
   Raid Devices : 1

 Avail Dev Size : 5851063680 (2790.00 GiB 2995.74 GB)
     Array Size : 5851063296 (2790.00 GiB 2995.74 GB)
  Used Dev Size : 5851063296 (2790.00 GiB 2995.74 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors

Interesting, they're the exact same size, even though the partitions are different

you did run mdadm --create on sda1 at some point, right?

yeah, I believe so. would I lose that after a reboot though?

nope, mdadm --create writes to the disk

ah I see

do you have parted on there? To try and print the two partition tables?

or gparted

(parted /dev/sda p ; parted /dev/sdb p)

$ (parted /dev/sda p ; parted /dev/sdb p)
Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      4840MB  3001GB  2996GB                     boot

Model: ATA ST3000DM001-9YN1 (scsi)
Disk /dev/sdb: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name  Flags
 1      131kB   2550MB  2550MB  ext4                  raid

That is an interesting partitioning setup they have there...

Ok, trying to figure out how to get the actual start sector (of your newly formatted volume) out of device mapper

ok... I think I figured it out

the command is dmsetup table

root@Zia:/etc/lvm/backup# dmsetup table Zia-root
0 195305472 linear 9:127 11715584

that is saying that my logical volume Zia-root (which would be volume group Zia, lv root) starts at 11715584

It's Synology Hybrid Raid, their own custom implementation I think

so, now, I just need to figure out how to convert 1152 to a block number... and then we can use losetup and mount the filesystem

cool, you're a real life saver

Which RAID device is that currently mapped to (which /dev/mdX number)?

dd if=/dev/md127 bs=512 skip=$((1152+2)) count=1 | xxd

use the right number instead of 127

and then look at this line:

0000030: 124a aa50 1400 1900 53ef 0100 0100 0000 .J.P....S.......

and let's hope that you see the 53ef

to be clear, the raid device that /dev/sda1 (the one we're trying to recover data from) is mapped to

hmm, it's not mapped to an mdX

maybe I didn't use mdadm --create on it. Although I really thought I did.

wait, if you didn't this may be real easy

mdadm --assemble -v --read-only /dev/md5 /dev/sda1

I just made up md5, just make sure it isn't already in use

it didn't like the --read-only option so I removed it, and got this:

$ mdadm --assemble -v /dev/md5 /dev/sda1
mdadm: looking for devices for /dev/md5
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has no superblock - assembly aborted

check /proc/mdstat ... see if its already in an array

nope

$ cat /proc/mdstat
Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4]
md3 : active raid1 sdb5[0]
      2925531648 blocks super 1.2 [1/1] [U]

md1 : active raid1 sdb2[0]
      2097088 blocks [2/1] [U_]

md0 : active raid1 sdb1[0]
      2490176 blocks [2/1] [U_]

unused devices: <none>

hmmm, maybe its from the testdisk running, not sure. Anyway, you can try that dd line on sda1... if it doesn't work (you don't get the magic number) add in a +2048 and see if that gets the magic number

got it with the +2048

0000030: 1419 a550 1a00 1f00 53ef 0100 0100 0000  ...P....S.......

ok, that's good news, you've found the ext4 filesystem

yay

so, run losetup -f to find a free loop device

$ losetup -f
/dev/loop0

then losetup /dev/loop0 -o $((512*(1152+2048))) -r /dev/sda1

that should set up a read-only (-r) mapping

then you could try, for example, e2fsck -n /dev/loop0

or be optimistic and mount -text4 /dev/loop0 /mnt (or wherever)

well, add an -o ro in there, though it should be default due to the r/o loop device

GENIUS!

it worked?

it worked, thank you so much

And next time, ask before attempting data recovery... :-P

I'm going to post an answer attempting to summarize.

lol I will

thanks again, I really appreciate the whole thing.

And, BTW, its more expensive, but the normal first thing to do when trying data recovery like this is "make a copy of the disks, and work on the copy".

yeah, I just didn't have another 3TB drive (nor could I afford one right now) to copy to

I'll be more careful in the future ;-)

Ok, I plopped in my summary answer. Feel free to edit it.

I'm sure I don't have to tell you of the importance of backups, either :-P

Also, FYI, when you ran mdadm --create (assuming you actually did so), it wrote to the disk, there is a chance that scribbled on one of your files. Also, whatever failure caused the initial problem may have left corruption. So, there is a chance that one or more of the files you're copying off may be corrupt.

No idea what type of files you're storing, but beware that corruption potential. If you have trusted copies of important files, you'll want to compare them to check.

Anyway, I'm off.

It's mostly dev stuff, almost all of it was backed up (some of it is backups of what I have on my laptop) except for about a weeks worth of work, but there were also some rips of my daughter's dance recital DVD that I didn't have backed up. Once again, thank you :-)