Discussion between Deltik and SalmonKiller

Stack Exchange

05:23

1

Try specifying the number of network block devices to initialize using the nbds_max parameter: modprobe nbd nbds_max=8 Example: deltik@node51 [~]$ sudo rmmod nbd rmmod: ERROR: Module nbd is not currently loaded deltik@node51 [~]$ ls -l /dev/nbd* ls: cannot access '/dev/nbd*': No such file or...

SalmonKiller

Hmm. I tried doing this, and then doing sudo rmmod nbd. It actually removed the nbd from somewhere (I don't know where). Given that i'm running the whole system on google cloud platform, any tips on where the nbd might be?

Deltik

@SalmonKiller: Do you have the stock kernel that ships with Debian 8? Does nbd: registered device at major show up in /var/log/syslog?

SalmonKiller

Nope, it doesn't show up in /var/log/syslog.

Deltik

@SalmonKiller: And what kernel are you running?

SalmonKiller

3.16.0-4-amd64 is the kernel

Deltik

05:23

@SalmonKiller: Assuming you're logging normally, the nbd kernel module is not reaching this line in the source code. You should get network block devices because every failure case above that line would return an error code. Do you have folders at /sys/block/nbd*?

SalmonKiller

Yeah, i have those directories

Deltik

It sounds like the kernel module is registering the block devices, since they show up in /sys/block.

SalmonKiller

Yup.

Deltik

It could be that you have some strange udev rules that are preventing nbd* device files from being created in /dev.

SalmonKiller

hmm. I guess it's something that google cloud platform controls.

Deltik

05:32

What virtualization technology does Google Cloud compute use?

(I've never used them myself.)

SalmonKiller

Hehe. I'm totally new to this server world, so I might be wrong, but it looks like gcloud would be the answer to that question.

Deltik

If you install virt-what and run it, the command will tell you what virtualization technology you're using.

SalmonKiller

kvm

Deltik

Okay, so you can customize udev rules when you're using KVM.

I'm wondering if systemd screwed something up: github.com/systemd/systemd/pull/2422/commits/…

SalmonKiller

05:48

but that commit is already merged into production

Deltik

You probably have the older revision, like I do. But I'm not sure how it relates to your issue.

What output do you get when you run sudo dmesg | grep nbd?

SalmonKiller

[ 4791.180642] nbd: registered device at major 43

Deltik

Okay, that definitely means that the nbd module registered block devices.

Now, we just need to figure out why /dev/nbd0 and the 15 others like it don't exist.

SalmonKiller

probably mounted somewhere else

i asked another question on superuser

7

Q: VirtualBox Fatal: no bootable medium found! System Halted

I just download and then tried to add new virtual machine. But it gives me an error. I couldn't find the solution. it seems that VM can not find the OS. I do not know how to resolve. Error is: Fatal: no bootable medium found! System Halted. I have Vista and latest version of Sun VirtualBox.

virtualbox

Deltik

If it's somewhere else, I'd think that udev set it up. What is your output of this command?: sudo grep -ir nbd /etc/udev/ /lib/udev/

SalmonKiller

05:52

/lib/udev/rules.d/99-systemd.rules:# Ignore nbd devices in the "add" event, with "change" the nbd is ready
/lib/udev/rules.d/99-systemd.rules:ACTION=="add", SUBSYSTEM=="block", KERNEL=="nbd*", ENV{SYSTEMD_READY}="0"
/lib/udev/rules.d/60-persistent-storage.rules:KERNEL=="fd*|mtd*|nbd*|gnbd*|btibm*|dm-*|md*|zram*|mmcblk[0-9]*rpmb", GOTO="persistent_storage_end"

Deltik

Are you sure that's the right question on Super User?

Also, I don't have that line for /lib/udev/rules.d/60-persistent-storage.rules on my system. I'm looking into this now…

SalmonKiller

sorry i posted the wrong question

the one from super user wasn't mine

Deltik

I have an idea. What's the output of mount | grep 'udev'?

And one more command, just to be sure: df /dev

SalmonKiller

output of first is nothing

output of second is `Filesystem 1K-blocks Used Available Use% Mounted on
tmpfs 304352 0 304352 0% /dev`

Deltik

Okay, we might be getting somewhere now.

What is the output of df -T /dev?

SalmonKiller

06:00

same as before, `Filesystem Type 1K-blocks Used Available Use% Mounted on
tmpfs tmpfs 304352 0 304352 0% /dev`

Deltik

It's a little different this time

But

I think I figured it out.

The second column should show devtmpfs, not tmpfs.

SalmonKiller

i see

how it's different now

Deltik

Does this command work?: umount -v /dev

If yes, we can fix your problem in one more command.

SalmonKiller

after doing sudo umount -v /dev

it told me target was busy

umount: /dev: target is busy
(In some cases useful info about processes that
use the device is found by lsof(8) or fuser(1).)

Deltik

Hmm… lemme think.

SalmonKiller

06:04

here are some steps

stackoverflow.com/questions/7878707/…

it looks like they are risky though

Deltik

Bad idea. Don't do that.

Is udev running? Check with this command: systemctl status udev.service

SalmonKiller

This is just weird. This is what i'm getting: Failed to get D-Bus connection: Unknown error -1

Deltik

I can't remember… does Debian Jessie use systemd?

It sounds like systemd isn't running, so try this command instead: service udev status

SalmonKiller

Hmm. this is what i'm getting

udev does not support containers, not started ... (warning).

Also, just running udev returns -bash: udev: command not found

Deltik

Huh…

That suggests that maybe you are running a containerized operating system.

SalmonKiller

06:11

I think I am. Sorry if i forgot to mention that. Also, if I run just systemd it returns Trying to run as user instance, but the system has not been booted with systemd.

Deltik

Nevertheless, try this:

sudo rmmod nbd
sudo mount -t devtmpfs none /dev
sudo modprobe nbd
ls /dev/nbd*

SalmonKiller

WOHOOOOO

it worked

Deltik

Heh

SalmonKiller

wait, so what did that 2nd line do?

Deltik

Well, that's good, but it doesn't explain why devtmpfs wasn't in use in the first place.

SalmonKiller

06:13

devtmpfs is another version of the tmpfs file system?

Deltik

Okay, so there's a mount on /dev already, but it's tmpfs instead of devtmpfs. You need devtmpfs, so the second command puts a mount over that mount so that new accesses to /dev go to the devtmpfs.

I'll update my answer to reflect these findings.

SalmonKiller

Cool. Awesome, thanks for you time mate.

Deltik

You're welcome

Deltik

06:31

Okay, @SalmonKiller, I updated my answer for your perusal: serverfault.com/a/787572/193677

Transcript for

Discussion between Deltik and SalmonK…