Both Requires and After have been given. The devices themselves are changed frequently, and so referencing them in the unit descriptor is not feasible. I wish the mount to activate when no more devices will be attached through the initialization of the iSCSI service.

@brainchild What exactly do you mean by changed though? I suspect the service (more precisely its corresponding iscsiadm command) is non-blocking / asynchronous. Even if it does wait on certain level, I doubt that it can wait until the SCSI disk driver has done probing /enumerating all the targets.

@TomYan, The volume itself spans multiple physical devices. The devices associated with the logical volume are changed frequently, sometimes added, and sometimes removed. Of course mounting volumes on the network block devices must occur synchronously only after all devices are scanned. Otherwise, each boot sequence would inevitably be bound to a race condition, as is the premise of my issue.

Of course mounting volumes on the network block devices must occur synchronously only after all devices are scanned. What I mean is, the iscsi service acts like someone who plug a bunch of disks to a machine, and even if the ExecStart command return only after it has finished plugging all the drives, I am not quite sure there's any facilities for it (in user space) to get notified by the kernel space that all of the disks it has plugged has been enumerated.

Btw, what kind of volume are we talking about here? Is it BTRFS? To be frank, it sounds to me that the "synchronization"/"waiting" should be done by the filesystem driver...(for example, apparently there's btrfs device ready)

@TomYan, Btrfs scans devices currently registered in the kernel that match a UUID. If no device is found with metadata matching the ID, or if the metadata written to any matching device refers to devices not present, then mounting fails. The mount operation does not wait for devices to become available. Such behavior has never been expected. Device discovery must complete before mounting is attempted.

@brainchild that's why I think you probably need the mount unit to pull a custom service that run corresponding btrfs device ready (I haven't really tested how exactly it works though, like if it wait indefinitely or merely exit with different return code to reflects the readiness of the required devices). (Unfortunately though, apparently it takes a device node instead of a UUID or so as its argument, which means you still need to change the service from time to time unless there's at least a single device that is essentially "fixed".)

@TomYan: The issue is not one appropriately addressed with the file system tools. The storage controller drivers (e.g. iSCSI) resolve when the device scan is complete. The file system may only address whether a mount is viable.

AFAIK, there's two level of "device scan" when it comes to iSCSI. Just as I said, the client side program might be able to make sure it "insert"/"virtualize" every targets/drives before it quits, but that doesn't mean it can talk to the generic scsi disk driver and wait until that done probing all the drives it inserted. (Also, I'm not seeing why you even bother insisting otherwise, because if that's true, you probably won't be stuck in this problem/dilemma? I mean what was your goal to ask here then? Also, take a look at the man page of btrfs-device for its ready subcommand.)

@brainchild Also, this isn't even really about iSCSI AFAICT, because it would just be the same case if e.g. a physical storage device is slow in responding to device probing/enumeration.

@TomYan, The issue is particular to iSCSI, because mounting in general is deferred until after scanning of physical block devices is complete.

@brainchild If you say so. Try to define "scanning is complete" when "all devices" and the time that "they are here/ready" is not logically deterministic. Remember the btrfs driver is what that is responsible for "assembling" your "volume" here btw, which makes it the only logically sensical "station" to wait at. (You can only get away from the race by start scanning when the devices are actually there, and the generic scsi disk driver is what makes them so, not anything iscsi.)

@TomYan, Scanning for devices is a set of events that concludes at some definite point in time. It is only after such point that mounting should be attempted. The mounting operation should begin after the device controller (e.g. iSCSI) has completed scanning for all devices.

@brainchild it really doesn't have anything to do with "device controller". No device controller would tell you anything like "I have scanned all devices", as "all" doesn't make sense in this context. I've given you facts/logics (instead of wild guesses / "faith") in my comments and an answer how things work to help you / get you out of the problem you bumped into, if you for whatever reason don't believe in the info, just drop it. There's no point to argue. I'm not the one who can't get things to work here.

@TomYan, Sorry, but your assertions are hard for me to understand as accurate. For a local SCSI controller and disk holding a file system, the boot sequence would ensure the device being registered in the kernel before an attempt to mount. Otherwise, all Linux systems, as a matter of course, would fail to boot intermittently, due to such a race condition. If a SCSI device is would be discovered at boot, then it will be so before the mounts are processed. The difference for network devices is simply that certain operations must follow loading the network stack.

@brainchild: The boot sequence only needs to wait for that specific device to appear; it doesn't need to concern itself with the entire enumeration being "done" or the specifics of the bus or controller that provides the device. When the initramfs needs to mount /dev/md1 on /newroot, it waits only for /dev/md1 to become "ready", which might be assembled incrementally as hotplug events arrive. And when the initramfs needs to mount a Btrfs array, it can likewise wait for Btrfs to mark that specific array as complete. I.e. do not think of the boot process as following a strict linear order.

@brainchild also you have been over-generalizing "registered in the kernel". There are different components (drivers at different levels/layers) in the kernel, and a device don't get registered "atomically" at all of them. Neither does an HBA or its driver needs the driver at some upper layer (e.g. sd / btrfs) to "report back" to the lower layer (or userspace program that talks with it) that "all devices you exposed to us have been enumerated", and iscsi does not make an exception, just as you said. (Literally you own argument contradicts with your belief.)

@brainchild in a systemd system, the only thing that "holds" attempt of mounting is the unit ordering between the mount unit and the device unit, and the iscsi unit does NOT declare finished only until all related device unit/node are created, at least not by default. (And even if/when it does, technically it still doesn't guarantee the btrfs driver have done scanning all of them.)

@u1686_grawity, You write, "The boot sequence only needs to wait for that specific device to appear". I respond, 1) a failed mount may be reported only after all devices have been scanned, and 2) by now a common use case is mounting a multi-device volume identified by not by device addresses but by identifier written on the device, that is, a UUID. In fact, for the latter case, only one device of the group may be given by its address.

@TomYan, Device registration in the kernel may occur through various layers, but relevant to the current discussion is a device being present in devpts and available for mounting. The boot process is sequenced to prevent race conditions for mounting local volumes.

@brainchild there's no such thing as all devices, since you can plug one in / it can respond at any time the current discussion is a device being present in devpts and available for mounting which is not the case for btrfs multi-device volume; see the comment of grawity on my first answer The boot process is sequenced to prevent race conditions for mounting local volumes or see my second answer

@TomYan, If the kernel is simply waiting for device availability in order to mount, and if arrays are handled through a workaround handled by the file system, then why is mounting failing in my case for an incomplete array, based on the device becoming available only in the future with respect to the attempted mount?

@brainchild I have no idea what you mean by kernel waits for device availability, no one said anything like that, perhaps you misunderstood some of my/our statements. workaround handled by the file system it's not even quite a workaround; the btrfs driver is what's responsible for this "array" your are talking about (don't forget you actually created the multi-device volume with btrfs, unless you are talking about some other array). Therefore, there has to be a mechanism to wait until the btrfs driver has registered the necessary devices (as per a specific volume) to itself.

@brainchild and the mechanism is the BTRFS_IOC_DEVICES_READY ioctl (which is implemented in the btrfs driver), which is made use of by a udev builtin (which in turn is used for the "systemd-ready guard" udev rule to hold mount attempt) and btrfs device ready. And the rest of the story has been explained in my second answer.

@TomYan, Let me try again. Does the mount attempt block until all devices are available for the target volume? If so, then why is it failing in my case, during certain boot sequences, according to the race condition, because one of more of the devices is not yet available? That the mount fails suggests that the sequence is designed such that all devices should be available when mounting is attempted.

@brainchild Does the mount attempt block until all devices are available neither does the kernel or the actual mount program does anything like that. But since we are (probably?) talking about mounting handled by systemd, systemd does wait until a device unit is up before it starts its corresponding mount unit. Usually the device unit will be declared up once the device node is there, but it would be a wrong thing to do for btrfs since the node is created even when just one of the devices of a multi-dev volume is there, hence the udev stuff we mentioned. (And see my second answer, again)

By device node I really mean the fs uuid symlink, which is linked to the first device of the multi-dev volume enumerated. And my second answer tells you why systemd didn't "work as expected" for you