Open Stack

On Mon, 13 Aug 2012, Vishvananda Ishaya wrote:

> Hey Everyone,
>
> Resulting Issues
> ----------------
>
> a) The device name only makes sense for linux. FreeBSD will select
> different device names, and windows doesn't even use device names. In
> addition xen uses /dev/xvda and kvm uses /dev/vda
>
> b) The device sent in kvm will not match where it actually shows up. We
> can consistently guess where it will show up if the guest kernel is >=
> 3.2, otherwise we are likely to be wrong, and it may change on a reboot
> anyway
>
> Long term solutions
> ------------------
>
> We probably shouldn't expose a device path, it should be a device number. This is probably the right change long term, but short term we need to make the device name make sense somehow. I want to delay the long term until after the summit, and come up with something that works short-term with our existing parameters and usage.
>
> The first proposal I have is to make the device parameter optional. The system will automatically generate a valid device name that will be accurate for xen and kvm with guest kernel 3.2, but will likely be wrong for old kvm guests in some situations. I think this is definitely an improvement and only a very minor change to an extension api (making a parameter optional, and returning the generated value of the parameter).
>
> (review at https://review.openstack.org/#/c/10908/)
>
> The second proposal I have is to use a feature of kvm attach and set the
> device serial number. We can set it to the same value as the device
> parameter. This means that a device attached to /dev/vdb may not always
> be at /dev/vdb (with old kvm guests), but it will at least show up at
> /dev/disk/by-id/virtio-vdb consistently.

This is the right way to do this.
Expose 'serial-number' (or some other name for it) in the API, attach the
device with that serial number and get out of the way.

If the user doesn't provide you one, then create a unique one (at least
for that guest) and return it.  For many use cases, a user attaches a
disk, ssh's in, finds the new disk, and uses it.  Don't burden them with
coming up with a naming/uuid scheme for this parameter if they dont want
to.

Does xen have anything like this?  Can you set the serial number of the
xen block device?

> (review coming soon)
>
> First question: should we return this magic path somewhere via the api?
> It would be pretty easy to have horizon generate it but it might be nice
> to have it show up. If we do return it, do we mangle the device to
> always show the consistent one, or do we return it as another parameter?
> guest_device perhaps?

>From the api perspective, I think it makes most sense to call it what it
is.  Don't make any promises or allusions to what the guest OS will do
with it.

> Second question: what should happen if someone specifies /dev/xvda
> against a kvm cloud or /dev/vda against a xen cloud?
> I see two options:
> a) automatically convert it to the right value and return it
> b) fail with an error message

In EC2, this fails with an error message.  I think this is more correct.
The one issue here is that you really cannot, and should not attempt to
guess or know what the guest has named devices.  Thats why we're we have
this problem in the first place.

So, I dont have strong feelings either way on this.  Its broken to pass
'device=' and assume that means something.

> Third question: what do we do if someone specifies a device value to a
> kvm cloud that we know will not work. For example the vm has /dev/vda
> and /dev/vdb and they request an attach at /dev/vdf. In this case we
> know that it will likely show up at /dev/vdc. I see a few options here
> and none of them are amazing:
>
> a) let the attach go through as is.
>   advantages: it will allow scripts to work without having to manually find the next device.
>   disadvantages: the device name will never be correct in the guest
> b) automatically modify the request to attach at /dev/vdc and return it
>   advantages: the device name will be correct some of the time (kvm guests with newer kernels)
>   disadvantages: sometimes the name is wrong anyway. The user may not expect the device number to change
> c) fail and say, the next disk must be attached at /dev/vdc:
>   advantages: explicit
>   disadvantages: painful, incompatible, and the place we say to attach may be incorrect anyway (kvm guests with old kernels)

I vote 'a'.
Just be stupid. Play it simple, don't believe that you can understand what
device naming convention the guest kernel and udev have decided upon.

Heres an example.  Do you know what happens if I attach 26 devices?
/dev/vd[a-z], then what?  I'm pretty sure it goes to /dev/vd[a-z][a-z],
but its not worth you trying to know that.  That convention may not be
followed for xen block devices.  At one point (maybe only with scsi
attached disks, letters were never re-used, so an attach, detach, attach
would end up going /dev/vdb, dev/vdc, /dev/vdd).

There is no binary api that linux and udev promise on this, so anything
that is there now is subject to change at any future point.  And as you
admitted, the past was different also.

Just dont bother.  Let them specify whatever they want, expose in
documentation the "right way" to do this.

> The second proposal earlier will at least give us a consistent name to
> find the volume in all these cases, although b) means we have to check
> the return value to find out what that consistent location is like we do
> when we don't pass in a device.
>
> I hope everything is clear, but if more explanation is needed please let
> me know. If anyone has alternative/better proposals please tell me. The
> last question I think is the most important.

I'm really looking forward to the long term solution.

Thanks for raising this.