Open Stack

Hello,

I just resolved an issue where migrating instances with iSCSI volumes would
occasionally fail. There's a bug report here:

https://bugs.launchpad.net/nova/+bug/1423772

The core cause ended up being libvirt transferring the volume paths
verbatim. For example, take the situation where:

compute node 1: has luns 0-5 in use for volumes
compute node 2: has luns 0-2 in use for volumes
instance 1: hosted on compute node 1 with a volume using lun 5

If this instance was moved from node 1 to 2, it would fail with:

Live Migration failure: Failed to open file
'/dev/disk/by-path/ip-192.168.1.1:blah-blah-lun-5': No such file or
directory

Meanwhile, compute node 2 tried creating an iSCSI connection at lun 3,
which is the next lun in line for use.

Two other situations would happen:

* If lun 5 was already in use on the destination node, the migration would
think it's available and try connecting to it.
* If lun 5 was the natural next available lun, the migration would work and
everything would be fine.

I found this blueprint which resolves the problem:

https://blueprints.launchpad.net/nova/+spec/iscsi-live-migration-different-target
https://review.openstack.org/#/c/137466/

The patch will rewrite the libvirt xml file to use the correct luns on the
destination server rather than the luns from the source server.

Since I'm using Icehouse, I backported the changes -- fortunately it was
pretty easy.

This environment jumped from Grizzly to Icehouse. I'm not sure if it was
just dumb luck that I never saw this on Grizzly or something changed that
introduced it between Grizzly and Icehouse.

Anyway, I wanted to post a message because iSCSI-backed volumes are fairly
common.

Thanks,
Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20150319/a5fbeb9a/attachment.html>