Open Stack

Hello,

I'm trying to live-migrate a volume-based instance and it fails. I have 
successfully done this in Havana, but Icehouse either works differently 
or I'm missing something.

I'm using Centos6.5 with KVM, local firewalls and selinux are disabled 
on all nodes.

I have a controller, 3 compute nodes (compute1 to compute3) and 3 block 
storage nodes (block1 to block3).

I'm creating an instance using "boot-from-image-creates-new-volume"; the 
instance gets created on compute2 using a volume created on block2 and 
works fine, although I can see

2014-05-21 16:31:48.487 1462 WARNING nova.virt.libvirt.driver 
[req-e53e2353-2073-4a09-8d3b-94298a982e2d 
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478] 
[instance: a20aa279-4701-47c7-9be1-ae5846a00d91] File injection into a 
boot from volume instance is not supported

I have made the modifications to libvirtd.conf, nova.conf and 
/etc/sysconfig/libvirtd suggested here: 
http://kimizhang.wordpress.com/2013/08/26/openstack-vm-live-migration/

 From the CLI, I do

nova live-migration a20aa279-4701-47c7-9be1-ae5846a00d91 compute3.gbif.clu

With havana this used to work; with Icehouse, the instance ends up on 
compute3, with status 'Active' but power state is 'Paused'. No errors in 
the compute or libvirtd logs of source compute node, but on destination 
(compute3) comput.log says:
2014-05-21 17:00:37.135 1467 WARNING nova.compute.manager [-] [instance: 
a20aa279-4701-47c7-9be1-ae5846a00d91] Instance is paused unexpectedly. 
Ignore.


Instance stopped with  'nova stop a20aa279-4701-47c7-9be1-ae5846a00d91'.

Trying to start with 'instance start' results in the following erros in 
libvirtd.log on compute3:

2014-05-22 09:53:19.396+0000: 1443: error : virFileOpenAs:1093 : failed 
to open file 
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1': 
No such device or address
2014-05-22 09:53:19.396+0000: 1443: error : 
virStorageFileGetMetadataRecurse:939 : cannot open file 
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1': 
No such device or address

with corresponding errors in compute3's compute.log (see below).

The thing is, on compute3, we have:
[root at compute3 nova]# ls -alh 
/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1
lrwxrwxrwx 1 root root 9 May 22 11:49 
/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1 
-> ../../sdb

so the symlink exists....

'Regular' migration works on the installation with this type of volume 
based instances.

I was wondering if anyone could shed any light on this.

Thanks!

Best regards,
Andrei

Compute.log full errors:

2014-05-22 11:53:17.668 1467 ERROR glanceclient.common.http [-] Request 
returned failure status.
2014-05-22 11:53:17.668 1467 WARNING nova.compute.utils 
[req-5c09ada2-2072-4858-91a7-063b3d6224ed 
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478] 
[instance: a20aa279-4701-47c7-9be1-ae5846a00d91] Can't access image : 
Image  could not be found.
2014-05-22 11:53:19.398 1467 ERROR nova.virt.libvirt.driver 
[req-5c09ada2-2072-4858-91a7-063b3d6224ed 
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478] An 
error occurred while trying to launch a defined domain with xml: <domain 
type='kvm'>
   <name>instance-00000015</name>
   <uuid>a20aa279-4701-47c7-9be1-ae5846a00d91</uuid>
   <memory unit='KiB'>2097152</memory>
   <currentMemory unit='KiB'>2097152</currentMemory>
   <vcpu placement='static'>1</vcpu>
   <sysinfo type='smbios'>
     <system>
       <entry name='manufacturer'>Red Hat Inc.</entry>
       <entry name='product'>OpenStack Nova</entry>
       <entry name='version'>2014.1-2.el6</entry>
       <entry name='serial'>96ccf268-2a3e-11dc-a7d0-000ea68f7399</entry>
       <entry name='uuid'>a20aa279-4701-47c7-9be1-ae5846a00d91</entry>
     </system>
   </sysinfo>
   <os>
     <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
     <boot dev='hd'/>
     <smbios mode='sysinfo'/>
   </os>
   <features>
     <acpi/>
     <apic/>
   </features>
   <cpu mode='host-model'>
     <model fallback='allow'/>
   </cpu>
   <clock offset='utc'>
     <timer name='pit' tickpolicy='delay'/>
     <timer name='rtc' tickpolicy='catchup'/>
     <timer name='hpet' present='no'/>
   </clock>
   <on_poweroff>destroy</on_poweroff>
   <on_reboot>restart</on_reboot>
   <on_crash>destroy</on_crash>
   <devices>
     <emulator>/usr/libexec/qemu-kvm</emulator>
     <disk type='block' device='disk'>
       <driver name='qemu' type='raw' cache='none'/>
       <source 
dev='/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1'/>
       <target dev='vda' bus='virtio'/>
<serial>b897a227-e7ca-4179-a17d-3e00700557a4</serial>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x04' 
function='0x0'/>
     </disk>
     <controller type='usb' index='0'>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x01' 
function='0x2'/>
     </controller>
     <interface type='bridge'>
       <mac address='fa:16:3e:78:0f:fa'/>
       <source bridge='br100'/>
       <model type='virtio'/>
       <filterref filter='nova-instance-instance-00000015-fa163e780ffa'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x03' 
function='0x0'/>
     </interface>
     <serial type='file'>
       <source 
path='/var/lib/nova/instances/a20aa279-4701-47c7-9be1-ae5846a00d91/console.log'/>
       <target port='0'/>
     </serial>
     <serial type='pty'>
       <target port='1'/>
     </serial>
     <console type='file'>
       <source 
path='/var/lib/nova/instances/a20aa279-4701-47c7-9be1-ae5846a00d91/console.log'/>
       <target type='serial' port='0'/>
     </console>
     <input type='tablet' bus='usb'/>
     <input type='mouse' bus='ps2'/>
     <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' 
keymap='en-us'>
       <listen type='address' address='0.0.0.0'/>
     </graphics>
     <video>
       <model type='cirrus' vram='9216' heads='1'/>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x02' 
function='0x0'/>
     </video>
     <memballoon model='virtio'>
       <address type='pci' domain='0x0000' bus='0x00' slot='0x05' 
function='0x0'/>
     </memballoon>
   </devices>
</domain>

2014-05-22 11:53:19.896 1467 ERROR oslo.messaging.rpc.dispatcher [-] 
Exception during message handling: cannot open file 
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1': 
No such device or address
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
Traceback (most recent call last):
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", 
line 133, in _dispatch_and_reply
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
incoming.message))
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", 
line 176, in _dispatch
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher return 
self._do_dispatch(endpoint, method, ctxt, args)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", 
line 122, in _do_dispatch
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher result 
= getattr(endpoint, method)(ctxt, **new_args)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher payload)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", 
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher return 
f(self, context, *args, **kw)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 280, in 
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher pass
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", 
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 266, in 
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher return 
function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 333, in 
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 309, in 
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher e, 
sys.exc_info())
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", 
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296, in 
decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher return 
function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2300, 
in start_instance
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
self._power_on(context, instance)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2288, 
in _power_on
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
block_device_info)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
2128, in power_on
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
self._hard_reboot(context, instance, network_info, block_device_info)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
2092, in _hard_reboot
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
vifs_already_plugged=True)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
3644, in _create_domain_and_network
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
power_on=power_on)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
3547, in _create_domain
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
domain.XMLDesc(0))
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", 
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
3542, in _create_domain
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
domain.createWithFlags(launch_flags)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in doit
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher result 
= proxy_call(self._autowrap, f, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in 
proxy_call
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher rv = 
execute(f,*args,**kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher rv = 
meth(*args,**kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File 
"/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in 
createWithFlags
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher if ret 
== -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher 
libvirtError: cannot open file 
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1': 
No such device or address
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
2014-05-22 11:53:19.898 1467 ERROR oslo.messaging._drivers.common [-] 
Returning exception cannot open file 
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1': 
No such device or address to caller
2014-05-22 11:53:19.898 1467 ERROR oslo.messaging._drivers.common [-] 
['Traceback (most recent call last):\n', '  File 
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", 
line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File 
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", 
line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, 
ctxt, args)\n', '  File 
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", 
line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, 
**new_args)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in 
wrapped\n    payload)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", 
line 68, in __exit__\n    six.reraise(self.type_, self.value, 
self.tb)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in 
wrapped\n    return f(self, context, *args, **kw)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 280, in 
decorated_function\n    pass\n', '  File 
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", 
line 68, in __exit__\n    six.reraise(self.type_, self.value, 
self.tb)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 266, in 
decorated_function\n    return function(self, context, *args, 
**kwargs)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 333, in 
decorated_function\n    function(self, context, *args, **kwargs)\n', '  
File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 
309, in decorated_function\n    e, sys.exc_info())\n', '  File 
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", 
line 68, in __exit__\n    six.reraise(self.type_, self.value, 
self.tb)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296, in 
decorated_function\n    return function(self, context, *args, 
**kwargs)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2300, 
in start_instance\n    self._power_on(context, instance)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2288, 
in _power_on\n    block_device_info)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
2128, in power_on\n    self._hard_reboot(context, instance, 
network_info, block_device_info)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
2092, in _hard_reboot\n    vifs_already_plugged=True)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
3644, in _create_domain_and_network\n    power_on=power_on)\n', ' File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
3547, in _create_domain\n    domain.XMLDesc(0))\n', '  File 
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", 
line 68, in __exit__\n    six.reraise(self.type_, self.value, 
self.tb)\n', '  File 
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 
3542, in _create_domain\n domain.createWithFlags(launch_flags)\n', '  
File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in 
doit\n    result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  
File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in 
proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File 
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in 
tworker\n    rv = meth(*args,**kwargs)\n', '  File 
"/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in 
createWithFlags\n    if ret == -1: raise libvirtError 
(\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: 
cannot open file 
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1': 
No such device or address\n"]