[Openstack] Icehouse live migration fails for volume based instances
ostck at gbif.org
ostck at gbif.org
Fri May 23 16:36:54 UTC 2014
Hi again,
I've dug deeper into this... those errors appear because the tgtd
configuration on /etc/cinder/volumes/ on the block server hosting the
instance's volume is erased during the live migration; "tgtadm --lld
iscsi --mode target --op show" does not show the volume any more - on
ANY of the block servers.
Basically the volume is crippled (but shown as ok with cinder list).
This seems like a bug, but I'll let smarter people decide ;-)
Let me explain...
Let's say we have an instance running on compute1 with all storage on a
volume which resides on block1. From the CLI we run nova live-migration
towards compute2. What happens, is the instance config gets moved to
compute2, the iSCSI target is discovered from compute2 (which is why
/dev/disk/by/path shows the volume as existing) BUT after this, the
iSCSI target definition on block1:/etc/cinder/volumes/ is deleted (and
tgtd reloaded) - as such, compute2 cannot work properly and fdisk -l on
compute2 does not show the disk device corresponding to the iSCSI volume.
I've circumvented it manually by making a backup of the volume
definition file from /etc/cinder/volumes/ and restoring it immediately
after the deletion then reloading tgtd, and the migration succeeded.
Not a real solution, but until this gets fixed....
Best regards,
Andrei
On 05/22/2014 12:10 PM, ostck at gbif.org wrote:
> Hello,
>
> I'm trying to live-migrate a volume-based instance and it fails. I
> have successfully done this in Havana, but Icehouse either works
> differently or I'm missing something.
>
> I'm using Centos6.5 with KVM, local firewalls and selinux are disabled
> on all nodes.
>
> I have a controller, 3 compute nodes (compute1 to compute3) and 3
> block storage nodes (block1 to block3).
>
> I'm creating an instance using "boot-from-image-creates-new-volume";
> the instance gets created on compute2 using a volume created on block2
> and works fine, although I can see
>
> 2014-05-21 16:31:48.487 1462 WARNING nova.virt.libvirt.driver
> [req-e53e2353-2073-4a09-8d3b-94298a982e2d
> 38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478]
> [instance: a20aa279-4701-47c7-9be1-ae5846a00d91] File injection into a
> boot from volume instance is not supported
>
> I have made the modifications to libvirtd.conf, nova.conf and
> /etc/sysconfig/libvirtd suggested here:
> http://kimizhang.wordpress.com/2013/08/26/openstack-vm-live-migration/
>
> From the CLI, I do
>
> nova live-migration a20aa279-4701-47c7-9be1-ae5846a00d91
> compute3.gbif.clu
>
> With havana this used to work; with Icehouse, the instance ends up on
> compute3, with status 'Active' but power state is 'Paused'. No errors
> in the compute or libvirtd logs of source compute node, but on
> destination (compute3) comput.log says:
> 2014-05-21 17:00:37.135 1467 WARNING nova.compute.manager [-]
> [instance: a20aa279-4701-47c7-9be1-ae5846a00d91] Instance is paused
> unexpectedly. Ignore.
>
>
> Instance stopped with 'nova stop a20aa279-4701-47c7-9be1-ae5846a00d91'.
>
> Trying to start with 'instance start' results in the following erros
> in libvirtd.log on compute3:
>
> 2014-05-22 09:53:19.396+0000: 1443: error : virFileOpenAs:1093 :
> failed to open file
> '/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
> No such device or address
> 2014-05-22 09:53:19.396+0000: 1443: error :
> virStorageFileGetMetadataRecurse:939 : cannot open file
> '/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
> No such device or address
>
> with corresponding errors in compute3's compute.log (see below).
>
> The thing is, on compute3, we have:
> [root at compute3 nova]# ls -alh
> /dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1
> lrwxrwxrwx 1 root root 9 May 22 11:49
> /dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1
> -> ../../sdb
>
> so the symlink exists....
>
> 'Regular' migration works on the installation with this type of volume
> based instances.
>
> I was wondering if anyone could shed any light on this.
>
> Thanks!
>
> Best regards,
> Andrei
>
> Compute.log full errors:
>
> 2014-05-22 11:53:17.668 1467 ERROR glanceclient.common.http [-]
> Request returned failure status.
> 2014-05-22 11:53:17.668 1467 WARNING nova.compute.utils
> [req-5c09ada2-2072-4858-91a7-063b3d6224ed
> 38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478]
> [instance: a20aa279-4701-47c7-9be1-ae5846a00d91] Can't access image :
> Image could not be found.
> 2014-05-22 11:53:19.398 1467 ERROR nova.virt.libvirt.driver
> [req-5c09ada2-2072-4858-91a7-063b3d6224ed
> 38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478] An
> error occurred while trying to launch a defined domain with xml:
> <domain type='kvm'>
> <name>instance-00000015</name>
> <uuid>a20aa279-4701-47c7-9be1-ae5846a00d91</uuid>
> <memory unit='KiB'>2097152</memory>
> <currentMemory unit='KiB'>2097152</currentMemory>
> <vcpu placement='static'>1</vcpu>
> <sysinfo type='smbios'>
> <system>
> <entry name='manufacturer'>Red Hat Inc.</entry>
> <entry name='product'>OpenStack Nova</entry>
> <entry name='version'>2014.1-2.el6</entry>
> <entry name='serial'>96ccf268-2a3e-11dc-a7d0-000ea68f7399</entry>
> <entry name='uuid'>a20aa279-4701-47c7-9be1-ae5846a00d91</entry>
> </system>
> </sysinfo>
> <os>
> <type arch='x86_64' machine='rhel6.5.0'>hvm</type>
> <boot dev='hd'/>
> <smbios mode='sysinfo'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> </features>
> <cpu mode='host-model'>
> <model fallback='allow'/>
> </cpu>
> <clock offset='utc'>
> <timer name='pit' tickpolicy='delay'/>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='hpet' present='no'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <devices>
> <emulator>/usr/libexec/qemu-kvm</emulator>
> <disk type='block' device='disk'>
> <driver name='qemu' type='raw' cache='none'/>
> <source
> dev='/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1'/>
> <target dev='vda' bus='virtio'/>
> <serial>b897a227-e7ca-4179-a17d-3e00700557a4</serial>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
> </disk>
> <controller type='usb' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x2'/>
> </controller>
> <interface type='bridge'>
> <mac address='fa:16:3e:78:0f:fa'/>
> <source bridge='br100'/>
> <model type='virtio'/>
> <filterref filter='nova-instance-instance-00000015-fa163e780ffa'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </interface>
> <serial type='file'>
> <source
> path='/var/lib/nova/instances/a20aa279-4701-47c7-9be1-ae5846a00d91/console.log'/>
> <target port='0'/>
> </serial>
> <serial type='pty'>
> <target port='1'/>
> </serial>
> <console type='file'>
> <source
> path='/var/lib/nova/instances/a20aa279-4701-47c7-9be1-ae5846a00d91/console.log'/>
> <target type='serial' port='0'/>
> </console>
> <input type='tablet' bus='usb'/>
> <input type='mouse' bus='ps2'/>
> <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'
> keymap='en-us'>
> <listen type='address' address='0.0.0.0'/>
> </graphics>
> <video>
> <model type='cirrus' vram='9216' heads='1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
> </video>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
> </memballoon>
> </devices>
> </domain>
>
> 2014-05-22 11:53:19.896 1467 ERROR oslo.messaging.rpc.dispatcher [-]
> Exception during message handling: cannot open file
> '/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
> No such device or address
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> Traceback (most recent call last):
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
> line 133, in _dispatch_and_reply
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> incoming.message))
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
> line 176, in _dispatch
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> return self._do_dispatch(endpoint, method, ctxt, args)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
> line 122, in _do_dispatch
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> result = getattr(endpoint, method)(ctxt, **new_args)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher payload)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> six.reraise(self.type_, self.value, self.tb)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> return f(self, context, *args, **kw)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 280,
> in decorated_function
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher pass
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> six.reraise(self.type_, self.value, self.tb)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 266,
> in decorated_function
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> return function(self, context, *args, **kwargs)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 333,
> in decorated_function
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> function(self, context, *args, **kwargs)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 309,
> in decorated_function
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher e,
> sys.exc_info())
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> six.reraise(self.type_, self.value, self.tb)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296,
> in decorated_function
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> return function(self, context, *args, **kwargs)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2300,
> in start_instance
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> self._power_on(context, instance)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2288,
> in _power_on
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> block_device_info)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 2128, in power_on
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> self._hard_reboot(context, instance, network_info, block_device_info)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 2092, in _hard_reboot
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> vifs_already_plugged=True)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 3644, in _create_domain_and_network
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> power_on=power_on)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 3547, in _create_domain
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> domain.XMLDesc(0))
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> six.reraise(self.type_, self.value, self.tb)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 3542, in _create_domain
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> domain.createWithFlags(launch_flags)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in doit
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> result = proxy_call(self._autowrap, f, *args, **kwargs)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in
> proxy_call
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher rv =
> execute(f,*args,**kwargs)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher rv =
> meth(*args,**kwargs)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
> "/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in
> createWithFlags
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher if
> ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
> dom=self)
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> libvirtError: cannot open file
> '/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
> No such device or address
> 2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
> 2014-05-22 11:53:19.898 1467 ERROR oslo.messaging._drivers.common [-]
> Returning exception cannot open file
> '/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
> No such device or address to caller
> 2014-05-22 11:53:19.898 1467 ERROR oslo.messaging._drivers.common [-]
> ['Traceback (most recent call last):\n', ' File
> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
> line 133, in _dispatch_and_reply\n incoming.message))\n', ' File
> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
> line 176, in _dispatch\n return self._do_dispatch(endpoint, method,
> ctxt, args)\n', ' File
> "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
> line 122, in _do_dispatch\n result = getattr(endpoint,
> method)(ctxt, **new_args)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in
> wrapped\n payload)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__\n six.reraise(self.type_, self.value,
> self.tb)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in
> wrapped\n return f(self, context, *args, **kw)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 280,
> in decorated_function\n pass\n', ' File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__\n six.reraise(self.type_, self.value,
> self.tb)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 266,
> in decorated_function\n return function(self, context, *args,
> **kwargs)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 333,
> in decorated_function\n function(self, context, *args,
> **kwargs)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 309,
> in decorated_function\n e, sys.exc_info())\n', ' File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__\n six.reraise(self.type_, self.value,
> self.tb)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296,
> in decorated_function\n return function(self, context, *args,
> **kwargs)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2300,
> in start_instance\n self._power_on(context, instance)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2288,
> in _power_on\n block_device_info)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 2128, in power_on\n self._hard_reboot(context, instance,
> network_info, block_device_info)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 2092, in _hard_reboot\n vifs_already_plugged=True)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 3644, in _create_domain_and_network\n power_on=power_on)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 3547, in _create_domain\n domain.XMLDesc(0))\n', ' File
> "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
> line 68, in __exit__\n six.reraise(self.type_, self.value,
> self.tb)\n', ' File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
> 3542, in _create_domain\n domain.createWithFlags(launch_flags)\n', '
> File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179,
> in doit\n result = proxy_call(self._autowrap, f, *args,
> **kwargs)\n', ' File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in
> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
> "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in
> tworker\n rv = meth(*args,**kwargs)\n', ' File
> "/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in
> createWithFlags\n if ret == -1: raise libvirtError
> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
> cannot open file
> '/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
> No such device or address\n"]
>
>
>
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
More information about the Openstack
mailing list