<div dir="ltr">Please guys, Any ideas. I have not been able to fix the issue till now.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 6, 2015 at 1:33 PM, Vijaya Bhaskar <span dir="ltr"><<a href="mailto:vijayabhaskar@sparksupport.com" target="_blank">vijayabhaskar@sparksupport.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi all,<br><br></div>I have an openstack setup with ceph as the storage backend and linuxbridge plugin for networking. When I try to take a snapshot of an instance that is running, the image is getting created, but it gets deleted as soon as the creation starts. When I checked the nova logs I found these:<br><br></div>logs<br>-------<br><br>2015-02-06 13:10:35.863 8030 AUDIT nova.compute.manager [req-b7effc42-8f02-433e-9500-f068b2a827b4 None] [instance: 172cd3ae-c4a2-4559-abcb-31e81132acba] instance snapshotting<br>2015-02-06 13:10:36.196 8030 INFO nova.compute.manager [-] [instance: 172cd3ae-c4a2-4559-abcb-31e81132acba] VM Paused (Lifecycle Event)<br>2015-02-06 13:10:36.382 8030 INFO nova.compute.manager [-] [instance: 172cd3ae-c4a2-4559-abcb-31e81132acba] During sync_power_state the instance has a pending task (image_snapshot). Skip.<br>2015-02-06 13:10:37.813 8030 INFO nova.compute.manager [-] [instance: 172cd3ae-c4a2-4559-abcb-31e81132acba] VM Stopped (Lifecycle Event)<br>2015-02-06 13:10:37.818 8030 INFO nova.virt.libvirt.driver [req-b7effc42-8f02-433e-9500-f068b2a827b4 None] [instance: 172cd3ae-c4a2-4559-abcb-31e81132acba] Beginning cold snapshot process<br>2015-02-06 13:10:37.822 8030 INFO oslo.messaging._drivers.impl_rabbit [req-b7effc42-8f02-433e-9500-f068b2a827b4 ] Connecting to AMQP server on controller1:5672<br>2015-02-06 13:10:37.840 8030 INFO oslo.messaging._drivers.impl_rabbit [req-b7effc42-8f02-433e-9500-f068b2a827b4 ] Connected to AMQP server on controller1:5672<br>2015-02-06 13:10:38.042 8030 INFO nova.compute.manager [-] [instance: 172cd3ae-c4a2-4559-abcb-31e81132acba] During sync_power_state the instance has a pending task (image_snapshot). Skip.<br>2015-02-06 13:10:40.647 8030 INFO nova.compute.manager [-] [instance: 172cd3ae-c4a2-4559-abcb-31e81132acba] During sync_power_state the instance has a pending task (image_pending_upload). Skip.<br>2015-02-06 13:10:46.309 8030 ERROR nova.virt.libvirt.driver [req-b7effc42-8f02-433e-9500-f068b2a827b4 None] Error launching a defined domain with XML: <domain type='kvm'><br> <name>instance-00000097</name><br> <uuid>172cd3ae-c4a2-4559-abcb-31e81132acba</uuid><br> <metadata><br> <nova:instance xmlns:nova="<a href="http://openstack.org/xmlns/libvirt/nova/1.0" target="_blank">http://openstack.org/xmlns/libvirt/nova/1.0</a>"><br> <nova:package version="2014.2.1"/><br> <nova:name>CIRROS-TEST</nova:name><br> <nova:creationTime>2015-02-06 07:40:12</nova:creationTime><br> <nova:flavor name="micro"><br> <nova:memory>256</nova:memory><br> <nova:disk>1</nova:disk><br> <nova:swap>0</nova:swap><br> <nova:ephemeral>0</nova:ephemeral><br> <nova:vcpus>1</nova:vcpus><br> </nova:flavor><br> <nova:owner/><br> <nova:root type="image" uuid="95b4a0ff-9007-440b-bcdf-0caeab529de4"/><br> </nova:instance><br> </metadata><br> <memory unit='KiB'>262144</memory><br> <currentMemory unit='KiB'>262144</currentMemory><br> <vcpu placement='static' cpuset='0'>1</vcpu><br> <sysinfo type='smbios'><br> <system><br> <entry name='manufacturer'>OpenStack Foundation</entry><br> <entry name='product'>OpenStack Nova</entry><br> <entry name='version'>2014.2.1</entry><br> <entry name='serial'>a7a9fb94-4349-9d21-ef9f-40fa6edf982f</entry><br> <entry name='uuid'>172cd3ae-c4a2-4559-abcb-31e81132acba</entry><br> </system><br> </sysinfo><br> <os><br> <type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type><br> <boot dev='hd'/><br> <smbios mode='sysinfo'/><br> </os><br> <features><br> <acpi/><br> <apic/><br> </features><br> <cpu><br> <topology sockets='1' cores='1' threads='1'/><br> </cpu><br> <clock offset='utc'><br> <timer name='pit' tickpolicy='delay'/><br> <timer name='rtc' tickpolicy='catchup'/><br> <timer name='hpet' present='no'/><br> </clock><br> <on_poweroff>destroy</on_poweroff><br> <on_reboot>restart</on_reboot><br> <on_crash>destroy</on_crash><br> <devices><br> <emulator>/usr/bin/qemu-system-x86_64</emulator><br> <disk type='network' device='disk'><br> <driver name='qemu' type='raw' cache='none'/><br> <auth username='cinder'><br> <secret type='ceph' uuid='a3c760c0-536b-4be7-97b8-d59b0948c050'/><br> </auth><br> <source protocol='rbd' name='vms/172cd3ae-c4a2-4559-abcb-31e81132acba_disk'><br> <host name='10.0.11.11' port='6789'/><br> <host name='10.0.11.12' port='6789'/><br> <host name='10.0.11.13' port='6789'/><br> </source><br> <target dev='vda' bus='virtio'/><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/><br> </disk><br> <controller type='usb' index='0'><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/><br> </controller><br> <controller type='pci' index='0' model='pci-root'/><br> <interface type='bridge'><br> <mac address='fa:16:3e:48:11:dd'/><br> <source bridge='brq6913fc1c-69'/><br> <target dev='tape3360180-4d'/><br> <model type='rtl8139'/><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/><br> </interface><br> <serial type='file'><br> <source path='/var/lib/nova/instances/172cd3ae-c4a2-4559-abcb-31e81132acba/console.log'/><br> <target port='0'/><br> </serial><br> <serial type='pty'><br> <target port='1'/><br> </serial><br> <console type='file'><br> <source path='/var/lib/nova/instances/172cd3ae-c4a2-4559-abcb-31e81132acba/console.log'/><br> <target type='serial' port='0'/><br> </console><br> <input type='tablet' bus='usb'/><br> <input type='mouse' bus='ps2'/><br> <input type='keyboard' bus='ps2'/><br> <graphics type='vnc' port='-1' autoport='yes' listen='10.0.11.17' keymap='en-us'><br> <listen type='address' address='10.0.11.17'/><br> </graphics><br> <video><br> <model type='cirrus' vram='9216' heads='1'/><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/><br> </video><br> <memballoon model='virtio'><br> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/><br> <stats period='10'/><br> </memballoon><br> </devices><br></domain><br><br>2015-02-06 13:10:46.935 8030 INFO nova.scheduler.client.report [req-b7effc42-8f02-433e-9500-f068b2a827b4 None] Compute_service record updated for ('compute', 'compute')<br>2015-02-06 13:10:46.939 8030 ERROR oslo.messaging.rpc.dispatcher [req-b7effc42-8f02-433e-9500-f068b2a827b4 ] Exception during message handling: Cannot get interface MTU on 'brq6913fc1c-69': No such device<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher incoming.message))<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher payload)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher pass<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 289, in decorated_function<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 331, in decorated_function<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 379, in decorated_function<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher instance=instance)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 369, in decorated_function<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher *args, **kwargs)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3027, in snapshot_instance<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher task_states.IMAGE_SNAPSHOT)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3058, in _snapshot_instance<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher update_task_state)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1733, in snapshot<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher new_dom = self._create_domain(domain=virt_dom)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4338, in _create_domain<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher LOG.error(err)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4329, in _create_domain<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher domain.createWithFlags(launch_flags)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher result = proxy_call(self._autowrap, f, *args, **kwargs)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher rv = execute(f,*args,**kwargs)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher rv = meth(*args,**kwargs)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/libvirt.py", line 896, in createWithFlags<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)<br>2015-02-06 13:10:46.939 8030 TRACE oslo.messaging.rpc.dispatcher libvirtError: Cannot get interface MTU on 'brq6913fc1c-69': No such device<br><br></div>The issue is that nova is trying to freeze the instance before snapshot(I think), but unable to resume the instance due to missing bridge , and the snapshot image is deleted automatically. However, the instances can suspend, soft reboot and hard reboot successfully.<br><br></div>Please help.<br></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Systems Engineer<br>SparkSupport<br></div>
</div>