[Openstack] Configuring Live Migration

Guilherme Russi luisguilherme.cr at gmail.com
Tue Jul 30 17:25:05 UTC 2013


Do you think this manual
http://docs.openstack.org/trunk/openstack-compute/admin/content/nova-uid-mismatch.html
can
help, I mean, I don't have problem to create the VMs, maybe these changes


 find / -uid 108 -exec chown nova {} \; # note the 108 here is the old
nova uid before the change
         find / -gid 120 -exec chgrp nova {} \;

they can screw up my creation that is working, another question, about the
directly migration with the virsh, if I upgrade the libvirt version, can it
break the stack working? I'm making a lot of mess with this :S


2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>

> Sure, thank you.
>
> cat /var/log/nova/nova-compute.log | grep -E ERROR
> 2013-07-30 13:48:08.074 ERROR nova.compute.manager
> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
> 5061e7a6-2482-4af2-97cf-0074326fb940] Unexpected error while running
> command.
> Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to
> ERROR
> 2013-07-30 13:48:08.747 ERROR nova.openstack.common.rpc.amqp
> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>
> cat /var/log/nova/nova-compute.log | grep -E TRACE
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> Traceback (most recent call last):
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
> 430, in _process_data
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp     rval
> = self.proxy.dispatch(ctxt, version, method, **args)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
> line 133, in dispatch
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> return getattr(proxyobj, method)(ctxt, **kwargs)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> temp_level, payload)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> return f(self, context, *args, **kw)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
> decorated_function
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp     pass
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
> decorated_function
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> return function(self, context, *args, **kwargs)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
> decorated_function
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> function(self, context, *args, **kwargs)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
> decorated_function
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp     e,
> sys.exc_info())
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
> decorated_function
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> return function(self, context, *args, **kwargs)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
> resize_instance
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> block_device_info)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
> in migrate_disk_and_power_off
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> inst_base_resize)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> self.gen.next()
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
> in migrate_disk_and_power_off
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp   File
> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> cmd=' '.join(cmd))
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
> ProcessExecutionError: Unexpected error while running command.
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp Command:
> ssh 10.3.77.48 mkdir -p
> /var/lib/nova/instances/5061e7a6-2482-4af2-97cf-0074326fb940
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp Exit
> code: 255
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp Stdout:
> ''
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp Stderr:
> 'Host key verification failed.\r\n'
> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>
>
>
>
>
> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>
>> Can you provide more information (errors and/or logs)?
>>
>> Filipe Manco
>> http://about.me/fmanco
>>
>>
>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>
>>> It didn't work :(
>>>
>>>
>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>
>>>> Ok, testing it.
>>>>
>>>>
>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>
>>>>> Basically edit the /etc/passwd and /etc/passwd- files and change
>>>>> /bin/false to /bin/bash on the nova user entry.
>>>>>
>>>>> I'm having another problem after this related to libvirt but I have no
>>>>> time to fix it know. Basically the migration ends but then libvirt isn't
>>>>> able to boot the machine again. Let me know about your results.
>>>>>
>>>>> Regards
>>>>>
>>>>> Filipe Manco
>>>>> http://about.me/fmanco
>>>>>
>>>>>
>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>
>>>>>> Ok, on my scenario I have 2 Compute nodes and 1 Controller + Network
>>>>>> node, how did you change your configs, I can try your idea.
>>>>>>
>>>>>>
>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>
>>>>>>> No I just wanted to know what is your scenario. About the mkdir I
>>>>>>> spend a couple of time looking at it and managed to fix it by changing the
>>>>>>> shell of the nova user to /bin/bash instead of /bin/false on all the
>>>>>>> compute nodes. You can try it.
>>>>>>>
>>>>>>> Filipe Manco
>>>>>>> http://about.me/fmanco
>>>>>>>
>>>>>>>
>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>
>>>>>>>> Hello Filipe, I followed the manual on the openstack documentation,
>>>>>>>> I shared the /var/lib/nova/instances on my controller, do I need to share
>>>>>>>> another folders?
>>>>>>>>
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> A couple of messages ago you shown an error where the mkdir
>>>>>>>>> command failed. At that time the keys were properly configured. Can you
>>>>>>>>> revert your changes?
>>>>>>>>>
>>>>>>>>> About the mkdir error, I had a similar one but I wasn't able to
>>>>>>>>> fix it yet. Do you have shared storage for your instances?
>>>>>>>>>
>>>>>>>>> I never used virsh directly because, AFAIK, openstack will not be
>>>>>>>>> informed of the changes and will not work correctly (but I may be wrong in
>>>>>>>>> this one).
>>>>>>>>>
>>>>>>>>> Filipe Manco
>>>>>>>>> http://about.me/fmanco
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>
>>>>>>>>>> I created the new key, but still getting error: here are the logs:
>>>>>>>>>>
>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>> 2013-07-30 11:37:51.097 ERROR nova.compute.manager
>>>>>>>>>> [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2
>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>> bed3c3be-31d6-42c3-9a7f-e40ef9c1c764] Unexpected error while running
>>>>>>>>>> command.
>>>>>>>>>>  Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>>>>>> vm_state to ERROR
>>>>>>>>>> 2013-07-30 11:37:52.463 ERROR nova.openstack.common.rpc.amqp
>>>>>>>>>> [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2
>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File
>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>> 430, in _process_data
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File
>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>> line 133, in dispatch
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>>>>>>> wrapped
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     temp_level, payload)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     self.gen.next()
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>>>>>>> wrapped
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     return f(self, context, *args, **kw)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 209, in decorated_function
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     pass
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     self.gen.next()
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 195, in decorated_function
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     return function(self, context, *args, **kwargs)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 260, in decorated_function
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     function(self, context, *args, **kwargs)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 237, in decorated_function
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     e, sys.exc_info())
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     self.gen.next()
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 224, in decorated_function
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     return function(self, context, *args, **kwargs)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 2354, in resize_instance
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     block_device_info)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>> 3467, in migrate_disk_and_power_off
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     inst_base_resize)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     self.gen.next()
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>> 3444, in migrate_disk_and_power_off
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in
>>>>>>>>>> execute
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>     cmd=' '.join(cmd))
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>> /var/lib/nova/instances/bed3c3be-31d6-42c3-9a7f-e40ef9c1c764
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Exit code: 255
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Stdout: ''
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>
>>>>>>>>>> I've been trying to migrate directly with virsh, but I got this
>>>>>>>>>> error and the solutions through the web don't help.
>>>>>>>>>>
>>>>>>>>>> virsh migrate --live instance-00000063 qemu+ssh://
>>>>>>>>>> 192.168.3.3/system
>>>>>>>>>> root at 192.168.3.3's password:
>>>>>>>>>> erro: internal error cannot load AppArmor profile
>>>>>>>>>> 'libvirt-bed3c3be-31d6-42c3-9a7f-e40ef9c1c764'
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thank you all.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/7/29 <yongiman at gmail.com>
>>>>>>>>>>
>>>>>>>>>> Hi Guilherme,
>>>>>>>>>>>
>>>>>>>>>>> I suugest making new ssh key and put the public key in the host
>>>>>>>>>>>
>>>>>>>>>>> # ssh-keygen (you will find id_rsa.pub in your .ssh directory)
>>>>>>>>>>>
>>>>>>>>>>> Copy the contents of this file and paste at the last line of
>>>>>>>>>>> authorized_keys in the host.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2013. 7. 29., at 오후 10:40, Guilherme Russi <
>>>>>>>>>>> luisguilherme.cr at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hello Filipe, I created the /var/lib/nova/.ssh/authorized_keys
>>>>>>>>>>> and its content is the keys from the know_hosts, is it right? I'm still
>>>>>>>>>>> getting the same error when I try to migrate a VM.
>>>>>>>>>>>
>>>>>>>>>>> Thank you.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> About the keys it's up to you. You can use the same or create a
>>>>>>>>>>>> new one. Using the same is easier.
>>>>>>>>>>>>
>>>>>>>>>>>> But the nova user public key should go into
>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys not into the know_hosts file. The easier
>>>>>>>>>>>> way (as you only have two compute nodes) to add the keys to the know_hosts
>>>>>>>>>>>> is to manually ssh and accept the key.
>>>>>>>>>>>>
>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, Must I create another ssh key to communicate both compute
>>>>>>>>>>>>> nodes or the current key can work? I've put the plublic key from both
>>>>>>>>>>>>> compute node in the know_hosts. Which permission should I apply? 644?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> First you need to check why this "Failed to add the host to
>>>>>>>>>>>>>> the list of known hosts (/var/lib/nova/.ssh/known_hosts)."
>>>>>>>>>>>>>> happens. It maybe file permissions or so. Then you need to generate an ssh
>>>>>>>>>>>>>> key for the user nova and add it to the other compute nodes on the
>>>>>>>>>>>>>> authorized_keys file (the password is not really necessary as soon as you
>>>>>>>>>>>>>> have the keys in place).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> sudo -u nova ssh 192.168.3.3
>>>>>>>>>>>>>>> The authenticity of host '192.168.3.3 (192.168.3.3)' can't
>>>>>>>>>>>>>>> be established.
>>>>>>>>>>>>>>> ECDSA key fingerprint is
>>>>>>>>>>>>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>>>>>>>>>>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>>>>>>>>>>>>> Failed to add the host to the list of known hosts
>>>>>>>>>>>>>>> (/var/lib/nova/.ssh/known_hosts).
>>>>>>>>>>>>>>> nova at 192.168.3.3's password:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But I don't know the user nova's password.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you try *sudo -u nova ssh <remote-compute-node>* form
>>>>>>>>>>>>>>>> the compute node currently running the VM and check if it works?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello Filipe,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>  I made the copy from my .ssh/know_hosts to places you
>>>>>>>>>>>>>>>>> told me, but I'm still getting the same error. Any other ideas?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This error means that you cannot ssh into the new compute
>>>>>>>>>>>>>>>>>> node because it's public key cannot be verified. Add the public keys of the
>>>>>>>>>>>>>>>>>> compute nodes to /etc/ssh/know_hosts file (or the nova one, usually under
>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/) and the problem should be gone.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ok, I'll look more carefully the log and it's cool this
>>>>>>>>>>>>>>>>>>> command from libvirt, I didn't know it, I'll try it and return to you.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you again.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>> That's confused me, "Unexpected error while running
>>>>>>>>>>>>>>>>>>>> command" will give us what command is running when this error occured, you
>>>>>>>>>>>>>>>>>>>> may check the log more carefully.
>>>>>>>>>>>>>>>>>>>> and you also can migrate the VM directly by libvirt
>>>>>>>>>>>>>>>>>>>> command line like below:
>>>>>>>>>>>>>>>>>>>> virsh  migrate --live --persistent --undefinesource
>>>>>>>>>>>>>>>>>>>> --verbose --domain ${instance-uuid} --desturi
>>>>>>>>>>>>>>>>>>>> qemu+tcp://${the-dest-host-ip}/system
>>>>>>>>>>>>>>>>>>>> then see the output result.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>  *发件人:*Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>> *发送时间:*2013-07-26 09:30
>>>>>>>>>>>>>>>>>>>> *主题:*Re: Re: Re: Re: Re: [Openstack] Configuring Live
>>>>>>>>>>>>>>>>>>>> Migration
>>>>>>>>>>>>>>>>>>>> *收件人:*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>> *抄送:*"openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>  When I enter the action to migrate, the only two
>>>>>>>>>>>>>>>>>>>> errors showen at this period are these two.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>> There may have more error logs at this minute
>>>>>>>>>>>>>>>>>>>>> 2013-07-25 17:28,
>>>>>>>>>>>>>>>>>>>>> you can get the backtrace near the error log you post.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>  *发件人:*Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>> *发送时间:*2013-07-26 04:48
>>>>>>>>>>>>>>>>>>>>> *主题:*Re: Re: Re: Re: [Openstack] Configuring Live
>>>>>>>>>>>>>>>>>>>>> Migration
>>>>>>>>>>>>>>>>>>>>> *收件人:*"Wangpan"<hzwangpan at corp.netease.com>,"
>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>> *抄送:*
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>  Hello guys, I'm getting problems when trying to
>>>>>>>>>>>>>>>>>>>>> migrate a VM.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I'm getting this error on my log:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>  2013-07-25 17:28:31.582 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>> [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>> 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while running
>>>>>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting
>>>>>>>>>>>>>>>>>>>>> instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>> 2013-07-25 17:28:32.212 ERROR
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> The user Wangpan was helping me, but I guess his hour
>>>>>>>>>>>>>>>>>>>>> is different from mine, can anybody five me an idea?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hello Wangpan, I've looking at the logs and I saw one
>>>>>>>>>>>>>>>>>>>>>> error with my ssh port, and then I fixed the port, but now I'm getting this
>>>>>>>>>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>  2013-07-25 14:09:50.388 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>> [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>> d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while running
>>>>>>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting
>>>>>>>>>>>>>>>>>>>>>> instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 14:09:51.005 ERROR
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> What "host key verification"? Where should I put this
>>>>>>>>>>>>>>>>>>>>>> key?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <luisguilherme.cr at gmail.com
>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Typed wrong here
>>>>>>>>>>>>>>>>>>>>>>> Em 25/07/2013 00:42, "Wangpan" <
>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>> chmod 777 -R /var/lib/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>> not the /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>  *发件人:*Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>> *发送时间:*2013-07-25 10:39
>>>>>>>>>>>>>>>>>>>>>>>> *主题:*Re: Re: Re: [Openstack] Configuring Live
>>>>>>>>>>>>>>>>>>>>>>>> Migration
>>>>>>>>>>>>>>>>>>>>>>>> *收件人:*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>> *抄送:*
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>  I've done chmod 777 -R /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Does it work?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/24 Wangpan <hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>> I saw this error "Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/'" in your logs.
>>>>>>>>>>>>>>>>>>>>>>>>> you may need to chown the instances
>>>>>>>>>>>>>>>>>>>>>>>>> path  '/var/lib/nova/instances/'
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人:*Guilherme Russi <luisguilherme.cr at gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间:*2013-07-25 10:08
>>>>>>>>>>>>>>>>>>>>>>>>> *主题:*Re: Re: [Openstack] Configuring Live
>>>>>>>>>>>>>>>>>>>>>>>>> Migration
>>>>>>>>>>>>>>>>>>>>>>>>> *收件人:*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>> *抄送:*
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>  Compute Node 01:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>  cat /var/log/nova/nova-*.log | grep -E ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.326 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.511 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.673 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.837 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.402 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.530 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.657 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.036 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.170 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n    if not
>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>  self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.791 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.925 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:07.567 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:34:02.320 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:23.618 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.087 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.220 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.224 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.225 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.636 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.729 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n    if not
>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>  self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:05.364 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:26.986 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:39.249 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:54.459 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:47.486 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:58.434 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:46:43.408 1694 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:26:43.627 1694 ERROR nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:46.003 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:55.194 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:53:42.789 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:54:05.061 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:05:44.677 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:06:14.408 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>>> [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>> 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>>>>>>> last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>  cat /var/log/nova/nova-*.log | grep -E TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007,
>>>>>>>>>>>>>>>>>>>>>>>>> in _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 44, in register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007,
>>>>>>>>>>>>>>>>>>>>>>>>> in _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 44, in register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007,
>>>>>>>>>>>>>>>>>>>>>>>>> in _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 44, in register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007,
>>>>>>>>>>>>>>>>>>>>>>>>> in _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 44, in register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007,
>>>>>>>>>>>>>>>>>>>>>>>>> in _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 44, in register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>   with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1510, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info=block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 2326, in to_xml
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> disk_info, rescue, block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 2244, in get_guest_config
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> inst_type):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 2051, in get_guest_storage_config
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> inst_type)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 2013, in get_guest_disk_config
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> self.disk_cachemode,
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 355, in disk_cachemode
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> if not self._supports_direct_io(CONF.instances_path):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1676, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> {'path': dirpath, 'ex': str(e)})
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1663, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     f
>>>>>>>>>>>>>>>>>>>>>>>>> = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in
>>>>>>>>>>>>>>>>>>>>>>>>> run_instance
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     do_run_instance()
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>> 242, in inner
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in
>>>>>>>>>>>>>>>>>>>>>>>>> do_run_instance
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     admin_password, is_first_time, node,
>>>>>>>>>>>>>>>>>>>>>>>>> instance)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self._set_instance_error_state(context,
>>>>>>>>>>>>>>>>>>>>>>>>> instance['uuid'])
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in
>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     filter_properties, bdms)
>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>  ...
>>>>>>>>>>
>>>>>>>>>> [Mensagem reduzida]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130730/5a8eb02d/attachment.html>


More information about the Openstack mailing list