[Openstack] Configuring Live Migration
Filipe Manco
filipe.manco at gmail.com
Tue Jul 30 16:05:28 UTC 2013
Basically edit the /etc/passwd and /etc/passwd- files and change /bin/false
to /bin/bash on the nova user entry.
I'm having another problem after this related to libvirt but I have no time
to fix it know. Basically the migration ends but then libvirt isn't able to
boot the machine again. Let me know about your results.
Regards
Filipe Manco
http://about.me/fmanco
2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
> Ok, on my scenario I have 2 Compute nodes and 1 Controller + Network node,
> how did you change your configs, I can try your idea.
>
>
> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>
>> No I just wanted to know what is your scenario. About the mkdir I spend a
>> couple of time looking at it and managed to fix it by changing the shell of
>> the nova user to /bin/bash instead of /bin/false on all the compute nodes.
>> You can try it.
>>
>> Filipe Manco
>> http://about.me/fmanco
>>
>>
>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>
>>> Hello Filipe, I followed the manual on the openstack documentation, I
>>> shared the /var/lib/nova/instances on my controller, do I need to share
>>> another folders?
>>>
>>> Regards.
>>>
>>>
>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>
>>>> Hi
>>>>
>>>> A couple of messages ago you shown an error where the mkdir command
>>>> failed. At that time the keys were properly configured. Can you revert your
>>>> changes?
>>>>
>>>> About the mkdir error, I had a similar one but I wasn't able to fix it
>>>> yet. Do you have shared storage for your instances?
>>>>
>>>> I never used virsh directly because, AFAIK, openstack will not be
>>>> informed of the changes and will not work correctly (but I may be wrong in
>>>> this one).
>>>>
>>>> Filipe Manco
>>>> http://about.me/fmanco
>>>>
>>>>
>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>
>>>>> I created the new key, but still getting error: here are the logs:
>>>>>
>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>> 2013-07-30 11:37:51.097 ERROR nova.compute.manager
>>>>> [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2
>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>> bed3c3be-31d6-42c3-9a7f-e40ef9c1c764] Unexpected error while running
>>>>> command.
>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>> vm_state to ERROR
>>>>> 2013-07-30 11:37:52.463 ERROR nova.openstack.common.rpc.amqp
>>>>> [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2
>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>
>>>>>
>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> Traceback (most recent call last):
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>>> line 430, in _process_data
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File
>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>> line 133, in dispatch
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>> wrapped
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> temp_level, payload)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>> wrapped
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> return f(self, context, *args, **kw)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>>>> in decorated_function
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> pass
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>>>> in decorated_function
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> return function(self, context, *args, **kwargs)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>>>> in decorated_function
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> function(self, context, *args, **kwargs)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>>>> in decorated_function
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> e, sys.exc_info())
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>>>> in decorated_function
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> return function(self, context, *args, **kwargs)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>>>> in resize_instance
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> block_device_info)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>> 3467, in migrate_disk_and_power_off
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> inst_base_resize)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>> 3444, in migrate_disk_and_power_off
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> cmd=' '.join(cmd))
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>> /var/lib/nova/instances/bed3c3be-31d6-42c3-9a7f-e40ef9c1c764
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp Exit
>>>>> code: 255
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> Stdout: ''
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>> 2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>
>>>>> I've been trying to migrate directly with virsh, but I got this error
>>>>> and the solutions through the web don't help.
>>>>>
>>>>> virsh migrate --live instance-00000063 qemu+ssh://192.168.3.3/system
>>>>> root at 192.168.3.3's password:
>>>>> erro: internal error cannot load AppArmor profile
>>>>> 'libvirt-bed3c3be-31d6-42c3-9a7f-e40ef9c1c764'
>>>>>
>>>>>
>>>>> Thank you all.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/7/29 <yongiman at gmail.com>
>>>>>
>>>>> Hi Guilherme,
>>>>>>
>>>>>> I suugest making new ssh key and put the public key in the host
>>>>>>
>>>>>> # ssh-keygen (you will find id_rsa.pub in your .ssh directory)
>>>>>>
>>>>>> Copy the contents of this file and paste at the last line of
>>>>>> authorized_keys in the host.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 2013. 7. 29., at 오후 10:40, Guilherme Russi <
>>>>>> luisguilherme.cr at gmail.com> wrote:
>>>>>>
>>>>>> Hello Filipe, I created the /var/lib/nova/.ssh/authorized_keys and
>>>>>> its content is the keys from the know_hosts, is it right? I'm still getting
>>>>>> the same error when I try to migrate a VM.
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>
>>>>>>> About the keys it's up to you. You can use the same or create a new
>>>>>>> one. Using the same is easier.
>>>>>>>
>>>>>>> But the nova user public key should go into
>>>>>>> /var/lib/nova/.ssh/authorized_keys not into the know_hosts file. The easier
>>>>>>> way (as you only have two compute nodes) to add the keys to the know_hosts
>>>>>>> is to manually ssh and accept the key.
>>>>>>>
>>>>>>> Filipe Manco
>>>>>>> http://about.me/fmanco
>>>>>>>
>>>>>>>
>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>
>>>>>>>> Ok, Must I create another ssh key to communicate both compute nodes
>>>>>>>> or the current key can work? I've put the plublic key from both compute
>>>>>>>> node in the know_hosts. Which permission should I apply? 644?
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>
>>>>>>>>> First you need to check why this "Failed to add the host to the
>>>>>>>>> list of known hosts (/var/lib/nova/.ssh/known_hosts)." happens.
>>>>>>>>> It maybe file permissions or so. Then you need to generate an ssh key for
>>>>>>>>> the user nova and add it to the other compute nodes on the authorized_keys
>>>>>>>>> file (the password is not really necessary as soon as you have the keys in
>>>>>>>>> place).
>>>>>>>>>
>>>>>>>>> Filipe Manco
>>>>>>>>> http://about.me/fmanco
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>
>>>>>>>>>> sudo -u nova ssh 192.168.3.3
>>>>>>>>>> The authenticity of host '192.168.3.3 (192.168.3.3)' can't be
>>>>>>>>>> established.
>>>>>>>>>> ECDSA key fingerprint is
>>>>>>>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>>>>>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>>>>>>>> Failed to add the host to the list of known hosts
>>>>>>>>>> (/var/lib/nova/.ssh/known_hosts).
>>>>>>>>>> nova at 192.168.3.3's password:
>>>>>>>>>>
>>>>>>>>>> But I don't know the user nova's password.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Can you try *sudo -u nova ssh <remote-compute-node>* form the
>>>>>>>>>>> compute node currently running the VM and check if it works?
>>>>>>>>>>>
>>>>>>>>>>> Filipe Manco
>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Hello Filipe,
>>>>>>>>>>>>
>>>>>>>>>>>> I made the copy from my .ssh/know_hosts to places you told me,
>>>>>>>>>>>> but I'm still getting the same error. Any other ideas?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> This error means that you cannot ssh into the new compute node
>>>>>>>>>>>>> because it's public key cannot be verified. Add the public keys of the
>>>>>>>>>>>>> compute nodes to /etc/ssh/know_hosts file (or the nova one, usually under
>>>>>>>>>>>>> /var/lib/nova/.ssh/) and the problem should be gone.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>
>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ok, I'll look more carefully the log and it's cool this
>>>>>>>>>>>>>> command from libvirt, I didn't know it, I'll try it and return to you.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you again.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/7/25 Wangpan <hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>> That's confused me, "Unexpected error while running command"
>>>>>>>>>>>>>>> will give us what command is running when this error occured, you may check
>>>>>>>>>>>>>>> the log more carefully.
>>>>>>>>>>>>>>> and you also can migrate the VM directly by libvirt command
>>>>>>>>>>>>>>> line like below:
>>>>>>>>>>>>>>> virsh migrate --live --persistent --undefinesource
>>>>>>>>>>>>>>> --verbose --domain ${instance-uuid} --desturi
>>>>>>>>>>>>>>> qemu+tcp://${the-dest-host-ip}/system
>>>>>>>>>>>>>>> then see the output result.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>> *发件人：*Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>> *发送时间：*2013-07-26 09:30
>>>>>>>>>>>>>>> *主题：*Re: Re: Re: Re: Re: [Openstack] Configuring Live
>>>>>>>>>>>>>>> Migration
>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>> *抄送：*"openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> When I enter the action to migrate, the only two errors
>>>>>>>>>>>>>>> showen at this period are these two.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/7/25 Wangpan <hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>> There may have more error logs at this minute 2013-07-25
>>>>>>>>>>>>>>>> 17:28,
>>>>>>>>>>>>>>>> you can get the backtrace near the error log you post.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>> *发件人：*Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>> *发送时间：*2013-07-26 04:48
>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: Re: [Openstack] Configuring Live Migration
>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com>,"
>>>>>>>>>>>>>>>> openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello guys, I'm getting problems when trying to migrate a
>>>>>>>>>>>>>>>> VM.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm getting this error on my log:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013-07-25 17:28:31.582 ERROR nova.compute.manager
>>>>>>>>>>>>>>>> [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>> 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while running
>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting
>>>>>>>>>>>>>>>> instance vm_state to ERROR
>>>>>>>>>>>>>>>> 2013-07-25 17:28:32.212 ERROR
>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The user Wangpan was helping me, but I guess his hour is
>>>>>>>>>>>>>>>> different from mine, can anybody five me an idea?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello Wangpan, I've looking at the logs and I saw one
>>>>>>>>>>>>>>>>> error with my ssh port, and then I fixed the port, but now I'm getting this
>>>>>>>>>>>>>>>>> error:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013-07-25 14:09:50.388 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>> [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>> d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while running
>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting
>>>>>>>>>>>>>>>>> instance vm_state to ERROR
>>>>>>>>>>>>>>>>> 2013-07-25 14:09:51.005 ERROR
>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> What "host key verification"? Where should I put this key?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Typed wrong here
>>>>>>>>>>>>>>>>>> Em 25/07/2013 00:42, "Wangpan" <
>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com> escreveu:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>> chmod 777 -R /var/lib/nova/instances
>>>>>>>>>>>>>>>>>>> not the /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>> *发件人：*Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-25 10:39
>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: [Openstack] Configuring Live Migration
>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've done chmod 777 -R /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Does it work?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/7/24 Wangpan <hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>> I saw this error "Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/'" in your logs.
>>>>>>>>>>>>>>>>>>>> you may need to chown the instances
>>>>>>>>>>>>>>>>>>>> path '/var/lib/nova/instances/'
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>> *发件人：*Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-25 10:08
>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: [Openstack] Configuring Live Migration
>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Compute Node 01:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-*.log | grep -E ERROR
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.326 1820 ERROR
>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.511 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.673 1820 ERROR
>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.837 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.402 1820 ERROR
>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.530 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.657 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>>>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while
>>>>>>>>>>>>>>>>>>>> checking direct I/O: '[Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.036 1685 ERROR
>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.170 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info=block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>> in to_xml\n disk_info, rescue, block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>> in get_guest_config\n inst_type):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n inst_type)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n self.disk_cachemode,\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n if not
>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>> self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.791 1685 ERROR
>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.925 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:07.567 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 ERROR
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:34:02.320 1685 ERROR
>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:23.618 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.087 1685 ERROR
>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.220 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.224 ERROR nova.virt.libvirt.driver
>>>>>>>>>>>>>>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while
>>>>>>>>>>>>>>>>>>>> checking direct I/O: '[Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.225 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.636 1694 ERROR
>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.729 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info=block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>> in to_xml\n disk_info, rescue, block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>> in get_guest_config\n inst_type):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n inst_type)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n self.disk_cachemode,\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n if not
>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>> self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:05.364 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:26.986 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:39.249 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:54.459 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:47.486 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:58.434 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>> 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:46:43.408 1694 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:26:43.627 1694 ERROR nova.manager [-]
>>>>>>>>>>>>>>>>>>>> Error during ComputeManager._run_image_cache_manager_pass: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:46.003 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:55.194 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>> 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:53:42.789 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:54:05.061 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>> 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:05:44.677 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:06:14.408 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>> 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most recent call
>>>>>>>>>>>>>>>>>>>> last):\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>> 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-*.log | grep -E TRACE
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> task(self, context)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> task(self, context)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> task(self, context)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> task(self, context)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Traceback (most recent
>>>>>>>>>>>>>>>>>>>> call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] block_device_info)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] os.makedirs(path)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] OSError: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Traceback (most recent
>>>>>>>>>>>>>>>>>>>> call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] block_device_info)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] os.makedirs(path)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] OSError: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Traceback (most recent
>>>>>>>>>>>>>>>>>>>> call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] block_device_info)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] os.makedirs(path)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] OSError: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in
>>>>>>>>>>>>>>>>>>>> periodic_tasks
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> task(self, context)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> storage_users.register_storage_use(CONF.instances_path, CONF.host)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE nova.manager
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Traceback (most recent
>>>>>>>>>>>>>>>>>>>> call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] block_device_info)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>> block_device_info=block_device_info)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>> in to_xml
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] disk_info, rescue,
>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>> in get_guest_config
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] inst_type):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] inst_type)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] self.disk_cachemode,
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>> in disk_cachemode
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] if not
>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>> in _supports_direct_io
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] {'path': dirpath,
>>>>>>>>>>>>>>>>>>>> 'ex': str(e)})
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] self.gen.next()
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>> in _supports_direct_io
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] f = os.open(testfile,
>>>>>>>>>>>>>>>>>>>> os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] OSError: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied: '/var/lib/nova/instances/.directio.test'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Traceback (most recent
>>>>>>>>>>>>>>>>>>>> call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] block_device_info)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] os.makedirs(path)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] OSError: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Traceback (most recent
>>>>>>>>>>>>>>>>>>>> call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>> _spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] block_device_info)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>> in spawn
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>> in _create_image
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] os.makedirs(path)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] OSError: [Errno 13]
>>>>>>>>>>>>>>>>>>>> Permission denied:
>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager
>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp temp_level, payload)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp pass
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in
>>>>>>>>>>>>>>>>>>>> run_instance
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp do_run_instance()
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>> 242, in inner
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in
>>>>>>>>>>>>>>>>>>>> do_run_instance
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp admin_password, is_first_time, node,
>>>>>>>>>>>>>>>>>>>> instance)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in
>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self._set_instance_error_state(context,
>>>>>>>>>>>>>>>>>>>> instance['uuid'])
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in
>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp filter_properties, bdms)
>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ...
>>>>>
>>>>> [Mensagem reduzida]
>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130730/e2f289ec/attachment.html>
More information about the Openstack
mailing list