Open Stack

Hello again Filipe, I've tried to change my compute nodes' UID and GID to
match those from my controller node, but when I restart the services, the
nova-compute service keep on stop/waiting, I tried to reboot the compute
node, but it keeps the same way. I've been checking if UID or GID was in
conflict, but they are not.
The UID and GID from my compute nodes are matching, but I need to match
them with ones from my controller node, right?

Thank you.


2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>

> Confirming the UIDs and GIDs, I had one of them that didn't match, then I
> choose another ID to match, is it right?
>
> Thank you.
>
> Regards.
>
>
> 2013/7/31 Filipe Manco <filipe.manco at gmail.com>
>
>> Answers inline.
>>
>> Filipe Manco
>> http://about.me/fmanco
>>
>>
>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>
>>> Right, just to check, is it known_hosts or know_hosts like you write?
>>>
>>
>> known_hosts you're right
>>
>>
>>> And another thing, I made the test sudo -u nova ssh my_compute_node and
>>> it stops in a password screen, is it right?
>>>
>>
>> No. It should be able to ssh without password. Check all files have
>> permissions 700 and belong to nova.
>>
>> And the last one, after making chown -R nova /var/lib/nova/ I'm getting
>>> error to create new VMs.
>>>
>>
>> Do you follow the guide about changing nova ids? Make sure that
>> /etc/passwd and /etc/passwd- (notice the -) match. The same for /etc/group
>> and /etc/group- or reboot your compute nodes.
>>
>> And don't forget to restart all your services including libvirt-bin.
>>
>>
>>>
>>> Thank you again.
>>>
>>>
>>> 2013/7/31 Filipe Manco <filipe.manco at gmail.com>
>>>
>>>> This is getting a little bit confusing so here are a recipe:
>>>>
>>>> 1. Delete everythin under */var/lib/nova/.ssh/*
>>>> 2. Run *ssh-keygen -N '' -f /var/lib/nova/.ssh/id_rsa*
>>>> 3. Run *cp **/var/lib/nova/.ssh/id_rsa.pub
>>>> /var/lib/nova/.ssh/authorized_keys
>>>> *4. For every compute node execute *ssh-keyscan
>>>> [node_fqdn],[node_ip_address] >> /var/lib/nova/.ssh/know_hosts*
>>>> 5. Copy the contents of */var/lib/nova/.ssh* to all your compute nodes
>>>>
>>>> Test it by running *sudo -u nova ssh [compute_node]*
>>>>
>>>> Filipe Manco
>>>> http://about.me/fmanco
>>>>
>>>>
>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>
>>>>> I did the chown -R nova /var/lib/nova/ but still the error, any idea?
>>>>> I put the chown permission in my two compute nodes and in my controller
>>>>> node.
>>>>>
>>>>> 2013-07-31 15:40:55.720 ERROR nova.compute.manager
>>>>> [req-86d4ddd9-8c85-4ee7-ba0a-d83e81f26818 c402785616534f2096b34ce132b7d3f2
>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>> f4af662d-ae55-4346-82cf-a997192706b5] Unexpected error while running
>>>>> command.
>>>>> Stderr: 'Permission denied, please try again.\r\nPermission denied,
>>>>> please try again.\r\nPermission denied (publickey,password).\r\n'. Setting
>>>>> instance vm_state to ERROR
>>>>> 2013-07-31 15:40:56.298 ERROR nova.openstack.common.rpc.amqp
>>>>> [req-86d4ddd9-8c85-4ee7-ba0a-d83e81f26818 c402785616534f2096b34ce132b7d3f2
>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>
>>>>>
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Traceback (most recent call last):
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>>> line 430, in _process_data
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File
>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>> line 133, in dispatch
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>> wrapped
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> temp_level, payload)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>> wrapped
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> return f(self, context, *args, **kw)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>>>> in decorated_function
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> pass
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>>>> in decorated_function
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> return function(self, context, *args, **kwargs)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>>>> in decorated_function
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> function(self, context, *args, **kwargs)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>>>> in decorated_function
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> e, sys.exc_info())
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>>>> in decorated_function
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> return function(self, context, *args, **kwargs)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>>>> in resize_instance
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> block_device_info)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>> 3467, in migrate_disk_and_power_off
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> inst_base_resize)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>> 3444, in migrate_disk_and_power_off
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> cmd=' '.join(cmd))
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>> /var/lib/nova/instances/f4af662d-ae55-4346-82cf-a997192706b5
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Exit code: 255
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Stdout: ''
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Stderr: 'Permission denied, please try again.\r\nPermission denied, please
>>>>> try again.\r\nPermission denied (publickey,password).\r\n'
>>>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>
>>>>>
>>>>>
>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>
>>>>>> chown -r nova. /var/lib/nova
>>>>>> :)
>>>>>>
>>>>>> *Razique Mahroua** - **Nuage & Co*
>>>>>> razique.mahroua at gmail.com
>>>>>> Tel : +33 9 72 37 94 15
>>>>>>
>>>>>>
>>>>>> Le 31 juil. 2013 à 19:36, Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>> a écrit :
>>>>>>
>>>>>> My config file is like that:
>>>>>>
>>>>>> ls -la /var/lib/nova/.ssh/config
>>>>>> -rw-r--r-- 1 nova root 25 Jul 31 10:49 /var/lib/nova/.ssh/config
>>>>>>
>>>>>>
>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>
>>>>>>> ok make sure the file 'config' belongs to the user nova and it'll be
>>>>>>> good :)
>>>>>>>
>>>>>>> *Razique Mahroua** - **Nuage & Co*
>>>>>>> razique.mahroua at gmail.com
>>>>>>> Tel : +33 9 72 37 94 15
>>>>>>>
>>>>>>> <NUAGECO-LOGO-Fblan_petit.jpg>
>>>>>>>
>>>>>>> Le 31 juil. 2013 à 18:11, Guilherme Russi <
>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>
>>>>>>> here is what happened:
>>>>>>>
>>>>>>>  ssh nova at 10.3.77.48
>>>>>>> The authenticity of host '10.3.77.48 (10.3.77.48)' can't be
>>>>>>> established.
>>>>>>> ECDSA key fingerprint is
>>>>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>>>>> Failed to add the host to the list of known hosts
>>>>>>> (/var/lib/nova/.ssh/known_hosts).
>>>>>>> nova at 10.3.77.48's password:
>>>>>>>
>>>>>>> I don't remember if I set up a password for nova user, I don't know
>>>>>>> which is it.
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>>
>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>
>>>>>>>> ok can you update the file /etc/passwd
>>>>>>>> and use /bin/sh as a shell for the nova user
>>>>>>>> then log as nova :
>>>>>>>> # su - nova
>>>>>>>>
>>>>>>>> after that, try to ssh into the other node :
>>>>>>>> # ssh nova at 10.3.77.48
>>>>>>>>
>>>>>>>> You shouldn't be asked for anything, you should just be connected
>>>>>>>> to the host
>>>>>>>>
>>>>>>>>
>>>>>>>> Le 31 juil. 2013 à 17:07, Guilherme Russi <
>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>
>>>>>>>> This is the nova-compute.log's content from my Compute node "Caos"
>>>>>>>> where the VM is first created.
>>>>>>>>
>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>> 2013-07-31 11:18:49.606 ERROR nova.compute.manager
>>>>>>>> [req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>> 96afb354-a6f8-49a8-9136-3fb2ef8bd321] Unexpected error while running
>>>>>>>> command.
>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>>>> vm_state to ERROR
>>>>>>>> 2013-07-31 11:18:50.245 ERROR nova.openstack.common.rpc.amqp
>>>>>>>> [req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>> 2013-07-31 11:50:47.134 ERROR nova.compute.manager
>>>>>>>> [req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>> 0c729d3b-4e5a-45b8-9e68-1382b321d77e] Unexpected error while running
>>>>>>>> command.
>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>>>> vm_state to ERROR
>>>>>>>> 2013-07-31 11:50:47.909 ERROR nova.openstack.common.rpc.amqp
>>>>>>>> [req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>
>>>>>>>>
>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Traceback (most recent call last):
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File
>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>> 430, in _process_data
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File
>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>> line 133, in dispatch
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>>>>> wrapped
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     temp_level, payload)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     self.gen.next()
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>>>>> wrapped
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     return f(self, context, *args, **kw)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 209, in decorated_function
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     pass
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     self.gen.next()
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 195, in decorated_function
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     return function(self, context, *args, **kwargs)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 260, in decorated_function
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     function(self, context, *args, **kwargs)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 237, in decorated_function
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     e, sys.exc_info())
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     self.gen.next()
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 224, in decorated_function
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     return function(self, context, *args, **kwargs)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 2354, in resize_instance
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     block_device_info)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>> 3467, in migrate_disk_and_power_off
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     inst_base_resize)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     self.gen.next()
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>> 3444, in migrate_disk_and_power_off
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in
>>>>>>>> execute
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     cmd=' '.join(cmd))
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>>>>> /var/lib/nova/instances/96afb354-a6f8-49a8-9136-3fb2ef8bd321
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Exit code: 255
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Stdout: ''
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Traceback (most recent call last):
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File
>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>> 430, in _process_data
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File
>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>> line 133, in dispatch
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>>>>> wrapped
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     temp_level, payload)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     self.gen.next()
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>>>>> wrapped
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     return f(self, context, *args, **kw)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 209, in decorated_function
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     pass
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     self.gen.next()
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 195, in decorated_function
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     return function(self, context, *args, **kwargs)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 260, in decorated_function
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     function(self, context, *args, **kwargs)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 237, in decorated_function
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     e, sys.exc_info())
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     self.gen.next()
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 224, in decorated_function
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     return function(self, context, *args, **kwargs)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>> 2354, in resize_instance
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     block_device_info)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>> 3467, in migrate_disk_and_power_off
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     inst_base_resize)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     self.gen.next()
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>> 3444, in migrate_disk_and_power_off
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in
>>>>>>>> execute
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>     cmd=' '.join(cmd))
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>>>>> /var/lib/nova/instances/0c729d3b-4e5a-45b8-9e68-1382b321d77e
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Exit code: 255
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Stdout: ''
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/7/31 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>
>>>>>>>>> What's your error know when you try to migrate?
>>>>>>>>>
>>>>>>>>>  Filipe Manco
>>>>>>>>> http://about.me/fmanco
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>
>>>>>>>>>> I still don't know what to do, I've checked again my nova.conf
>>>>>>>>>> from my compute nodes, checked the public key from authorized_keys, checked
>>>>>>>>>> the folder /var/lib/nova/instances mapped from my controller to my compute
>>>>>>>>>> nodes, and checked the uid/gid from nova and libvirt-qemu.
>>>>>>>>>>
>>>>>>>>>> Am I missing anything?
>>>>>>>>>>
>>>>>>>>>> Thank you all.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Yes, I was doing right so.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Pub, always the *public key*, the private key is…well private
>>>>>>>>>>>> and should NEVER be shared :)
>>>>>>>>>>>>
>>>>>>>>>>>> Le 31 juil. 2013 à 16:01, Guilherme Russi <
>>>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>> Just to check, the file to copy and insert into the
>>>>>>>>>>>> authorized_keys is from the id_rsa or id_rsa.pub?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Just make a correction on my path, it is in /var/lib/nova/.ssh
>>>>>>>>>>>>> not in /var/lib/.ssh like I sad before :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Le 31 juil. 2013 à 15:40, Guilherme Russi <
>>>>>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hello friends,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  First from JuanFra: I've alrealdy checked it, they are the
>>>>>>>>>>>>>> same.
>>>>>>>>>>>>>>  Second from Razique: Could you give me a strictly
>>>>>>>>>>>>>> explanation, I already have the folder in /var/lib/.ssh with the file
>>>>>>>>>>>>>> authorized_keys inside. This file config  must have only
>>>>>>>>>>>>>> StrictHostKeyChecking no inside it?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> yup that prevents the user nova from being prompted to type
>>>>>>>>>>>>>> "yes" if it's an unknown server
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you all.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hey Guilherme :)
>>>>>>>>>>>>>>> make sure :
>>>>>>>>>>>>>>> • To exchange pubkey of nova users amongst all nodes
>>>>>>>>>>>>>>> • Add in /var/lib a directory .ssh/ and create a "config"
>>>>>>>>>>>>>>> file with into : *StrictHostKeyChecking* no
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> *Razique Mahroua** - **Nuage & Co*
>>>>>>>>>>>>>>> razique.mahroua at gmail.com
>>>>>>>>>>>>>>> Tel : +33 9 72 37 94 15
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> <NUAGECO-LOGO-Fblan_petit.jpg>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Le 30 juil. 2013 à 19:08, Guilherme Russi <
>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sure, thank you.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.074 ERROR nova.compute.manager
>>>>>>>>>>>>>>> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>> 5061e7a6-2482-4af2-97cf-0074326fb940] Unexpected error while running
>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting
>>>>>>>>>>>>>>> instance vm_state to ERROR
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 ERROR nova.openstack.common.rpc.amqp
>>>>>>>>>>>>>>> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
>>>>>>>>>>>>>>> resize_instance
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     block_device_info)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     inst_base_resize)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     utils.execute('ssh', dest, 'mkdir',
>>>>>>>>>>>>>>> '-p', inst_base)
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error
>>>>>>>>>>>>>>> while running command.
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>>>>>>> /var/lib/nova/instances/5061e7a6-2482-4af2-97cf-0074326fb940
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Exit code: 255
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stdout: ''
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can you provide more information (errors and/or logs)?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> It didn't work :(
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ok, testing it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Basically edit the /etc/passwd and /etc/passwd- files
>>>>>>>>>>>>>>>>>>> and change /bin/false to /bin/bash on the nova user entry.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I'm having another problem after this related to libvirt
>>>>>>>>>>>>>>>>>>> but I have no time to fix it know. Basically the migration ends but then
>>>>>>>>>>>>>>>>>>> libvirt isn't able to boot the machine again. Let me know about your
>>>>>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Ok, on my scenario I have 2 Compute nodes and 1
>>>>>>>>>>>>>>>>>>>> Controller + Network node, how did you change your configs, I can try your
>>>>>>>>>>>>>>>>>>>> idea.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> No I just wanted to know what is your scenario. About
>>>>>>>>>>>>>>>>>>>>> the mkdir I spend a couple of time looking at it and managed to fix it by
>>>>>>>>>>>>>>>>>>>>> changing the shell of the nova user to /bin/bash instead of /bin/false on
>>>>>>>>>>>>>>>>>>>>> all the compute nodes. You can try it.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hello Filipe, I followed the manual on the openstack
>>>>>>>>>>>>>>>>>>>>>> documentation, I shared the /var/lib/nova/instances on my controller, do I
>>>>>>>>>>>>>>>>>>>>>> need to share another folders?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> A couple of messages ago you shown an error where
>>>>>>>>>>>>>>>>>>>>>>> the mkdir command failed. At that time the keys were properly configured.
>>>>>>>>>>>>>>>>>>>>>>> Can you revert your changes?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> About the mkdir error, I had a similar one but I
>>>>>>>>>>>>>>>>>>>>>>> wasn't able to fix it yet. Do you have shared storage for your instances?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I never used virsh directly because, AFAIK,
>>>>>>>>>>>>>>>>>>>>>>> openstack will not be informed of the changes and will not work correctly
>>>>>>>>>>>>>>>>>>>>>>> (but I may be wrong in this one).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I created the new key, but still getting error:
>>>>>>>>>>>>>>>>>>>>>>>> here are the logs:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:51.097 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>>>> [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>>>> bed3c3be-31d6-42c3-9a7f-e40ef9c1c764] Unexpected error while running
>>>>>>>>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>>>>>>>>>  Stderr: 'Host key verification failed.\r\n'.
>>>>>>>>>>>>>>>>>>>>>>>> Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 ERROR
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-c3441bfc-dcd7-4625-b701-716f152a4bfe
>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
>>>>>>>>>>>>>>>>>>>>>>>> resize_instance
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
>>>>>>>>>>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     inst_base_resize)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
>>>>>>>>>>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     utils.execute('ssh', dest, 'mkdir',
>>>>>>>>>>>>>>>>>>>>>>>> '-p', inst_base)
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error
>>>>>>>>>>>>>>>>>>>>>>>> while running command.
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/instances/bed3c3be-31d6-42c3-9a7f-e40ef9c1c764
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Exit code: 255
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stdout: ''
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I've been trying to migrate directly with virsh,
>>>>>>>>>>>>>>>>>>>>>>>> but I got this error and the solutions through the web don't help.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> virsh migrate --live instance-00000063 qemu+ssh://
>>>>>>>>>>>>>>>>>>>>>>>> 192.168.3.3/system
>>>>>>>>>>>>>>>>>>>>>>>> root at 192.168.3.3's password:
>>>>>>>>>>>>>>>>>>>>>>>> erro: internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-bed3c3be-31d6-42c3-9a7f-e40ef9c1c764'
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thank you all.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/29 <yongiman at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Guilherme,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I suugest making new ssh key and put the public
>>>>>>>>>>>>>>>>>>>>>>>>> key in the host
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> # ssh-keygen (you will find id_rsa.pub in your
>>>>>>>>>>>>>>>>>>>>>>>>> .ssh directory)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Copy the contents of this file and paste at the
>>>>>>>>>>>>>>>>>>>>>>>>> last line of authorized_keys in the host.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On 2013. 7. 29., at 오후 10:40, Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hello Filipe, I created the
>>>>>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys and its content is the keys from the
>>>>>>>>>>>>>>>>>>>>>>>>> know_hosts, is it right? I'm still getting the same error when I try to
>>>>>>>>>>>>>>>>>>>>>>>>> migrate a VM.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> About the keys it's up to you. You can use the
>>>>>>>>>>>>>>>>>>>>>>>>>> same or create a new one. Using the same is easier.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> But the nova user public key should go into
>>>>>>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys not into the know_hosts file. The easier
>>>>>>>>>>>>>>>>>>>>>>>>>> way (as you only have two compute nodes) to add the keys to the know_hosts
>>>>>>>>>>>>>>>>>>>>>>>>>> is to manually ssh and accept the key.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Ok, Must I create another ssh key to communicate
>>>>>>>>>>>>>>>>>>>>>>>>>>> both compute nodes or the current key can work? I've put the plublic key
>>>>>>>>>>>>>>>>>>>>>>>>>>> from both compute node in the know_hosts. Which permission should I apply?
>>>>>>>>>>>>>>>>>>>>>>>>>>> 644?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> First you need to check why this "Failed to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> add the host to the list of known hosts (/var/lib/nova/.ssh/known_hosts)."
>>>>>>>>>>>>>>>>>>>>>>>>>>>> happens. It maybe file permissions or so. Then you need to generate an ssh
>>>>>>>>>>>>>>>>>>>>>>>>>>>> key for the user nova and add it to the other compute nodes on the
>>>>>>>>>>>>>>>>>>>>>>>>>>>> authorized_keys file (the password is not really necessary as soon as you
>>>>>>>>>>>>>>>>>>>>>>>>>>>> have the keys in place).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> sudo -u nova ssh 192.168.3.3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The authenticity of host '192.168.3.3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (192.168.3.3)' can't be established.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ECDSA key fingerprint is
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to continue connecting
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (yes/no)? yes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Failed to add the host to the list of known
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hosts (/var/lib/nova/.ssh/known_hosts).
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova at 192.168.3.3's password:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> But I don't know the user nova's password.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try *sudo -u nova ssh
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> <remote-compute-node>* form the compute node
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently running the VM and check if it works?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Filipe,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  I made the copy from my .ssh/know_hosts to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> places you told me, but I'm still getting the same error. Any other ideas?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This error means that you cannot ssh into
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> the new compute node because it's public key cannot be verified. Add the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> public keys of the compute nodes to /etc/ssh/know_hosts file (or the nova
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> one, usually under /var/lib/nova/.ssh/) and the problem should be gone.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ok, I'll look more carefully the log and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's cool this command from libvirt, I didn't know it, I'll try it and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> return to you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you again.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That's confused me, "Unexpected error
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> while running command" will give us what command is running when this error
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> occured, you may check the log more carefully.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and you also can migrate the VM directly
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> by libvirt command line like below:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> virsh  migrate --live --persistent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --undefinesource --verbose --domain ${instance-uuid} --desturi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> qemu+tcp://${the-dest-host-ip}/system
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then see the output result.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-26 09:30
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*"openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  When I enter the action to migrate, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only two errors showen at this period are these two.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There may have more error logs at this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> minute 2013-07-25 17:28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you can get the backtrace near the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> error log you post.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-26 04:48
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>,"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Hello guys, I'm getting problems when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to migrate a VM.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm getting this error on my log:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  2013-07-25 17:28:31.582 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed.\r\n'. Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 17:28:32.212 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The user Wangpan was helping me, but I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guess his hour is different from mine, can anybody five me an idea?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Wangpan, I've looking at the logs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and I saw one error with my ssh port, and then I fixed the port, but now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm getting this error:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  2013-07-25 14:09:50.388 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed.\r\n'. Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 14:09:51.005 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What "host key verification"? Where
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should I put this key?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Typed wrong here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Em 25/07/2013 00:42, "Wangpan" <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chmod 777 -R /var/lib/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not the /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-25 10:39
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  I've done chmod 777 -R
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does it work?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/24 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I saw this error "Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/'" in your logs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you may need to chown the instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> path  '/var/lib/nova/instances/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-25 10:08
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Compute Node 01:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  cat /var/log/nova/nova-*.log |
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> grep -E ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.326 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.511 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.673 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.837 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.402 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.530 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.657 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.036 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.170 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n    if not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.791 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.925 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:07.567 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:34:02.320 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:23.618 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.087 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.220 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.224 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.225 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.636 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.729 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n    if not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:05.364 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:26.986 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:39.249 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:54.459 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:47.486 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:58.434 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:46:43.408 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:26:43.627 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:46.003 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:55.194 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:53:42.789 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:54:05.061 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:05:44.677 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:06:14.408 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  cat /var/log/nova/nova-*.log |
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> grep -E TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1510, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info=block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2326, in to_xml
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_info, rescue, block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2244, in get_guest_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inst_type):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2051, in get_guest_storage_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inst_type)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013, in get_guest_disk_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.disk_cachemode,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 355, in disk_cachemode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if not self._supports_direct_io(CONF.instances_path):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1676, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> {'path': dirpath, 'ex': str(e)})
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1663, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     f
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     do_run_instance()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 242, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do_run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     admin_password, is_first_time, node,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self._set_instance_error_state(context,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance['uuid'])
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     filter_properties, bdms)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ...
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [Mensagem reduzida]
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Mailing list:
>>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>>>>>> Post to     : openstack at lists.openstack.org
>>>>>>>>>>>>>>> Unsubscribe :
>>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/72c61bb4/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NUAGECO-LOGO-Fblan_petit.jpg
Type: image/jpeg
Size: 10122 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/72c61bb4/attachment.jpg>