[Openstack] Configuring Live Migration

Guilherme Russi luisguilherme.cr at gmail.com
Wed Jul 31 19:31:23 UTC 2013


Right, just to check, is it known_hosts or know_hosts like you write? And
another thing, I made the test sudo -u nova ssh my_compute_node and it
stops in a password screen, is it right?
And the last one, after making chown -R nova /var/lib/nova/ I'm getting
error to create new VMs.

Thank you again.


2013/7/31 Filipe Manco <filipe.manco at gmail.com>

> This is getting a little bit confusing so here are a recipe:
>
> 1. Delete everythin under */var/lib/nova/.ssh/*
> 2. Run *ssh-keygen -N '' -f /var/lib/nova/.ssh/id_rsa*
> 3. Run *cp **/var/lib/nova/.ssh/id_rsa.pub
> /var/lib/nova/.ssh/authorized_keys
> *4. For every compute node execute *ssh-keyscan
> [node_fqdn],[node_ip_address] >> /var/lib/nova/.ssh/know_hosts*
> 5. Copy the contents of */var/lib/nova/.ssh* to all your compute nodes
>
> Test it by running *sudo -u nova ssh [compute_node]*
>
> Filipe Manco
> http://about.me/fmanco
>
>
> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>
>> I did the chown -R nova /var/lib/nova/ but still the error, any idea? I
>> put the chown permission in my two compute nodes and in my controller node.
>>
>> 2013-07-31 15:40:55.720 ERROR nova.compute.manager
>> [req-86d4ddd9-8c85-4ee7-ba0a-d83e81f26818 c402785616534f2096b34ce132b7d3f2
>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>> f4af662d-ae55-4346-82cf-a997192706b5] Unexpected error while running
>> command.
>> Stderr: 'Permission denied, please try again.\r\nPermission denied,
>> please try again.\r\nPermission denied (publickey,password).\r\n'. Setting
>> instance vm_state to ERROR
>> 2013-07-31 15:40:56.298 ERROR nova.openstack.common.rpc.amqp
>> [req-86d4ddd9-8c85-4ee7-ba0a-d83e81f26818 c402785616534f2096b34ce132b7d3f2
>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>
>>
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> Traceback (most recent call last):
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>> 430, in _process_data
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>> line 133, in dispatch
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> return getattr(proxyobj, method)(ctxt, **kwargs)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> temp_level, payload)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> return f(self, context, *args, **kw)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>> decorated_function
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> pass
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>> decorated_function
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> return function(self, context, *args, **kwargs)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>> decorated_function
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> function(self, context, *args, **kwargs)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>> decorated_function
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp     e,
>> sys.exc_info())
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>> decorated_function
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> return function(self, context, *args, **kwargs)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
>> resize_instance
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> block_device_info)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
>> in migrate_disk_and_power_off
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> inst_base_resize)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> self.gen.next()
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
>> in migrate_disk_and_power_off
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp   File
>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> cmd=' '.join(cmd))
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> ProcessExecutionError: Unexpected error while running command.
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> Command: ssh 10.3.77.48 mkdir -p
>> /var/lib/nova/instances/f4af662d-ae55-4346-82cf-a997192706b5
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp Exit
>> code: 255
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> Stdout: ''
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>> Stderr: 'Permission denied, please try again.\r\nPermission denied, please
>> try again.\r\nPermission denied (publickey,password).\r\n'
>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>
>>
>>
>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>
>>> chown -r nova. /var/lib/nova
>>> :)
>>>
>>> *Razique Mahroua** - **Nuage & Co*
>>> razique.mahroua at gmail.com
>>> Tel : +33 9 72 37 94 15
>>>
>>>
>>> Le 31 juil. 2013 à 19:36, Guilherme Russi <luisguilherme.cr at gmail.com>
>>> a écrit :
>>>
>>> My config file is like that:
>>>
>>> ls -la /var/lib/nova/.ssh/config
>>> -rw-r--r-- 1 nova root 25 Jul 31 10:49 /var/lib/nova/.ssh/config
>>>
>>>
>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>
>>>> ok make sure the file 'config' belongs to the user nova and it'll be
>>>> good :)
>>>>
>>>> *Razique Mahroua** - **Nuage & Co*
>>>> razique.mahroua at gmail.com
>>>> Tel : +33 9 72 37 94 15
>>>>
>>>> <NUAGECO-LOGO-Fblan_petit.jpg>
>>>>
>>>> Le 31 juil. 2013 à 18:11, Guilherme Russi <luisguilherme.cr at gmail.com>
>>>> a écrit :
>>>>
>>>> here is what happened:
>>>>
>>>>  ssh nova at 10.3.77.48
>>>> The authenticity of host '10.3.77.48 (10.3.77.48)' can't be established.
>>>> ECDSA key fingerprint is
>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>> Failed to add the host to the list of known hosts
>>>> (/var/lib/nova/.ssh/known_hosts).
>>>> nova at 10.3.77.48's password:
>>>>
>>>> I don't remember if I set up a password for nova user, I don't know
>>>> which is it.
>>>>
>>>> Regards.
>>>>
>>>>
>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>
>>>>> ok can you update the file /etc/passwd
>>>>> and use /bin/sh as a shell for the nova user
>>>>> then log as nova :
>>>>> # su - nova
>>>>>
>>>>> after that, try to ssh into the other node :
>>>>> # ssh nova at 10.3.77.48
>>>>>
>>>>> You shouldn't be asked for anything, you should just be connected to
>>>>> the host
>>>>>
>>>>>
>>>>> Le 31 juil. 2013 à 17:07, Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>> a écrit :
>>>>>
>>>>> This is the nova-compute.log's content from my Compute node "Caos"
>>>>> where the VM is first created.
>>>>>
>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>> 2013-07-31 11:18:49.606 ERROR nova.compute.manager
>>>>> [req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>> 96afb354-a6f8-49a8-9136-3fb2ef8bd321] Unexpected error while running
>>>>> command.
>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state
>>>>> to ERROR
>>>>> 2013-07-31 11:18:50.245 ERROR nova.openstack.common.rpc.amqp
>>>>> [req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>> 2013-07-31 11:50:47.134 ERROR nova.compute.manager
>>>>> [req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>> 0c729d3b-4e5a-45b8-9e68-1382b321d77e] Unexpected error while running
>>>>> command.
>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state
>>>>> to ERROR
>>>>> 2013-07-31 11:50:47.909 ERROR nova.openstack.common.rpc.amqp
>>>>> [req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>
>>>>>
>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Traceback (most recent call last):
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>>> line 430, in _process_data
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File
>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>> line 133, in dispatch
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>> wrapped
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> temp_level, payload)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>> wrapped
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> return f(self, context, *args, **kw)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>>>> in decorated_function
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> pass
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>>>> in decorated_function
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> return function(self, context, *args, **kwargs)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>>>> in decorated_function
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> function(self, context, *args, **kwargs)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>>>> in decorated_function
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> e, sys.exc_info())
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>>>> in decorated_function
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> return function(self, context, *args, **kwargs)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>>>> in resize_instance
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> block_device_info)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>> 3467, in migrate_disk_and_power_off
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> inst_base_resize)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> self.gen.next()
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>> 3444, in migrate_disk_and_power_off
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> cmd=' '.join(cmd))
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>> /var/lib/nova/instances/96afb354-a6f8-49a8-9136-3fb2ef8bd321
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Exit code: 255
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Stdout: ''
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Traceback (most recent call last):
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>>> line 430, in _process_data
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File
>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>> line 133, in dispatch
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>> wrapped
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   temp_level, payload)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   self.gen.next()
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>> wrapped
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   return f(self, context, *args, **kw)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>>>> in decorated_function
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   pass
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   self.gen.next()
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>>>> in decorated_function
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   return function(self, context, *args, **kwargs)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>>>> in decorated_function
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   function(self, context, *args, **kwargs)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>>>> in decorated_function
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   e, sys.exc_info())
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   self.gen.next()
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>>>> in decorated_function
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   return function(self, context, *args, **kwargs)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>>>> in resize_instance
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   block_device_info)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>> 3467, in migrate_disk_and_power_off
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   inst_base_resize)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   self.gen.next()
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>> 3444, in migrate_disk_and_power_off
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>   cmd=' '.join(cmd))
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>> /var/lib/nova/instances/0c729d3b-4e5a-45b8-9e68-1382b321d77e
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Exit code: 255
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Stdout: ''
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/7/31 Filipe Manco <filipe.manco at gmail.com>
>>>>>
>>>>>> What's your error know when you try to migrate?
>>>>>>
>>>>>>  Filipe Manco
>>>>>> http://about.me/fmanco
>>>>>>
>>>>>>
>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>
>>>>>>> I still don't know what to do, I've checked again my nova.conf from
>>>>>>> my compute nodes, checked the public key from authorized_keys, checked the
>>>>>>> folder /var/lib/nova/instances mapped from my controller to my compute
>>>>>>> nodes, and checked the uid/gid from nova and libvirt-qemu.
>>>>>>>
>>>>>>> Am I missing anything?
>>>>>>>
>>>>>>> Thank you all.
>>>>>>>
>>>>>>>
>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>
>>>>>>>> Yes, I was doing right so.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>
>>>>>>>>> Pub, always the *public key*, the private key is…well private and
>>>>>>>>> should NEVER be shared :)
>>>>>>>>>
>>>>>>>>> Le 31 juil. 2013 à 16:01, Guilherme Russi <
>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>> Just to check, the file to copy and insert into the
>>>>>>>>> authorized_keys is from the id_rsa or id_rsa.pub?
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>
>>>>>>>>>> Just make a correction on my path, it is in /var/lib/nova/.ssh
>>>>>>>>>> not in /var/lib/.ssh like I sad before :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Le 31 juil. 2013 à 15:40, Guilherme Russi <
>>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>>
>>>>>>>>>>> Hello friends,
>>>>>>>>>>>
>>>>>>>>>>>  First from JuanFra: I've alrealdy checked it, they are the same.
>>>>>>>>>>>  Second from Razique: Could you give me a strictly explanation,
>>>>>>>>>>> I already have the folder in /var/lib/.ssh with the file authorized_keys
>>>>>>>>>>> inside. This file config  must have only StrictHostKeyChecking no
>>>>>>>>>>> inside it?
>>>>>>>>>>>
>>>>>>>>>>> yup that prevents the user nova from being prompted to type
>>>>>>>>>>> "yes" if it's an unknown server
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thank you all.
>>>>>>>>>>>
>>>>>>>>>>> Guilherme.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> Hey Guilherme :)
>>>>>>>>>>>> make sure :
>>>>>>>>>>>> • To exchange pubkey of nova users amongst all nodes
>>>>>>>>>>>> • Add in /var/lib a directory .ssh/ and create a "config" file
>>>>>>>>>>>> with into : *StrictHostKeyChecking* no
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> *Razique Mahroua** - **Nuage & Co*
>>>>>>>>>>>> razique.mahroua at gmail.com
>>>>>>>>>>>> Tel : +33 9 72 37 94 15
>>>>>>>>>>>>
>>>>>>>>>>>> <NUAGECO-LOGO-Fblan_petit.jpg>
>>>>>>>>>>>>
>>>>>>>>>>>> Le 30 juil. 2013 à 19:08, Guilherme Russi <
>>>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>> Sure, thank you.
>>>>>>>>>>>>
>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>>>> 2013-07-30 13:48:08.074 ERROR nova.compute.manager
>>>>>>>>>>>> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>> 5061e7a6-2482-4af2-97cf-0074326fb940] Unexpected error while running
>>>>>>>>>>>> command.
>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>>>>>>>> vm_state to ERROR
>>>>>>>>>>>> 2013-07-30 13:48:08.747 ERROR nova.openstack.common.rpc.amqp
>>>>>>>>>>>> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>>>>>
>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>> decorated_function
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>> decorated_function
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>> decorated_function
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>> decorated_function
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>> decorated_function
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
>>>>>>>>>>>> resize_instance
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     block_device_info)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     inst_base_resize)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     utils.execute('ssh', dest, 'mkdir',
>>>>>>>>>>>> '-p', inst_base)
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error
>>>>>>>>>>>> while running command.
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>>>> /var/lib/nova/instances/5061e7a6-2482-4af2-97cf-0074326fb940
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp Exit code: 255
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stdout: ''
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>> nova.openstack.common.rpc.amqp
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Can you provide more information (errors and/or logs)?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> It didn't work :(
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ok, testing it.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Basically edit the /etc/passwd and /etc/passwd- files and
>>>>>>>>>>>>>>>> change /bin/false to /bin/bash on the nova user entry.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I'm having another problem after this related to libvirt
>>>>>>>>>>>>>>>> but I have no time to fix it know. Basically the migration ends but then
>>>>>>>>>>>>>>>> libvirt isn't able to boot the machine again. Let me know about your
>>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ok, on my scenario I have 2 Compute nodes and 1 Controller
>>>>>>>>>>>>>>>>> + Network node, how did you change your configs, I can try your idea.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> No I just wanted to know what is your scenario. About the
>>>>>>>>>>>>>>>>>> mkdir I spend a couple of time looking at it and managed to fix it by
>>>>>>>>>>>>>>>>>> changing the shell of the nova user to /bin/bash instead of /bin/false on
>>>>>>>>>>>>>>>>>> all the compute nodes. You can try it.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hello Filipe, I followed the manual on the openstack
>>>>>>>>>>>>>>>>>>> documentation, I shared the /var/lib/nova/instances on my controller, do I
>>>>>>>>>>>>>>>>>>> need to share another folders?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> A couple of messages ago you shown an error where the
>>>>>>>>>>>>>>>>>>>> mkdir command failed. At that time the keys were properly configured. Can
>>>>>>>>>>>>>>>>>>>> you revert your changes?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> About the mkdir error, I had a similar one but I wasn't
>>>>>>>>>>>>>>>>>>>> able to fix it yet. Do you have shared storage for your instances?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I never used virsh directly because, AFAIK, openstack
>>>>>>>>>>>>>>>>>>>> will not be informed of the changes and will not work correctly (but I may
>>>>>>>>>>>>>>>>>>>> be wrong in this one).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I created the new key, but still getting error: here
>>>>>>>>>>>>>>>>>>>>> are the logs:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:51.097 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>> [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>> bed3c3be-31d6-42c3-9a7f-e40ef9c1c764] Unexpected error while running
>>>>>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>>>>>>  Stderr: 'Host key verification failed.\r\n'. Setting
>>>>>>>>>>>>>>>>>>>>> instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 ERROR
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-c3441bfc-dcd7-4625-b701-716f152a4bfe
>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
>>>>>>>>>>>>>>>>>>>>> resize_instance
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     block_device_info)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
>>>>>>>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     inst_base_resize)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
>>>>>>>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     utils.execute('ssh', dest, 'mkdir',
>>>>>>>>>>>>>>>>>>>>> '-p', inst_base)
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error
>>>>>>>>>>>>>>>>>>>>> while running command.
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/instances/bed3c3be-31d6-42c3-9a7f-e40ef9c1c764
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Exit code: 255
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stdout: ''
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I've been trying to migrate directly with virsh, but I
>>>>>>>>>>>>>>>>>>>>> got this error and the solutions through the web don't help.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> virsh migrate --live instance-00000063 qemu+ssh://
>>>>>>>>>>>>>>>>>>>>> 192.168.3.3/system
>>>>>>>>>>>>>>>>>>>>> root at 192.168.3.3's password:
>>>>>>>>>>>>>>>>>>>>> erro: internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>> 'libvirt-bed3c3be-31d6-42c3-9a7f-e40ef9c1c764'
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thank you all.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/7/29 <yongiman at gmail.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Guilherme,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I suugest making new ssh key and put the public key
>>>>>>>>>>>>>>>>>>>>>> in the host
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> # ssh-keygen (you will find id_rsa.pub in your .ssh
>>>>>>>>>>>>>>>>>>>>>> directory)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Copy the contents of this file and paste at the last
>>>>>>>>>>>>>>>>>>>>>> line of authorized_keys in the host.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 2013. 7. 29., at 오후 10:40, Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hello Filipe, I created the
>>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys and its content is the keys from the
>>>>>>>>>>>>>>>>>>>>>> know_hosts, is it right? I'm still getting the same error when I try to
>>>>>>>>>>>>>>>>>>>>>> migrate a VM.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> About the keys it's up to you. You can use the same
>>>>>>>>>>>>>>>>>>>>>>> or create a new one. Using the same is easier.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> But the nova user public key should go into
>>>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys not into the know_hosts file. The easier
>>>>>>>>>>>>>>>>>>>>>>> way (as you only have two compute nodes) to add the keys to the know_hosts
>>>>>>>>>>>>>>>>>>>>>>> is to manually ssh and accept the key.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Ok, Must I create another ssh key to communicate
>>>>>>>>>>>>>>>>>>>>>>>> both compute nodes or the current key can work? I've put the plublic key
>>>>>>>>>>>>>>>>>>>>>>>> from both compute node in the know_hosts. Which permission should I apply?
>>>>>>>>>>>>>>>>>>>>>>>> 644?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> First you need to check why this "Failed to add
>>>>>>>>>>>>>>>>>>>>>>>>> the host to the list of known hosts (/var/lib/nova/.ssh/known_hosts)."
>>>>>>>>>>>>>>>>>>>>>>>>> happens. It maybe file permissions or so. Then you need to generate an ssh
>>>>>>>>>>>>>>>>>>>>>>>>> key for the user nova and add it to the other compute nodes on the
>>>>>>>>>>>>>>>>>>>>>>>>> authorized_keys file (the password is not really necessary as soon as you
>>>>>>>>>>>>>>>>>>>>>>>>> have the keys in place).
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> sudo -u nova ssh 192.168.3.3
>>>>>>>>>>>>>>>>>>>>>>>>>> The authenticity of host '192.168.3.3
>>>>>>>>>>>>>>>>>>>>>>>>>> (192.168.3.3)' can't be established.
>>>>>>>>>>>>>>>>>>>>>>>>>> ECDSA key fingerprint is
>>>>>>>>>>>>>>>>>>>>>>>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to continue connecting
>>>>>>>>>>>>>>>>>>>>>>>>>> (yes/no)? yes
>>>>>>>>>>>>>>>>>>>>>>>>>> Failed to add the host to the list of known hosts
>>>>>>>>>>>>>>>>>>>>>>>>>> (/var/lib/nova/.ssh/known_hosts).
>>>>>>>>>>>>>>>>>>>>>>>>>> nova at 192.168.3.3's password:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> But I don't know the user nova's password.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try *sudo -u nova ssh
>>>>>>>>>>>>>>>>>>>>>>>>>>> <remote-compute-node>* form the compute node
>>>>>>>>>>>>>>>>>>>>>>>>>>> currently running the VM and check if it works?
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Filipe,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>  I made the copy from my .ssh/know_hosts to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> places you told me, but I'm still getting the same error. Any other ideas?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This error means that you cannot ssh into the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new compute node because it's public key cannot be verified. Add the public
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> keys of the compute nodes to /etc/ssh/know_hosts file (or the nova one,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> usually under /var/lib/nova/.ssh/) and the problem should be gone.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ok, I'll look more carefully the log and it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cool this command from libvirt, I didn't know it, I'll try it and return to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you again.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <hzwangpan at corp.netease.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That's confused me, "Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command" will give us what command is running when this error
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> occured, you may check the log more carefully.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and you also can migrate the VM directly by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> libvirt command line like below:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> virsh  migrate --live --persistent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --undefinesource --verbose --domain ${instance-uuid} --desturi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> qemu+tcp://${the-dest-host-ip}/system
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then see the output result.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人:*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间:*2013-07-26 09:30
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题:*Re: Re: Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人:*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送:*"openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  When I enter the action to migrate, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only two errors showen at this period are these two.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There may have more error logs at this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> minute 2013-07-25 17:28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you can get the backtrace near the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> error log you post.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人:*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间:*2013-07-26 04:48
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题:*Re: Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人:*"Wangpan"<hzwangpan at corp.netease.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >,"openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送:*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Hello guys, I'm getting problems when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to migrate a VM.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm getting this error on my log:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  2013-07-25 17:28:31.582 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed.\r\n'. Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 17:28:32.212 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The user Wangpan was helping me, but I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guess his hour is different from mine, can anybody five me an idea?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Wangpan, I've looking at the logs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and I saw one error with my ssh port, and then I fixed the port, but now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm getting this error:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  2013-07-25 14:09:50.388 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed.\r\n'. Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 14:09:51.005 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What "host key verification"? Where should
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I put this key?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Typed wrong here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Em 25/07/2013 00:42, "Wangpan" <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chmod 777 -R /var/lib/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not the /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人:*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间:*2013-07-25 10:39
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题:*Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人:*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送:*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  I've done chmod 777 -R
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does it work?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/24 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I saw this error "Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/'" in your logs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you may need to chown the instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> path  '/var/lib/nova/instances/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人:*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间:*2013-07-25 10:08
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题:*Re: Re: [Openstack] Configuring
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人:*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送:*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Compute Node 01:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  cat /var/log/nova/nova-*.log | grep
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -E ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.326 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.511 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.673 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.837 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.402 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.530 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.657 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.036 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.170 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n    if not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.791 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.925 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:07.567 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:34:02.320 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:23.618 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.087 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.220 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.224 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.225 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.636 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.729 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n    if not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:05.364 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:26.986 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:39.249 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:54.459 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:47.486 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:58.434 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:46:43.408 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:26:43.627 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:46.003 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:55.194 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:53:42.789 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:54:05.061 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:05:44.677 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:06:14.408 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  cat /var/log/nova/nova-*.log | grep
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -E TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1510, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info=block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2326, in to_xml
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_info, rescue, block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2244, in get_guest_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inst_type):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2051, in get_guest_storage_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inst_type)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013, in get_guest_disk_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.disk_cachemode,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 355, in disk_cachemode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if not self._supports_direct_io(CONF.instances_path):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1676, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> {'path': dirpath, 'ex': str(e)})
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1663, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     f
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     do_run_instance()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 242, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do_run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     admin_password, is_first_time, node,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self._set_instance_error_state(context,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance['uuid'])
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     filter_properties, bdms)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ...
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [Mensagem reduzida]
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Mailing list:
>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>>> Post to     : openstack at lists.openstack.org
>>>>>>>>>>>> Unsubscribe :
>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/863e4965/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NUAGECO-LOGO-Fblan_petit.jpg
Type: image/jpeg
Size: 10122 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/863e4965/attachment.jpg>


More information about the Openstack mailing list