[Openstack] Configuring Live Migration
Guilherme Russi
luisguilherme.cr at gmail.com
Wed Jul 31 17:32:33 UTC 2013
Now the error has changed
cat /var/log/nova/nova-compute.log | grep -E ERROR
2013-07-31 14:30:04.759 ERROR nova.compute.manager
[req-2b1fca53-da5b-4bf5-ba44-9730e2c23a01 c402785616534f2096b34ce132b7d3f2
d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
1a61c59c-942b-446e-9635-c97b0516e1d0] Unexpected error while running
command.
Stderr: 'Failed to add the host to the list of known hosts
(/var/lib/nova/.ssh/known_hosts).\r\nPermission denied, please try
again.\r\nPermission denied, please try again.\r\nPermission denied
(publickey,password).\r\n'. Setting instance vm_state to ERROR
2013-07-31 14:30:05.536 ERROR nova.openstack.common.rpc.amqp
[req-2b1fca53-da5b-4bf5-ba44-9730e2c23a01 c402785616534f2096b34ce132b7d3f2
d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
cat /var/log/nova/nova-compute.log | grep -E ERROR
2013-07-31 11:18:49.606 ERROR nova.compute.manager
[req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
96afb354-a6f8-49a8-9136-3fb2ef8bd321] Unexpected error while running
command.
Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to
ERROR
2013-07-31 11:18:50.245 ERROR nova.openstack.common.rpc.amqp
[req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
2013-07-31 11:50:47.134 ERROR nova.compute.manager
[req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
0c729d3b-4e5a-45b8-9e68-1382b321d77e] Unexpected error while running
command.
Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to
ERROR
2013-07-31 11:50:47.909 ERROR nova.openstack.common.rpc.amqp
[req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
2013-07-31 14:30:04.759 ERROR nova.compute.manager
[req-2b1fca53-da5b-4bf5-ba44-9730e2c23a01 c402785616534f2096b34ce132b7d3f2
d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
1a61c59c-942b-446e-9635-c97b0516e1d0] Unexpected error while running
command.
Stderr: 'Failed to add the host to the list of known hosts
(/var/lib/nova/.ssh/known_hosts).\r\nPermission denied, please try
again.\r\nPermission denied, please try again.\r\nPermission denied
(publickey,password).\r\n'. Setting instance vm_state to ERROR
2013-07-31 14:30:05.536 ERROR nova.openstack.common.rpc.amqp
[req-2b1fca53-da5b-4bf5-ba44-9730e2c23a01 c402785616534f2096b34ce132b7d3f2
d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
root at caos:~# cat /var/log/nova/nova-compute.log | grep -E TRACE
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
Traceback (most recent call last):
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
430, in _process_data
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp rval
= self.proxy.dispatch(ctxt, version, method, **args)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 133, in dispatch
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
return getattr(proxyobj, method)(ctxt, **kwargs)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
temp_level, payload)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
return f(self, context, *args, **kw)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
decorated_function
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp pass
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
decorated_function
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
decorated_function
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
function(self, context, *args, **kwargs)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
decorated_function
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp e,
sys.exc_info())
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
decorated_function
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
resize_instance
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
block_device_info)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
in migrate_disk_and_power_off
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
inst_base_resize)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
in migrate_disk_and_power_off
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
cmd=' '.join(cmd))
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
ProcessExecutionError: Unexpected error while running command.
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp Command:
ssh 10.3.77.48 mkdir -p
/var/lib/nova/instances/96afb354-a6f8-49a8-9136-3fb2ef8bd321
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp Exit
code: 255
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp Stdout:
''
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp Stderr:
'Host key verification failed.\r\n'
2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
Traceback (most recent call last):
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
430, in _process_data
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp rval
= self.proxy.dispatch(ctxt, version, method, **args)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 133, in dispatch
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
return getattr(proxyobj, method)(ctxt, **kwargs)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
temp_level, payload)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
return f(self, context, *args, **kw)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
decorated_function
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp pass
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
decorated_function
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
decorated_function
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
function(self, context, *args, **kwargs)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
decorated_function
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp e,
sys.exc_info())
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
decorated_function
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
resize_instance
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
block_device_info)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
in migrate_disk_and_power_off
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
inst_base_resize)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
in migrate_disk_and_power_off
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
cmd=' '.join(cmd))
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
ProcessExecutionError: Unexpected error while running command.
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp Command:
ssh 10.3.77.48 mkdir -p
/var/lib/nova/instances/0c729d3b-4e5a-45b8-9e68-1382b321d77e
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp Exit
code: 255
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp Stdout:
''
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp Stderr:
'Host key verification failed.\r\n'
2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
Traceback (most recent call last):
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
430, in _process_data
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp rval
= self.proxy.dispatch(ctxt, version, method, **args)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
line 133, in dispatch
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
return getattr(proxyobj, method)(ctxt, **kwargs)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
temp_level, payload)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
return f(self, context, *args, **kw)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
decorated_function
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp pass
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
decorated_function
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
decorated_function
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
function(self, context, *args, **kwargs)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
decorated_function
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp e,
sys.exc_info())
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
decorated_function
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
return function(self, context, *args, **kwargs)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
resize_instance
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
block_device_info)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
in migrate_disk_and_power_off
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
inst_base_resize)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
self.gen.next()
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
in migrate_disk_and_power_off
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp File
"/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
cmd=' '.join(cmd))
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
ProcessExecutionError: Unexpected error while running command.
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp Command:
ssh 10.3.77.48 mkdir -p
/var/lib/nova/instances/1a61c59c-942b-446e-9635-c97b0516e1d0
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp Exit
code: 255
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp Stdout:
''
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp Stderr:
'Failed to add the host to the list of known hosts
(/var/lib/nova/.ssh/known_hosts).\r\nPermission denied, please try
again.\r\nPermission denied, please try again.\r\nPermission denied
(publickey,password).\r\n'
2013-07-31 14:30:05.536 19528 TRACE nova.openstack.common.rpc.amqp
2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
> Just to check: chown nova /var/lib/nova/.ssh/config?
>
> Thank you :)
>
>
> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>
>> ok make sure the file 'config' belongs to the user nova and it'll be good
>> :)
>>
>> *Razique Mahroua** - **Nuage & Co*
>> razique.mahroua at gmail.com
>> Tel : +33 9 72 37 94 15
>>
>>
>> Le 31 juil. 2013 à 18:11, Guilherme Russi <luisguilherme.cr at gmail.com> a
>> écrit :
>>
>> here is what happened:
>>
>> ssh nova at 10.3.77.48
>> The authenticity of host '10.3.77.48 (10.3.77.48)' can't be established.
>> ECDSA key fingerprint is e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>> Are you sure you want to continue connecting (yes/no)? yes
>> Failed to add the host to the list of known hosts
>> (/var/lib/nova/.ssh/known_hosts).
>> nova at 10.3.77.48's password:
>>
>> I don't remember if I set up a password for nova user, I don't know which
>> is it.
>>
>> Regards.
>>
>>
>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>
>>> ok can you update the file /etc/passwd
>>> and use /bin/sh as a shell for the nova user
>>> then log as nova :
>>> # su - nova
>>>
>>> after that, try to ssh into the other node :
>>> # ssh nova at 10.3.77.48
>>>
>>> You shouldn't be asked for anything, you should just be connected to the
>>> host
>>>
>>>
>>> Le 31 juil. 2013 à 17:07, Guilherme Russi <luisguilherme.cr at gmail.com>
>>> a écrit :
>>>
>>> This is the nova-compute.log's content from my Compute node "Caos" where
>>> the VM is first created.
>>>
>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>> 2013-07-31 11:18:49.606 ERROR nova.compute.manager
>>> [req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>> 96afb354-a6f8-49a8-9136-3fb2ef8bd321] Unexpected error while running
>>> command.
>>> Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state
>>> to ERROR
>>> 2013-07-31 11:18:50.245 ERROR nova.openstack.common.rpc.amqp
>>> [req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>> 2013-07-31 11:50:47.134 ERROR nova.compute.manager
>>> [req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>> 0c729d3b-4e5a-45b8-9e68-1382b321d77e] Unexpected error while running
>>> command.
>>> Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state
>>> to ERROR
>>> 2013-07-31 11:50:47.909 ERROR nova.openstack.common.rpc.amqp
>>> [req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>
>>>
>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> Traceback (most recent call last):
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>> line 430, in _process_data
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File
>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>> line 133, in dispatch
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> return getattr(proxyobj, method)(ctxt, **kwargs)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>> wrapped
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> temp_level, payload)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>> wrapped
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> return f(self, context, *args, **kw)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>> in decorated_function
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> pass
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>> in decorated_function
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> return function(self, context, *args, **kwargs)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>> in decorated_function
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> function(self, context, *args, **kwargs)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>> in decorated_function
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> e, sys.exc_info())
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>> in decorated_function
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> return function(self, context, *args, **kwargs)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>> in resize_instance
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> block_device_info)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>> 3467, in migrate_disk_and_power_off
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> inst_base_resize)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>> 3444, in migrate_disk_and_power_off
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> cmd=' '.join(cmd))
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> ProcessExecutionError: Unexpected error while running command.
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> Command: ssh 10.3.77.48 mkdir -p
>>> /var/lib/nova/instances/96afb354-a6f8-49a8-9136-3fb2ef8bd321
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp Exit
>>> code: 255
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> Stdout: ''
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> Stderr: 'Host key verification failed.\r\n'
>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> Traceback (most recent call last):
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>> line 430, in _process_data
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File
>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>> line 133, in dispatch
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> return getattr(proxyobj, method)(ctxt, **kwargs)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>> wrapped
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> temp_level, payload)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>> wrapped
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> return f(self, context, *args, **kw)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>> in decorated_function
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> pass
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>> in decorated_function
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> return function(self, context, *args, **kwargs)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>> in decorated_function
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> function(self, context, *args, **kwargs)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>> in decorated_function
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> e, sys.exc_info())
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>> in decorated_function
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> return function(self, context, *args, **kwargs)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>> in resize_instance
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> block_device_info)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>> 3467, in migrate_disk_and_power_off
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> inst_base_resize)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>> 3444, in migrate_disk_and_power_off
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> cmd=' '.join(cmd))
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> ProcessExecutionError: Unexpected error while running command.
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> Command: ssh 10.3.77.48 mkdir -p
>>> /var/lib/nova/instances/0c729d3b-4e5a-45b8-9e68-1382b321d77e
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp Exit
>>> code: 255
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> Stdout: ''
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>> Stderr: 'Host key verification failed.\r\n'
>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>
>>>
>>>
>>>
>>>
>>> 2013/7/31 Filipe Manco <filipe.manco at gmail.com>
>>>
>>>> What's your error know when you try to migrate?
>>>>
>>>> Filipe Manco
>>>> http://about.me/fmanco
>>>>
>>>>
>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>
>>>>> I still don't know what to do, I've checked again my nova.conf from my
>>>>> compute nodes, checked the public key from authorized_keys, checked the
>>>>> folder /var/lib/nova/instances mapped from my controller to my compute
>>>>> nodes, and checked the uid/gid from nova and libvirt-qemu.
>>>>>
>>>>> Am I missing anything?
>>>>>
>>>>> Thank you all.
>>>>>
>>>>>
>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>
>>>>>> Yes, I was doing right so.
>>>>>>
>>>>>>
>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>
>>>>>>> Pub, always the *public key*, the private key is…well private and
>>>>>>> should NEVER be shared :)
>>>>>>>
>>>>>>> Le 31 juil. 2013 à 16:01, Guilherme Russi <
>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>
>>>>>>> Just to check, the file to copy and insert into the authorized_keys
>>>>>>> is from the id_rsa or id_rsa.pub?
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>
>>>>>>>> Just make a correction on my path, it is in /var/lib/nova/.ssh not
>>>>>>>> in /var/lib/.ssh like I sad before :)
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Le 31 juil. 2013 à 15:40, Guilherme Russi <
>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>
>>>>>>>>> Hello friends,
>>>>>>>>>
>>>>>>>>> First from JuanFra: I've alrealdy checked it, they are the same.
>>>>>>>>> Second from Razique: Could you give me a strictly explanation, I
>>>>>>>>> already have the folder in /var/lib/.ssh with the file authorized_keys
>>>>>>>>> inside. This file config must have only StrictHostKeyChecking no
>>>>>>>>> inside it?
>>>>>>>>>
>>>>>>>>> yup that prevents the user nova from being prompted to type "yes"
>>>>>>>>> if it's an unknown server
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thank you all.
>>>>>>>>>
>>>>>>>>> Guilherme.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>
>>>>>>>>>> Hey Guilherme :)
>>>>>>>>>> make sure :
>>>>>>>>>> • To exchange pubkey of nova users amongst all nodes
>>>>>>>>>> • Add in /var/lib a directory .ssh/ and create a "config" file
>>>>>>>>>> with into : *StrictHostKeyChecking* no
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Razique Mahroua** - **Nuage & Co*
>>>>>>>>>> razique.mahroua at gmail.com
>>>>>>>>>> Tel : +33 9 72 37 94 15
>>>>>>>>>>
>>>>>>>>>> <NUAGECO-LOGO-Fblan_petit.jpg>
>>>>>>>>>>
>>>>>>>>>> Le 30 juil. 2013 à 19:08, Guilherme Russi <
>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> Sure, thank you.
>>>>>>>>>>
>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>> 2013-07-30 13:48:08.074 ERROR nova.compute.manager
>>>>>>>>>> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>> 5061e7a6-2482-4af2-97cf-0074326fb940] Unexpected error while running
>>>>>>>>>> command.
>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>>>>>> vm_state to ERROR
>>>>>>>>>> 2013-07-30 13:48:08.747 ERROR nova.openstack.common.rpc.amqp
>>>>>>>>>> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>>>
>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>> 430, in _process_data
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File
>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>> line 133, in dispatch
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>>>>>>> wrapped
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> temp_level, payload)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> self.gen.next()
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>>>>>>> wrapped
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> return f(self, context, *args, **kw)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 209, in decorated_function
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> pass
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> self.gen.next()
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 195, in decorated_function
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> return function(self, context, *args, **kwargs)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 260, in decorated_function
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> function(self, context, *args, **kwargs)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 237, in decorated_function
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> e, sys.exc_info())
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> self.gen.next()
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 224, in decorated_function
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> return function(self, context, *args, **kwargs)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line
>>>>>>>>>> 2354, in resize_instance
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> block_device_info)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>> 3467, in migrate_disk_and_power_off
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> inst_base_resize)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> self.gen.next()
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>> 3444, in migrate_disk_and_power_off
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in
>>>>>>>>>> execute
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> cmd=' '.join(cmd))
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>> /var/lib/nova/instances/5061e7a6-2482-4af2-97cf-0074326fb940
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Exit code: 255
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Stdout: ''
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE nova.openstack.common.rpc.amqp
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Can you provide more information (errors and/or logs)?
>>>>>>>>>>>
>>>>>>>>>>> Filipe Manco
>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>> It didn't work :(
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Ok, testing it.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Basically edit the /etc/passwd and /etc/passwd- files and
>>>>>>>>>>>>>> change /bin/false to /bin/bash on the nova user entry.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm having another problem after this related to libvirt but
>>>>>>>>>>>>>> I have no time to fix it know. Basically the migration ends but then
>>>>>>>>>>>>>> libvirt isn't able to boot the machine again. Let me know about your
>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ok, on my scenario I have 2 Compute nodes and 1 Controller +
>>>>>>>>>>>>>>> Network node, how did you change your configs, I can try your idea.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> No I just wanted to know what is your scenario. About the
>>>>>>>>>>>>>>>> mkdir I spend a couple of time looking at it and managed to fix it by
>>>>>>>>>>>>>>>> changing the shell of the nova user to /bin/bash instead of /bin/false on
>>>>>>>>>>>>>>>> all the compute nodes. You can try it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hello Filipe, I followed the manual on the openstack
>>>>>>>>>>>>>>>>> documentation, I shared the /var/lib/nova/instances on my controller, do I
>>>>>>>>>>>>>>>>> need to share another folders?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> A couple of messages ago you shown an error where the
>>>>>>>>>>>>>>>>>> mkdir command failed. At that time the keys were properly configured. Can
>>>>>>>>>>>>>>>>>> you revert your changes?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> About the mkdir error, I had a similar one but I wasn't
>>>>>>>>>>>>>>>>>> able to fix it yet. Do you have shared storage for your instances?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I never used virsh directly because, AFAIK, openstack
>>>>>>>>>>>>>>>>>> will not be informed of the changes and will not work correctly (but I may
>>>>>>>>>>>>>>>>>> be wrong in this one).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I created the new key, but still getting error: here are
>>>>>>>>>>>>>>>>>>> the logs:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:51.097 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>> [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>> bed3c3be-31d6-42c3-9a7f-e40ef9c1c764] Unexpected error while running
>>>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting
>>>>>>>>>>>>>>>>>>> instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 ERROR
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-c3441bfc-dcd7-4625-b701-716f152a4bfe
>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp temp_level, payload)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp pass
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return function(self, context, *args,
>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp e, sys.exc_info())
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return function(self, context, *args,
>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
>>>>>>>>>>>>>>>>>>> resize_instance
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp block_device_info)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
>>>>>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp inst_base_resize)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
>>>>>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp utils.execute('ssh', dest, 'mkdir',
>>>>>>>>>>>>>>>>>>> '-p', inst_base)
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp cmd=' '.join(cmd))
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error
>>>>>>>>>>>>>>>>>>> while running command.
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>>>>>>>>>>> /var/lib/nova/instances/bed3c3be-31d6-42c3-9a7f-e40ef9c1c764
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Exit code: 255
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stdout: ''
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've been trying to migrate directly with virsh, but I
>>>>>>>>>>>>>>>>>>> got this error and the solutions through the web don't help.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> virsh migrate --live instance-00000063 qemu+ssh://
>>>>>>>>>>>>>>>>>>> 192.168.3.3/system
>>>>>>>>>>>>>>>>>>> root at 192.168.3.3's password:
>>>>>>>>>>>>>>>>>>> erro: internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>> 'libvirt-bed3c3be-31d6-42c3-9a7f-e40ef9c1c764'
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thank you all.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/7/29 <yongiman at gmail.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Guilherme,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I suugest making new ssh key and put the public key in
>>>>>>>>>>>>>>>>>>>> the host
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> # ssh-keygen (you will find id_rsa.pub in your .ssh
>>>>>>>>>>>>>>>>>>>> directory)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Copy the contents of this file and paste at the last
>>>>>>>>>>>>>>>>>>>> line of authorized_keys in the host.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 2013. 7. 29., at 오후 10:40, Guilherme Russi <
>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello Filipe, I created the
>>>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys and its content is the keys from the
>>>>>>>>>>>>>>>>>>>> know_hosts, is it right? I'm still getting the same error when I try to
>>>>>>>>>>>>>>>>>>>> migrate a VM.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> About the keys it's up to you. You can use the same or
>>>>>>>>>>>>>>>>>>>>> create a new one. Using the same is easier.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> But the nova user public key should go into
>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys not into the know_hosts file. The easier
>>>>>>>>>>>>>>>>>>>>> way (as you only have two compute nodes) to add the keys to the know_hosts
>>>>>>>>>>>>>>>>>>>>> is to manually ssh and accept the key.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Ok, Must I create another ssh key to communicate both
>>>>>>>>>>>>>>>>>>>>>> compute nodes or the current key can work? I've put the plublic key from
>>>>>>>>>>>>>>>>>>>>>> both compute node in the know_hosts. Which permission should I apply? 644?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> First you need to check why this "Failed to add the
>>>>>>>>>>>>>>>>>>>>>>> host to the list of known hosts (/var/lib/nova/.ssh/known_hosts)."
>>>>>>>>>>>>>>>>>>>>>>> happens. It maybe file permissions or so. Then you need to generate an ssh
>>>>>>>>>>>>>>>>>>>>>>> key for the user nova and add it to the other compute nodes on the
>>>>>>>>>>>>>>>>>>>>>>> authorized_keys file (the password is not really necessary as soon as you
>>>>>>>>>>>>>>>>>>>>>>> have the keys in place).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> sudo -u nova ssh 192.168.3.3
>>>>>>>>>>>>>>>>>>>>>>>> The authenticity of host '192.168.3.3
>>>>>>>>>>>>>>>>>>>>>>>> (192.168.3.3)' can't be established.
>>>>>>>>>>>>>>>>>>>>>>>> ECDSA key fingerprint is
>>>>>>>>>>>>>>>>>>>>>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to continue connecting
>>>>>>>>>>>>>>>>>>>>>>>> (yes/no)? yes
>>>>>>>>>>>>>>>>>>>>>>>> Failed to add the host to the list of known hosts
>>>>>>>>>>>>>>>>>>>>>>>> (/var/lib/nova/.ssh/known_hosts).
>>>>>>>>>>>>>>>>>>>>>>>> nova at 192.168.3.3's password:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> But I don't know the user nova's password.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Can you try *sudo -u nova ssh
>>>>>>>>>>>>>>>>>>>>>>>>> <remote-compute-node>* form the compute node
>>>>>>>>>>>>>>>>>>>>>>>>> currently running the VM and check if it works?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Filipe,
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I made the copy from my .ssh/know_hosts to
>>>>>>>>>>>>>>>>>>>>>>>>>> places you told me, but I'm still getting the same error. Any other ideas?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> This error means that you cannot ssh into the
>>>>>>>>>>>>>>>>>>>>>>>>>>> new compute node because it's public key cannot be verified. Add the public
>>>>>>>>>>>>>>>>>>>>>>>>>>> keys of the compute nodes to /etc/ssh/know_hosts file (or the nova one,
>>>>>>>>>>>>>>>>>>>>>>>>>>> usually under /var/lib/nova/.ssh/) and the problem should be gone.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ok, I'll look more carefully the log and it's
>>>>>>>>>>>>>>>>>>>>>>>>>>>> cool this command from libvirt, I didn't know it, I'll try it and return to
>>>>>>>>>>>>>>>>>>>>>>>>>>>> you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you again.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That's confused me, "Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command" will give us what command is running when this error
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> occured, you may check the log more carefully.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and you also can migrate the VM directly by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> libvirt command line like below:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> virsh migrate --live --persistent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --undefinesource --verbose --domain ${instance-uuid} --desturi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> qemu+tcp://${the-dest-host-ip}/system
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then see the output result.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-26 09:30
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*"openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> When I enter the action to migrate, the only
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> two errors showen at this period are these two.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There may have more error logs at this minute
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 17:28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you can get the backtrace near the error log
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you post.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-26 04:48
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: Re: [Openstack] Configuring
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com>,"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello guys, I'm getting problems when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to migrate a VM.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm getting this error on my log:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 17:28:31.582 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 17:28:32.212 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The user Wangpan was helping me, but I guess
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> his hour is different from mine, can anybody five me an idea?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Wangpan, I've looking at the logs and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I saw one error with my ssh port, and then I fixed the port, but now I'm
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> getting this error:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 14:09:50.388 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 14:09:51.005 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What "host key verification"? Where should I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> put this key?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Typed wrong here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Em 25/07/2013 00:42, "Wangpan" <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chmod 777 -R /var/lib/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not the /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-25 10:39
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: [Openstack] Configuring
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I've done chmod 777 -R
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does it work?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/24 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I saw this error "Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/'" in your logs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you may need to chown the instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> path '/var/lib/nova/instances/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-25 10:08
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: [Openstack] Configuring
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Compute Node 01:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-*.log | grep -E
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.326 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.511 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.673 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.837 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.402 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.530 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.657 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.036 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.170 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info=block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n disk_info, rescue, block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n inst_type):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n inst_type)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n self.disk_cachemode,\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n if not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.791 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.925 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:07.567 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:34:02.320 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:23.618 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.087 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.220 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n admin_pass=admin_password)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n os.makedirs(path)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.224 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.225 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.636 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.729 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info=block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n disk_info, rescue, block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n inst_type):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n inst_type)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n self.disk_cachemode,\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n if not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:05.364 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:26.986 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:39.249 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:54.459 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:47.486 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:58.434 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:46:43.408 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:26:43.627 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:46.003 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:55.194 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:53:42.789 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:54:05.061 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:05:44.677 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:06:14.408 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n set_access_ip=set_access_ip)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n self.gen.next()\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n block_device_info)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> rv = meth(*args,**kwargs)\n', ' File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-*.log | grep -E
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1510, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info=block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2326, in to_xml
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_info, rescue, block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2244, in get_guest_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inst_type):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2051, in get_guest_storage_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inst_type)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013, in get_guest_disk_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.disk_cachemode,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 355, in disk_cachemode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if not self._supports_direct_io(CONF.instances_path):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1676, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> {'path': dirpath, 'ex': str(e)})
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1663, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] f
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp temp_level, payload)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp do_run_instance()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 242, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do_run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp admin_password, is_first_time, node,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self._set_instance_error_state(context,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance['uuid'])
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp filter_properties, bdms)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [Mensagem reduzida]
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Mailing list:
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>> Post to : openstack at lists.openstack.org
>>>>>>>>>> Unsubscribe :
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/f84b8a9e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NUAGECO-LOGO-Fblan_petit.jpg
Type: image/jpeg
Size: 10122 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/f84b8a9e/attachment.jpg>
More information about the Openstack
mailing list
