Open Stack

Answers inline.

Filipe Manco
http://about.me/fmanco


2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>

> Right, just to check, is it known_hosts or know_hosts like you write?
>

known_hosts you're right


> And another thing, I made the test sudo -u nova ssh my_compute_node and it
> stops in a password screen, is it right?
>

No. It should be able to ssh without password. Check all files have
permissions 700 and belong to nova.

And the last one, after making chown -R nova /var/lib/nova/ I'm getting
> error to create new VMs.
>

Do you follow the guide about changing nova ids? Make sure that /etc/passwd
and /etc/passwd- (notice the -) match. The same for /etc/group and
/etc/group- or reboot your compute nodes.

And don't forget to restart all your services including libvirt-bin.


>
> Thank you again.
>
>
> 2013/7/31 Filipe Manco <filipe.manco at gmail.com>
>
>> This is getting a little bit confusing so here are a recipe:
>>
>> 1. Delete everythin under */var/lib/nova/.ssh/*
>> 2. Run *ssh-keygen -N '' -f /var/lib/nova/.ssh/id_rsa*
>> 3. Run *cp **/var/lib/nova/.ssh/id_rsa.pub
>> /var/lib/nova/.ssh/authorized_keys
>> *4. For every compute node execute *ssh-keyscan
>> [node_fqdn],[node_ip_address] >> /var/lib/nova/.ssh/know_hosts*
>> 5. Copy the contents of */var/lib/nova/.ssh* to all your compute nodes
>>
>> Test it by running *sudo -u nova ssh [compute_node]*
>>
>> Filipe Manco
>> http://about.me/fmanco
>>
>>
>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>
>>> I did the chown -R nova /var/lib/nova/ but still the error, any idea? I
>>> put the chown permission in my two compute nodes and in my controller node.
>>>
>>> 2013-07-31 15:40:55.720 ERROR nova.compute.manager
>>> [req-86d4ddd9-8c85-4ee7-ba0a-d83e81f26818 c402785616534f2096b34ce132b7d3f2
>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>> f4af662d-ae55-4346-82cf-a997192706b5] Unexpected error while running
>>> command.
>>> Stderr: 'Permission denied, please try again.\r\nPermission denied,
>>> please try again.\r\nPermission denied (publickey,password).\r\n'. Setting
>>> instance vm_state to ERROR
>>> 2013-07-31 15:40:56.298 ERROR nova.openstack.common.rpc.amqp
>>> [req-86d4ddd9-8c85-4ee7-ba0a-d83e81f26818 c402785616534f2096b34ce132b7d3f2
>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>
>>>
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> Traceback (most recent call last):
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>> line 430, in _process_data
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> rval = self.proxy.dispatch(ctxt, version, method, **args)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File
>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>> line 133, in dispatch
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> return getattr(proxyobj, method)(ctxt, **kwargs)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>> wrapped
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> temp_level, payload)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>> wrapped
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> return f(self, context, *args, **kw)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>> in decorated_function
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> pass
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>> in decorated_function
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> return function(self, context, *args, **kwargs)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>> in decorated_function
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> function(self, context, *args, **kwargs)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>> in decorated_function
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> e, sys.exc_info())
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>> in decorated_function
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> return function(self, context, *args, **kwargs)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>> in resize_instance
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> block_device_info)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>> 3467, in migrate_disk_and_power_off
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> inst_base_resize)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> self.gen.next()
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>> 3444, in migrate_disk_and_power_off
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> cmd=' '.join(cmd))
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> ProcessExecutionError: Unexpected error while running command.
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> Command: ssh 10.3.77.48 mkdir -p
>>> /var/lib/nova/instances/f4af662d-ae55-4346-82cf-a997192706b5
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp Exit
>>> code: 255
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> Stdout: ''
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>> Stderr: 'Permission denied, please try again.\r\nPermission denied, please
>>> try again.\r\nPermission denied (publickey,password).\r\n'
>>> 2013-07-31 15:40:56.298 19528 TRACE nova.openstack.common.rpc.amqp
>>>
>>>
>>>
>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>
>>>> chown -r nova. /var/lib/nova
>>>> :)
>>>>
>>>> *Razique Mahroua** - **Nuage & Co*
>>>> razique.mahroua at gmail.com
>>>> Tel : +33 9 72 37 94 15
>>>>
>>>>
>>>> Le 31 juil. 2013 à 19:36, Guilherme Russi <luisguilherme.cr at gmail.com>
>>>> a écrit :
>>>>
>>>> My config file is like that:
>>>>
>>>> ls -la /var/lib/nova/.ssh/config
>>>> -rw-r--r-- 1 nova root 25 Jul 31 10:49 /var/lib/nova/.ssh/config
>>>>
>>>>
>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>
>>>>> ok make sure the file 'config' belongs to the user nova and it'll be
>>>>> good :)
>>>>>
>>>>> *Razique Mahroua** - **Nuage & Co*
>>>>> razique.mahroua at gmail.com
>>>>> Tel : +33 9 72 37 94 15
>>>>>
>>>>> <NUAGECO-LOGO-Fblan_petit.jpg>
>>>>>
>>>>> Le 31 juil. 2013 à 18:11, Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>> a écrit :
>>>>>
>>>>> here is what happened:
>>>>>
>>>>>  ssh nova at 10.3.77.48
>>>>> The authenticity of host '10.3.77.48 (10.3.77.48)' can't be
>>>>> established.
>>>>> ECDSA key fingerprint is
>>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>>> Are you sure you want to continue connecting (yes/no)? yes
>>>>> Failed to add the host to the list of known hosts
>>>>> (/var/lib/nova/.ssh/known_hosts).
>>>>> nova at 10.3.77.48's password:
>>>>>
>>>>> I don't remember if I set up a password for nova user, I don't know
>>>>> which is it.
>>>>>
>>>>> Regards.
>>>>>
>>>>>
>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>
>>>>>> ok can you update the file /etc/passwd
>>>>>> and use /bin/sh as a shell for the nova user
>>>>>> then log as nova :
>>>>>> # su - nova
>>>>>>
>>>>>> after that, try to ssh into the other node :
>>>>>> # ssh nova at 10.3.77.48
>>>>>>
>>>>>> You shouldn't be asked for anything, you should just be connected to
>>>>>> the host
>>>>>>
>>>>>>
>>>>>> Le 31 juil. 2013 à 17:07, Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>> a écrit :
>>>>>>
>>>>>> This is the nova-compute.log's content from my Compute node "Caos"
>>>>>> where the VM is first created.
>>>>>>
>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>> 2013-07-31 11:18:49.606 ERROR nova.compute.manager
>>>>>> [req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>> 96afb354-a6f8-49a8-9136-3fb2ef8bd321] Unexpected error while running
>>>>>> command.
>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>> vm_state to ERROR
>>>>>> 2013-07-31 11:18:50.245 ERROR nova.openstack.common.rpc.amqp
>>>>>> [req-017b4047-9ec0-4412-9f9b-6660625f2221 c402785616534f2096b34ce132b7d3f2
>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>> 2013-07-31 11:50:47.134 ERROR nova.compute.manager
>>>>>> [req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>> 0c729d3b-4e5a-45b8-9e68-1382b321d77e] Unexpected error while running
>>>>>> command.
>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>> vm_state to ERROR
>>>>>> 2013-07-31 11:50:47.909 ERROR nova.openstack.common.rpc.amqp
>>>>>> [req-91960dc0-9f6d-44cf-8abb-5b23c95f9850 c402785616534f2096b34ce132b7d3f2
>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>
>>>>>>
>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Traceback (most recent call last):
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>>>> line 430, in _process_data
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File
>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>> line 133, in dispatch
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>>> wrapped
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   temp_level, payload)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   self.gen.next()
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>>> wrapped
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   return f(self, context, *args, **kw)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   pass
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   self.gen.next()
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   return function(self, context, *args, **kwargs)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   function(self, context, *args, **kwargs)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   e, sys.exc_info())
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   self.gen.next()
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   return function(self, context, *args, **kwargs)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>>>>> in resize_instance
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   block_device_info)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>> 3467, in migrate_disk_and_power_off
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   inst_base_resize)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   self.gen.next()
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>> 3444, in migrate_disk_and_power_off
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   cmd=' '.join(cmd))
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>>> /var/lib/nova/instances/96afb354-a6f8-49a8-9136-3fb2ef8bd321
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Exit code: 255
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Stdout: ''
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>>> 2013-07-31 11:18:50.245 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Traceback (most recent call last):
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py",
>>>>>> line 430, in _process_data
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   rval = self.proxy.dispatch(ctxt, version, method, **args)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File
>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>> line 133, in dispatch
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   return getattr(proxyobj, method)(ctxt, **kwargs)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in
>>>>>> wrapped
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   temp_level, payload)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   self.gen.next()
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in
>>>>>> wrapped
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   return f(self, context, *args, **kw)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   pass
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   self.gen.next()
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   return function(self, context, *args, **kwargs)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   function(self, context, *args, **kwargs)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   e, sys.exc_info())
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   self.gen.next()
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224,
>>>>>> in decorated_function
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   return function(self, context, *args, **kwargs)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354,
>>>>>> in resize_instance
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   block_device_info)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>> 3467, in migrate_disk_and_power_off
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   inst_base_resize)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   self.gen.next()
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>> 3444, in migrate_disk_and_power_off
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>   cmd=' '.join(cmd))
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> ProcessExecutionError: Unexpected error while running command.
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Command: ssh 10.3.77.48 mkdir -p
>>>>>> /var/lib/nova/instances/0c729d3b-4e5a-45b8-9e68-1382b321d77e
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Exit code: 255
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Stdout: ''
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>> Stderr: 'Host key verification failed.\r\n'
>>>>>> 2013-07-31 11:50:47.909 19528 TRACE nova.openstack.common.rpc.amqp
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/7/31 Filipe Manco <filipe.manco at gmail.com>
>>>>>>
>>>>>>> What's your error know when you try to migrate?
>>>>>>>
>>>>>>>  Filipe Manco
>>>>>>> http://about.me/fmanco
>>>>>>>
>>>>>>>
>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>
>>>>>>>> I still don't know what to do, I've checked again my nova.conf from
>>>>>>>> my compute nodes, checked the public key from authorized_keys, checked the
>>>>>>>> folder /var/lib/nova/instances mapped from my controller to my compute
>>>>>>>> nodes, and checked the uid/gid from nova and libvirt-qemu.
>>>>>>>>
>>>>>>>> Am I missing anything?
>>>>>>>>
>>>>>>>> Thank you all.
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>
>>>>>>>>> Yes, I was doing right so.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>
>>>>>>>>>> Pub, always the *public key*, the private key is…well private
>>>>>>>>>> and should NEVER be shared :)
>>>>>>>>>>
>>>>>>>>>> Le 31 juil. 2013 à 16:01, Guilherme Russi <
>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>
>>>>>>>>>> Just to check, the file to copy and insert into the
>>>>>>>>>> authorized_keys is from the id_rsa or id_rsa.pub?
>>>>>>>>>>
>>>>>>>>>> Thank you.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/7/31 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Just make a correction on my path, it is in /var/lib/nova/.ssh
>>>>>>>>>>> not in /var/lib/.ssh like I sad before :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Le 31 juil. 2013 à 15:40, Guilherme Russi <
>>>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>>>
>>>>>>>>>>>> Hello friends,
>>>>>>>>>>>>
>>>>>>>>>>>>  First from JuanFra: I've alrealdy checked it, they are the
>>>>>>>>>>>> same.
>>>>>>>>>>>>  Second from Razique: Could you give me a strictly explanation,
>>>>>>>>>>>> I already have the folder in /var/lib/.ssh with the file authorized_keys
>>>>>>>>>>>> inside. This file config  must have only StrictHostKeyChecking no
>>>>>>>>>>>> inside it?
>>>>>>>>>>>>
>>>>>>>>>>>> yup that prevents the user nova from being prompted to type
>>>>>>>>>>>> "yes" if it's an unknown server
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you all.
>>>>>>>>>>>>
>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2013/7/31 Razique Mahroua <razique.mahroua at gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey Guilherme :)
>>>>>>>>>>>>> make sure :
>>>>>>>>>>>>> • To exchange pubkey of nova users amongst all nodes
>>>>>>>>>>>>> • Add in /var/lib a directory .ssh/ and create a "config" file
>>>>>>>>>>>>> with into : *StrictHostKeyChecking* no
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> *Razique Mahroua** - **Nuage & Co*
>>>>>>>>>>>>> razique.mahroua at gmail.com
>>>>>>>>>>>>> Tel : +33 9 72 37 94 15
>>>>>>>>>>>>>
>>>>>>>>>>>>> <NUAGECO-LOGO-Fblan_petit.jpg>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Le 30 juil. 2013 à 19:08, Guilherme Russi <
>>>>>>>>>>>>> luisguilherme.cr at gmail.com> a écrit :
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sure, thank you.
>>>>>>>>>>>>>
>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>>>>> 2013-07-30 13:48:08.074 ERROR nova.compute.manager
>>>>>>>>>>>>> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>> 5061e7a6-2482-4af2-97cf-0074326fb940] Unexpected error while running
>>>>>>>>>>>>> command.
>>>>>>>>>>>>> Stderr: 'Host key verification failed.\r\n'. Setting instance
>>>>>>>>>>>>> vm_state to ERROR
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 ERROR nova.openstack.common.rpc.amqp
>>>>>>>>>>>>> [req-bb59c9e8-8ef3-4461-8a51-01166ccc758d c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
>>>>>>>>>>>>>
>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
>>>>>>>>>>>>> resize_instance
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     block_device_info)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     inst_base_resize)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     utils.execute('ssh', dest, 'mkdir',
>>>>>>>>>>>>> '-p', inst_base)
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error
>>>>>>>>>>>>> while running command.
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>>>>> /var/lib/nova/instances/5061e7a6-2482-4af2-97cf-0074326fb940
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Exit code: 255
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stdout: ''
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>>>>> 2013-07-30 13:48:08.747 6254 TRACE
>>>>>>>>>>>>> nova.openstack.common.rpc.amqp
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can you provide more information (errors and/or logs)?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> It didn't work :(
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ok, testing it.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Basically edit the /etc/passwd and /etc/passwd- files and
>>>>>>>>>>>>>>>>> change /bin/false to /bin/bash on the nova user entry.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I'm having another problem after this related to libvirt
>>>>>>>>>>>>>>>>> but I have no time to fix it know. Basically the migration ends but then
>>>>>>>>>>>>>>>>> libvirt isn't able to boot the machine again. Let me know about your
>>>>>>>>>>>>>>>>> results.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ok, on my scenario I have 2 Compute nodes and 1
>>>>>>>>>>>>>>>>>> Controller + Network node, how did you change your configs, I can try your
>>>>>>>>>>>>>>>>>> idea.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> No I just wanted to know what is your scenario. About
>>>>>>>>>>>>>>>>>>> the mkdir I spend a couple of time looking at it and managed to fix it by
>>>>>>>>>>>>>>>>>>> changing the shell of the nova user to /bin/bash instead of /bin/false on
>>>>>>>>>>>>>>>>>>> all the compute nodes. You can try it.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hello Filipe, I followed the manual on the openstack
>>>>>>>>>>>>>>>>>>>> documentation, I shared the /var/lib/nova/instances on my controller, do I
>>>>>>>>>>>>>>>>>>>> need to share another folders?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 2013/7/30 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> A couple of messages ago you shown an error where the
>>>>>>>>>>>>>>>>>>>>> mkdir command failed. At that time the keys were properly configured. Can
>>>>>>>>>>>>>>>>>>>>> you revert your changes?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> About the mkdir error, I had a similar one but I
>>>>>>>>>>>>>>>>>>>>> wasn't able to fix it yet. Do you have shared storage for your instances?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I never used virsh directly because, AFAIK, openstack
>>>>>>>>>>>>>>>>>>>>> will not be informed of the changes and will not work correctly (but I may
>>>>>>>>>>>>>>>>>>>>> be wrong in this one).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 2013/7/30 Guilherme Russi <luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I created the new key, but still getting error: here
>>>>>>>>>>>>>>>>>>>>>> are the logs:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E ERROR
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:51.097 ERROR nova.compute.manager
>>>>>>>>>>>>>>>>>>>>>> [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2
>>>>>>>>>>>>>>>>>>>>>> d532a4fc2e9e4b5f83b6dec7085237e5] [instance:
>>>>>>>>>>>>>>>>>>>>>> bed3c3be-31d6-42c3-9a7f-e40ef9c1c764] Unexpected error while running
>>>>>>>>>>>>>>>>>>>>>> command.
>>>>>>>>>>>>>>>>>>>>>>  Stderr: 'Host key verification failed.\r\n'.
>>>>>>>>>>>>>>>>>>>>>> Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 ERROR
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-c3441bfc-dcd7-4625-b701-716f152a4bfe
>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> cat /var/log/nova/nova-compute.log | grep -E TRACE
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in
>>>>>>>>>>>>>>>>>>>>>> resize_instance
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     block_device_info)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467,
>>>>>>>>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     inst_base_resize)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444,
>>>>>>>>>>>>>>>>>>>>>> in migrate_disk_and_power_off
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     utils.execute('ssh', dest, 'mkdir',
>>>>>>>>>>>>>>>>>>>>>> '-p', inst_base)
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error
>>>>>>>>>>>>>>>>>>>>>> while running command.
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Command: ssh 10.3.77.48 mkdir -p
>>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/instances/bed3c3be-31d6-42c3-9a7f-e40ef9c1c764
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Exit code: 255
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stdout: ''
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Stderr: 'Host key verification failed.\r\n'
>>>>>>>>>>>>>>>>>>>>>> 2013-07-30 11:37:52.463 6254 TRACE
>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I've been trying to migrate directly with virsh, but
>>>>>>>>>>>>>>>>>>>>>> I got this error and the solutions through the web don't help.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> virsh migrate --live instance-00000063 qemu+ssh://
>>>>>>>>>>>>>>>>>>>>>> 192.168.3.3/system
>>>>>>>>>>>>>>>>>>>>>> root at 192.168.3.3's password:
>>>>>>>>>>>>>>>>>>>>>> erro: internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>> 'libvirt-bed3c3be-31d6-42c3-9a7f-e40ef9c1c764'
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thank you all.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 2013/7/29 <yongiman at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Guilherme,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I suugest making new ssh key and put the public key
>>>>>>>>>>>>>>>>>>>>>>> in the host
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> # ssh-keygen (you will find id_rsa.pub in your .ssh
>>>>>>>>>>>>>>>>>>>>>>> directory)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Copy the contents of this file and paste at the last
>>>>>>>>>>>>>>>>>>>>>>> line of authorized_keys in the host.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 2013. 7. 29., at 오후 10:40, Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hello Filipe, I created the
>>>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys and its content is the keys from the
>>>>>>>>>>>>>>>>>>>>>>> know_hosts, is it right? I'm still getting the same error when I try to
>>>>>>>>>>>>>>>>>>>>>>> migrate a VM.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> About the keys it's up to you. You can use the same
>>>>>>>>>>>>>>>>>>>>>>>> or create a new one. Using the same is easier.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> But the nova user public key should go into
>>>>>>>>>>>>>>>>>>>>>>>> /var/lib/nova/.ssh/authorized_keys not into the know_hosts file. The easier
>>>>>>>>>>>>>>>>>>>>>>>> way (as you only have two compute nodes) to add the keys to the know_hosts
>>>>>>>>>>>>>>>>>>>>>>>> is to manually ssh and accept the key.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Ok, Must I create another ssh key to communicate
>>>>>>>>>>>>>>>>>>>>>>>>> both compute nodes or the current key can work? I've put the plublic key
>>>>>>>>>>>>>>>>>>>>>>>>> from both compute node in the know_hosts. Which permission should I apply?
>>>>>>>>>>>>>>>>>>>>>>>>> 644?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> First you need to check why this "Failed to add
>>>>>>>>>>>>>>>>>>>>>>>>>> the host to the list of known hosts (/var/lib/nova/.ssh/known_hosts)."
>>>>>>>>>>>>>>>>>>>>>>>>>> happens. It maybe file permissions or so. Then you need to generate an ssh
>>>>>>>>>>>>>>>>>>>>>>>>>> key for the user nova and add it to the other compute nodes on the
>>>>>>>>>>>>>>>>>>>>>>>>>> authorized_keys file (the password is not really necessary as soon as you
>>>>>>>>>>>>>>>>>>>>>>>>>> have the keys in place).
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> sudo -u nova ssh 192.168.3.3
>>>>>>>>>>>>>>>>>>>>>>>>>>> The authenticity of host '192.168.3.3
>>>>>>>>>>>>>>>>>>>>>>>>>>> (192.168.3.3)' can't be established.
>>>>>>>>>>>>>>>>>>>>>>>>>>> ECDSA key fingerprint is
>>>>>>>>>>>>>>>>>>>>>>>>>>> e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
>>>>>>>>>>>>>>>>>>>>>>>>>>> Are you sure you want to continue connecting
>>>>>>>>>>>>>>>>>>>>>>>>>>> (yes/no)? yes
>>>>>>>>>>>>>>>>>>>>>>>>>>> Failed to add the host to the list of known
>>>>>>>>>>>>>>>>>>>>>>>>>>> hosts (/var/lib/nova/.ssh/known_hosts).
>>>>>>>>>>>>>>>>>>>>>>>>>>> nova at 192.168.3.3's password:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> But I don't know the user nova's password.
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Can you try *sudo -u nova ssh
>>>>>>>>>>>>>>>>>>>>>>>>>>>> <remote-compute-node>* form the compute node
>>>>>>>>>>>>>>>>>>>>>>>>>>>> currently running the VM and check if it works?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Filipe,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  I made the copy from my .ssh/know_hosts to
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> places you told me, but I'm still getting the same error. Any other ideas?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Filipe Manco <filipe.manco at gmail.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> This error means that you cannot ssh into the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> new compute node because it's public key cannot be verified. Add the public
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> keys of the compute nodes to /etc/ssh/know_hosts file (or the nova one,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> usually under /var/lib/nova/.ssh/) and the problem should be gone.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Filipe Manco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://about.me/fmanco
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/26 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Ok, I'll look more carefully the log and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> it's cool this command from libvirt, I didn't know it, I'll try it and
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> return to you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you again.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> That's confused me, "Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command" will give us what command is running when this error
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> occured, you may check the log more carefully.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and you also can migrate the VM directly by
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> libvirt command line like below:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> virsh  migrate --live --persistent
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> --undefinesource --verbose --domain ${instance-uuid} --desturi
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> qemu+tcp://${the-dest-host-ip}/system
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> then see the output result.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-26 09:30
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*"openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  When I enter the action to migrate, the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> only two errors showen at this period are these two.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> There may have more error logs at this
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> minute 2013-07-25 17:28,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you can get the backtrace near the
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> error log you post.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-26
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-26 04:48
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<hzwangpan at corp.netease.com
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >,"openstack at lists.launchpad.net"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> openstack at lists.launchpad.net>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Hello guys, I'm getting problems when
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> trying to migrate a VM.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm getting this error on my log:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  2013-07-25 17:28:31.582 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed.\r\n'. Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 17:28:32.212 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> The user Wangpan was helping me, but I
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> guess his hour is different from mine, can anybody five me an idea?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Hello Wangpan, I've looking at the logs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> and I saw one error with my ssh port, and then I fixed the port, but now
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I'm getting this error:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  2013-07-25 14:09:50.388 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> running command.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Stderr: 'Host key verification
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> failed.\r\n'. Setting instance vm_state to ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25 14:09:51.005 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-6930a2e2-60e7-421f-a560-26a626ecebcb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> What "host key verification"? Where
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> should I put this key?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Guilherme.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/25 Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Typed wrong here
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Em 25/07/2013 00:42, "Wangpan" <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> chmod 777 -R /var/lib/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> not the /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-25 10:39
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: Re: [Openstack]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Configuring Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  I've done chmod 777 -R
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> /var/log/nova/instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Does it work?
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013/7/24 Wangpan <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> I saw this error "Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/'" in your logs.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> you may need to chown the instances
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> path  '/var/lib/nova/instances/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-25
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Wangpan
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  *发件人：*Guilherme Russi <
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> luisguilherme.cr at gmail.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *发送时间：*2013-07-25 10:08
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *主题：*Re: Re: [Openstack] Configuring
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Live Migration
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *收件人：*"Wangpan"<
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> hzwangpan at corp.netease.com>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> *抄送：*
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  Compute Node 01:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  cat /var/log/nova/nova-*.log | grep
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -E ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.326 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:29.511 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.673 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:36.837 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.402 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:39.530 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.657 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.036 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:32.170 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n    if not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.791 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:50.925 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:07.567 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp [req-43b99187-3d4b-4f42-8ee5-eab58a448d05
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Exception during message handling
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:34:02.320 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:23.618 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.087 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:38:49.220 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    admin_pass=admin_password)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_image\n    fileutils.ensure_tree(basepath(suffix=\'\'))\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 29, in ensure_tree\n    os.makedirs(path)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/os.py", line 157, in makedirs\n    mkdir(name,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mode)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:41:15.017 1685 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 12:45:36.353 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.224 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> denied: '/var/lib/nova/instances/.directio.test''
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:09.225 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.636 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.virt.libvirt.driver [-] [instance:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy, instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disappeared.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:13:21.729 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info=block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in to_xml\n    disk_info, rescue, block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_config\n    inst_type):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_storage_config\n    inst_type)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in get_guest_disk_config\n    self.disk_cachemode,\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in disk_cachemode\n    if not
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self._supports_direct_io(CONF.instances_path):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    {\'path\': dirpath, \'ex\': str(e)})\n', '
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _supports_direct_io\n    f = os.open(testfile, os.O_CREAT | os.O_WRONLY
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 13:25:36.702 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:05:37.095 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:45:37.319 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:05.364 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 14:47:26.986 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 15:25:37.532 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:05:37.847 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:39.249 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:14:54.459 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:47.486 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:43:58.434 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 16:46:43.408 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:26:43.627 1694 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [Errno 13] Permission denied: '/var/lock/nova'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:46.003 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:47:55.194 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:53:42.789 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 17:54:05.061 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:05:44.677 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 18:06:14.408 ERROR
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> recent call last):\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance\n    set_access_ip=set_access_ip)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    LOG.exception(_(\'Instance failed to spawn\'),
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File "/usr/lib/python2.7/contextlib.py", line 24,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in __exit__\n    self.gen.next()\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in spawn\n    block_device_info)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain_and_network\n    domain = self._create_domain(xml,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance=instance)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _create_domain\n    domain.createWithFlags(launch_flags)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  result = proxy_call(self._autowrap, f, *args, **kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> proxy_call\n    rv = execute(f,*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>    rv = meth(*args,**kwargs)\n', '  File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> createWithFlags\n    if ret == -1: raise libvirtError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> internal error cannot load AppArmor profile
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  cat /var/log/nova/nova-*.log | grep
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -E TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:03:57.262 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 08:43:57.655 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 09:23:57.816 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:03:57.993 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:20:21.767 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:33:17.279 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:40:26.722 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File "/usr/lib/python2.7/dist-packages/nova/manager.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 241, in periodic_tasks
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     task(self, context)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_image_cache_manager_pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     storage_users.register_storage_use(CONF.instances_path,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> CONF.host)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 228, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> register_storage_use
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager     with open(id_path, 'w') as f:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager IOError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/compute_nodes'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 10:43:58.194 1820 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.manager
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1510, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info=block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2326, in to_xml
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> disk_info, rescue, block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2244, in get_guest_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inst_type):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2051, in get_guest_storage_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> inst_type)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013, in get_guest_disk_config
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.disk_cachemode,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 355, in disk_cachemode
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> if not self._supports_direct_io(CONF.instances_path):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1676, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> {'path': dirpath, 'ex': str(e)})
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1663, in _supports_direct_io
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]     f
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/.directio.test'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:03:19.658 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:22:38.103 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> in _spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> block_device_info)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1517, in spawn
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> admin_pass=admin_password)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 1748, in _create_image
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> fileutils.ensure_tree(basepath(suffix=''))
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 29, in ensure_tree
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> os.makedirs(path)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> File "/usr/lib/python2.7/os.py", line 157, in makedirs
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> mkdir(name, mode)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> OSError: [Errno 13] Permission denied:
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:26:00.874 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp Traceback (most recent call last):
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 430, in _process_data
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     rval = self.proxy.dispatch(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> version, method, **args)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 133, in dispatch
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     temp_level, payload)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     pass
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     e, sys.exc_info())
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> decorated_function
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     return function(self, context, *args,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     do_run_instance()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 242, in inner
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     retval = f(*args, **kwargs)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> do_run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     admin_password, is_first_time, node,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self._set_instance_error_state(context,
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> instance['uuid'])
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/contextlib.py",
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> line 24, in __exit__
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     self.gen.next()
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> _run_instance
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp     filter_properties, bdms)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> 2013-07-24 11:27:08.045 1685 TRACE
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> nova.openstack.common.rpc.amqp   File
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  ...
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [Mensagem reduzida]
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Mailing list:
>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>>>> Post to     : openstack at lists.openstack.org
>>>>>>>>>>>>> Unsubscribe :
>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/c8524321/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: NUAGECO-LOGO-Fblan_petit.jpg
Type: image/jpeg
Size: 10122 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130731/c8524321/attachment.jpg>