[Openstack] Configuring Live Migration
James R Penick
penick at yahoo-inc.com
Tue Jul 30 22:41:22 UTC 2013
Did you put the new public key on the source host, or destination host authorized_keys? It should be on the destination host.
:)=
-----Original Message-----
From: Guilherme Russi [luisguilherme.cr at gmail.com]
Received: Tuesday, 30 Jul 2013, 15:35
To: 한승진 [yongiman at gmail.com]; openstack [openstack at lists.openstack.org]
CC: {openstack at lists.launchpad.net} [openstack at lists.launchpad.net]
Subject: Re: [Openstack] Configuring Live Migration
I created the new key, but still getting error: here are the logs:
cat /var/log/nova/nova-compute.log | grep -E ERROR
2013-07-30 11:37:51.097 ERROR nova.compute.manager [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: bed3c3be-31d6-42c3-9a7f-e40ef9c1c764] Unexpected error while running command.
Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to ERROR
2013-07-30 11:37:52.463 ERROR nova.openstack.common.rpc.amqp [req-c3441bfc-dcd7-4625-b701-716f152a4bfe c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
cat /var/log/nova/nova-compute.log | grep -E TRACE
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in decorated_function
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp pass
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in decorated_function
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2354, in resize_instance
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp block_device_info)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3467, in migrate_disk_and_power_off
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp inst_base_resize)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3444, in migrate_disk_and_power_off
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp utils.execute('ssh', dest, 'mkdir', '-p', inst_base)
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 239, in execute
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp cmd=' '.join(cmd))
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp Command: ssh 10.3.77.48 mkdir -p /var/lib/nova/instances/bed3c3be-31d6-42c3-9a7f-e40ef9c1c764
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp Exit code: 255
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp Stdout: ''
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp Stderr: 'Host key verification failed.\r\n'
2013-07-30 11:37:52.463 6254 TRACE nova.openstack.common.rpc.amqp
I've been trying to migrate directly with virsh, but I got this error and the solutions through the web don't help.
virsh migrate --live instance-00000063 qemu+ssh://192.168.3.3/system<http://192.168.3.3/system>
root at 192.168.3.3<mailto:root at 192.168.3.3>'s password:
erro: internal error cannot load AppArmor profile 'libvirt-bed3c3be-31d6-42c3-9a7f-e40ef9c1c764'
Thank you all.
2013/7/29 <yongiman at gmail.com<mailto:yongiman at gmail.com>>
Hi Guilherme,
I suugest making new ssh key and put the public key in the host
# ssh-keygen (you will find id_rsa.pub in your .ssh directory)
Copy the contents of this file and paste at the last line of authorized_keys in the host.
On 2013. 7. 29., at 오후 10:40, Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>> wrote:
Hello Filipe, I created the /var/lib/nova/.ssh/authorized_keys and its content is the keys from the know_hosts, is it right? I'm still getting the same error when I try to migrate a VM.
Thank you.
2013/7/26 Filipe Manco <filipe.manco at gmail.com<mailto:filipe.manco at gmail.com>>
About the keys it's up to you. You can use the same or create a new one. Using the same is easier.
But the nova user public key should go into /var/lib/nova/.ssh/authorized_keys not into the know_hosts file. The easier way (as you only have two compute nodes) to add the keys to the know_hosts is to manually ssh and accept the key.
Filipe Manco
http://about.me/fmanco
2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
Ok, Must I create another ssh key to communicate both compute nodes or the current key can work? I've put the plublic key from both compute node in the know_hosts. Which permission should I apply? 644?
Thank you.
2013/7/26 Filipe Manco <filipe.manco at gmail.com<mailto:filipe.manco at gmail.com>>
First you need to check why this "Failed to add the host to the list of known hosts (/var/lib/nova/.ssh/known_hosts)." happens. It maybe file permissions or so. Then you need to generate an ssh key for the user nova and add it to the other compute nodes on the authorized_keys file (the password is not really necessary as soon as you have the keys in place).
Filipe Manco
http://about.me/fmanco
2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
sudo -u nova ssh 192.168.3.3
The authenticity of host '192.168.3.3 (192.168.3.3)' can't be established.
ECDSA key fingerprint is e3:ca:f7:dd:1e:1f:e3:e3:17:f8:52:09:a4:41:e1:40.
Are you sure you want to continue connecting (yes/no)? yes
Failed to add the host to the list of known hosts (/var/lib/nova/.ssh/known_hosts).
nova at 192.168.3.3<mailto:nova at 192.168.3.3>'s password:
But I don't know the user nova's password.
2013/7/26 Filipe Manco <filipe.manco at gmail.com<mailto:filipe.manco at gmail.com>>
Can you try sudo -u nova ssh <remote-compute-node> form the compute node currently running the VM and check if it works?
Filipe Manco
http://about.me/fmanco
2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
Hello Filipe,
I made the copy from my .ssh/know_hosts to places you told me, but I'm still getting the same error. Any other ideas?
Thank you.
2013/7/26 Filipe Manco <filipe.manco at gmail.com<mailto:filipe.manco at gmail.com>>
Hi
This error means that you cannot ssh into the new compute node because it's public key cannot be verified. Add the public keys of the compute nodes to /etc/ssh/know_hosts file (or the nova one, usually under /var/lib/nova/.ssh/) and the problem should be gone.
Regards
Filipe Manco
http://about.me/fmanco
2013/7/26 Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
Ok, I'll look more carefully the log and it's cool this command from libvirt, I didn't know it, I'll try it and return to you.
Thank you again.
2013/7/25 Wangpan <hzwangpan at corp.netease.com<mailto:hzwangpan at corp.netease.com>>
That's confused me, "Unexpected error while running command" will give us what command is running when this error occured, you may check the log more carefully.
and you also can migrate the VM directly by libvirt command line like below:
virsh migrate --live --persistent --undefinesource --verbose --domain ${instance-uuid} --desturi qemu+tcp://${the-dest-host-ip}/system
then see the output result.
2013-07-26
________________________________
Wangpan
________________________________
发件人：Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
发送时间：2013-07-26 09:30
主题：Re: Re: Re: Re: Re: [Openstack] Configuring Live Migration
收件人："Wangpan"<hzwangpan at corp.netease.com<mailto:hzwangpan at corp.netease.com>>
抄送："openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>"<openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>>
When I enter the action to migrate, the only two errors showen at this period are these two.
2013/7/25 Wangpan <hzwangpan at corp.netease.com<mailto:hzwangpan at corp.netease.com>>
There may have more error logs at this minute 2013-07-25 17:28,
you can get the backtrace near the error log you post.
2013-07-26
________________________________
Wangpan
________________________________
发件人：Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
发送时间：2013-07-26 04:48
主题：Re: Re: Re: Re: [Openstack] Configuring Live Migration
收件人："Wangpan"<hzwangpan at corp.netease.com<mailto:hzwangpan at corp.netease.com>>,"openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>"<openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>>
抄送：
Hello guys, I'm getting problems when trying to migrate a VM.
I'm getting this error on my log:
2013-07-25 17:28:31.582 ERROR nova.compute.manager [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 919f13d2-8116-4714-b2ec-d2e0f3d3c910] Unexpected error while running command.
Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to ERROR
2013-07-25 17:28:32.212 ERROR nova.openstack.common.rpc.amqp [req-07d2c8b0-80c3-4ae8-8faf-226a099500e4 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
The user Wangpan was helping me, but I guess his hour is different from mine, can anybody five me an idea?
Thank you.
Guilherme.
2013/7/25 Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
Hello Wangpan, I've looking at the logs and I saw one error with my ssh port, and then I fixed the port, but now I'm getting this error:
2013-07-25 14:09:50.388 ERROR nova.compute.manager [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: d268d24b-96fc-4450-88d4-19ba83cd8892] Unexpected error while running command.
Stderr: 'Host key verification failed.\r\n'. Setting instance vm_state to ERROR
2013-07-25 14:09:51.005 ERROR nova.openstack.common.rpc.amqp [req-6930a2e2-60e7-421f-a560-26a626ecebcb c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
What "host key verification"? Where should I put this key?
Thank you.
Guilherme.
2013/7/25 Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
Typed wrong here
Em 25/07/2013 00:42, "Wangpan" <hzwangpan at corp.netease.com<mailto:hzwangpan at corp.netease.com>> escreveu:
chmod 777 -R /var/lib/nova/instances
not the /var/log/nova/instances
2013-07-25
________________________________
Wangpan
________________________________
发件人：Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
发送时间：2013-07-25 10:39
主题：Re: Re: Re: [Openstack] Configuring Live Migration
收件人："Wangpan"<hzwangpan at corp.netease.com<mailto:hzwangpan at corp.netease.com>>
抄送：
I've done chmod 777 -R /var/log/nova/instances
Does it work?
2013/7/24 Wangpan <hzwangpan at corp.netease.com<mailto:hzwangpan at corp.netease.com>>
I saw this error "Permission denied: '/var/lib/nova/instances/'" in your logs.
you may need to chown the instances path '/var/lib/nova/instances/'
2013-07-25
________________________________
Wangpan
________________________________
发件人：Guilherme Russi <luisguilherme.cr at gmail.com<mailto:luisguilherme.cr at gmail.com>>
发送时间：2013-07-25 10:08
主题：Re: Re: [Openstack] Configuring Live Migration
收件人："Wangpan"<hzwangpan at corp.netease.com<mailto:hzwangpan at corp.netease.com>>
抄送：
Compute Node 01:
cat /var/log/nova/nova-*.log | grep -E ERROR
2013-07-24 08:03:57.262 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 08:43:57.655 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 09:23:57.816 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 10:03:57.993 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 10:20:21.767 ERROR nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Instance failed to spawn
2013-07-24 10:20:29.326 1820 ERROR nova.virt.libvirt.driver [-] [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] During wait destroy, instance disappeared.
2013-07-24 10:20:29.511 ERROR nova.compute.manager [req-c4b2d1db-676c-4398-a5b8-582a76ec6c35 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n os.makedirs(path)\n', ' File "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'\n"]
2013-07-24 10:33:17.279 ERROR nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Instance failed to spawn
2013-07-24 10:33:36.673 1820 ERROR nova.virt.libvirt.driver [-] [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] During wait destroy, instance disappeared.
2013-07-24 10:33:36.837 ERROR nova.compute.manager [req-f10c5f14-b3ad-4c13-b20f-c53f57379163 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n os.makedirs(path)\n', ' File "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'\n"]
2013-07-24 10:40:26.722 ERROR nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Instance failed to spawn
2013-07-24 10:40:39.402 1820 ERROR nova.virt.libvirt.driver [-] [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] During wait destroy, instance disappeared.
2013-07-24 10:40:39.530 ERROR nova.compute.manager [req-5d0289b3-36d7-4046-996c-a1decb922066 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n os.makedirs(path)\n', ' File "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'\n"]
2013-07-24 10:43:58.194 1820 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 11:03:19.657 ERROR nova.virt.libvirt.driver [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test''
2013-07-24 11:03:19.658 ERROR nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Instance failed to spawn
2013-07-24 11:03:32.036 1685 ERROR nova.virt.libvirt.driver [-] [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] During wait destroy, instance disappeared.
2013-07-24 11:03:32.170 ERROR nova.compute.manager [req-5441aa83-cc8a-4d25-a17e-9e4f63448451 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510, in spawn\n block_device_info=block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326, in to_xml\n disk_info, rescue, block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in get_guest_config\n inst_type):\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051, in get_guest_storage_config\n inst_type)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013, in get_guest_disk_config\n self.disk_cachemode,\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355, in disk_cachemode\n if not self._supports_direct_io(CONF.instances_path):\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676, in _supports_direct_io\n {\'path\': dirpath, \'ex\': str(e)})\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663, in _supports_direct_io\n f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test'\n"]
2013-07-24 11:22:38.103 ERROR nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Instance failed to spawn
2013-07-24 11:22:50.791 1685 ERROR nova.virt.libvirt.driver [-] [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] During wait destroy, instance disappeared.
2013-07-24 11:22:50.925 ERROR nova.compute.manager [req-26522972-c8be-4be5-83d1-f45412718d94 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n os.makedirs(path)\n', ' File "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'\n"]
2013-07-24 11:26:00.874 ERROR nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Instance failed to spawn
2013-07-24 11:27:07.567 ERROR nova.compute.manager [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n os.makedirs(path)\n', ' File "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'\n"]
2013-07-24 11:27:08.045 ERROR nova.openstack.common.rpc.amqp [req-43b99187-3d4b-4f42-8ee5-eab58a448d05 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Exception during message handling
2013-07-24 11:34:02.320 1685 ERROR nova.virt.libvirt.driver [-] [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] During wait destroy, instance disappeared.
2013-07-24 11:38:23.618 ERROR nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 69a2c512-bea1-46de-91be-c33a07799981] Instance failed to spawn
2013-07-24 11:38:49.087 1685 ERROR nova.virt.libvirt.driver [-] [instance: 69a2c512-bea1-46de-91be-c33a07799981] During wait destroy, instance disappeared.
2013-07-24 11:38:49.220 ERROR nova.compute.manager [req-d9052cd7-12d2-4df6-94c2-4ed3891e6bbb c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 69a2c512-bea1-46de-91be-c33a07799981] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image\n fileutils.ensure_tree(basepath(suffix=\'\'))\n', ' File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree\n os.makedirs(path)\n', ' File "/usr/lib/python2.7/os.py", line 157, in makedirs\n mkdir(name, mode)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'\n"]
2013-07-24 11:41:15.017 1685 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 12:45:36.353 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 13:13:09.224 ERROR nova.virt.libvirt.driver [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] Error on '/var/lib/nova/instances' while checking direct I/O: '[Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test''
2013-07-24 13:13:09.225 ERROR nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Instance failed to spawn
2013-07-24 13:13:21.636 1694 ERROR nova.virt.libvirt.driver [-] [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] During wait destroy, instance disappeared.
2013-07-24 13:13:21.729 ERROR nova.compute.manager [req-4d081c1f-ad11-4b89-904d-700291b47894 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510, in spawn\n block_device_info=block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326, in to_xml\n disk_info, rescue, block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in get_guest_config\n inst_type):\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051, in get_guest_storage_config\n inst_type)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013, in get_guest_disk_config\n self.disk_cachemode,\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355, in disk_cachemode\n if not self._supports_direct_io(CONF.instances_path):\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676, in _supports_direct_io\n {\'path\': dirpath, \'ex\': str(e)})\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663, in _supports_direct_io\n f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)\n', "OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test'\n"]
2013-07-24 13:25:36.702 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:05:37.095 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:45:37.319 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:47:05.364 ERROR nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Instance failed to spawn
2013-07-24 14:47:26.986 ERROR nova.compute.manager [req-4b74d886-c5cc-4e28-ac42-824ba815f7ac c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n domain = self._create_domain(xml, instance=instance)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'\n"]
2013-07-24 15:25:37.532 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 16:05:37.847 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 16:14:39.249 ERROR nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Instance failed to spawn
2013-07-24 16:14:54.459 ERROR nova.compute.manager [req-e0716278-f864-4f70-962a-bf74559fc4c9 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n domain = self._create_domain(xml, instance=instance)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'\n"]
2013-07-24 16:43:47.486 ERROR nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Instance failed to spawn
2013-07-24 16:43:58.434 ERROR nova.compute.manager [req-5d9b9575-a6eb-4324-b975-78dba69c24ef c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n domain = self._create_domain(xml, instance=instance)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-00497f97-a0fe-4a51-b7f6-6901d0d34791'\n"]
2013-07-24 16:46:43.408 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 17:26:43.627 1694 ERROR nova.manager [-] Error during ComputeManager._run_image_cache_manager_pass: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 17:47:46.003 ERROR nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Instance failed to spawn
2013-07-24 17:47:55.194 ERROR nova.compute.manager [req-370ac716-203d-43aa-a146-709297c715b3 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 8a4e0c84-07ec-40ab-9585-5433dff92b64] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n domain = self._create_domain(xml, instance=instance)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-8a4e0c84-07ec-40ab-9585-5433dff92b64'\n"]
2013-07-24 17:53:42.789 ERROR nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Instance failed to spawn
2013-07-24 17:54:05.061 ERROR nova.compute.manager [req-153949aa-d5a7-473f-b79b-38616ae77441 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: d3bfc215-7e24-40d2-b6f9-a1542a2b9d05] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n domain = self._create_domain(xml, instance=instance)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-d3bfc215-7e24-40d2-b6f9-a1542a2b9d05'\n"]
2013-07-24 18:05:44.677 ERROR nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Instance failed to spawn
2013-07-24 18:06:14.408 ERROR nova.compute.manager [req-60486a96-54a1-4472-846b-b57209c68921 c402785616534f2096b34ce132b7d3f2 d532a4fc2e9e4b5f83b6dec7085237e5] [instance: 005f7640-a8ce-429a-ba91-85e7db1d86ac] Error: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 834, in _run_instance\n set_access_ip=set_access_ip)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1093, in _spawn\n LOG.exception(_(\'Instance failed to spawn\'), instance=instance)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn\n block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network\n domain = self._create_domain(xml, instance=instance)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain\n domain.createWithFlags(launch_flags)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit\n result = proxy_call(self._autowrap, f, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker\n rv = meth(*args,**kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags\n if ret == -1: raise libvirtError (\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError: internal error cannot load AppArmor profile 'libvirt-005f7640-a8ce-429a-ba91-85e7db1d86ac'\n"]
cat /var/log/nova/nova-*.log | grep -E TRACE
2013-07-24 08:03:57.262 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 08:03:57.262 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 08:03:57.262 1820 TRACE nova.manager task(self, context)
2013-07-24 08:03:57.262 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 08:03:57.262 1820 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 08:03:57.262 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 08:03:57.262 1820 TRACE nova.manager retval = f(*args, **kwargs)
2013-07-24 08:03:57.262 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 08:03:57.262 1820 TRACE nova.manager with open(id_path, 'w') as f:
2013-07-24 08:03:57.262 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 08:03:57.262 1820 TRACE nova.manager
2013-07-24 08:43:57.655 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 08:43:57.655 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 08:43:57.655 1820 TRACE nova.manager task(self, context)
2013-07-24 08:43:57.655 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 08:43:57.655 1820 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 08:43:57.655 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 08:43:57.655 1820 TRACE nova.manager retval = f(*args, **kwargs)
2013-07-24 08:43:57.655 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 08:43:57.655 1820 TRACE nova.manager with open(id_path, 'w') as f:
2013-07-24 08:43:57.655 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 08:43:57.655 1820 TRACE nova.manager
2013-07-24 09:23:57.816 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 09:23:57.816 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 09:23:57.816 1820 TRACE nova.manager task(self, context)
2013-07-24 09:23:57.816 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 09:23:57.816 1820 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 09:23:57.816 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 09:23:57.816 1820 TRACE nova.manager retval = f(*args, **kwargs)
2013-07-24 09:23:57.816 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 09:23:57.816 1820 TRACE nova.manager with open(id_path, 'w') as f:
2013-07-24 09:23:57.816 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 09:23:57.816 1820 TRACE nova.manager
2013-07-24 10:03:57.993 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 10:03:57.993 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 10:03:57.993 1820 TRACE nova.manager task(self, context)
2013-07-24 10:03:57.993 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 10:03:57.993 1820 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 10:03:57.993 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 10:03:57.993 1820 TRACE nova.manager retval = f(*args, **kwargs)
2013-07-24 10:03:57.993 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 10:03:57.993 1820 TRACE nova.manager with open(id_path, 'w') as f:
2013-07-24 10:03:57.993 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 10:03:57.993 1820 TRACE nova.manager
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] Traceback (most recent call last):
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] block_device_info)
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] admin_pass=admin_password)
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] os.makedirs(path)
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] mkdir(name, mode)
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/55215431-0a42-4fc8-a642-c08d3cc5e2bc/'
2013-07-24 10:20:21.767 1820 TRACE nova.compute.manager [instance: 55215431-0a42-4fc8-a642-c08d3cc5e2bc]
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] Traceback (most recent call last):
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] block_device_info)
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] admin_pass=admin_password)
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] os.makedirs(path)
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] mkdir(name, mode)
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/ca733a02-766e-46e1-9a15-bd2a12b10ebd/'
2013-07-24 10:33:17.279 1820 TRACE nova.compute.manager [instance: ca733a02-766e-46e1-9a15-bd2a12b10ebd]
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] Traceback (most recent call last):
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] block_device_info)
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] admin_pass=admin_password)
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] os.makedirs(path)
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] mkdir(name, mode)
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/3b15a4e9-fb2c-452a-8045-16d1f58ebf73/'
2013-07-24 10:40:26.722 1820 TRACE nova.compute.manager [instance: 3b15a4e9-fb2c-452a-8045-16d1f58ebf73]
2013-07-24 10:43:58.194 1820 TRACE nova.manager Traceback (most recent call last):
2013-07-24 10:43:58.194 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 10:43:58.194 1820 TRACE nova.manager task(self, context)
2013-07-24 10:43:58.194 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 10:43:58.194 1820 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 10:43:58.194 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 10:43:58.194 1820 TRACE nova.manager retval = f(*args, **kwargs)
2013-07-24 10:43:58.194 1820 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 10:43:58.194 1820 TRACE nova.manager with open(id_path, 'w') as f:
2013-07-24 10:43:58.194 1820 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 10:43:58.194 1820 TRACE nova.manager
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] Traceback (most recent call last):
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] block_device_info)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510, in spawn
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] block_device_info=block_device_info)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326, in to_xml
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] disk_info, rescue, block_device_info)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in get_guest_config
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] inst_type):
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051, in get_guest_storage_config
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] inst_type)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013, in get_guest_disk_config
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] self.disk_cachemode,
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355, in disk_cachemode
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] if not self._supports_direct_io(CONF.instances_path):
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676, in _supports_direct_io
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] {'path': dirpath, 'ex': str(e)})
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] self.gen.next()
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663, in _supports_direct_io
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test'
2013-07-24 11:03:19.658 1685 TRACE nova.compute.manager [instance: 40c3fb5c-61d9-491a-9bfa-cac824a2887d]
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] Traceback (most recent call last):
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] block_device_info)
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] admin_pass=admin_password)
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] os.makedirs(path)
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] mkdir(name, mode)
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3/'
2013-07-24 11:22:38.103 1685 TRACE nova.compute.manager [instance: fd3ab09d-fc9f-4ee9-b5f5-72540893ecd3]
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] Traceback (most recent call last):
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] block_device_info)
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] admin_pass=admin_password)
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] os.makedirs(path)
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] mkdir(name, mode)
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/b60db5a8-92ea-4a81-bd53-d73dc2019b0b/'
2013-07-24 11:26:00.874 1685 TRACE nova.compute.manager [instance: b60db5a8-92ea-4a81-bd53-d73dc2019b0b]
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 430, in _process_data
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp temp_level, payload)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 94, in wrapped
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 209, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp pass
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 195, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 260, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 237, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 224, in decorated_function
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1210, in run_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp do_run_instance()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 242, in inner
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1209, in do_run_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp admin_password, is_first_time, node, instance)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 865, in _run_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self._set_instance_error_state(context, instance['uuid'])
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 857, in _run_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp filter_properties, bdms)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 897, in _reschedule_or_reraise
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self._log_original_error(exc_info, instance_uuid)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self.gen.next()
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 892, in _reschedule_or_reraise
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self._shutdown_instance(context, instance, bdms)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1222, in _shutdown_instance
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp network_info = self._get_instance_nw_info(context, instance)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 667, in _get_instance_nw_info
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp instance, conductor_api=self.conductor_api)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 359, in get_instance_nw_info
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp result = self._get_instance_nw_info(context, instance, networks)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 367, in _get_instance_nw_info
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp nw_info = self._build_network_info_model(context, instance, networks)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py", line 778, in _build_network_info_model
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp data = client.list_ports(**search_opts)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 107, in with_params
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp ret = self.function(instance, *args, **kwargs)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 255, in list_ports
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp **_params)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 996, in list
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp for r in self._pagination(collection, path, **params):
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 1009, in _pagination
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp res = self.get(path, params=params)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 982, in get
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp headers=headers, params=params)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 967, in retry_request
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp headers=headers, params=params)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 912, in do_request
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp self._handle_fault_response(status_code, replybody)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 893, in _handle_fault_response
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp exception_handler_v20(status_code, des_error_body)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/quantumclient/v2_0/client.py", line 87, in exception_handler_v20
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp message=message)
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp QuantumClientException: Request Timeout
2013-07-24 11:27:08.045 1685 TRACE nova.openstack.common.rpc.amqp
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] Traceback (most recent call last):
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] block_device_info)
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1517, in spawn
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] admin_pass=admin_password)
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1748, in _create_image
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] fileutils.ensure_tree(basepath(suffix=''))
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] os.makedirs(path)
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] mkdir(name, mode)
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/69a2c512-bea1-46de-91be-c33a07799981/'
2013-07-24 11:38:23.618 1685 TRACE nova.compute.manager [instance: 69a2c512-bea1-46de-91be-c33a07799981]
2013-07-24 11:41:15.017 1685 TRACE nova.manager Traceback (most recent call last):
2013-07-24 11:41:15.017 1685 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 11:41:15.017 1685 TRACE nova.manager task(self, context)
2013-07-24 11:41:15.017 1685 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 11:41:15.017 1685 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 11:41:15.017 1685 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 11:41:15.017 1685 TRACE nova.manager retval = f(*args, **kwargs)
2013-07-24 11:41:15.017 1685 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 11:41:15.017 1685 TRACE nova.manager with open(id_path, 'w') as f:
2013-07-24 11:41:15.017 1685 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 11:41:15.017 1685 TRACE nova.manager
2013-07-24 12:45:36.353 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 12:45:36.353 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 12:45:36.353 1694 TRACE nova.manager task(self, context)
2013-07-24 12:45:36.353 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 12:45:36.353 1694 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 12:45:36.353 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 228, in inner
2013-07-24 12:45:36.353 1694 TRACE nova.manager retval = f(*args, **kwargs)
2013-07-24 12:45:36.353 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/virt/storage_users.py", line 44, in register_storage_use
2013-07-24 12:45:36.353 1694 TRACE nova.manager with open(id_path, 'w') as f:
2013-07-24 12:45:36.353 1694 TRACE nova.manager IOError: [Errno 13] Permission denied: '/var/lib/nova/instances/compute_nodes'
2013-07-24 12:45:36.353 1694 TRACE nova.manager
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] Traceback (most recent call last):
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] block_device_info)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1510, in spawn
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] block_device_info=block_device_info)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2326, in to_xml
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] disk_info, rescue, block_device_info)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in get_guest_config
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] inst_type):
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2051, in get_guest_storage_config
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] inst_type)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2013, in get_guest_disk_config
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] self.disk_cachemode,
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 355, in disk_cachemode
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] if not self._supports_direct_io(CONF.instances_path):
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1676, in _supports_direct_io
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] {'path': dirpath, 'ex': str(e)})
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] self.gen.next()
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1663, in _supports_direct_io
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] f = os.open(testfile, os.O_CREAT | os.O_WRONLY | os.O_DIRECT)
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/.directio.test'
2013-07-24 13:13:09.225 1694 TRACE nova.compute.manager [instance: f105be37-d5d9-4234-ba4e-3897d60e97bb]
2013-07-24 13:25:36.702 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 13:25:36.702 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 13:25:36.702 1694 TRACE nova.manager task(self, context)
2013-07-24 13:25:36.702 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 13:25:36.702 1694 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 13:25:36.702 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 13:25:36.702 1694 TRACE nova.manager fileutils.ensure_tree(local_lock_path)
2013-07-24 13:25:36.702 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 13:25:36.702 1694 TRACE nova.manager os.makedirs(path)
2013-07-24 13:25:36.702 1694 TRACE nova.manager File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 13:25:36.702 1694 TRACE nova.manager mkdir(name, mode)
2013-07-24 13:25:36.702 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 13:25:36.702 1694 TRACE nova.manager
2013-07-24 14:05:37.095 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 14:05:37.095 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 14:05:37.095 1694 TRACE nova.manager task(self, context)
2013-07-24 14:05:37.095 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 14:05:37.095 1694 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 14:05:37.095 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 14:05:37.095 1694 TRACE nova.manager fileutils.ensure_tree(local_lock_path)
2013-07-24 14:05:37.095 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 14:05:37.095 1694 TRACE nova.manager os.makedirs(path)
2013-07-24 14:05:37.095 1694 TRACE nova.manager File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 14:05:37.095 1694 TRACE nova.manager mkdir(name, mode)
2013-07-24 14:05:37.095 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:05:37.095 1694 TRACE nova.manager
2013-07-24 14:45:37.319 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 14:45:37.319 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 14:45:37.319 1694 TRACE nova.manager task(self, context)
2013-07-24 14:45:37.319 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 14:45:37.319 1694 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 14:45:37.319 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 14:45:37.319 1694 TRACE nova.manager fileutils.ensure_tree(local_lock_path)
2013-07-24 14:45:37.319 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 14:45:37.319 1694 TRACE nova.manager os.makedirs(path)
2013-07-24 14:45:37.319 1694 TRACE nova.manager File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 14:45:37.319 1694 TRACE nova.manager mkdir(name, mode)
2013-07-24 14:45:37.319 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 14:45:37.319 1694 TRACE nova.manager
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] Traceback (most recent call last):
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] block_device_info)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] block_device_info)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] domain = self._create_domain(xml, instance=instance)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] domain.createWithFlags(launch_flags)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] rv = execute(f,*args,**kwargs)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] rv = meth(*args,**kwargs)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601] libvirtError: internal error cannot load AppArmor profile 'libvirt-748277e2-e3b5-4385-9c0e-c8e0ddefa601'
2013-07-24 14:47:05.364 1694 TRACE nova.compute.manager [instance: 748277e2-e3b5-4385-9c0e-c8e0ddefa601]
2013-07-24 15:25:37.532 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 15:25:37.532 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 15:25:37.532 1694 TRACE nova.manager task(self, context)
2013-07-24 15:25:37.532 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 15:25:37.532 1694 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 15:25:37.532 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 15:25:37.532 1694 TRACE nova.manager fileutils.ensure_tree(local_lock_path)
2013-07-24 15:25:37.532 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 15:25:37.532 1694 TRACE nova.manager os.makedirs(path)
2013-07-24 15:25:37.532 1694 TRACE nova.manager File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 15:25:37.532 1694 TRACE nova.manager mkdir(name, mode)
2013-07-24 15:25:37.532 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 15:25:37.532 1694 TRACE nova.manager
2013-07-24 16:05:37.847 1694 TRACE nova.manager Traceback (most recent call last):
2013-07-24 16:05:37.847 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/manager.py", line 241, in periodic_tasks
2013-07-24 16:05:37.847 1694 TRACE nova.manager task(self, context)
2013-07-24 16:05:37.847 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4007, in _run_image_cache_manager_pass
2013-07-24 16:05:37.847 1694 TRACE nova.manager storage_users.register_storage_use(CONF.instances_path, CONF.host)
2013-07-24 16:05:37.847 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 210, in inner
2013-07-24 16:05:37.847 1694 TRACE nova.manager fileutils.ensure_tree(local_lock_path)
2013-07-24 16:05:37.847 1694 TRACE nova.manager File "/usr/lib/python2.7/dist-packages/nova/openstack/common/fileutils.py", line 29, in ensure_tree
2013-07-24 16:05:37.847 1694 TRACE nova.manager os.makedirs(path)
2013-07-24 16:05:37.847 1694 TRACE nova.manager File "/usr/lib/python2.7/os.py", line 157, in makedirs
2013-07-24 16:05:37.847 1694 TRACE nova.manager mkdir(name, mode)
2013-07-24 16:05:37.847 1694 TRACE nova.manager OSError: [Errno 13] Permission denied: '/var/lock/nova'
2013-07-24 16:05:37.847 1694 TRACE nova.manager
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] Traceback (most recent call last):
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] block_device_info)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] block_device_info)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2435, in _create_domain_and_network
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] domain = self._create_domain(xml, instance=instance)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2396, in _create_domain
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] domain.createWithFlags(launch_flags)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 187, in doit
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] result = proxy_call(self._autowrap, f, *args, **kwargs)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 147, in proxy_call
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] rv = execute(f,*args,**kwargs)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 76, in tworker
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] rv = meth(*args,**kwargs)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 711, in createWithFlags
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913] libvirtError: internal error cannot load AppArmor profile 'libvirt-23cf38cc-797e-40f2-a995-6e181493c913'
2013-07-24 16:14:39.249 1694 TRACE nova.compute.manager [instance: 23cf38cc-797e-40f2-a995-6e181493c913]
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] Traceback (most recent call last):
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1089, in _spawn
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] block_device_info)
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1520, in spawn
2013-07-24 16:43:47.486 1694 TRACE nova.compute.manager [instance: 00497f97-a0fe-4a51-b7f6-6901d0d34791]
...
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack at lists.launchpad.net<mailto:openstack at lists.launchpad.net>
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130730/8b565bc0/attachment.html>
More information about the Openstack
mailing list