<div dir="ltr">Hi Steven,<div><br></div><div style>Yeah, something definitely funny with the image.</div><div style><br></div><div style>The partition that holds /var/lib/nova/instances isn't out of disk space, is it? </div>
<div style><br></div><div style>Could you try importing the small cirros image and see if you're able to launch that? You can use the following snippet:</div><div style><br></div><div style><div>. /root/openrc </div>
<div>wget <a href="https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img">https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img</a></div><div>glance add name='cirros image' is_public=true container_format=bare disk_format=qcow2 < cirros-0.3.0-x86_64-disk.img</div>
<div><br></div><div style>Joe</div></div><div style><br></div><div style><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Mar 19, 2013 at 3:48 PM, Steven Barnabas <span dir="ltr"><<a href="mailto:sbarnabas@frontporch.com" target="_blank">sbarnabas@frontporch.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div>drwxr-xr-x 2 nova nova 4096 Jan 30 08:42 buckets</div>
<div>drwxr-xr-x 8 nova nova 4096 Feb 22 13:54 CA</div>
<div>drwxr-xr-x 2 nova nova 4096 Jan 30 08:42 images</div>
<div>drwxr-xr-x 2 nova nova 4096 Jan 30 08:42 instances</div>
<div>drwxr-xr-x 2 nova nova 4096 Jan 30 08:42 keys</div>
<div>drwxr-xr-x 2 nova nova 4096 Jan 30 08:42 networks</div>
<div>-rw-r----- 1 nova nova 158720 Feb 22 13:54 nova.sqlite</div>
<div>drwxr-xr-x 2 nova nova 4096 Jan 30 08:42 tmp</div>
<div><br>
</div>
<div>Looks like there is permission there. There is no instance folder under /instances. I deleted the instance and tried creating a new one with a new name. Same 10 min thing then an error. I checked the nova-compute log this time and I am receiving a
completely different error now:</div>
<div><br>
</div>
<div>
<div>2013-03-19 13:31:53 ERROR nova.compute.manager [req-8d23bdc9-80db-4bd7-bd63-e712b1c0615e aa0e0626b30140fd87f26225a202bf58 fd28b03caed749d8b8b014ec350d9f92] [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Instance failed to spawn</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Traceback (most recent call last):</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 747, in _spawn</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] block_device_info)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] temp_level, payload)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] self.gen.next()</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] return f(*args, **kw)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1090, in spawn</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] admin_pass=admin_password)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1356, in _create_image</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] project_id=instance['project_id'])</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 126, in cache</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] *args, **kwargs)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 173, in create_image</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] prepare_template(target=base, *args, **kwargs)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 796, in inner</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] retval = f(*args, **kwargs)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 117, in call_if_not_exists</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] fetch_func(target=target, *args, **kwargs)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 440, in fetch_image</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] images.fetch_to_raw(context, image_id, target, user_id, project_id)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 98, in fetch_to_raw</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] staged)</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 210, in execute</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] cmd=' '.join(cmd))</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] ProcessExecutionError: Unexpected error while running command.</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Command: qemu-img convert -O raw /var/lib/nova/instances/_base/2986816d8cbac0ea14d12b014f770563ab0a0517.part /var/lib/nova/instances/_base/2986816d8cbac0ea14d12b014f770563ab0a0517.converted</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Exit code: 1</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Stdout: ''</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Stderr: 'qemu-img: error while reading sector 131072: Invalid argument\n'</div>
<div>2013-03-19 13:31:53 23291 TRACE nova.compute.manager [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] </div>
<div>2013-03-19 13:31:53 DEBUG nova.utils [req-8d23bdc9-80db-4bd7-bd63-e712b1c0615e aa0e0626b30140fd87f26225a202bf58 fd28b03caed749d8b8b014ec350d9f92] Got semaphore "compute_resources" for method "abort_resource_claim"... inner /usr/lib/python2.7/dist-packages/nova/utils.py:765</div>
<div>2013-03-19 13:31:53 INFO nova.compute.resource_tracker [req-8d23bdc9-80db-4bd7-bd63-e712b1c0615e aa0e0626b30140fd87f26225a202bf58 fd28b03caed749d8b8b014ec350d9f92] Aborting claim: [Claim c75434ec-677f-4b9e-b5f8-8ab93bcdf972: 16384 MB memory, 170 GB disk,
4 VCPUS]</div>
<div>2013-03-19 13:31:53 DEBUG nova.compute.manager [req-8d23bdc9-80db-4bd7-bd63-e712b1c0615e aa0e0626b30140fd87f26225a202bf58 fd28b03caed749d8b8b014ec350d9f92] [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Deallocating network for instance _deallocate_network
/usr/lib/python2.7/dist-packages/nova/compute/manager.py:773</div>
<div>2013-03-19 13:31:53 DEBUG nova.network.quantumv2.api [req-8d23bdc9-80db-4bd7-bd63-e712b1c0615e aa0e0626b30140fd87f26225a202bf58 fd28b03caed749d8b8b014ec350d9f92] deallocate_for_instance() for PL7 deallocate_for_instance /usr/lib/python2.7/dist-packages/nova/network/quantumv2/api.py:171</div>
<div>2013-03-19 13:31:53 DEBUG nova.compute.manager [req-8d23bdc9-80db-4bd7-bd63-e712b1c0615e aa0e0626b30140fd87f26225a202bf58 fd28b03caed749d8b8b014ec350d9f92] [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Re-scheduling instance: attempt 1 _reschedule /usr/lib/python2.7/dist-packages/nova/compute/manager.py:579</div>
<div>2013-03-19 13:31:53 DEBUG nova.utils [req-8d23bdc9-80db-4bd7-bd63-e712b1c0615e aa0e0626b30140fd87f26225a202bf58 fd28b03caed749d8b8b014ec350d9f92] Got semaphore "compute_resources" for method "update_usage"... inner /usr/lib/python2.7/dist-packages/nova/utils.py:765</div>
<div>2013-03-19 13:31:53 23291 DEBUG nova.openstack.common.rpc.amqp [-] Making asynchronous cast on scheduler... cast /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py:377</div>
<div>2013-03-19 13:31:53 ERROR nova.compute.manager [req-8d23bdc9-80db-4bd7-bd63-e712b1c0615e aa0e0626b30140fd87f26225a202bf58 fd28b03caed749d8b8b014ec350d9f92] [instance: c75434ec-677f-4b9e-b5f8-8ab93bcdf972] Build error: ['Traceback (most recent call last):\n',
' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 494, in _run_instance\n injected_files, admin_password)\n', ' File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 747, in _spawn\n block_device_info)\n', ' File
"/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped\n temp_level, payload)\n', ' File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n self.gen.next()\n', ' File "/usr/lib/python2.7/dist-packages/nova/exception.py",
line 92, in wrapped\n return f(*args, **kw)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1090, in spawn\n admin_pass=admin_password)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line
1356, in _create_image\n project_id=instance[\'project_id\'])\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 126, in cache\n *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py",
line 173, in create_image\n prepare_template(target=base, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 796, in inner\n retval = f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py",
line 117, in call_if_not_exists\n fetch_func(target=target, *args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 440, in fetch_image\n images.fetch_to_raw(context, image_id, target, user_id, project_id)\n',
' File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 98, in fetch_to_raw\n staged)\n', ' File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 210, in execute\n cmd=\' \'.join(cmd))\n', "ProcessExecutionError: Unexpected error
while running command.\nCommand: qemu-img convert -O raw /var/lib/nova/instances/_base/2986816d8cbac0ea14d12b014f770563ab0a0517.part /var/lib/nova/instances/_base/2986816d8cbac0ea14d12b014f770563ab0a0517.converted\nExit code: 1\nStdout: ''\nStderr: 'qemu-img:
error while reading sector 131072: Invalid argument\\n'\n"]</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Kind of seems like it does not like the image?</div>
<div><br>
</div>
<div>
<div>Steven Barnabas<br>
Network Engineer <br>
Front Porch, Inc.<br>
<a href="tel:209-288-5580" value="+12092885580" target="_blank">209-288-5580</a><br>
<a href="tel:209-652-7733" value="+12096527733" target="_blank">209-652-7733</a> mobile<br>
<a href="http://www.frontporch.com/" target="_blank">www.frontporch.com</a></div>
<div><br>
</div>
<br>
</div>
<br>
<div>
<div>On Mar 18, 2013, at 7:04 PM, Joe Topjian <<a href="mailto:joe.topjian@cybera.ca" target="_blank">joe.topjian@cybera.ca</a>> wrote:</div>
<br>
<blockquote type="cite"><span style="font-family:Helvetica;font-size:medium;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:-webkit-auto;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;display:inline!important;float:none"><span> </span>/var/lib/nova/instances</span></blockquote>
</div>
<br>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Joe Topjian<div>Systems Administrator</div><div>Cybera Inc.</div><div><br></div><div><a href="http://www.cybera.ca" target="_blank">www.cybera.ca</a></div><div>
<br></div><div><font color="#666666"><span>Cybera</span><span> is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.</span></font></div>
</div>