We're creating a (huge) temp file, uploading it, and then deleting it. So really we should be streaming the snapshot direct to the destination (glance?)<div><div><br></div><div>Checking the code, we are writing it sequentially (particularly if we're writing in raw):</div>
<div><a href="https://github.com/qemu/QEMU/blob/master/qemu-img.c">https://github.com/qemu/QEMU/blob/master/qemu-img.c</a></div><div><br></div><div><br></div><div>But there's more...</div><div><div><br></div><div>> qemu-img --help</div>
<div>qemu-img version 1.0, Copyright (c) 2004-2008 Fabrice Bellard</div></div><div>...</div><div><div>Supported formats: vvfat vpc vmdk vdi <b>sheepdog</b> <b>rbd</b> raw host_cdrom host_floppy host_device file qed qcow2 qcow parallels <b>nbd iscsi</b> dmg <b>tftp ftps ftp https http</b> cow cloop bochs blkverify blkdebug</div>
</div><div><br></div><div><br></div><div>So it looks like we really want a "Supported format: glance" there (particularly as there's already http support in block/curl.c) :-) I guess we could then even do crazy things like booting direct from glance?</div>
<div><br></div><div>Or, if we don't want to get back into C, we could at least optimize the case where glance is backed by Ceph, and stream direct to a Ceph file, and then hand that file to Glance.</div><div><br></div>
<div>Justin<br><br><br><br>
<br><br><div class="gmail_quote">On Fri, Mar 16, 2012 at 9:11 AM, Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com">jaypipes@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Stackers,<br>
<br>
So, in diagnosing a few things on TryStack yesterday, I ran into an interesting problem with snapshotting that I'm hoping to get some advice on.<br>
<br>
== The Problem ==<br>
<br>
The TryStack codebase is Diablo, however the code involved in this particular problem I believe is the same in Essex...<br>
<br>
The issue that was happening was a user was attempting to snapshot a tiny instance (512MB/1-core) through the dashboard. The dashboard returned and noted that a snapshot was created and was in Queued status.<br>
<br>
The snapshot never goes out of Queued status, and so I logged into the compute node that housed the instance in question to see if I could figure out what was going on.<br>
<br>
Grepping through the compute log, I found the following:<br>
<br>
(nova.rpc): TRACE: Traceback (most recent call last):<br>
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-<u></u>packages/nova/rpc/impl_kombu.<u></u>py", line 628, in _process_data<br>
(nova.rpc): TRACE: rval = node_func(context=ctxt, **node_args)<br>
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-<u></u>packages/nova/exception.py", line 100, in wrapped<br>
(nova.rpc): TRACE: return f(*args, **kw)<br>
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-<u></u>packages/nova/compute/manager.<u></u>py", line 687, in snapshot_instance<br>
(nova.rpc): TRACE: self.driver.snapshot(context, instance_ref, image_id)<br>
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-<u></u>packages/nova/exception.py", line 100, in wrapped<br>
(nova.rpc): TRACE: return f(*args, **kw)<br>
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-<u></u>packages/nova/virt/libvirt/<u></u>connection.py", line 479, in snapshot<br>
(nova.rpc): TRACE: utils.execute(*qemu_img_cmd)<br>
(nova.rpc): TRACE: File "/usr/lib/python2.7/dist-<u></u>packages/nova/utils.py", line 190, in execute<br>
(nova.rpc): TRACE: cmd=' '.join(cmd))<br>
(nova.rpc): TRACE: ProcessExecutionError: Unexpected error while running command.<br>
(nova.rpc): TRACE: Command: qemu-img convert -f qcow2 -O raw -s e7ba4fb5f6f04f99b07d1d222ada02<u></u>19 /opt/openstack/nova/instances/<u></u>instance-00000548/disk /tmp/tmpIuOQo0/<u></u>e7ba4fb5f6f04f99b07d1d222ada02<u></u>19<br>
(nova.rpc): TRACE: Exit code: 1<br>
(nova.rpc): TRACE: Stdout: ''<br>
(nova.rpc): TRACE: Stderr: 'qemu-img: error while writing\n'<br>
<br>
QEMU was unhelpfully returning a vague error message of "error while writing".<br>
<br>
It turned out, after speaking with a couple folks on IRC (thx vishy and rmk!) that the snapshot process (qemu-img convert ... above) is storing the output of the process (the snapshot) in a temporary directory created using tempfile.mkdtemp() in the nova/virt/libvirt/connection.<u></u>py file.<br>
<br>
As it turns out, the base operating system we install on our compute nodes in TryStack has a (very) small root partition -- only 2GB in size (we use the devstack build_pxe_env.sh script to create the base Ubuntu image that is netbooted on the compute nodes.<br>
<br>
Looking at the free disk space on the compute node in question, the problem was apparent:<br>
<br>
root@freecloud102:/var/log/<u></u>nova# df -h<br>
Filesystem Size Used Avail Use% Mounted on<br>
/dev/ram0 2.0G 1.4G 535M 73% /<br>
devtmpfs 48G 240K 48G 1% /dev<br>
none 48G 0 48G 0% /dev/shm<br>
none 48G 212K 48G 1% /var/run<br>
none 48G 0 48G 0% /var/lock<br>
/dev/md0 5.4T 93G 5.1T 2% /opt/openstack<br>
<br>
There simply isn't enough free space on the root partition (which is where /tmp is housed) for the snapshot to be created.<br>
<br>
== Possible Solutions ==<br>
<br>
So, there are a number of solutions that we can work on here, and I'm wondering what the preference would be. Here are the solutions I have come up with, along with a no-brainer improvement to Nova that would help in diagnosing this problem:<br>
<br>
The no-brainer: Detect before attempting a snapshot that there is enough space on a device to perform the operation, and if not, throw a useful error message up the stack<br>
<br>
Solutions to the disk space problem:<br>
<br>
(1) Silly Jay, change the damn size of the root partition in your PXE base OS install!<br>
<br>
Now, I'm no expert in creating customized base disk images, but from looking at the build_pxe_env.sh script in devstack [1], it seems pretty trivial to change the ramdisk_size parameter in the startup options to something larger than 2109600. We could do this and reimage the compute nodes one by one.<br>
<br>
(2) Make the location in which the snapshot is made configurable.<br>
<br>
Right now, as mentioned above, tempfile.mkdtemp() is used, which creates a directory in the user's TMPDIR (typically /tmp, which is usually on the root partition).<br>
<br>
We could add an option (--libvirt-snapshot-dir?) that would allow nova-compute to override where that snapshot is built.<br>
<br>
(3) Change the user (running nova-compute) TMPDIR setting to something different than /tmp on the root partition).<br>
<br>
Thoughts?<br>
-jay<br>
<br>
[1] <a href="https://github.com/openstack-dev/devstack/blob/stable/diablo/tools/build_pxe_env.sh" target="_blank">https://github.com/openstack-<u></u>dev/devstack/blob/stable/<u></u>diablo/tools/build_pxe_env.sh</a><br>
<br>
______________________________<u></u>_________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~<u></u>openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/<u></u>ListHelp</a><br>
</blockquote></div><br></div></div>