Open Stack

Hi Preston, some great questions in here. Some comments inline, but 
tl;dr my answer is "yes, we need to be doing a much better job thinking 
about how I/O intensive operations affect other things running on 
providers of compute and block storage resources"

On 10/19/2014 06:41 AM, Preston L. Bannister wrote:
> OK, I am fairly new here (to OpenStack). Maybe I am missing something.
> Or not.
>
> Have a DevStack, running in a VM (VirtualBox), backed by a single flash
> drive (on my current generation MacBook). Could be I have something off
> in my setup.
>
> Testing nova backup - first the existing implementation, then my (much
> changed) replacement.
>
> Simple scripts for testing. Create images. Create instances (five). Run
> backup on all instances.
>
> Currently found in:
> https://github.com/dreadedhill-work/stack-backup/tree/master/backup-scripts
>
> First time I started backups of all (five) instances, load on the
> Devstack VM went insane, and all but one backup failed. Seems that all
> of the backups were performed immediately (or attempted), without any
> sort of queuing or load management. Huh. Well, maybe just the backup
> implementation is naive...

Yes, you are exactly correct. There is no queuing behaviour for any of 
the "backup" operations (I put "backup" operations in quotes because IMO 
it is silly to refer to them as backup operations, since all they are 
doing really is a snapshot action against the instance/volume -- and 
then attempting to be a poor man's cloud cron).

The backup is initiated from the admin_actions API extension here:

https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/contrib/admin_actions.py#L297

which calls the nova.compute.api.API.backup() method here:

https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2031

which, after creating some image metadata in Glance for the snapshot, 
calls the compute RPC API here:

https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L759

Which sends an RPC asynchronous message to the compute node to execute 
the instance snapshot and "rotate backups":

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2969

That method eventually calls the blocking snapshot() operation on the 
virt driver:

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L3041

And it is the nova.virt.libvirt.Driver.snapshot() method that is quite 
"icky", with lots of logic to determine the type of snapshot to do and 
how to do it:

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1607

The gist of the driver's snapshot() method calls 
ImageBackend.snapshot(), which is responsible for doing the actual 
snapshot of the instance:

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1685

and then once the snapshot is done, the method calls to the Glance API 
to upload the snapshotted disk image to Glance:

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1730-L1734

All of which is I/O intensive and AFAICT, mostly done in a blocking 
manner, with no queuing or traffic control measures, so as you correctly 
point out, if the compute node daemon receives 5 backup requests, it 
will go ahead and do 5 snapshot operations and 5 uploads to Glance all 
as fast as it can. It will do it in 5 different eventlet greenthreads, 
but there are no designs in place to prioritize the snapshotting I/O 
lower than active VM I/O.

> I will write on this at greater length, but backup should interfere as
> little as possible with foreground processing. Overloading a host is
> entirely unacceptable.

Agree with you completely.

> Replaced the backup implementation so it does proper queuing (among
> other things). Iterating forward - implementing and testing.

Is this code up somewhere we can take a look at?

> Fired off snapshots on five Cinder volumes (attached to five instances).
> Again the load shot very high. Huh. Well, in a full-scale OpenStack
> setup, maybe storage can handle that much I/O more gracefully ... or
> not. Again, should taking snapshots interfere with foreground activity?
> I would say, most often not. Queuing and serializing snapshots would
> strictly limit the interference with foreground. Also, very high end
> storage can perform snapshots *very* quickly, so serialized snapshots
> will not be slow. My take is that the default behavior should be to
> queue and serialize all heavy I/O operations, with non-default
> allowances for limited concurrency.
>
> Cleaned up (which required reboot/unstack/stack and more). Tried again.
>
> Ran two test backups (which in the current iteration create Cinder
> volume snapshots). Asked Cinder to delete the snapshots. Again, very
> high load factors, and in "top" I can see two long-running "dd"
> processes. (Given I have a single disk, more than one "dd" is not good.)
>
> Running too many heavyweight operations against storage can lead to
> thrashing. Queuing can strictly limit that load, and insure better and
> reliable performance. I am not seeing evidence of this thought in my
> OpenStack testing.
>
> So far it looks like there is no thought to managing the impact of disk
> intensive management operations. Am I missing something?

Nope, I think you pretty much hit the nail on the head. Very much 
appreciate your thoughts above on the matter and I look forward to 
working with you to address the problems. If you haven't already, get to 
know the Rally contributor team, who are working to write benchmarks and 
stress tests that show these kinds of issues.

The Rally team hangs out on freenode IRC #openstack-rally channel.

All the best,
-jay