Open Stack

On Wed, Nov 27, 2013 at 02:45:22PM +0000, Edward Hope-Morley wrote:
> Moving this to the ml as requested, would appreciate
> comments/thoughts/feedback.
> 
> So, I recently proposed a small patch to the oslo rpc code (initially in
> oslo-incubator then moved to oslo.messaging) which extends the existing
> support for limiting the rpc thread pool so that concurrent requests can
> be limited based on type/method. The blueprint and patch are here:
> 
> https://blueprints.launchpad.net/oslo.messaging/+spec/rpc-concurrency-control
> 
> The basic idea is that if you have server with limited resources you may
> want restrict operations that would impact those resources e.g. live
> migrations on a specific hypervisor or volume formatting on particular
> volume node. This patch allows you, admittedly in a very crude way, to
> apply a fixed limit to a set of rpc methods. I would like to know
> whether or not people think this is sort of thing would be useful or
> whether it alludes to a more fundamental issue that should be dealt with
> in a different manner.

Based on this description of the problem I have some observations

 - I/O load from the guest OS itself is just as important to consider
   as I/O load from management operations Nova does for a guest. Both
   have the capability to impose denial-of-service on a host. IIUC, the
   flavour specs have the ability to express resource constraints for
   the virtual machines to prevent a guest OS initiated DOS-attack

 - I/O load from live migration is attributable to the running
   virtual machine. As such I'd expect that any resource controls
   associated with the guest (from the flavour specs) should be
   applied to control the load from live migration.

   Unfortunately life isn't quite this simple with KVM/libvirt
   currently. For networking we've associated each virtual TAP
   device with traffic shaping filters. For migration you have
   to set a bandwidth cap explicitly via the API. For network
   based storage backends, you don't directly control network
   usage, but instead I/O operations/bytes. Ultimately though
   there should be a way to enforce limits on anything KVM does,
   similarly I expect other hypervisors can do the same

 - I/O load from operations that Nova does on behalf of a guest
   that may be running, or may yet to be launched. These are not
   directly known to the hypervisor, so existing resource limits
   won't apply. Nova however should have some capability for
   applying resource limits to I/O intensive things it does and
   somehow associate them with the flavour limits  or some global
   per user cap perhaps.

> Thoughts?

Overall I think that trying to apply caps on the number of API calls
that can be made is not really a credible way to avoid users inflicting
DOS attack on the host OS. Not least because it does nothing to control
what a guest OS itself may do. If you do caps based on num of APIs calls
in a time period, you end up having to do an extremely pessistic
calculation - basically have to consider the worst case for any single
API call, even if most don't hit the worst case. This is going to hurt
scalability of the system as a whole IMHO.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|