Open Stack

On Wed, Nov 27, 2013 at 05:39:30PM +0000, Edward Hope-Morley wrote:
> On 27/11/13 15:49, Daniel P. Berrange wrote:
> > On Wed, Nov 27, 2013 at 02:45:22PM +0000, Edward Hope-Morley wrote:
> >> Moving this to the ml as requested, would appreciate
> >> comments/thoughts/feedback.
> >>
> >> So, I recently proposed a small patch to the oslo rpc code (initially in
> >> oslo-incubator then moved to oslo.messaging) which extends the existing
> >> support for limiting the rpc thread pool so that concurrent requests can
> >> be limited based on type/method. The blueprint and patch are here:
> >>
> >> https://blueprints.launchpad.net/oslo.messaging/+spec/rpc-concurrency-control
> >>
> >> The basic idea is that if you have server with limited resources you may
> >> want restrict operations that would impact those resources e.g. live
> >> migrations on a specific hypervisor or volume formatting on particular
> >> volume node. This patch allows you, admittedly in a very crude way, to
> >> apply a fixed limit to a set of rpc methods. I would like to know
> >> whether or not people think this is sort of thing would be useful or
> >> whether it alludes to a more fundamental issue that should be dealt with
> >> in a different manner.
> > Based on this description of the problem I have some observations
> >
> >  - I/O load from the guest OS itself is just as important to consider
> >    as I/O load from management operations Nova does for a guest. Both
> >    have the capability to impose denial-of-service on a host. IIUC, the
> >    flavour specs have the ability to express resource constraints for
> >    the virtual machines to prevent a guest OS initiated DOS-attack
> >
> >  - I/O load from live migration is attributable to the running
> >    virtual machine. As such I'd expect that any resource controls
> >    associated with the guest (from the flavour specs) should be
> >    applied to control the load from live migration.
> >
> >    Unfortunately life isn't quite this simple with KVM/libvirt
> >    currently. For networking we've associated each virtual TAP
> >    device with traffic shaping filters. For migration you have
> >    to set a bandwidth cap explicitly via the API. For network
> >    based storage backends, you don't directly control network
> >    usage, but instead I/O operations/bytes. Ultimately though
> >    there should be a way to enforce limits on anything KVM does,
> >    similarly I expect other hypervisors can do the same
> >
> >  - I/O load from operations that Nova does on behalf of a guest
> >    that may be running, or may yet to be launched. These are not
> >    directly known to the hypervisor, so existing resource limits
> >    won't apply. Nova however should have some capability for
> >    applying resource limits to I/O intensive things it does and
> >    somehow associate them with the flavour limits  or some global
> >    per user cap perhaps.
> >
> >> Thoughts?
> > Overall I think that trying to apply caps on the number of API calls
> > that can be made is not really a credible way to avoid users inflicting
> > DOS attack on the host OS. Not least because it does nothing to control
> > what a guest OS itself may do. If you do caps based on num of APIs calls
> > in a time period, you end up having to do an extremely pessistic
> > calculation - basically have to consider the worst case for any single
> > API call, even if most don't hit the worst case. This is going to hurt
> > scalability of the system as a whole IMHO.
> >
> > Regards,
> > Daniel
> Daniel, thanks for this, these are all valid points and essentially tie
> with the fundamental issue of dealing with DOS attacks but for this bp I
> actually want to stay away from this area i.e. this is not intended to
> solve any tenant-based attack issues in the rpc layer (although that
> definitely warrants a discussion e.g. how do we stop a single tenant
> from consuming the entire thread pool with requests) but rather I'm
> thinking more from a QOS perspective i.e. to allow an admin to account
> for a resource bias e.g. slow RAID controller, on a given node (not
> necessarily Nova/HV) which could be alleviated with this sort of crude
> rate limiting. Of course one problem with this approach is that
> blocked/limited requests still reside in the same pool as other requests
> so if we did want to use this it may be worth considering offloading
> blocked requests or giving them their own pool altogether.
> 
> ...or maybe this is just pie in the sky after all.

I don't think it is valid to ignore tenant-based attacks in this. You
have a single resource here and it can be consumed by the tenant
OS, by the VM associated with the tenant or by Nova itself. As such,
IMHO adding rate limiting to Nova APIs alone is a non-solution because
you've still left it wide open to starvation by any number of other
routes which are arguably even more critical to address than the API
calls.

Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|