Open Stack

Hi,

On Wed, 2013-11-27 at 14:45 +0000, Edward Hope-Morley wrote:
> Moving this to the ml as requested, would appreciate
> comments/thoughts/feedback.

Thanks, I too would appreciate input from others.

> So, I recently proposed a small patch to the oslo rpc code (initially in
> oslo-incubator then moved to oslo.messaging) which extends the existing
> support for limiting the rpc thread pool so that concurrent requests can
> be limited based on type/method. The blueprint and patch are here:
> 
> https://blueprints.launchpad.net/oslo.messaging/+spec/rpc-concurrency-control
> 
> The basic idea is that if you have server with limited resources you may
> want restrict operations that would impact those resources e.g. live
> migrations on a specific hypervisor or volume formatting on particular
> volume node. This patch allows you, admittedly in a very crude way, to
> apply a fixed limit to a set of rpc methods. I would like to know
> whether or not people think this is sort of thing would be useful or
> whether it alludes to a more fundamental issue that should be dealt with
> in a different manner.

Just to be clear for everyone what we're talking about. Your patch means
that if an operator sees that requests to the 'foo' and 'bar' RPC
methods for a given service are overwhelming the capacity of the
machine, you can throttle them by adding e.g.

  concurrency_control_enabled = true
  concurrency_control_actions = foo,bar
  concurrency_control_limit = 2

to the service's configuration file.

If you accept the premise of what's required here, I think you really
want to have e.g. a json policy file which can control the concurrency
limit on each method individually:

{
    "compute": {
        "baseapi": {
            "ping": 10
        },
        "": {
            "foo": 1,
            "bar": 2
        }
    }
}

but that starts feeling pretty ridiculous.

My concern is that we're avoiding addressing a more fundamental issue
here. From IRC:

 <markmc> "avoid specific concurrent operations from consuming too many
          system resources and starving other less resource intensive
          actions"
 <markmc> I'd like us to think about whether we can come up with a
          solution that fixes the problem for people, without them
          having to mess with this type of configuration
 <markmc> but yeah ... if we can't figure out a way of doing that, there
          is an argument for giving operators and interim workaround
 <markmc> I wouldn't be in favour of an interim fix without first
          exploring the options for a more fundamental fix
 <markmc> this isn't easily removable later, because once people start
          to rely on it we would need to put it through a deprecation
          period to remove it
 <markmc> also, an interim solution like this takes away the pressure on
          us to find a more fundamental solution ... and we may wind up
          never doing that


So, I guess my first question is ... what specific RPC methods have you
seen issues with and feel you need to throttle?

Thanks,
Mark.