[openstack-dev] [all][massively distributed][architecture]Coordination between actions/WGs

Clint Byrum clint at fewbar.com
Mon Sep 5 22:54:25 UTC 2016


Excerpts from Flavio Percoco's message of 2016-09-05 12:08:04 +0200:
> On 02/09/16 10:56 -0700, Clint Byrum wrote:
> >Excerpts from Ken Giusti's message of 2016-09-02 11:05:51 -0400:
> >> On Thu, Sep 1, 2016 at 4:53 PM, Ian Wells <ijw.ubuntu at cack.org.uk> wrote:
> >> > On 1 September 2016 at 06:52, Ken Giusti <kgiusti at gmail.com> wrote:
> >> >>
> >> >> On Wed, Aug 31, 2016 at 3:30 PM, Ian Wells <ijw.ubuntu at cack.org.uk> wrote:
> >> >
> >> >> > I have opinions about other patterns we could use, but I don't want to
> >> >> > push
> >> >>
> >> >> > my solutions here, I want to see if this is really as much of a problem
> >> >> > as
> >> >> > it looks and if people concur with my summary above.  However, the right
> >> >> > approach is most definitely to create a new and more fitting set of oslo
> >> >> > interfaces for communication patterns, and then to encourage people to
> >> >> > move
> >> >> > to the new ones from the old.  (Whether RabbitMQ is involved is neither
> >> >> > here
> >> >> > nor there, as this is really a question of Oslo APIs, not their
> >> >> > implementation.)
> >> >> >
> >> >>
> >> >> Hmmmmmm... maybe.   Message bus technology is varied, and so is it's
> >> >> behavior.  There are brokerless, point-to-point backends supported by
> >> >> oslo.messaging [1],[2] which will exhibit different
> >> >> capabilities/behaviors from the traditional broker-based
> >> >> store-and-forward backend (e.g. message acking end-to-end vs to the
> >> >> intermediary).
> >> >
> >> >
> >> > The important thing is that you shouldn't have to look behind the curtain.
> >> > We can offer APIs that are driven by the implementation (designed for test,
> >> > and trivial to implement correctly given handy open source projects we know
> >> > and trust) and the choice of design will therefore be dependent on the
> >> > backend mechanisms we consider for use to implement the API.  APIs are
> >> > always a point of negotiation between what the caller needs and what can be
> >> > implemented in a practical amount of time.  But *I do not care* whether
> >> > you're using rabbits or carrier pigeons just so long as what you have
> >> > documented that the API promises me is actually true.  I *do not expect* to
> >> > have to read RabbitMQ ior AMQP documentation to work out what behaviour I
> >> > should expect for my messaging.  And its behaviour should be consistent if I
> >> > have a choice of messaging backends.
> >> >
> >>
> >> And I agree totally - this is the way it _should_ be.  And to get
> >> there we do have to address the ambiguities in the existing API, as
> >> well as extend it so applications can explicitly state their service
> >> needs.
> >>
> >> My point is that the API also has to be _backend_ agnostic.  That
> >> really hasn't been the priority it should be IMHO.  The current API as
> >> it stands leaks too much of the backend behavior past the API.
> >>
> >> For example here's where we are with the current API: a majority of
> >> deployments are broker based - applications using oslo.messaging  have
> >> come to rely _indirectly_ on the behavioral side effects of using a
> >> broker backend.  In fact RabbitMQ's operational characteristics have
> >> become the de-facto "correct" behavior.  Any other backend that
> >> doesn't exhibit exactly the same behavior as RabbitMQ is considered
> >> buggy.   Consider qpidd for example - simple differences in default
> >> queue lifecycle and default flow control settings resulted in
> >> messaging behavior different from RabbitMQ.  These were largely
> >> considered bugs in qpidd.  I think this played a large part in the
> >> lack of adoption of qpidd.
> >>
> >> And qpidd is the same type of messaging backend as rabbitmq - a
> >> broker.  Imagine what deployer's are going to hit when they attempt to
> >> use a completely different technology - non-brokered backends like
> >> Zeromq or message routing.
> >>
> >> Improving the API as you describe will go a long way to solving this
> >> situation.  And believe me I agree 100% that this API work needs to be
> >> done.
> >>
> >> But the API redesign should be done in a backend-agnostic manner.  We
> >> (the oslo.messaging devs) have to ensure that _required_ API features
> >> cannot be tied to any one backend implementation.  For example things
> >> like notification pools are trivial to support for broker backends,
> >> but hard/impossible for point to point distributed technologies.  It
> >> must be clear to the application devs that using those optional
> >> features that cannot be effectively implemented for a given backend
> >> basically forces the deployer's hand.
> >>
> >> My point is yes we need to improve that API but it should be done in a
> >> backend agnostic way. There are currently features/behaviors that
> >> essentially require a broker back end.  We should avoid making such
> >> features mandatory elements of the API and ensure that the API users
> >> are well aware of the consequences for deployers when using such
> >> features.
> >>
> >
> >All of what you say is true.
> >
> >However, I want us to also consider the cost of being so modular at the
> >RPC level.
> >
> >Yes it's nice that we have RabbitMQ and ZeroMQ as options, But do we
> >actually need these options? Could we just migrate to ZeroMQ, or HTTP/2,
> >gRPC, thrift, etc.? Then we could tell deployers "good news, you don't
> >need that component anymore, we factored it out" rather than "hey look
> >here, more deployment choices, good luck!"
> 
> Based on the last OPs midcycle's feedback, the answer to the above question
> would probably be "just keep rabbitmq". Turns out that the complains about
> Rabbitmq dropped significantely for whatever reason (Either it becaome more
> stable or OPs just learned how to cope with Rabbits craziness).
> 

We've all seen the cute picture of the lego figures pushing a cart with
square wheels, refusing to take time to switch to round wheels, saying
something like "No thanks, we're too busy, we got this".

Ops are amazing, and will work with what they're given and produce
amazing things. But that doesn't mean we should stop improving.

> We should probably start by asking ourselves who's really being bitten by the
> messaging bus right now? Large (and please, let's not bikeshed on what a Large
> Cloud is) Clouds? Small Clouds? New Clouds? Everyone? 
> 

Complexity is really hard to measure in both magnitude and effect. But we
can reason that the removal of an unnecessary component of the system
will make it simpler, and thus should make it simpler to debug, and
cheaper to build.

> The we can start asking ourselves things like: Would a change of the
> API/underlying technology help them? Why? How? What technology exactly and why?
> What technology would make their lives simpler and why?
> 
> I'm not saying this research/work is not useful/important (in fact, I've been
> advocating for it for almost 2 years now) but I do want us to be more careful
> and certainly I don't think this change should be anything but transparent for
> every deployment out there.
> 

Transparent would be impossible. It's entirely likely that clouds built
today have rabbitmq reachable by all nodes, but compute and control
plane unreachable by eachother. So you have to make sure people have
a router or proxy in the same place as RabbitMQ. Change isn't free,
but if we can figure out how to ask them to replace a stateful service
(RabbitMQ) with a stateless service (Like an L3 router), I think that's
the kind of change operators will welcome given enough lead time and
flexibility to make the transition smooth.

> To answer this topic more directly. As much as being opinionated would help
> driving focus and providing a better result here, I believe we are not there yet
> and I also believe a backend agnostic API would be more benefitial to begin
> with. We're not going to move 98% of the OpenStack deployments out there off of
> rabbitmq just like that.
> 

I think we should try. It can take several releases, but I think it's ok
to make new requirements on upgrade as long as there's a live upgrade
path.



More information about the OpenStack-dev mailing list