Open Stack

On Fri, 2016-11-04 at 10:45 -0400, Ken Giusti wrote:
> On Wed, Nov 2, 2016 at 8:11 PM, Joshua Harlow <harlowja at fastmail.com>
> wrote:
> > 
> > Hi folks,
> > 
> > There was a bunch of chatter at the summit about how there are
> > really two
> > different types of (oslo) messaging usage that exist in openstack
> > and how
> > they need not be backed by the same solution type (rabbitmq, qpid,
> > kafka...).
> > 
> > For those that were not at the oslo sessions:
> > 
> > https://wiki.openstack.org/wiki/Design_Summit/Ocata/Etherpads#Oslo
> > 
> > The general gist was though that we need to make sure people really
> > do know
> > that there are two very different types of messaging usage in
> > openstack and
> > to ensure that operators (and developers) are picking the right
> > backing
> > technology for each type.
> > 
> > So some questions naturally arise out of this.
> > 
> > * Where are the best practices with regard to selection of the best
> > backend
> > type for rpc (and one for notifications); is this something
> > oslo.messaging
> > should work through (or can the docs team and operator group also
> > help in
> > making this)?
> > 
> > * What are the tradeoffs in using the same (or different)
> > technology for rpc
> > and notifications?
> > 
> I think the olso.messaging team should take the lead here and educate
> users as to what the options are, and  how the two supported
> messaging
> services (RPC and Notifications) differ with respect to backend
> requirements.   These topics really should be part of the
> oslo.messaging 'Theory of Operations' documentation that was
> discussed
> during the Arch WG summit meeting.
> 
> Currently the biggest functional difference between the backends is
> the support of store-and-forward (e.g. queueing) verses point-to-
> point
> message transfer.  Oslo could at least explain the pros and cons of
> each approach with respect to the RPC and Notification services so
> that folks understand what the tradeoffs and advantages are in the
> first place.
> 
> Furthermore the team should also document the functional differences
> between the various choices of backends.  For instance it would be
> useful to understand how the two supported point-to-point backends
> (zmq and dispatch router) differ in both behavior and recommended
> deployment.
> 
> 
> > 
> > * Is it even possible for all oslo.messaging consuming projects to
> > be able
> > to choose 2 different backends, are consuming projects consuming
> > the library
> > correctly so that they can use 2 different backends?
> > 
> > * Is devstack able to run with say kafka for notifications and
> > rabbitmq for
> > rpc (if not, is there any work item the oslo group can help with to
> > make
> > this possible) so that we can ensure and test that all projects can
> > work
> > correctly with appropriate (and possibly different) backends?
> > 

If the backend is considered a black box, the goal should be that from
the application's perspective, there is no functional difference
between the use of one backend over another. Rather, the differences
will be non-functional in nature (e.g. quality of service, scalability,
resilience, maintainability, ec.). This is what oslo.messaging will
need to help explain to users so that they will understand the
operational differences between deploying one backend versus another
(e.g. will they be managing a cluster of brokers or a mesh of routers).

The near term challenge will be to make it easy to configure the use of
alternative backends. The introduction of the transport_url
configuration option should help but most of the existing configuration
management frameworks continue to use the legacy (soon to be
deprecated) rpc_backend, host, port and auth configuration options. For
the most part, the configurations have "hard coded" the use of rabbit.

As an example of what will be required, an effort to update the
openstack puppet modules to introduce the transport_url configuration
options took place during Newton. This uses the the puppet-oslo
module that encapsulates the oslo.messaging configuration.

https://review.openstack.org/#/c/321818/

With this change across the core services, a packstack scenario was
prototyped that setup AMQP 1.0 backend for RPC and rabbit for
notifications. The resulting deployment was able to successfully pass
the  tempest smoke tests, etc.

A key enabler for the use of dual backends in practice will be to work
across the spectrum of installation/configuration management frameworks
to adopt the transport_url(s) options. This is a fair bit of scope but
once done, the oslo.messaging backend plug-in model can be fully
leveraged.

> > * Any other messaging, arch-wg work that we (oslo or others) can
> > help out
> > with to make sure that projects (and operators) are using the right
> > technology for the right use (and not just defaulting to RPC over
> > rabbitmq
> > because it exists, when in reality something else might be a better
> > choice)?
> > 
> Ultimately there should be recommendations for which backends are
> optimal for a range of different deployment scenarios, but at this
> point we really don't have enough data and experience with these
> backends to create such recommendations.
> 
> > 
> > * More(?)
> > 
> > Just wanted to get this conversation started, because afaik it's
> > one that
> > has not been widely circulated (and operators have been setting up
> > rabbitmq
> > in various HA and clustered and ... modes, when in reality thinking
> > through
> > what and how it is used may be more appropriate); this also applies
> > to
> > developers since some technical solutions in openstack seem to be
> > created
> > due to (in-part) rabbitmq shortcomings (cells v1 afaik was *in*
> > part created
> > due to scaling issues).
> > 
> > -Josh
> > 
> > ___________________________________________________________________
> > _______
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsu
> > bscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
>