Open Stack

Excerpts from Flavio Percoco's message of 2014-09-12 00:22:35 -0700:
> On 09/12/2014 03:29 AM, Clint Byrum wrote:
> > Excerpts from Zane Bitter's message of 2014-09-11 15:21:26 -0700:
> >> On 09/09/14 19:56, Clint Byrum wrote:
> >>> Excerpts from Samuel Merritt's message of 2014-09-09 16:12:09 -0700:
> >>>> On 9/9/14, 12:03 PM, Monty Taylor wrote:
> >>>>> On 09/04/2014 01:30 AM, Clint Byrum wrote:
> >>>>>> Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700:
> >>>>>>> Greetings,
> >>>>>>>
> >>>>>>> Last Tuesday the TC held the first graduation review for Zaqar. During
> >>>>>>> the meeting some concerns arose. I've listed those concerns below with
> >>>>>>> some comments hoping that it will help starting a discussion before the
> >>>>>>> next meeting. In addition, I've added some comments about the project
> >>>>>>> stability at the bottom and an etherpad link pointing to a list of use
> >>>>>>> cases for Zaqar.
> >>>>>>>
> >>>>>>
> >>>>>> Hi Flavio. This was an interesting read. As somebody whose attention has
> >>>>>> recently been drawn to Zaqar, I am quite interested in seeing it
> >>>>>> graduate.
> >>>>>>
> >>>>>>> # Concerns
> >>>>>>>
> >>>>>>> - Concern on operational burden of requiring NoSQL deploy expertise to
> >>>>>>> the mix of openstack operational skills
> >>>>>>>
> >>>>>>> For those of you not familiar with Zaqar, it currently supports 2 nosql
> >>>>>>> drivers - MongoDB and Redis - and those are the only 2 drivers it
> >>>>>>> supports for now. This will require operators willing to use Zaqar to
> >>>>>>> maintain a new (?) NoSQL technology in their system. Before expressing
> >>>>>>> our thoughts on this matter, let me say that:
> >>>>>>>
> >>>>>>>       1. By removing the SQLAlchemy driver, we basically removed the
> >>>>>>> chance
> >>>>>>> for operators to use an already deployed "OpenStack-technology"
> >>>>>>>       2. Zaqar won't be backed by any AMQP based messaging technology for
> >>>>>>> now. Here's[0] a summary of the research the team (mostly done by
> >>>>>>> Victoria) did during Juno
> >>>>>>>       3. We (OpenStack) used to require Redis for the zmq matchmaker
> >>>>>>>       4. We (OpenStack) also use memcached for caching and as the oslo
> >>>>>>> caching lib becomes available - or a wrapper on top of dogpile.cache -
> >>>>>>> Redis may be used in place of memcached in more and more deployments.
> >>>>>>>       5. Ceilometer's recommended storage driver is still MongoDB,
> >>>>>>> although
> >>>>>>> Ceilometer has now support for sqlalchemy. (Please correct me if I'm
> >>>>>>> wrong).
> >>>>>>>
> >>>>>>> That being said, it's obvious we already, to some extent, promote some
> >>>>>>> NoSQL technologies. However, for the sake of the discussion, lets assume
> >>>>>>> we don't.
> >>>>>>>
> >>>>>>> I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't
> >>>>>>> keep avoiding these technologies. NoSQL technologies have been around
> >>>>>>> for years and we should be prepared - including OpenStack operators - to
> >>>>>>> support these technologies. Not every tool is good for all tasks - one
> >>>>>>> of the reasons we removed the sqlalchemy driver in the first place -
> >>>>>>> therefore it's impossible to keep an homogeneous environment for all
> >>>>>>> services.
> >>>>>>>
> >>>>>>
> >>>>>> I whole heartedly agree that non traditional storage technologies that
> >>>>>> are becoming mainstream are good candidates for use cases where SQL
> >>>>>> based storage gets in the way. I wish there wasn't so much FUD
> >>>>>> (warranted or not) about MongoDB, but that is the reality we live in.
> >>>>>>
> >>>>>>> With this, I'm not suggesting to ignore the risks and the extra burden
> >>>>>>> this adds but, instead of attempting to avoid it completely by not
> >>>>>>> evolving the stack of services we provide, we should probably work on
> >>>>>>> defining a reasonable subset of NoSQL services we are OK with
> >>>>>>> supporting. This will help making the burden smaller and it'll give
> >>>>>>> operators the option to choose.
> >>>>>>>
> >>>>>>> [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/
> >>>>>>>
> >>>>>>>
> >>>>>>> - Concern on should we really reinvent a queue system rather than
> >>>>>>> piggyback on one
> >>>>>>>
> >>>>>>> As mentioned in the meeting on Tuesday, Zaqar is not reinventing message
> >>>>>>> brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack
> >>>>>>> flavor on top. [0]
> >>>>>>>
> >>>>>>
> >>>>>> I think Zaqar is more like SMTP and IMAP than AMQP. You're not really
> >>>>>> trying to connect two processes in real time. You're trying to do fully
> >>>>>> asynchronous messaging with fully randomized access to any message.
> >>>>>>
> >>>>>> Perhaps somebody should explore whether the approaches taken by large
> >>>>>> scale IMAP providers could be applied to Zaqar.
> >>>>>>
> >>>>>> Anyway, I can't imagine writing a system to intentionally use the
> >>>>>> semantics of IMAP and SMTP. I'd be very interested in seeing actual use
> >>>>>> cases for it, apologies if those have been posted before.
> >>>>>
> >>>>> It seems like you're EITHER describing something called XMPP that has at
> >>>>> least one open source scalable backend called ejabberd. OR, you've
> >>>>> actually hit the nail on the head with bringing up SMTP and IMAP but for
> >>>>> some reason that feels strange.
> >>>>>
> >>>>> SMTP and IMAP already implement every feature you've described, as well
> >>>>> as retries/failover/HA and a fully end to end secure transport (if
> >>>>> installed properly) If you don't actually set them up to run as a public
> >>>>> messaging interface but just as a cloud-local exchange, then you could
> >>>>> get by with very low overhead for a massive throughput - it can very
> >>>>> easily be run on a single machine for Sean's simplicity, and could just
> >>>>> as easily be scaled out using well known techniques for public cloud
> >>>>> sized deployments?
> >>>>>
> >>>>> So why not use existing daemons that do this? You could still use the
> >>>>> REST API you've got, but instead of writing it to a mongo backend and
> >>>>> trying to implement all of the things that already exist in SMTP/IMAP -
> >>>>> you could just have them front to it. You could even bypass normal
> >>>>> delivery mechanisms and do neat things with local injection.
> >>>>>
> >>>>> I don't care about the NoSQL question on its own. Mongo is fine. Redis
> >>>>> is fine. I don't think either has any features for this use case that
> >>>>> make a licks worth of difference compared to MySQL or Postgres, but I
> >>>>> also don't think they are a PROBLEM in an of themselves.
> >>>>>
> >>>>> The main thing I care about here is every description I've heard of what
> >>>>> zaqar wants to do (which does seem to be getting clearer through this
> >>>>> thread) is still well implemented somewhere as an existing scalable
> >>>>> service. Is zaqar actually Rabbit with a REST interface? Is it ejabberd
> >>>>> with a rest interface? Or is it IMAP/SMTP with a REST interface. You'll
> >>>>> note that probably nobody would think a single server that wanted to be
> >>>>> both Rabbit AND IMAP/SMTP is a good idea ... at least this is one of the
> >>>>> reasons why we all think Microsoft Exchange is a pile of garbage, no?
> >>>>>
> >>>>> I also worry about the fact that one description of zaqar was used to
> >>>>> communicate a need for divergent requirements (it needs to be a
> >>>>> high-volume fast message broker/queue - which, btw, sounds more like
> >>>>> Rabbit/oslo.messaging and less like what Clint describes above) ... and
> >>>>> that's why it wants to use falcon and not pecan and why it wants to use
> >>>>> mongo and not SQL. And then what we're doing it reimplementing something
> >>>>> like rabbit except in python (again, given as the justification for
> >>>>> deviating from how other bits of OpenStack work)
> >>>>>
> >>>>> BUT - if that's not actually what zaqar is - if it isn't a rabbit
> >>>>> replacement and doesn't need to do massive high volume sub-second
> >>>>> queuing because what it's actually modeling is a message subscription
> >>>>> service that's closer to email than to anything else, then there is
> >>>>> nothing about the components that are happily used in the rest of
> >>>>> OpenStack that should be precluded from being used. A REST api written
> >>>>> in pecan should be fine ... as should an SQL backend, because 99% of all
> >>>>> operations are going to be primary key lookups where even a moderately
> >>>>> tuned database should be absolutely fine at keeping up.
> >>>>>
> >>>>> So which is it? Because it sounds like to me it's a thing that actually
> >>>>> does NOT need to diverge in technology in any way, but that I've been
> >>>>> told that it needs to diverge because it's delivering a different set of
> >>>>> features - and I'm pretty sure if it _is_ the thing that needs to
> >>>>> diverge in technology because of its feature set, then it's a thing I
> >>>>> don't think we should be implementing in python in OpenStack because it
> >>>>> already exists and it's called AMQP.
> >>>>
> >>>> Whether Zaqar is more like AMQP or more like email is a really strange
> >>>> metric to use for considering its inclusion.
> >>>>
> >>>> Let me put on my web application developer's hat. Whenever I've worked
> >>>> on a web app, I've invariably wound up needing HTTP servers, background
> >>>> workers, and some sort of queue to connect the two.
> >>>>
> >>>> I've done the thing where I've stored queue entries in the app's
> >>>> database and had the workers poll for jobs; the load adds up
> >>>> surprisingly fast, and it's got some bad positive-feedback failure
> >>>> modes. However, it is nice and durable, so my app doesn't lose messages.
> >>>>
> >>>> I've done the thing where I've thrown together a VM and stuck Redis or
> >>>> rabbitmq or beanstalkd on it. That gets me nice, fast queues, but no
> >>>> semblance of reliability. If that one VM dies, all my queued messages
> >>>> are lost.
> >>>>
> >>>> Then there's Zaqar, which is this nice HTTP API that I can use for my
> >>>> application's queues. I go and make a couple of POST requests and now
> >>>> I've got some queues for my application to use. My app servers POST
> >>>> messages to their queues, and my background workers sit and make GET
> >>>> requests for messages to process. I can have the whole thing up and
> >>>> running in a few hours. Better yet, I barely have to monitor the thing.
> >>>> I can poll for queue stats every few minutes and alert if the queue gets
> >>>> too full, but that's all I've got to do. I don't have to worry about my
> >>>> queue VM going into swap, or my queue VM's NIC getting saturated, or
> >>>> kernel panics, or automatically promoting rabbitmq slaves to masters, or
> >>>> waking up at 3 AM to fix my app when I lose messages during a rabbitmq
> >>>> promotion, or any of that stuff. Using Zaqar means I can just worry
> >>>> about my application and leave all that other garbage to my cloud provider.
> >>>
> >>> What you just described is the queue pattern I spoke of.
> >>>
> >>> It does not require random access by message ID in any way shape or
> >>> form. It is also well served by AMQP. I wonder if people are still
> >>> confused by Zaqar's API and architecture because this would be fine for
> >>> an architecture if what you describe above were the requirements:
> >>>
> >>> https://www.dropbox.com/s/yonloa9ytlf8fdh/ZaqarQueueOnly.png?dl=0
> >>>
> >>> Just stick a REST shim in front of AMQP that enforces tenant permissions
> >>> and maps logical "zaqar queues" to whatever the backend serving queue is.
> >>>
> >>> So why would we need a NoSQL database for the data itself if all we are
> >>> doing is shoving messages in and taking them out the other end?
> >>
> >> This was one of the ideas suggested by the TC at the last graduation 
> >> review (or was it the incubation review? I forget). The Zaqar team 
> >> looked into it and this was the outcome (for now):
> >>
> >> http://blog.flaper87.com/post/marconi-amqp-see-you-later/
> >>
> > 
> > I've just re-read that, and it still feels the same as the first time:
> > this feels like justification for the current body of work, instead of
> > analysis for the necessity for the current feature set or feasibility of
> > scaling it.
> 
> To be fair, the analysis was done based one the current feature-set, the
> amount of work required, the project goals in mind and the feasibility
> of doing something like that in terms of time and short-term goals.
> Still, the use cases were kept in mind and that's the reason why the
> post keeps the doors open for future evaluation (at least that was the
> goal :P)
> 
> > "The third motivation is something that Marconi brings to traditional
> > brokers. With Marconi, it's possible to add per-queue sharding
> > capabilities to traditional brokers. By using Marconi's pools and flavors
> > it is possible to create separate clusters of storage that will be used
> > cooperatively based on the pool settings."
> > 
> > This statement does not seem use-case driven. It appears to be supporting
> > the current implementation which, I assume, was done to work around
> > the problem of databases not being able to scale like a queue. Users
> > will just want to be able to reason about the messaging guarantees like
> > "delivered at least once" and durability.
> 
> This is not correct. Zaqar is not just trying to address use cases from
> a user perspective but also the operator's.
> 
> Pools and flavors were not added because of databases not scaling as
> message-brokers. Furthermore, one of the use-cases when we started
> talking about adding pools was that scaling brokers is not as simple as
> it is presented.
> 
> Pools and flavors have different use cases. Pools in particular were
> added to add better scaling capabilities to Zaqar. In addition to that,
> pools also allow operators to isolate clusters for specific queues or
> tenants if necessary.
> 
> Flavors, instead, are a user-facing feature. They allow users to choose
> on what *type* of pool they want their messages. They can choose that
> based on the flavor capabilities, which are basically the pool
> capabilities. For instance, a user may choose between sending messages
> to a `in-memory` pool to have a higher throughput or to a durable pool.
> 
> The above obviously requires Zaqar to be deployed on top of different
> storage and it also requires pools to be configured by the admin. But
> both features address use-cases, for both operators and users, which we
> consider valuable. Both features are optional.
> 

This was a fantastic explanation, thank you. I have read a few of the
documents on it, but what you said above succinctly explains how use
cases are extremely well served by these features. Thank you.

> > Meanwhile there are quite a few message brokers that can scale in the
> > same way, that Zaqar wouldn't even have to model.
> > 
> > "The API was designed to be browsable but without sacrificing the
> > "messaging" feel of every queuing system. New things are being added to
> > make it more message based without sacrificing browsability."
> > 
> > This browsability is, at its core, the thing that concerns me and probably
> > others who are expressing their opinions. It's not that we don't think
> > it can be done, but we don't think it should be done when there are so
> > many use cases solved by a regular old message queue.
> > 
> 
> I'm going to, probably, oversimplify here so bare with me. Browsability
> is just another way to access messages. It makes it easier for users to
> stream messages out of Zaqar if that's what works best for them. I
> understand that what I'm describing is precisely what a queue does but
> in Zaqar it's a way for users to follow-their-nose and keep consuming
> messages.
> 
> I'm not saying it couldn't be revisited, though. I'm trying to explain
> that it adds support for a common way to access messages.
> 

Browsability (my spell checker doesn't think that is a word. ;) is
nothing new to the email user. It's a well known problem space really.

> > The fact that sessions are a part of AMQP makes it complicated to
> > emulate over HTTP when messages need to be ACK'd. However, it seems
> > pretty straight forward to me to have session ID as part of the API, and
> > to have an RPC bus like we do now for our usual OpenStack services that
> > has a topic per session. Then have the API frontends speaking over that
> > bus, pushing ACK's down that topic to a backend engine that maintains
> > the session. This is double-messaging for everything, which I wish I
> > could think of a better solution for, but it is still lighter weight
> > than putting all of the messages into a database.
> 
> I'm not sure I follow what you suggested here so I'll wait before I
> comment. What I didn't understand is whether you were thinking to have
> that solution as part of Zaqar or as a way to achieve browsability
> without Zaqar.
> 

I'm only addressing the difficulty that AMQP would have with getting ACK's
back to an original session. That paragraph is meant to highlight a key
problem with consumption of messages reliably from existing brokers via
an HTTP API. If we can't use ACK's for reliably delivery, we can't have
reliably delivery with queue semantics. It's a key part of AMQP that
helps the brokers operate efficiently. If they had to look messages up
by ID without an open session, they'd be back in the same problem space
as Zaqar: looking them up in a database.

> > Finally, two of the main sections in that article state "Store and
> > Forward" and "Message access by ID" are important to Zaqar. Those two
> > concepts are SMTP and IMAP in a nutshell. It still kind of boggles
> > my mind that these weren't considered as backends, given that their
> > protocols even match reasonably well to the parts of Zaqar's API that
> > pertain to messages. They may not be sexy, but I'd be quite surprised
> > if there wasn't an SMTP+IMAP combination out there that can scale at
> > least as cost effectively as Redis.
> 
> FWIW, the article doesn't say "Message access by ID" is important. It
> says that it's there now and we can't simply remove it because there are
> people already using it.
> 

I think it has become clear, that it should be proposed for deprecation
as soon as a v2 API is published and available without those semantics.

> In the other hand, store-and-forward is indeed, as of now, a key and
> important point in Zaqar's architecture. The choices the team has made
> for the storage drivers have been questioned a lot since its inception.
> At the incubation meeting the feedback was that mongodb's AGPL license
> was not good, hence sqlalchemy was added. At the first graduation
> attempt the sqlalchemy driver didn't turn out to be as good as we hoped,
> hence redis was added. And now IMAP. Don't get me wrong, I'm not trying
> to be mean/jerk, really. What I'm trying to get at is that behind the
> choices the team has made there were technical aspects that were taken
> under consideration. I know you're not saying there weren't but I think
> it's worth highlighting this.

I also don't mean to suggest that you _must_ try IMAP. I meant to say,
you might find it quite easy to map the driver requirements to IMAP,
and that there are quite a few very scalable IMAP services available as
storing lots of messsages is an old problem that has been solved for a
long time.

Apologies if my message dismissed your previous contributions, which are
significant and impressive. I only mean to say: wow, that looks like
email to me.