Open Stack

So this came up briefly at the tripleo sprint, and since I can't seem
to find a /why/ document
(https://wiki.openstack.org/wiki/Marconi/Incubation#Raised_Questions_.2B_Answers
and https://wiki.openstack.org/wiki/Marconi#Design don't supply this)
we decided at the TC meeting that I should raise it here.

Firstly, let me check my facts :) - Marconi is backed by a modular
'storage' layer which places some conceptual design constraints on the
storage backends that are possible (e.g. I rather expect a 0mq
implementation to be very tricky, at best (vs the RPC style front end
https://wiki.openstack.org/wiki/Marconi/specs/zmq/api/v1 )), and has a
hybrid control/data plane API implementation where one can call into
it to make queues etc, and to consume them.

The API for the queues is very odd from a queueing perspective -
https://wiki.openstack.org/wiki/Marconi/specs/api/v1#Get_a_Specific_Message
- you don't subscribe to the queue, you enumerate and ask for a single
message.

And the implementations in tree are mongodb (which is at best
contentious, due to the AGPL and many folks reasonable concerns about
it), and mysq.

My desires around Marconi are:
 - to make sure the queue we have is suitable for use by OpenStack
itself: we have a very strong culture around consolidating technology
choices, and it would be extremely odd to have Marconi be something
that isn't suitable to replace rabbitmq etc as the queue abstraction
in the fullness of time.
 - to make sure that deployers with scale / performance needs can have
that met by Marconi
 - to make my life easy as a deployer ;)

So my questions are:
 - why isn't the API a queue friendly API (e.g. like
https://github.com/twitter/kestrel - kestrel which uses the memcache
API, puts put into the queue, gets get from the queue). The current
API looks like pretty much the worst case scenario there - CRUD rather
than submit/retrieve with blocking requests (e.g. longpoll vs poll).
 - wouldn't it be better to expose other existing implementations of
HTTP message queues like nova does with hypervisors, rather than
creating our own one? E.g. HTTPSQS, RestMQ, Kestrel, queues.io.
   - or even do what Trove does and expose the actual implementation directly?
 - whats the plan to fix the API?
 - is there a plan / desire to back onto actual queue services (e.g.
AMQP, $anyof the http ones above, etc)
 - what is the current performance -  how many usecs does it take to
put a message, and get one back, in real world use? How many
concurrent clients can a single Marconi API server with one backing
server deliver today?

As background, 'implement a message queue in a SQL DB' is such a
horrid antipattern its been a standing joke in many organisations I've
been in - and yet we're preparing to graduate *exactly that* which is
frankly perplexing.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud