Open Stack

On Wed, 2014-03-19 at 10:17 +1300, Robert Collins wrote:
> So this came up briefly at the tripleo sprint, and since I can't seem
> to find a /why/ document
> (https://wiki.openstack.org/wiki/Marconi/Incubation#Raised_Questions_.2B_Answers
> and https://wiki.openstack.org/wiki/Marconi#Design don't supply this)

I think we need a slight reset on this discussion. The way this email
was phrased gives a strong sense of "Marconi is a dumb idea, it's going
to take a lot to persuade me otherwise".

That's not a great way to start a conversation, but it's easy to
understand - a TC member sees a project on the cusp of graduating and,
when they finally get a chance to look closely at it, a number of things
don't make much sense. "Wait! Stop! WTF!" is a natural reaction if you
think a bad decision is about to be made.

We've all got to understand how pressurized a situation these graduation
and incubation discussions are. Projects put an immense amount of work
into proving themselves worthy of being an integrated project, they get
fairly short bursts of interaction with the TC, TC members aren't
necessarily able to do a huge amount of due diligence in advance and yet
TC members are really, really keen to avoid either undermining a healthy
project around some cool new technology or undermining OpenStack by
including an unhealthy project or sub-par technology.

And then there's the time pressure where a decision has to be made by a
certain date and if that decision is "not this time", the six months
delay until the next chance for a positive decision can be really
draining on motivation and momentum when everybody had been so focused
on getting a positive decision this time around.

We really need cool heads here and, above all, to try our best to assume
good faith, intentions and ability on both sides.


Some of the questions Robert asked are common questions and I know they
were discussed during the incubation review. However, the questions
persist and it's really important that TC members (and the community at
large) feel they can stand behind the answers to those questions. If I'm
chatting to someone and they ask me "why does OpenStack need to
implement its own messaging broker?", I need to have a good answer.

How about we do our best to put the implications for the graduation
decision aside for a bit and focus on collaboratively pulling together a
FAQ that everyone can buy into? The "raised questions and answers"
section of the incubation review linked above is a good start, but I
think we can take this email as feedback that those questions and
answers need much improvement.

This could be a good pattern for all new projects - if the TC and the
new project can't work together to draft a solid FAQ like this, then
it's not a good sign for the project.

See below for my attempt to summarize the questions and how we might go
about answering them. Is this a reasonable start?

Mark.


Why isn't Marconi simply an API for provisioning and managing AMQP, Kestrel,
ZeroMQ, etc. brokers and queues? Why is a new broker implementation needed?

 => I'm not sure I can summarize the answer here - the need for a HTTP data
    plane API, the need for multi-tenancy, etc.? Maybe a table listing the
    required features and whether they're provided by these existing solutions.

    Maybe there's also an element of "we think we can do a better job". If so,
    the point probably worth addressing is "OpenStack shouldn't attempt to write
    a new database, or a new hypervisor, or a new SDN controller, or a new block
    storage implementation ... so why should we write a implement a new message
    broker? If this is just a bad analogy, explain why?

Implementing a message queue using an SQL DB seems like a bad idea, why is
Marconi doing that?

 => Perhaps explain why MongoDB is a good storage technology for this use case
    and the SQLalchemy driver is just a toy.

Marconi's default driver depends on MongoDB which is licensed under the AGPL.
This license is currently a no-go for some organizations, so what plans does
Marconi have to implement another production-ready storage driver that supports
all API features?

 => Discuss the Redis driver plans?

Is Marconi designed to be suitable for use by OpenStack itself?

 => Discuss that it's not currently in scope and why not. In what way does the
    OpenStack use case differ from the applications Marconi's current API
    focused on?

How should a client subscribe to a queue?

 => Discuss that it's not by GET /messages but instead POST /claims?limit=N