Open Stack

Excerpts from Flavio Percoco's message of 2014-03-19 03:01:19 -0700:
> FWIW, I think there's a value on having an sqlalchemy driver. It's
> helpful for newcomers, it integrates perfectly with the gate and I
> don't want to impose other folks what they should or shouldn't use in
> production. Marconi may be providing a data API but it's still
> non-opinionated and it wants to support other drivers - or at least provide
> a nice way to implement them. Working on sqlalchemy instead of amqp (or
> redis) was decided in the incubation meeting.
> 
> But again, It's an optional driver that we're talking about here. As
> of now, our recommended driver is mongodb's and as I already mentioned
> in this email, we'll start working on an amqp's one, which will likely
> become the recommended one. There's also support for redis.
> 
> As already mentioned, we have plans to complete the redis driver and
> write an amqp based one and let them both live in the code base.
> Having support for different storage dirvers makes marconi's sharding
> feature more valuable.
> 
> 

Just to steer this back to technical development discussions a bit:

I suggest the sqla driver be removed. It will never be useful as a queue
backend. It will confuse newcomers because they'll see the schema and
think that it will work and then use it, and then they find out that SQL
is just not suitable for queueing about the time that they're taking a
fire extinguisher to their rack.

"Just use Redis" is pretty interesting as a counter to the concerns
MongoDB's license situation. Redis, AFAIK, does not have many of the
features that make MongoDB attractive for backing a queue. The primary
one that I would cite is sharding. While MongoDB will manage sharding
for you, Redis works more like Memcached when you want to partition[1].
This is particularly problematic for an operational _storage_ product
as that means if you want to offline a node, you are going to have to
consider what kind of partitioning Marconi has used, and how it will
affect the availability and durability of the data.

All of this to say, if Marconi is going to be high scale, I agree that
SQL can't be used, and even that MongoDB, on technical abilities alone,
makes some sense. But I think what might be simpler is if Marconi just
shifted focus to make the API more like AMQP, and used AMQP on its
backend. This allows cloud operators to deploy what they're used to for
OpenStack, and would still give users something they're comfortable with
(an HTTP API) to consume it.

[1] http://redis.io/topics/partitioning