Open Stack

Hi,

Marconi manages its own sharding  (doesn't rely on mongoDB's own sharding)
in order to have more control on where data is stored. Sharding is done
based on project_id + queue_id and stored in a catalog. Since Marconi
manages it's own shards, it can use the same logic with any storage. If it
was redis, scaling wouldn't be any different than having mongoDB as
backend.

Marconi (with some work) can also offer different backends at the same time
to provide different performance / durability options to it's users. And
users here are not operators but actual customers/users that are using the
queuing service.

MongoDB seems to be a good choice as a storage backend as it doesn't need
VRRP during failover which makes it much easier to deploy on top of
OpenStack compute at times when moving a VIP can' t be done via VRRP. MySQL
for example would require a VIP in order to endure a failed master. monoDB
is relatively easier to manage (scale) when you have to migrate whole data
from one cluster to another.

I don't think RDBM is a bad idea but might not be practical. Mysql without
sql interface can be fast;
https://blogs.oracle.com/mysqlinnodb/entry/mysql_5_7_3_deep

best wishes,
Oz


On Thu, Mar 20, 2014 at 2:56 PM, Clint Byrum <clint at fewbar.com> wrote:

> Excerpts from Flavio Percoco's message of 2014-03-19 03:01:19 -0700:
> > FWIW, I think there's a value on having an sqlalchemy driver. It's
> > helpful for newcomers, it integrates perfectly with the gate and I
> > don't want to impose other folks what they should or shouldn't use in
> > production. Marconi may be providing a data API but it's still
> > non-opinionated and it wants to support other drivers - or at least
> provide
> > a nice way to implement them. Working on sqlalchemy instead of amqp (or
> > redis) was decided in the incubation meeting.
> >
> > But again, It's an optional driver that we're talking about here. As
> > of now, our recommended driver is mongodb's and as I already mentioned
> > in this email, we'll start working on an amqp's one, which will likely
> > become the recommended one. There's also support for redis.
> >
> > As already mentioned, we have plans to complete the redis driver and
> > write an amqp based one and let them both live in the code base.
> > Having support for different storage dirvers makes marconi's sharding
> > feature more valuable.
> >
> >
>
> Just to steer this back to technical development discussions a bit:
>
> I suggest the sqla driver be removed. It will never be useful as a queue
> backend. It will confuse newcomers because they'll see the schema and
> think that it will work and then use it, and then they find out that SQL
> is just not suitable for queueing about the time that they're taking a
> fire extinguisher to their rack.
>
> "Just use Redis" is pretty interesting as a counter to the concerns
> MongoDB's license situation. Redis, AFAIK, does not have many of the
> features that make MongoDB attractive for backing a queue. The primary
> one that I would cite is sharding. While MongoDB will manage sharding
> for you, Redis works more like Memcached when you want to partition[1].
> This is particularly problematic for an operational _storage_ product
> as that means if you want to offline a node, you are going to have to
> consider what kind of partitioning Marconi has used, and how it will
> affect the availability and durability of the data.
>
> All of this to say, if Marconi is going to be high scale, I agree that
> SQL can't be used, and even that MongoDB, on technical abilities alone,
> makes some sense. But I think what might be simpler is if Marconi just
> shifted focus to make the API more like AMQP, and used AMQP on its
> backend. This allows cloud operators to deploy what they're used to for
> OpenStack, and would still give users something they're comfortable with
> (an HTTP API) to consume it.
>
> [1] http://redis.io/topics/partitioning
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140320/a5faebea/attachment.html>