<div dir="ltr"><div>Hi,</div><div><br></div>Marconi manages its own sharding (doesn't rely on mongoDB's own sharding) in order to have more control on where data is stored. Sharding is done based on project_id + queue_id and stored in a catalog. Since Marconi manages it's own shards, it can use the same logic with any storage. If it was redis, scaling wouldn't be any different than having mongoDB as backend. <div>
<br></div><div>Marconi (with some work) can also offer different backends at the same time to provide different performance / durability options to it's users. And users here are not operators but actual customers/users that are using the queuing service. </div>
<div><br></div><div>MongoDB seems to be a good choice as a storage backend as it doesn't need VRRP during failover which makes it much easier to deploy on top of OpenStack compute at times when moving a VIP can' t be done via VRRP. MySQL for example would require a VIP in order to endure a failed master. monoDB is relatively easier to manage (scale) when you have to migrate whole data from one cluster to another.</div>
<div><br></div><div>I don't think RDBM is a bad idea but might not be practical. Mysql without sql interface can be fast;</div><div><a href="https://blogs.oracle.com/mysqlinnodb/entry/mysql_5_7_3_deep">https://blogs.oracle.com/mysqlinnodb/entry/mysql_5_7_3_deep</a><br>
</div><div><br></div><div>best wishes,</div><div>Oz</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Mar 20, 2014 at 2:56 PM, Clint Byrum <span dir="ltr"><<a href="mailto:clint@fewbar.com" target="_blank">clint@fewbar.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Excerpts from Flavio Percoco's message of 2014-03-19 03:01:19 -0700:<br>
> FWIW, I think there's a value on having an sqlalchemy driver. It's<br>
> helpful for newcomers, it integrates perfectly with the gate and I<br>
> don't want to impose other folks what they should or shouldn't use in<br>
> production. Marconi may be providing a data API but it's still<br>
> non-opinionated and it wants to support other drivers - or at least provide<br>
> a nice way to implement them. Working on sqlalchemy instead of amqp (or<br>
> redis) was decided in the incubation meeting.<br>
><br>
> But again, It's an optional driver that we're talking about here. As<br>
> of now, our recommended driver is mongodb's and as I already mentioned<br>
> in this email, we'll start working on an amqp's one, which will likely<br>
> become the recommended one. There's also support for redis.<br>
><br>
> As already mentioned, we have plans to complete the redis driver and<br>
> write an amqp based one and let them both live in the code base.<br>
> Having support for different storage dirvers makes marconi's sharding<br>
> feature more valuable.<br>
><br>
><br>
<br>
Just to steer this back to technical development discussions a bit:<br>
<br>
I suggest the sqla driver be removed. It will never be useful as a queue<br>
backend. It will confuse newcomers because they'll see the schema and<br>
think that it will work and then use it, and then they find out that SQL<br>
is just not suitable for queueing about the time that they're taking a<br>
fire extinguisher to their rack.<br>
<br>
"Just use Redis" is pretty interesting as a counter to the concerns<br>
MongoDB's license situation. Redis, AFAIK, does not have many of the<br>
features that make MongoDB attractive for backing a queue. The primary<br>
one that I would cite is sharding. While MongoDB will manage sharding<br>
for you, Redis works more like Memcached when you want to partition[1].<br>
This is particularly problematic for an operational _storage_ product<br>
as that means if you want to offline a node, you are going to have to<br>
consider what kind of partitioning Marconi has used, and how it will<br>
affect the availability and durability of the data.<br>
<br>
All of this to say, if Marconi is going to be high scale, I agree that<br>
SQL can't be used, and even that MongoDB, on technical abilities alone,<br>
makes some sense. But I think what might be simpler is if Marconi just<br>
shifted focus to make the API more like AMQP, and used AMQP on its<br>
backend. This allows cloud operators to deploy what they're used to for<br>
OpenStack, and would still give users something they're comfortable with<br>
(an HTTP API) to consume it.<br>
<br>
[1] <a href="http://redis.io/topics/partitioning" target="_blank">http://redis.io/topics/partitioning</a><br>
<div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>