Open Stack

Excerpts from Ozgur Akan's message of 2014-03-20 14:18:27 -0700:
> Hi,
> 
> Marconi manages its own sharding  (doesn't rely on mongoDB's own sharding)
> in order to have more control on where data is stored. Sharding is done
> based on project_id + queue_id and stored in a catalog. Since Marconi
> manages it's own shards, it can use the same logic with any storage. If it
> was redis, scaling wouldn't be any different than having mongoDB as
> backend.
> 

Cool. Said catalog is duplicated globally then?

> Marconi (with some work) can also offer different backends at the same time
> to provide different performance / durability options to it's users. And
> users here are not operators but actual customers/users that are using the
> queuing service.
>

Right, sort of like with AMQP how you can ask for reliable delivery or
not?

> MongoDB seems to be a good choice as a storage backend as it doesn't need
> VRRP during failover which makes it much easier to deploy on top of
> OpenStack compute at times when moving a VIP can' t be done via VRRP. MySQL
> for example would require a VIP in order to endure a failed master. monoDB
> is relatively easier to manage (scale) when you have to migrate whole data
> from one cluster to another.
>

Using Galera, MySQL doesn't require a VIP approach either.

> I don't think RDBM is a bad idea but might not be practical. Mysql without
> sql interface can be fast;
> https://blogs.oracle.com/mysqlinnodb/entry/mysql_5_7_3_deep
> 

The SQL isn't the only problem, and speed isn't the same as scalability
(Fast: Ferrari, Scalable: Bullet Train). You also have MVCC. In InnoDB,
just inserting, updating, and deleting millions of tiny rows in a
concurrent fashion will tie up threads and mutexes, and bog down InnoDB
with millions of tiny transactions.

The linked blog is dealing entirely with scaling excessive tiny reads,
which is important, but not really the problem Marconi faces. There's no
point in discussing how to try and make MySQL or any other MVCC database
work well as a queue backend.

IMO, Look at how QPID and RabbitMQ do durable messaging... that is
the model to copy. But that is why the original email was "why isn't
Marconi just provisioning brokers?" because those brokers have already
implemented this and it seems wasteful to try and do it again.