Open Stack

Hey Clint,

Thanks for reading, some comments in-line:

On 09/04/2014 10:30 AM, Clint Byrum wrote:
> Excerpts from Flavio Percoco's message of 2014-09-04 00:08:47 -0700:

[snip]

>> - Concern on should we really reinvent a queue system rather than
>> piggyback on one
>>
>> As mentioned in the meeting on Tuesday, Zaqar is not reinventing message
>> brokers. Zaqar provides a service akin to SQS from AWS with an OpenStack
>> flavor on top. [0]
>>
> 
> I think Zaqar is more like SMTP and IMAP than AMQP. You're not really
> trying to connect two processes in real time. You're trying to do fully
> asynchronous messaging with fully randomized access to any message.
> 
> Perhaps somebody should explore whether the approaches taken by large
> scale IMAP providers could be applied to Zaqar.
> 
> Anyway, I can't imagine writing a system to intentionally use the
> semantics of IMAP and SMTP. I'd be very interested in seeing actual use
> cases for it, apologies if those have been posted before.
> 
>> Some things that differentiate Zaqar from SQS is it's capability for
>> supporting different protocols without sacrificing multi-tenantcy and
>> other intrinsic features it provides. Some protocols you may consider
>> for Zaqar are: STOMP, MQTT.
>>
>> As far as the backend goes, Zaqar is not re-inventing it either. It sits
>> on top of existing storage technologies that have proven to be fast and
>> reliable for this task. The choice of using NoSQL technologies has a lot
>> to do with this particular thing and the fact that Zaqar needs a storage
>> capable of scaling, replicating and good support for failover.
>>
> 
> What's odd to me is that other systems like Cassandra and Riak are not
> being discussed. There are well documented large scale message storage
> systems on both, and neither is encumbered by the same licensing FUD
> as MongoDB.

FWIW, they both have been discussed. As far as Cassandra goes, we raised
the red flag after reading reading this post[0]. The post itself may be
obsolete already but I don't think I have enough knowledge about
Cassandra to actually figure this out. Some folks have come to us asking
for a Cassandra driver and they were interested in contributing/working
on one. I really hope that will happen someday, although it'll certainly
happen as an external driver. Riak, on the other hand, was certainly a
good candidate. What made us go with MongoDB and Redis is they're both
good for the job, they are both likely already deployed in OpenStack
clouds and we have enough knowledge to provide support and maintenance
for both drivers.

As a curious note, ElasticSearch and Swift have also been brought up
several times as valid stores for Zaqar. I haven't thought about this
throughly but I think they'd do a good job.

[0]
http://www.datastax.com/dev/blog/cassandra-anti-patterns-queues-and-queue-like-datasets

> Anyway, again if we look at this as a place to storage and retrieve
> messages, and not as a queue, then talking about databases, instead of
> message brokers, makes a lot more sense.
> 
>>
>> - concern on the maturity of the NoQSL not AGPL backend (Redis)
>>
>> Redis backend just landed and I've been working on a gate job for it
>> today. Although it hasn't been tested in production, if Zaqar graduates,
>> it still has a full development cycle to be tested and improved before
>> the first integrated release happens.
>>
> 
> I'd be quite interested to see how it is expected to scale. From my very
> quick reading of the driver, it only supports a single redis server. No
> consistent hash ring or anything like that.

Indeed, support for redis-cluster is in our roadmap[0]. As of now, it
can be scaled by using Zaqar pools. You can create several pools of
redis nodes, that you can balance between queues.

The next series of benchmarks will be done on this new Redis driver. I
hope those will be ready soon.

[0] https://blueprints.launchpad.net/zaqar/+spec/redis-pool


>> # Use Cases
>>
>> In addition to the aforementioned concerns and comments, I also would
>> like to share an etherpad that contains some use cases that other
>> integrated projects have for Zaqar[0]. The list is not exhaustive and
>> it'll contain more information before the next meeting.
>>
>> [0] https://etherpad.openstack.org/p/zaqar-integrated-projects-use-cases
>>
> 
> Just taking a look, there are two basic applications needed:
> 
> 1) An inbox. Horizon wants to know when snapshots are done. Heat wants
> to know what happened during a stack action. Etc.
> 
> 2) A user-focused message queue. Heat wants to push data to agents.
> Swift wants to synchronize processes when things happen.
> 
> To me, #1 is Zaqar as it is today. #2 is the one that I worry may not
> be served best by bending #1 onto it.

Push semantics are being developed. We've had enough discussions that
have helped preparing the ground for it. However, I believe both use
cases could be covered by Zaqar as-is.

Could you elaborate a bit more on #2? Especially on why you think Zaqar
as is can't serve this specific case?

Also, feel free to add use-cases to that etherpad if there's anything
missing. :D

Cheers,
Flavio

-- 
@flaper87
Flavio Percoco