Open Stack

Hi Li Ma,

Thank you very much for your reply

On 06.03.15 05:01, Li Ma wrote:
> Hi all, actually I'm writing the same mail topic for zeromq driver,
> but I haven't done it yet. Thank you for proposing this topic,
> ozamiatin.
>
> 1. ZeroMQ functionality
>
> Actually I proposed a session topic in the coming summit to show our
> production system, named 'Distributed Messaging System for OpenStack
> at Scale'. I don't know whether it will be allowed to present.
> Otherwise, if it is possible, I can share my experience in design
> summit.
>
> Currently, AWCloud (the company I'm working) deployed more than 20
> private clouds and 3 public clouds for our customers in production,
> scaling from 10 to 500 physical nodes without any performance issue.
> The performance dominates all the existing drivers in every aspect.
> All is using ZeroMQ driver. We started improving ZeroMQ driver in
> Icehouse and currently the modified driver has switched to
> oslo.messaging.
>
> As all knows, ZeroMQ has been unmaintainable for long. My colleagues
> and I continuously contribute patches to upstream. The progress is a
> little bit slow because we are doing everything just in our spare time
> and the review procedure is also not efficient.
>
> Here are two important patches [1], [2], for matchmaker redis. When
> they are landed, I think ZeroMQ driver is capable of running in small
> deployments.
It's good to hear you have a living deployment with zmq driver.
Is there a big divergence between your production and upstream versions
of the driver? Besides [1] and [2] fixes for redis we have [5] and [6]
critical multi-backend related issues for using the driver in real-world 
deployment.
> The only functionality for large-scale deployment that lacks in the
> current upstream codebase is socket pool scheduling (to provide
> lifecycle management, including recycle and reuse zeromq sockets). It
> was done several months ago and we are willing to contribute. I plan
> to propose a blueprint in the next release.
Pool, recycle and reuse sounds good for performance.
We also need a refactoring of the driver to reduce redundant entities
or reconsider them (like ZmqClient or InternalContext) and to reduce 
code replications (like with topics).
There is also some topics management needed.
Clear code == less bugs == easy understand == easy contribute.
We need a discussion (with related spec and UMLs) about what the driver 
architecture should be (I'm in progress with that).
>
> 2. Why ZeroMQ matters for OpenStack
>
> ZeroMQ is the only driver that depends on a stable library not an open
> source product. This is the most important thing that comes up my
> mind. When we deploy clouds with RabbitMQ or Qpid, we need
> comprehensive knowledge from their community, from deployment best
> practice to performance tuning for different scales. As an open source
> product, no doubt that bugs are always there. You need to push lots of
> things in different communities rather than OpenStack community.
> Finally, it is not that working, you all know it, right?
>
> ZeroMQ library itself is just encapsulation of sockets and is stable
> enough and widely used in large-scale cluster communication for long.
> We can build our own messaging system for inter-component RPC. We can
> improve it for OpenStack and have the governance for codebase. We
> don't need to rely on different products out of the community.
> Actually, only ZeroMQ provides the possibility.
>
> IMO, we can just keep it and improve it and finally it becomes another
> choice for operators.
Zmq is also an opensource product and it has it's own community,
but I agree that rabbit is a complicated "blackbox" software we depend on.
While zmq is just a library and it is much more simpler (more reliable) 
inside.
Zmq driver is more lower-level than rabbit and qpid therefore it 
provides more flexibility.
By now it is the only driver where brokerless implementation is possible.
>
> 3. ZeroMQ integration
>
> I've been working on the integration of ZeroMQ and DevStack for a
> while and actually it is working right now. I updated the deployment
> guide [3].
That's true it works! :)
> I think it is the time to bring a non-voting gate for ZeroMQ and we
> can make the functional tests work.
You can turn it with 'check experimental'. It is broken now.
>
> 4. ZeroMQ blueprints
>
> We'd love to provide blueprints to improve ZeroMQ, as ozamiatin does.
> According to my estimation, ZeroMQ can be another choice for
> production in 1-2 release cycles due to bp review and patch review
> procedure.
>
> 5. ZeroMQ discussion
>
> Here I'd like to say sorry for this driver. Due to spare time and
> timezone, I'm not available for IRC or other meeting or discussions.
> But if it is possible, should we create a subgroup for ZeroMQ and
> schedule meetings for it? If we can schedule in advance or at a fixed
> date & time, I'm in.
That's great idea
+1 for zmq subgroup and meetings
> 6. Feedback to ozamiatin's suggestions
>
> I'm with you in most all the proposals, but for packages, I think we
> can just separate all the components in a sub-directory. This step is
> enough at the current stage.
>
> Packaging the components are complicated. I don't think it is possible
> for oslo.messaging to break into two packages, like oslo.messaging and
> oslo.messaging.zeromq. And I cannot see the benefit clearly.
Subfolder is actually what I mean (python package like '_drivers')
it should stay in oslo.messaging. Separate package like 
oslo.messaging.zeromq is overkill.
As Doug proposed we can do consistently to AMQP-driver.
>
> For priorities, I think the number 1, 6 and 7 have the high priority,
> especially 7. Because ZeroMQ is pretty new for everyone, we do need
> more paper work to promote and introduce it to the community. By the
> way, I made a wiki before and everyone is welcome to update it [4].
>
> [1] https://review.openstack.org/#/c/152471/
> [2] https://review.openstack.org/#/c/155673/
> [3] https://review.openstack.org/#/c/130943/
> [4] https://wiki.openstack.org/wiki/ZeroMQ
[5] https://bugs.launchpad.net/oslo.messaging/+bug/1282297
[6] https://bugs.launchpad.net/ubuntu/+source/oslo.messaging/+bug/1381972
>
> Thanks a lot,
> Li Ma (Nick)
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Regards,
Oleksii Zamiatin