[openstack-dev] [oslo] zeromq work for kilo

James Page james.page at ubuntu.com
Thu Sep 18 07:52:03 UTC 2014


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 17/09/14 15:34, Doug Hellmann wrote:
> This thread [1] has turned more “future focused", so I’m moving the
> conversation to the -dev list where we usually have those sorts of
> discussions.
[...]
>> I'd also like to document the current design of the ZMQ driver -
>> Doug - where is the best place todo this? I thought in the source
>> tree somewhere.
> 
> The documentation in the oslo.messaging repository [2] would be a
> good place to start for that. If we decide deployers/operators need
> the information we can either refer to it from the guides managed
> by the documentation team or we can move/copy the information. How
> about if you start a new drivers subdirectory there, and add
> information about zmq. We can have other driver authors provide
> similar detail about their drivers in the same directory.
> 
> [2]
> http://git.openstack.org/cgit/openstack/oslo.messaging/tree/doc/source

OK
> 
- - I'll raise a review with some appropriate design documentation
sometime in the next few weeks.

>> 2) a list of the critical bugs that need to be fixed + any
>> existing patches associated with those bugs, so they can be
>> reviewed early in kilo
[...]
> That’s a good list, and shorter than I expected. I have added these
> bugs to the next-kilo milestone.

Thanks

>> 3) an analysis of what it would take to be able to run
>> functional tests for zeromq on our CI infrastructure, not
>> necessarily the full tempest run or devstack-gate job, probably
>> functional tests we place in the tree with the driver (we will be
>> doing this for all of the drivers) + besides writing new
>> functional tests, we need to bring the unit tests for zeromq into
>> the oslo.messaging repository
>> 
>> Kapil Thangavelu started work on both functional tests for the
>> ZMQ driver last week; the output from the sprint is here:
>> 
>> https://github.com/ostack-musketeers/oslo.messaging
>> 
>> it covers the ZMQ driver (including messaging through the
>> zmq-receiver proxy) and the associated MatchMakers (local, ring,
>> redis) at a varying levels of coverage, but I feel it moves
>> things in the right direction - Kapil's going to raise a review
>> for this in the next couple of days.
>> 
>> Doug - has any structure been agreed within the oslo.messaging
>> tree for unit/functional test splits? Right now we have them all
>> in one place.
> 
> I think we will want them split up, but we don’t have an agreed
> existing structure for that. I would like to see a test framework
> of some sort that defines the tests in a way that can be used to
> run the same functional for all of the drivers as separate jobs
> (with appropriate hooks for ensuring the needed services are
> running, etc.). Setting that up warrants its own spec, because
> there are going to be quite a few details to work out. We will also
> need to participate in the larger conversation about how to set up
> those functional test jobs to be consistent with the other
> projects.

I guess we can put then up for review as is and then refactor to
support any future framework changes that come along.

>> 4) and some improvements that we would like to make longer term
>> 
>> a) Connection re-use on outbound messaging avoiding the current
>> tcp setup overhead for every sent message.  This may also bring
>> further performance benefits due to underlying messaging batching
>> in ZMQ.
> 
> This sounds like it would be a good thing to do, but making what we
> have work correctly and testing it feels more important for now.

Agreed - these are all futures.

>> b) Moving from tcp PUSH/PULL sockets between servers to
>> DEALER/DEALER (or something similar) to allow for heartbeating
>> and more immediate failure detection
> 
> I would need to understand how much of a rewrite that represents
> before commenting further.

Sorry - that item was a little hand-wavey - it could be minimal in
terms of impact but I would probably see it tied to a) so it might
involve the zmq-receiver moving towards zmq-proxy for both inbound and
outbound tcp connections.

>> 
>> c) Crypto support
> 
> There are some other discussions about adding crypto to messaging,
> and I hope we can do that without having to touch each driver, if
> possible.

That would be even better IMHO.

- -- 
James Page
Ubuntu and Debian Developer
james.page at ubuntu.com
jamespage at debian.org
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJUGo8jAAoJEL/srsug59jDAMMP/0RCc3ElFC/E/UqZJrY+X2oE
YMB87TM06AX26wS2uKe7F7xEAsv+vMTcSK/FC13xGfd+xBTT13pHquBQjeHwUx2C
+2Fbp6QoLwbvZF69IWL0E/8M/KueGth+JrHRpPqQqK5OcuwmDbDOeq2eychDMg1w
pkpXF0PPsa0tBRo3VQUHc43po9fAqT9Vpf8nG1M5vNTuOcCnrrmGg3DOFCzjVlG1
u2Mfo3HNIL48ZdkrSLHMk+7V1j4qHdU8ADxKaE3aPbxAaSieTlsKtf4pxJSM9Ikg
6WBsCChIVZA62Ox9xRZntidWdHTKcc4Lsv/K9ZhnFzST6OoWvEkpNxol1g/YX9lV
30TUdVBhNGm+ZU+2J8vxJ5suyVD0oW/J2hZGwhomi3tpKk9euKb5HMe5Ln0Fy1k9
63//XE8IzGIkNzwnBd4qnEubo1hg7Jfuw30uij9jJ85IdWRrpXiiei4XKZNJsAgD
/RsKzQt+BwlcztcWoQ0RX3OuOonQXb+5EgfqBce8DDHs+xiGJ6ven8gUi744Mpjc
XzsyfuHLPFsQNS7vO0Qkdgw4Q5o9Es/HliMeVdSRNHn+YraAmJHFTZMIu12wcRhT
FYtgoh2aFoCshOD9O5Yte4xAuvF25cB3HdBIPqySN/n0f2q8DEQr4heaeY/D1yyr
1ce8CaKYNUrOG5qeVApQ
=CkAe
-----END PGP SIGNATURE-----



More information about the OpenStack-dev mailing list