[openstack-dev] [oslo][nova] Messaging: everything can talk to everything, and that is a bad thing

Flavio Percoco flavio at redhat.com
Tue Mar 22 13:15:13 UTC 2016

On 21/03/16 21:43 -0400, Adam Young wrote:
>I had a good discussion with the Nova folks in IRC today.
>My goal was to understand what could talk to what, and the short 
>according to dansmith
>" any node in nova land has to be able to talk to the queue for any 
>other one for the most part: compute->compute, compute->conductor, 
>conductor->compute, api->everything. There might be a few exceptions, 
>but not worth it, IMHO, in the current architecture."
>Longer conversation is here:
> http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2016-03-21.log.html#t2016-03-21T17:54:27
>Right now, the message queue is a nightmare.  All sorts of sensitive 
>information flows over the message queue: Tokens (including admin) are 
>the most obvious.  Every piece of audit data. All notifications and 
>all control messages.
>Before we continue down the path of "anything can talk to anything" 
>can we please map out what needs to talk to what, and why?  Many of 
>the use cases seem to be based on something that should be kicked off 
>by the conductor, such as "migrate, resize, live-migrate" and it 
>sounds like there are plans to make that happen.
>So, let's assume we can get to the point where, if node 1 needs to 
>talk to node 2, it will do so only via the conductor.  With that in 
>place, we can put an access control rule in place:

I don't think this is going to scale well. Eventually, this will require
evolving the conductor to some sort of message scheduler, which is pretty much
what the message bus is supposed to do.

>1.  Compute nodes can only read from the queue 
>2.  Compute nodes can only write to response queues in the RPC vhost
>3.  Compute nodes can only write to notification queus in the 
>notification host.
>I know that with AMQP, we should be able to identify the writer of a 
>message.  This means that each compute node should have its own user.  
>I have identified how to do that for Rabbit and QPid.  I assume for 
>0mq is would make sense to use ZAP (http://rfc.zeromq.org/spec:27) but 
>I'd rather the 0mq maintainers chime in here.

NOTE: Gentle reminder that qpidd has been removed from oslo.messaging.

I think you can configure rabbit, amqp1 and other technologies to do what you're
suggesting here without much trouble. TBH, I'm not sure how many chances would
be required in Nova (or even oslo.messaging) but I'd dare to say non are

>I think it is safe (and sane) to have the same use on the compute node 
>communicate with  Neutron, Nova, and Ceilometer.  This will avoid a 
>false sense of security: if one is compromised, they are all going to 
>be compromised.  Plan accordingly.
>Beyond that, we should have message broker users for each of the 
>components that is a client of the broker.
>Applications that run on top of the cloud, and that do not get 
>presence on the compute nodes, should have their own VHost.  I see 
>Sahara on my Tripleo deploy, but I assume there are others.  Either 
>they completely get their own vhost, or the apps should share one 
>separate from the RPC/Notification vhosts we currently have.  Even 
>Heat might fall into this category.
>Note that those application users can be allowed to read from the 
>notification queues if necessary.  They just should not be using the 
>same vhost for their own traffic.
>Please tell me if/where I am blindingly wrong in my analysis.

I guess my question is: Have you identified things that need to be changed in
any of the projects for this to be possible? Or is it a pure deployment

I'd argue that any change (assuming changes are required) are likely to happen
in specific projects (Nova, Neutron, etc) and that once this scenario is
supported, it'll remain a deployment choice to follow it or not. If I want my
undercloud services to use a single vhost and a single user, I must be able to
do that. The proposal in this email complicates deployments significantly,
despite it making sense from a security stand point.

One more thing. Depending on the messaging technology, having different virtual
hosts may have an impact on the performance when running under huge loads given
the fact that the data will be partitioned differently and, therefore,
written/read differently. I don't have good data at hand about this, sorry.


Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160322/a3fdf9c7/attachment.pgp>

More information about the OpenStack-dev mailing list