Open Stack

Hi Justin,

My though process is to go back to basics. To perform discovery there is no getting away from the fact that you have to start with a well-known address that your peers can access on the network. The second part is a service/protocol accessible at that address that can perform the discovery. So the questions are: what well-known addresses can I reach? And is that a suitable place to implement the service/protocol.

The metadata service is different to the others in that it can be accessed without credentials (correct me if I'm wrong), so it is the only possibility out of the openstack services if you do not want to have credentials on the peer instances. If that is not the case then the other services are options. All services require security groups and/or networks to be configured appropriately to access them.

(Yes, the question "can all instances access the same metadata service" did really mean are they all local. Sorry for being unclear. But I think your answer is yes, they are, right?)

Implementing the peer discovery in the instances themselves requires some kind of multicast or knowing a list of addresses to try. In both cases either the actual addresses or some name resolved through a naming service would do. Whatever is starting your instances does have access to at least nova, so it can find out if there are any running instances and what their addresses are. These could be used as the addresses they try first. These are the way that internet p2p services work and they work in the cloud.

So there are options. The metadata service is a good place in terms of accessibility, but may not be for other reasons. In particular, the lack of credentials relates to the fact it is only allowed to see its own information. Making that more dynamic and including information about other things in the system might change the security model slightly. Secondly, is it the purpose of the metadata server to do this job? That's more a matter of choice.

Personally, I think no, this is not the right place.

Paul.



From: Justin Santa Barbara [mailto:justin at fathomdb.com]
Sent: 24 January 2014 21:01
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Nova] bp proposal: discovery of peer instances through metadata service

Murray, Paul (HP Cloud Services)  wrote:

Multicast is not generally used over the internet, so the comment about removing multicast is not really justified, and any of the approaches that work there could be used.

I think multicast/broadcast is commonly used 'behind the firewall', but I'm happy to hear of any other alternatives that you would recommend - particularly if they can work on the cloud!


I agree that the metadata service is a sensible alternative. Do you imagine your instances all having access to the same metadata service? Is there something more generic and not tied to the architecture of a single openstack deployment?


Not sure I understand - doesn't every Nova instance has access to the metadata service, and they all connect to the same back-end database?  Has anyone not deployed the metadata service?  It is not cross-region / cross-provider - is that what you mean?  In terms of implementation (https://review.openstack.org/#/c/68825/) it is supposed to be the same as if you had done a list-instances call on the API provider.  I know there's been talk of federation here; when this happens it would be awesome to have a cross-provider view (optionally, probably).

Although this is a simple example, it is also the first of quite a lot of useful primitives that are commonly provided by configuration services. As it is possible to do what you want by other means (including using an implementation that has multicast within subnets - I'm sure neutron does actually have this), it seems that this makes less of a special case and rather a requirement for a more general notification service?

I don't see any other solution offering as easy a solution for users (either the developer of the application or the person that launches the instances).  If every instance had an automatic keystone token/trust with read-only access to its own project, that would be great.  If Heat intercepted every Nova call and added metadata, that would be great.  If Marconi offered every instance a 'broadcast' queue where it could reach all its peers, and we had a Keystone trust for that, that would be great.  But, those are all 12 month projects, and even if you built them and they were awesome they still wouldn't get deployed on all the major clouds, so I _still_ couldn't rely on them as an application developer.

My hope is to find something that every cloud can be comfortable deploying, that solves discovery just as broadcast/multicast solves it on typical LANs.  It may be that anything other than IP addresses will make e.g. HP public cloud uncomfortable; if so then I'll tweak it to just be IPs.  Finding an acceptable solution for everyone is the most important thing to me.  I am very open to any alternatives that will actually get deployed!

One idea I had: I could return a flat list of IPs, as JSON objects:

[
{ ip: '1.2.3.4' },
{ ip: '1.2.3.5' },
{ ip: '1.2.3.6' }
]

If e.g. it turns out that security groups are really important, then we can just pop the extra attribute into the same data format without breaking the API:

...
{ ip: '1.2.3.4', security_groups: [ 'sg1', 'sg2' ] }
...

Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140127/aaecbf85/attachment.html>