Open Stack

ceilometer cares. we listen to all notifications and build Measurement 
and Event data from (some of) them. to be honest, i don't know if 
changes in nova notifications have/are broken in ceilometer because 
quite frankly there are far too many notifications to test and track.

as a consumer of data, we don't really care what the format of the 
entire payload is. we just care that the select attributes we do care 
about are present and in some known position.

in ceilometer, we have two mapping files which define the notifications 
we are interested in and how we to normalise them[1][2]. currently, they 
are two files contained within ceilometer which we load at runtime. that 
said, at the summit we talked about how the projects should own their 
own definitions rather than ceilometer[3]. essentially, producer and 
consumer schemas are tightly coupled.

the basic premise was that we as consumers have no idea what is getting 
produced, but the projects do know and they know exactly what is useful 
in what they are producing. so if nova owned their own meters.yaml and 
events.yaml, nova devs could easily edit the appropriate file themselves 
when they add/modify/remove notification payloads. if a new metric is 
added to a notification, it can easily be added to meters.yaml; if a 
metric value is moved, meters.yaml can be updated for old and new locations.

using each projects definition files, ceilometer would load them all in 
and as a consumer, not have to worry about what every single change that 
happens outside of its domain. it's a different take on it: instead of 
having ownership on consumer to interpret accurately, the ownership is 
on producer to ensure consumer can understand.

we are working on a POC currently as there some logistics to be thought 
out still(ie. how ceilometer knows where to look for each projects 
definition files) but using this idea, we wouldn't even need to know 
what version of notification we were getting, just that we are told that 
we should look "either here or here...". if there's interest in 
exploring this idea or if there are any glaring issues please let us know.

[1] 
https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml
[2] 
https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml
[3] https://etherpad.openstack.org/p/mitaka-telemetry-cross-project

On 19/11/2015 5:05 PM, Ryan Rossiter wrote:
> Reading through [1] I started getting worries in the back of my head 
> about versioning these notifications. The main concern being how can 
> the consumer know about the versions and what's different between 
> them? Because these versioned notification payloads hold live nova 
> objects, there can be a lot of rug-pulling going on underneath these 
> notifications. If the payload doesn't pin itself to a certain level of 
> the object, a consumer can never be guaranteed the version of the 
> payload's object they will be receiving. I ran through a few of the 
> scenarios about irregular versions in the notifications subteam 
> meeting on Tuesday [2].
>
> My question is.... do we care about the consumer? Or is it a case of 
> "the consumer is always right" so we need to make sure we hand them 
> super consistent, well-defined blobs across the wire? Consumers will 
> have no idea of nova object internals, unless they feel like `python 
> -c import nova`. I do think we get one piece of help from o.vo though. 
> When the object is serialized, it hands the version with the object. 
> So consumers can look at the object and say "oh, I got 1.4 I know what 
> to do with this". But... they will have to implement their own compat 
> logic. Everyone will have to implement their own compat logic.
>
> We could expose a new API for getting the schema for a specific 
> version of a notification, so a consumer will know what they're 
> getting with their notifications. But I think that made mriedem 
> nauseous. We could make an oslo library that stuffs a shim in between 
> o.vo and nova's notifications to help out with compat/versioning, but 
> that sounds like a lot of work, especially because the end goal is 
> still not clearly defined.
>
> Thoughts?
>
> [1] https://review.openstack.org/#/c/247024
> [2] 
> http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29
>

-- 
gord