Open Stack

On 11/19/2015 5:52 PM, gord chung wrote:
> ceilometer cares. we listen to all notifications and build Measurement
> and Event data from (some of) them. to be honest, i don't know if
> changes in nova notifications have/are broken in ceilometer because
> quite frankly there are far too many notifications to test and track.
>
> as a consumer of data, we don't really care what the format of the
> entire payload is. we just care that the select attributes we do care
> about are present and in some known position.
>
> in ceilometer, we have two mapping files which define the notifications
> we are interested in and how we to normalise them[1][2]. currently, they
> are two files contained within ceilometer which we load at runtime. that
> said, at the summit we talked about how the projects should own their
> own definitions rather than ceilometer[3]. essentially, producer and
> consumer schemas are tightly coupled.
>
> the basic premise was that we as consumers have no idea what is getting
> produced, but the projects do know and they know exactly what is useful
> in what they are producing. so if nova owned their own meters.yaml and
> events.yaml, nova devs could easily edit the appropriate file themselves
> when they add/modify/remove notification payloads. if a new metric is
> added to a notification, it can easily be added to meters.yaml; if a
> metric value is moved, meters.yaml can be updated for old and new
> locations.
>
> using each projects definition files, ceilometer would load them all in
> and as a consumer, not have to worry about what every single change that
> happens outside of its domain. it's a different take on it: instead of
> having ownership on consumer to interpret accurately, the ownership is
> on producer to ensure consumer can understand.
>
> we are working on a POC currently as there some logistics to be thought
> out still(ie. how ceilometer knows where to look for each projects
> definition files) but using this idea, we wouldn't even need to know
> what version of notification we were getting, just that we are told that
> we should look "either here or here...". if there's interest in
> exploring this idea or if there are any glaring issues please let us know.
>
> [1]
> https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml
>
> [2]
> https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml
>
> [3] https://etherpad.openstack.org/p/mitaka-telemetry-cross-project
>
> On 19/11/2015 5:05 PM, Ryan Rossiter wrote:
>> Reading through [1] I started getting worries in the back of my head
>> about versioning these notifications. The main concern being how can
>> the consumer know about the versions and what's different between
>> them? Because these versioned notification payloads hold live nova
>> objects, there can be a lot of rug-pulling going on underneath these
>> notifications. If the payload doesn't pin itself to a certain level of
>> the object, a consumer can never be guaranteed the version of the
>> payload's object they will be receiving. I ran through a few of the
>> scenarios about irregular versions in the notifications subteam
>> meeting on Tuesday [2].
>>
>> My question is.... do we care about the consumer? Or is it a case of
>> "the consumer is always right" so we need to make sure we hand them
>> super consistent, well-defined blobs across the wire? Consumers will
>> have no idea of nova object internals, unless they feel like `python
>> -c import nova`. I do think we get one piece of help from o.vo though.
>> When the object is serialized, it hands the version with the object.
>> So consumers can look at the object and say "oh, I got 1.4 I know what
>> to do with this". But... they will have to implement their own compat
>> logic. Everyone will have to implement their own compat logic.
>>
>> We could expose a new API for getting the schema for a specific
>> version of a notification, so a consumer will know what they're
>> getting with their notifications. But I think that made mriedem
>> nauseous. We could make an oslo library that stuffs a shim in between
>> o.vo and nova's notifications to help out with compat/versioning, but
>> that sounds like a lot of work, especially because the end goal is
>> still not clearly defined.
>>
>> Thoughts?
>>
>> [1] https://review.openstack.org/#/c/247024
>> [2]
>> http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29
>>
>>
>

I don't think nova is going to maintain it's own meters/events yaml 
files in the form that ceilometer necessarily expects/uses today, since 
there are other things out there that consume nova notifications besides 
ceilometer. But I think you're thinking along the same lines of what 
Ryan was thinking, i.e. somehow nova providing a schema or some way for 
the consumer to digest/interpret what they are getting, but I don't know 
what that would look like or how it would be used by the consumer.

-- 

Thanks,

Matt Riedemann