Open Stack

On 19/11/15 08:53 PM, Matt Riedemann wrote:
>
>
> On 11/19/2015 5:52 PM, gord chung wrote:
>> ceilometer cares. we listen to all notifications and build Measurement
>> and Event data from (some of) them. to be honest, i don't know if
>> changes in nova notifications have/are broken in ceilometer because
>> quite frankly there are far too many notifications to test and track.
>>
>> as a consumer of data, we don't really care what the format of the
>> entire payload is. we just care that the select attributes we do care
>> about are present and in some known position.
>>
>> in ceilometer, we have two mapping files which define the notifications
>> we are interested in and how we to normalise them[1][2]. currently, they
>> are two files contained within ceilometer which we load at runtime. that
>> said, at the summit we talked about how the projects should own their
>> own definitions rather than ceilometer[3]. essentially, producer and
>> consumer schemas are tightly coupled.
>>
>> the basic premise was that we as consumers have no idea what is getting
>> produced, but the projects do know and they know exactly what is useful
>> in what they are producing. so if nova owned their own meters.yaml and
>> events.yaml, nova devs could easily edit the appropriate file themselves
>> when they add/modify/remove notification payloads. if a new metric is
>> added to a notification, it can easily be added to meters.yaml; if a
>> metric value is moved, meters.yaml can be updated for old and new
>> locations.
>>
>> using each projects definition files, ceilometer would load them all in
>> and as a consumer, not have to worry about what every single change that
>> happens outside of its domain. it's a different take on it: instead of
>> having ownership on consumer to interpret accurately, the ownership is
>> on producer to ensure consumer can understand.
>>
>> we are working on a POC currently as there some logistics to be thought
>> out still(ie. how ceilometer knows where to look for each projects
>> definition files) but using this idea, we wouldn't even need to know
>> what version of notification we were getting, just that we are told that
>> we should look "either here or here...". if there's interest in
>> exploring this idea or if there are any glaring issues please let us 
>> know.
>>
>> [1]
>> https://github.com/openstack/ceilometer/blob/master/ceilometer/meter/data/meters.yaml 
>>
>>
>> [2]
>> https://github.com/openstack/ceilometer/blob/master/etc/ceilometer/event_definitions.yaml 
>>
>>
>> [3] https://etherpad.openstack.org/p/mitaka-telemetry-cross-project
>>
>> On 19/11/2015 5:05 PM, Ryan Rossiter wrote:
>>> Reading through [1] I started getting worries in the back of my head
>>> about versioning these notifications. The main concern being how can
>>> the consumer know about the versions and what's different between
>>> them? Because these versioned notification payloads hold live nova
>>> objects, there can be a lot of rug-pulling going on underneath these
>>> notifications. If the payload doesn't pin itself to a certain level of
>>> the object, a consumer can never be guaranteed the version of the
>>> payload's object they will be receiving. I ran through a few of the
>>> scenarios about irregular versions in the notifications subteam
>>> meeting on Tuesday [2].
>>>
>>> My question is.... do we care about the consumer? Or is it a case of
>>> "the consumer is always right" so we need to make sure we hand them
>>> super consistent, well-defined blobs across the wire? Consumers will
>>> have no idea of nova object internals, unless they feel like `python
>>> -c import nova`. I do think we get one piece of help from o.vo though.
>>> When the object is serialized, it hands the version with the object.
>>> So consumers can look at the object and say "oh, I got 1.4 I know what
>>> to do with this". But... they will have to implement their own compat
>>> logic. Everyone will have to implement their own compat logic.
>>>
>>> We could expose a new API for getting the schema for a specific
>>> version of a notification, so a consumer will know what they're
>>> getting with their notifications. But I think that made mriedem
>>> nauseous. We could make an oslo library that stuffs a shim in between
>>> o.vo and nova's notifications to help out with compat/versioning, but
>>> that sounds like a lot of work, especially because the end goal is
>>> still not clearly defined.
>>>
>>> Thoughts?
>>>
>>> [1] https://review.openstack.org/#/c/247024
>>> [2]
>>> http://eavesdrop.openstack.org/irclogs/%23openstack-meeting-alt/%23openstack-meeting-alt.2015-11-17.log.html#t2015-11-17T20:22:29 
>>>
>>>
>>>
>>
>
> I don't think nova is going to maintain it's own meters/events yaml 
> files in the form that ceilometer necessarily expects/uses today, 
> since there are other things out there that consume nova notifications 
> besides ceilometer. But I think you're thinking along the same lines 
> of what Ryan was thinking, i.e. somehow nova providing a schema or 
> some way for the consumer to digest/interpret what they are getting, 
> but I don't know what that would look like or how it would be used by 
> the consumer.
>

just to clarify, the idea doesn't involve tailoring the notification 
payload to ceilometer, just that if a producer is producing a 
notification it knows contains a useful datapoint, the producer should 
tell someone explicitly 'this datapoint exists'.

one question i have is: who is the intended recipient(s) of the 
notifications? surely, there should be some target(s) as you need to 
ensure the message has useful data in the payload for the consumer(s)...

when we talked about notification schemas months/years ago[1][2], it was 
discussed that maybe the service catalog or something would capture an 
endpoint where schemas can be retrieved -- this is one way. but to be 
honest, in the notification data flow, producers are the initiator, not 
the consumers. similar to when a manager runs through the day's tasks 
(yay! summer job grunt work): while it's nice to hear the whole story 
(be told the schema), it's even better if they just tell me what i need 
to know (consumerA, you want attribute x, consumerB you want attribute y).

a big reason i want to give these event/metric definitions to the 
projects is because that's exactly what people are asking for. there 
have been multiple times where devs complain it makes no sense that for 
them to add metric data for their own project, they need to edit 
ceilometer -- i agree with this 100%. ceilometer devs are not experts in 
these projects, the projects own devs are.

[1] http://lists.openstack.org/pipermail/openstack-dev/2014-July/039858.html
[2] 
http://lists.openstack.org/pipermail/openstack-dev/2015-January/054708.html

cheers,

-- 
gord