Open Stack

tl;dr: Having a minimum payload standard enables more and more robust
services on both sides of notification bus. Expanded earnest optimism
below.

On Thu, 10 Jul 2014, Eoghan Glynn wrote:

> So it's the standardization of notification payload formats that's
> being proposed.

Generally that payload can be represented as a sequence of
dictionaries of key value pairs that, in the Ceilometer context,
is translated into a Sample. In a general sense is an event with an
actor, a volume, a unit, and some associated metadata.

As clarified elsewhere in the thread we already, in a sort of squishy
way, have a standard format for the envelope of a notification. We do
not have a standard format for the payload. I think we should (more on
that below).

> Well, the notifications emitted are intended to be general-purpose
> in nature.

The general purpose is that they are notifications of some info,
somewhat like described above.

> Ceilometer, StackTack OTOH put those notifications to a specific
> (and slightly different) purpose.
>
> Having the services emit exactly the sample format that ceilometer
> desired would save ceilometer itself doing the transformation, but
> would make the notifications less useful to other consumers.

I'm afraid I'll have to respectfully disagree on "less useful" and in
the spirit of good discussion present my case a bit more fully:

Having a large body of sevices emit a _known_ and well described sample
format (i.e. a "protocol") will make it possible for _many_ services,
including but not limited to Ceilometer and StackTach, to consume
those samples.

Enabling many services to publish samples notifications and many other
services to consume those samples (especially those services and samples
we haven't thought of or even imagined yet) is exactly why I'm arguing
this point.

If the payload had a guaranteed base form with known structure and keys
then consumption becomes easier. The structure doesn't have to limit
what _can_ be there, it should describe what must be there to be
adequate: if there are multiple samples, how is that sequence
represented (a list); in each sample what are the required fields (name,
time, unit, volume to guess at a few); in the parent framing what are
the required fields (e.g source, type).

I just started the code for processing of notifications from Ironic.
Conceptually they are the same as notifications from Nova but the
actual form of the payload is completely different. This means I have to
write a different processor for that payload. And now so does StackTach
if they want to handle it.

That seems completely ridiculous when all that is happening with the
data is that it is getting a bit sliced and diced and stored.

If I had a GeneralNotificationProcessor in Ceilo, all I would need to
do is, in config, say that I wanted to start accepting sample notifications
of the type that Ironic produces and wham, they're in there. StackTach
could do much the same (plus whatever else extra they like).

Yes, this moves the onus of creating well-formed metrics to notifying
systems, but that is good: It is providing exactly the sort of clean
and easy to test contract at the boundaries that leads to good
neighbors and easy testing.

>> Wouldn't the metering process be a lot easier if there was a
>> standardized package for a "sample" and anyone with the proper
>> credentials could drop a sample on the bus with the right exchange with
>> the right topic and anything (e.g. Ceilometer, StackTack, the
>> NewShinyMeteringShiz) that wants to consider itself a metering store can
>> consume it and hey presto.
>
> Problem is, the requirements of Ceilometer, StackTack, and indeed
> NewShinyMeteringShiz, are probably gonna be different.

Is that really a problem or do we fear it is? Don't they all want a
thing that has the same basic form and info (an event-like thingie
with a value).

>> If people are going to have to write a bunch of new tests and
>> related code to get notifications healthier why not make
>> notifications for metrics _healthy_ and available to any system
>> without needing to write a bunch of code on both sides of the bus.
>
> The intent was loose-coupling between the notifier and the notified.

Which intent? The current code certainly does not represent my
understanding of loose-coupling. It is, however, often a very
overloaded term so we may be meaning different things.

> These notifications pre-exist ceilometer, where never intended for
> ceilometer's exclusive use, and are in current use by frameworks
> other than ceilometer, each with differing requirements.

Yes, I understand most of that. The part I'm not convinced on is the
"differing requirements". If there are standard "events" then lots of
different (de-coupled) things can handle them, whatever their
requirements might be.

If I'm missing something critical about these differing requirements,
please enlighten me.

Having a standard notification payload format would of course mean change,
but we know that flexible metering/auditing is very important for the
OpenStack universe. Your argument seems to be that having such a
standard, predisposed to ceilometer, would limit flexibility and lose
capability. I'm suggesting that it would (after the hump of admittedly
quite a lot of work) increase flexiblity and save resources for
focusing on other capabilities (actually using the gathered data to do
interesting things).

>> Currently Ceilometer is required to know far too much about the
>> notifications it receives and that knowledge is being represented is
>> code. That is a BadThing™. I'm sure there are plenty of reasons for
>> why it has turned out that way, but if there is an opportunity for
>> change...?
>
> Not really, I don't think, TBH.

Take my example from above, the processing of Ironic notifications.
I think it is weird that I had to write code for that. Does it not
seem odd to you? If we had a standard format, we could have said,
in response to the initial proposal, "looks good but if you make the
data look like _this_ that would be totally wicked sweet!"

-- 
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent