Open Stack

> tl;dr: Having a minimum payload standard enables more and more robust
> services on both sides of notification bus. 

Yes, that's exactly the point of this thread.

> We do not have a standard format for the payload. I think we should
> (more on that below).

Again, such standardization is exactly the point of this thread.

But I guess you're suggesting not only that we version/schematize
individual notification payloads, but that we do so in a way that's
global across event types and emitters?
 
> > Well, the notifications emitted are intended to be general-purpose
> > in nature.
> 
> The general purpose is that they are notifications of some info,
> somewhat like described above.

Well, yeah.
  
> If the payload had a guaranteed base form with known structure and keys
> then consumption becomes easier. The structure doesn't have to limit
> what _can_ be there, it should describe what must be there to be
> adequate: if there are multiple samples, how is that sequence
> represented (a list); in each sample what are the required fields (name,
> time, unit, volume to guess at a few); in the parent framing what are
> the required fields (e.g source, type).
> 
> I just started the code for processing of notifications from Ironic.
> Conceptually they are the same as notifications from Nova but the
> actual form of the payload is completely different. This means I have to
> write a different processor for that payload. And now so does StackTach
> if they want to handle it.

So for the purposes of grounding the discussion, can you give an
example of what the Ironic notification payload might look like in
a perfect world?

(Just a link to a paste of a hand-crafted example would suffice)
 
> Yes, this moves the onus of creating well-formed metrics to notifying
> systems, but that is good: It is providing exactly the sort of clean
> and easy to test contract at the boundaries that leads to good
> neighbors and easy testing.

OK, so the thing to note is that the set of potential consumers is
not limited to services with a metrics-oriented view of the world.

So for example it's been proposed, or is actually a reality, that
at least the following different types of beasts are interested in
consuming notifications:

 * services with a metrics-oriented PoV (e.g. Ceilometer)

 * services with an events-oriented PoV (e.g. StackTach)

 * services with a UX-oriented PoV (e.g. Horizon)

 * services with an entitlement-enforcement PoV (e.g. Katello)

I'm not sure that "moving the onus of creating well-formed metrics to
notifying systems" is feasible, when many of the actual and potential
consumers of these data don't actually have a strictly metrics-oriented
perspective.

> > Problem is, the requirements of Ceilometer, StackTack, and indeed
> > NewShinyMeteringShiz, are probably gonna be different.
> 
> Is that really a problem or do we fear it is? Don't they all want a
> thing that has the same basic form and info (an event-like thingie
> with a value).

Well I think it is a real problem, given the breath of consumers as
described above.

But in an case, I'm bit confused by where this line of reasoning is
headed. We seem to have gone from the notifying service directly
emitting well-formed samples that ceilometer can just directly
persist, to more generic "event-like thingies".

Have I misread that point about *directly* emitting samples?

> Having a standard notification payload format would of course mean change,
> but we know that flexible metering/auditing is very important for the
> OpenStack universe. Your argument seems to be that having such a
> standard, predisposed to ceilometer, would limit flexibility and lose
> capability.

Yep, that is exactly my point. Predisposition of the notification format
to ceilometer's needs is what concerns me. As opposed to the notion of
standardization/schematization/versioning which is the explicit goal of
the discussion.

> Take my example from above, the processing of Ironic notifications.
> I think it is weird that I had to write code for that. Does it not
> seem odd to you?

OK, so have we strayed into an orthogonal concern here?

i.e. not that ceilometer requires some translation to be done, but
that this translation must be hand-craft in Python code as opposed
to being driven declaratively via some configured mapping rules? 

Cheers,
Eoghan