Open Stack

On Fri, 11 Jul 2014, Eoghan Glynn wrote:

> But I guess you're suggesting not only that we version/schematize
> individual notification payloads, but that we do so in a way that's
> global across event types and emitters?

That's partially correct. I'm suggesting the we consider standardizing a
general format for notification payloads across emitters. _Not_
individual notification payloads.

Schematizing individual payloads strikes me as heavyweight[1] when it
_might_[2] be easier to declare that "a list of dicts with certain
required fields" is sufficient to represent a multiplicity of value-
oriented events which are consumed as any of metric, event,
visualization data-point, etc.

Elsewhere there's been some discussion of ActivityStreams as a
possible model. It may be, they have the same basic idea: "stuff that
happens" can be represented by a sequence of relatively stupid event-
like things.

The intial versions of ActivityStreams was based on atom+xml but later
people realized "this would be a ton easier if it was some well known
and well formed JSON".

Take that line of evolution a little bit further and you've got "a list
of dicts with certain required fields".

(more below)

[1] A useful turn of phrase learned in this message:
http://lists.openstack.org/pipermail/openstack-dev/2014-July/039941.html
[2] Emphasis on the might, meaning might be worthy of consideration at
least as a strawman to help illuminate concerns and benefits.


> So for the purposes of grounding the discussion, can you give an
> example of what the Ironic notification payload might look like in
> a perfect world?

I can't give you a perfect example because I haven't had an
opportunity to understand all the needs and issues but a rough quickie
to try to illustrate the point is here:

     http://paste.openstack.org/show/86071/

The main difference there is that the value and the unit are provided
in known fields at a known descent into the data structure, rather than
needing to be extracted from strings that are values at custom keys.

I'd rather we not get hung up on the details of the representation as
that's the way rabbits go. If the concept has merit the representation
can follow.

>> Yes, this moves the onus of creating well-formed metrics to notifying
>> systems, but that is good: It is providing exactly the sort of clean
>> and easy to test contract at the boundaries that leads to good
>> neighbors and easy testing.
>
> OK, so the thing to note is that the set of potential consumers is
> not limited to services with a metrics-oriented view of the world.

I'm of the opinion that it _may_ be possible that "metrics-oriented"
is a subclass of something more generic that might be representable.
The CADF folk seem to think so (although I'm not suggesting we go down
that road, c.f. "heavyweight").

> * services with a metrics-oriented PoV (e.g. Ceilometer)
> * services with an events-oriented PoV (e.g. StackTach)
> * services with a UX-oriented PoV (e.g. Horizon)
> * services with an entitlement-enforcement PoV (e.g. Katello)

These all sound like there are in at least the same ballbark. I think
the common term would be "events" and a metric is a type of event?

> But in an case, I'm bit confused by where this line of reasoning is
> headed. We seem to have gone from the notifying service directly
> emitting well-formed samples that ceilometer can just directly
> persist, to more generic "event-like thingies".
>
> Have I misread that point about *directly* emitting samples?

No, you are seeing the evolution of my thinking: as I write about this
stuff and try to understand it, it becomes more clear (to me) that
samples ought to be treated as event-like thingies.

>> Having a standard notification payload format would of course mean change,
>> but we know that flexible metering/auditing is very important for the
>> OpenStack universe. Your argument seems to be that having such a
>> standard, predisposed to ceilometer, would limit flexibility and lose
>> capability.
>
> Yep, that is exactly my point. Predisposition of the notification format
> to ceilometer's needs is what concerns me. As opposed to the notion of
> standardization/schematization/versioning which is the explicit goal of
> the discussion.

Okay, and my proposal is not to have a standard predisposed to
ceilometer. It is to make a limited standard for a large class of
notifications and _not_ schematize individual notification types (more
on this aspect below) and then change ceilometer so it can use the plan.

>> Take my example from above, the processing of Ironic notifications.
>> I think it is weird that I had to write code for that. Does it not
>> seem odd to you?
>
> OK, so have we strayed into an orthogonal concern here?

I don't think so, it was supposed to be an example of the cost of there
not being an existing general standard. If there were such a standard I
wouldn't have had to write any code, only the Ironic folk would and I
would have had the free time to help them. Less code == good!

Similarly if there are individual schema for the various notification
types, every time someone wants to make a new notification they need
to get that schema validated and various agents and actors need to be
informed of its existence and then the new thing needs to be
integrated. That is limiting. If the contract can be managed at a higher
layer of abstraction more agents and actors can contribute.

> i.e. not that ceilometer requires some translation to be done, but
> that this translation must be hand-craft in Python code as opposed
> to being driven declaratively via some configured mapping rules?

I don't think there needs to be much, if any, mapping on the consumption
side of the notification process if there is a standard form in which
those notifications are emitted. In those cases where pipeline
transformation needs to be done ( multiple value gathering) the pipeline
can consume certain notifications and then emit more as the result of
the transformation.

-- 
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent