Open Stack

On Tue, 15 Jul 2014, Mark McLoughlin wrote:

> So you're proposing that all payloads should contain something like:
> [...] a class, type, id, value, unit and a space to put additional metadata.

That's the gist of it, but I'm only presenting that as a way to get
somebody else to point out what's wrong with it so we can get
closer to what's actually needed...

> On the subject of "notifications as a contract", calling the additional
> metadata field 'extra' suggests to me that there are no stability
> promises being made about those fields. Was that intentional?

...and as you point out, if everything that doesn't fit in the
"known" fields goes in 'extra' then the goal of contractual
stability may be lost.

What I think that shows us is that what we probably want is three
levels of contract. Currently we have one:

* There is a thing called a notification and it has a small number
   of top-level fields include 'payload'

At the second level would be:

* There is a list of things _in_ the payload which are events and
   they have some known general struture that allows injestion (as
   data) by lots of consumers.

And the third level would be:

* Each event has an internal structure (below the general structure)
   which is based on its type. In the simplest cases (some meters for
   example) a third level could either not be necessary or at least
   very small[1]. This is the badly named "extras" above.

Basically: If people are willing to pay the price (in terms of changes)
for contractual stability may as well get some miles out of it.
Three layers of abstraction means there can be three distinct levels
in applications or tools, each of which are optimized to a different
level of the topology: transporting messages, persisting/publishing
messages, extracting meaning from messages.

That's kind of present already, but it is done in a way that
requires a lot of duplication of knowledge between producer and
consumer and within different parts of the consumer. Which makes
effective testing and scaling more complex that it ought to be.

I know from various IRC chatter that this is a perennial topic round
these parts, frequently foundering, but perhaps each time we get a
bit further, learn a bit more?

In any case, as Eoghan and I stated elsewhere in the thread I'm
going to try to drive this forward, but I'm not going to rush it as
there's no looming deadline.

[1] Part of the reason I drove us off into the so-called weeds
earlier in the thread is a sense  that a large number of
events/samples/notification-payloads are capable of being classed as
nearly the same thing (except for their name and units) and thus
would not warrant their own specific schema. These are "just" events
that should have a very similar structure. If that structure is well
known and well accepted we will likely find that many existing
events can be bent to fit that structure for the sake of less code
and more reuse. As part of this process I'll try to figure out if this
is true .

-- 
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent