Open Stack

> Thanks for dusting this off. Versioning and published schemas for
> notifications are important to the StackTach team.  It would be nice to
> get this resolved. We're happy to help out.

Great!

> > A great outcome would include some or all of the following:
> >
> >  1. more complete in-tree test coverage of notification logic on the
> >     producer side
> 
> Ultimately this is the core problem. A breaking change in the
> notifications caused tests to fail in other systems. Should we be adding
> more tests or simply add version checking at the lower levels (like the
> first pass of RPC versioning did)?

Yes, that would definitely help.
 
> >  2. versioned notification payloads to protect consumers from breaking
> >     changes in payload format
> Yep, like RPC the biggies are:
> 1. removal of fields from notifications
> 2. change in semantics of a particular field
> 3. addition of new fields (not a biggie)

True, true and true.
 
> The urgency for notifications is a little different than RPC where there
> is a method on the other end expecting a certain format. Notifications
> consumers have to be a little more forgiving when things don't come in
> as expected.

Yes some tolerance can be expected, but whole point would be get away
from the current free-fir-all.
 
> This isn't a justification for breaking changes. Just stating that we
> have some leeway.

Sure.
 
> I guess it really comes down to systems that are using notifications for
> critical synchronization vs. purely informational.

Agreed. That's what I was getting at with the advisory versus actionable
distinction.

> >  3. external discoverability of which event types a service is emitting
> These questions can be saved for later, but ...
> 
> Is the use-case that a downstream system can learn which queue to
> subscribe to programmatically?
> 
> Is this a nice-to-have?
> 
> Would / should this belong in a metadata service?

Possibly. The idea was to facilitate the drive towards discoverability
for testing purposes.

> >  4. external discoverability of which event types a service is consuming
> 
> Isn't this what the topic queues are for? Consumers should only
> subscribe to the topics they're interested in.

Yeah, that one is questionable all right.

Again it relates back to the motivating discussion with QA  around 
branchless Tempest. The idea was to make it externally discoverable
which notifications that ceilometer was currently capable of and
configured to consume.

But on reflection it would be far more meaningful to surface which 
meters ceilo is currently capable of and configured to *produce*.

> > If you're thinking that sounds like a substantial chunk of cross-project
> > work & co-ordination, you'd be right :)
> 
> Perhaps notification schemas should be broken out into a separate
> repo(s)? That way we can test independent of the publishing system. For
> example, our notigen event simulator [5] could use it.
>
> These could just be dependent libraries/plugins to oslo.messaging.

That is an interesting idea, it would allow for the schemas to be
easily depended upon on the consumer side also for validation purposes.

> >  * acquiescence from non-integrated projects that currently consume
> >    these notifications
> >
> >    (we shouldn't, as good citizens, simply pull the rug out from under
> >    projects such as StackTach without discussion upfront)
> We'll adapt StackTach.v2 accordingly. StackTach.v3 is far less impacted
> by notification changes since they are offloaded and processed in a
> secondary step. Breaking changes will just stall the processing. I
> suspect .v3 will be in place before .v2 is affected.
> 
> Adding version handling to Stack-Distiller (our notification->event
> translator) should be pretty easy (and useful) [6]

Cool, sounds like this would not be disruptive to StackTach.
 
> > Any opinions on how desirable or necessary this is, and how the
> > detailed mechanics might work, would be welcome.
> 
> A published set of schemas would be very useful for StackTach, we'd love
> to help out in any way possible. In the near-term we have to press on
> under the assumption notification definitions are fragile.

Yes, the time horizon for this would be beyond the near-time.
 
> > Apologies BTW if this has already been discussed and rejected as
> > unworkable. I see a stalled versioned-notifications BP[2] and some
> > references to the CADF versioning scheme in the LP fossil-record.
> > Also an inconclusive ML thread from 2012[3], and a related grizzly
> > summit design session[4], but it's unclear to me whether these
> > aspirations got much traction in the end.
> 
> I'd really like to see the CADF work adopted vs. other, hand-rolled,
> schema solutions.

Yes, I need to learn more about CADF, but the general principle of
avoiding bicycle-reinvention applies.

Cheers,
Eoghan