Open Stack

On Fri, 11 Jul 2014, Eoghan Glynn wrote:

> A notification of compute.instance.create.start is naturally going to
> carry different types of data than a volume.snapshot.delete.end for
> example, but of course we'd seek to accommodate that difference within
> a generic structure as far as possible.

Is it going to carry different types of data or different values on
identical keys. The goal would be that the "meaning" of a notification
for compute.instance.create.start and volume.snapshot.delete.end
should be found on the same set of keys. That's all "the proposal" is.

>> Elsewhere there's been some discussion of ActivityStreams as a
>> possible model.
>
> Is that the social media syndication format you're talking about?
>
> That's been discussed elsewhere as possible model for openstack
> notifications? I missed that, can you provide a link?

I'm making something of a leap from

   http://lists.openstack.org/pipermail/openstack-dev/2014-July/039965.html

which links to

   https://wiki.openstack.org/wiki/NotificationSystem

which mentions PUSH, which, in the time since that NotificationSystem
document was written has evolved to be the transport for
ActivityStreams, where activities are events.

>> It may be, they have the same basic idea: "stuff that
>> happens" can be represented by a sequence of relatively stupid event-
>> like things.
>
> OK, I might be missing something here, but we seem to have a close
> approximation of that already:
>
>  stuff happens ==> events pop out on the bus
>
> So is your point that our events aren't dumb enough?

Events are too different in from, from event to event.

> (e.g. encode too much structure, or carry too much data, or require
> too much interpretation?)

Of these three, "require too much interpretation" is perhaps the best
match.

>> The intial versions of ActivityStreams was based on atom+xml but later
>> people realized "this would be a ton easier if it was some well known
>> and well formed JSON".
>
> Are you suggesting here we adopt atom+json?

No, I'm just charting the path from a complex over-schematized format
(atom+xml) through to one which is easier to consume (atom+json) to
one which is even easier to consume (a well known dictionary).

> Has the notification lost any expressive power? Not much by the looks
> of it.
>
> Is it a good idea to drop for example the error range on the floor
> because ceilometer doesn't know what to do with it?

The error range isn't being dropped: it would be in the extras because it
is of its own special meaning. If there's agreement that "error_range"
is a useful general key, then that would be a part of the standard
somewhere.

> Dunno, that would depend on whether (a) the delta value reported by
> IPMI is actually meaningful and (b) some other consumer can do something
> smart with it.

As I worried before, this concrete example seems to have diverted you to
focussing on the details of what is a complete strawman made in a few
minutes. Whether the delta value is dropped is an unknown at this time.

It might be that it could make sense for the producer to emit multiple
events where it now emits one. I don't really think that's that
important to the discussion, right now.

> I don't know enough about IPMI to comment on (a). My instinct would be
> to leave the door open for (b).

I don't reckon the door is being shut in any way.

> But overall, it doesn't look like too radical a change in format.

No, it's not. What it is is a minor change that "everyone" could adopt
and thus be able to play in the party.

> The metric-oriented view is subset/specialized aspect of something
> more generic.

Okay, so we agree on that. Can we agree that it might be possible to
represent that generic thing with a common grammar?

>> No, you are seeing the evolution of my thinking: as I write about this
>> stuff and try to understand it, it becomes more clear (to me) that
>> samples ought to be treated as event-like thingies.
>
> OK, so that shift makes it a little difficult to follow and engage with.

This is a mailing list for a distributed project, it is the sole place
where we can get some reasonable simulation of "having a chat" with a
wide body of interested parties. The point of having a chat is to
explore, think, and exchange and hone ideas. I certainly hope you
don't expect me and everyone else only to write to the list when we
have a fully researched proposal. This (apart from summits and
mid-cycles) is the most accessible place we have to which we can go to
create the information that allows us to do the compare and contrast
the eventually leads to creating proposals.

> OK, so perhaps I misread your earlier comments on samples and assumed
> a predisposition to ceilometer's requirements was being proposed?

It seems that way. Throughout I've been saying notifications should be
easy for anyone to create and anyone to consume and have confidence
they are doing it right.

> Can you start fleshing out exactly what you mean by a standard not
> necessarily predisposed to ceilometer, still sufficiently close to
> eliminate the need for a mapping layer on the ceilometer side?

That's what I'm trying to do here in this discussion here on this list
because as far as I'm able to tell that's the one way to ensure that
the idea has some validity and I'm not missing important details. I
thought I was saying "how about this general idea, any merit? Help me
flesh it out?" but apparently it came across as something different.

That's a shame. If I were made of less stern stuff I might be put off
trying to share ideas and make improvements: the hoops I'm leaping through
are making me a feel bit "whoa, man, really I don't want to have to try
this hard just to discuss an idea".

> And then please run this by the other stake-holders (e.g. StackTach)
> for a sanity check to confirm that it would complicate their lives
> unnecessarily.

I would have thought that was what I was already doing by posting in
this thread based on the title and the participants thus far, but
I'm assuming you mean something else?

>> I don't think so, it was supposed to be an example of the cost of there
>> not being an existing general standard. If there were such a standard I
>> wouldn't have had to write any code, only the Ironic folk would and I
>> would have had the free time to help them. Less code == good!
>
> Yes less code is great as long as we don't sacrifice the flexibility
> that we and other consumers need.

This ("flexibility") is a topic for a different conversation, I don't
want to throw us off this thread.

>> Similarly if there are individual schema for the various notification
>> types, every time someone wants to make a new notification they need
>> to get that schema validated and various agents and actors need to be
>> informed of its existence and then the new thing needs to be
>> integrated. That is limiting. If the contract can be managed at a higher
>> layer of abstraction more agents and actors can contribute.
>
> I don't really know what a "higher layer of abstraction" actually means
> when we talk about schema. More generic, less specific? Something else?

More generic and less specific. The scheme is for NotificationEvent,
not ComputeInstanceCreateStartNotificationEvent.

> OK just being devil's advocate for a second, it sounds like a truism
> to state we wouldn't need mapping logic if the mapping was done elsewhere.

Of course. The reason for doing that is because the publisher should
be the source of authority on what the mapping from
specific-thing-that-happened-here to NotificationEvent, not
Ceilometer. If it is Ceilometer then it means that StackTach must also
keep its own mapping. And so does every other consumer. The general
provision here is to get more DRY about information authority.

> If you can figure out how to do that in a concrete way ... then great,
> I'd be receptive, let's hear some proposals.

I've made a very limited one, how's it sound? Does it need an
implementation in order to warrant further discussion? Or would it
be better to toss it around a bit more? It makes no sense to me to
formalize something that has no potential legs.

Thanks for sticking through this.

-- 
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent