Open Stack

On Tue, 15 Jul 2014, Sandy Walsh wrote:

> This looks like a particular schema for one event-type (let's say
> "foo.sample").  It's hard to extrapolate this one schema to a generic
> set of common metadata applicable to all events. Really the only common
> stuff we can agree on is the stuff already there: tenant, user, server,
> message_id, request_id, timestamp, event_type, etc.

This is pretty much what I'm trying to figure out. We can,
relatively agree on a small set of stuff (like what you mention).
Presumably there are three more sets:

* special keys that could be changed to something more generally
   meaningful if we tried hard enough

* special keys that really are special and must be saved as such

* special keys that nobody cares about and can be tossed

Everybody thinks their own stuff is special[1] but it is often the case
that it's not.

In your other message you linked to http://paste.openstack.org/show/54140/
which shows some very complicated payloads (but only gets through
the first six events).

Is there related data (even speculative) for how many of those keys
are actually used?

And just looking at the paste (and the problem) generally, does it make
sense for the accessors in the dictionaries (the keys) to be terms which
are specific to the producer? Obviously that will increase the
appearance of disjunction between different events. A different
representation might not be as problematic.

Or maybe I'm completely wrong, just thinking out loud.

> This way, we can keep important notifications in a priority queue and
> handle them accordingly (since they hold important data), but let the
> samples get routed across less-reliable transports (like UDP) via the
> RoutingNotifier.

Presumably a more robust, uh, contract, for notifications, will
allow them to be dispatched (and re-dispatched) more effectively.

> Also, send the samples one-at-a-time and let them either a) drop on the
> floor (udp) or b) let the aggregator roll them up into something smaller
> (sliding window, etc). Making these large notifications contain a list
> of samples means we had to store state somewhere on the server until
> transmission time. Ideally something we wouldn't want to rely on.

I've wondered about this too. Is there history for why some of the
notifications which include samples are rolled up lists instead of
fired off one at a time. Seems like that will hurt parallelism
opportunities?

[1] There's vernacular here that I'd prefer to use but this is a
family mailing list.

-- 
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent