Open Stack

> I'd have a discussion on how to handle the counter relationship in
> multiple publisher.
> 
> 	This is also a partially reply to Doug's comments in
> 	https://review.openstack.org/#/c/18020/ "
> "The main issue is I don't understand the way the pipeline manager
> organizes its mapping of pipelines around a tuple of counter names,
> instead of tracking which pipelines want data from each individual
> counter".
> 
> 	There are two type potential relationship between counters. One is
> 	counters from the same pollster, another one is counters from
> 	different pollsters. I don't think we need consider counters from
> 	notification event (am I right?).
> 
> 1. Counters from the same pollster
> 	For counters from the same pollster, one example is counters from
> 	NetPollster, which include packets/bytes for each vNIC. The CW
> 	publisher need to calculate the overall packets/bytes from all
> 	vNIC, or some publisher need to calculate the average packet size
> 	of each vNIC.
> 	I considered 3 solutions for it.
> 	The first solution is to add a sequence_id field to counter, and all
> 	counters from the same NetPollster's emission have the same
> 	sequence_id. A transformer is used to cache all related counter
> 	till all related counter is received, e.g. a new sequence_id is
> 	received. 	However, the difficulties happens to the last round
> 	counters, since we will not receive any new counter, thus no idea
> 	if all the related counters received. Also we may need maintain the
> 	"sequence_id" if we will support multi-thread pollster.
> 
> 	The second solution is similar to first solution, but using
> 	timestamp instead of the sequence_id. If the timestamp with very
> 	near, they are assumed from same emission. It has the issues as
> 	first solution, also this method is not that accurate and may have
> 	corner case.
> 
> 	The third solution, also the solution in current patch, is to pass
> 	the counters as an iterator to pipelines, so that pipeline will
> 	always get these related counters as an iterator. One tricky thing
> 	here is, possibly some pipeline cares only part of these counters
> 	(e.g. one publisher only want to get the sent packets from all
> 	vNIC), thus a filter is needed to filter out un-needed counters.
> 	This is basically the logic of _setup_counters_mapping().
> 	With this solution, the pipeline interface is in fact counter array,
> 	instead of counter, and I think that's the major concern from Doug.
> 	Originally I thought it's ok since the pollster emits counter
> 	array, originally it's the main loop that decouples the array and
> 	now it's the pipeline decouples them. But I'm not quite sure.
> 
> 2. Counters from different pollster
> 	I don't think we need consider relationship among counters from
> 	different pollster, thus current patch does not support it. Or I'd
> 	prefer the customer of the publisher , like management software,
> 	cares for it, since there will be no difference.
> 	But if we do want to support it, the first solution above be more
> 	promising, although maintain the sequence_id among different
> 	pollster will be tricky.
> 
> Any idea/suggestion?

Slightly playing devil's advocate here, but could we simplify
this by requiring that the source of the these data also emits
the aggregate counter? (as well as the individual counters)

In a sense the source (the commute NetPollster in your example
above) is best placed to calculate the aggregate counter (say
the sum of rx/tx packets over all vNICs) without delving into
complicated sequence ID or timestamp-based schemes in order to
delegate this responsibility to the transformer chain.

In terms of duplication, the main things we want to avoid are
unnecessary measurements and duplicate metering data. The former
is not an issue in this case, whereas we can avoid both by the
metering publisher dropping the aggregate counter and the CW
transformer/publisher dropping the individual vNIC counters.

One counter-argument I guess is that there may be other usecases
as yet unenvisaged where the source pollster cannot easily
calculate the aggregate counter.

Cheers,
Eoghan