Open Stack

I'd have a discussion on how to handle the counter relationship in multiple publisher.

	This is also a partially reply to Doug's comments in https://review.openstack.org/#/c/18020/ "
"The main issue is I don't understand the way the pipeline manager organizes its mapping of pipelines around a tuple of counter names, instead of tracking which pipelines want data from each individual counter".

	There are two type potential relationship between counters. One is counters from the same pollster, another one is counters from different pollsters. I don't think we need consider counters from notification event (am I right?).

1. Counters from the same pollster
	For counters from the same pollster, one example is counters from NetPollster, which include packets/bytes for each vNIC. The CW publisher need to calculate the overall packets/bytes from all vNIC, or some publisher need to calculate the average packet size of each vNIC.
	I considered 3 solutions for it. 
	The first solution is to add a sequence_id field to counter, and all counters from the same NetPollster's emission have the same sequence_id. A transformer is used to cache all related counter till all related counter is received, e.g. a new sequence_id is received. 	However, the difficulties happens to the last round counters, since we will not receive any new counter, thus no idea if all the related counters received. Also we may need maintain the "sequence_id" if we will support multi-thread pollster.

	The second solution is similar to first solution, but using timestamp instead of the sequence_id. If the timestamp with very near, they are assumed from same emission. It has the issues as first solution, also this method is not that accurate and may have corner case.

	The third solution, also the solution in current patch, is to pass the counters as an iterator to pipelines, so that pipeline will always get these related counters as an iterator. One tricky thing here is, possibly some pipeline cares only part of these counters (e.g. one publisher only want to get the sent packets from all vNIC), thus a filter is needed to filter out un-needed counters. This is basically the logic of _setup_counters_mapping(). 
	With this solution, the pipeline interface is in fact counter array, instead of counter, and I think that's the major concern from Doug. Originally I thought it's ok since the pollster emits counter array, originally it's the main loop that decouples the array and now it's the pipeline decouples them. But I'm not quite sure.

2. Counters from different pollster
	I don't think we need consider relationship among counters from different pollster, thus current patch does not support it. Or I'd prefer the customer of the publisher , like management software, cares for it, since there will be no difference.
	But if we do want to support it, the first solution above be more promising, although maintain the sequence_id among different pollster will be tricky.

Any idea/suggestion?

Thanks
--jyh