Open Stack

On Sat, Jan 5, 2013 at 6:32 AM, Jiang, Yunhong <yunhong.jiang at intel.com>wrote:

> I'd have a discussion on how to handle the counter relationship in
> multiple publisher.
>
>         This is also a partially reply to Doug's comments in
> https://review.openstack.org/#/c/18020/ "
> "The main issue is I don't understand the way the pipeline manager
> organizes its mapping of pipelines around a tuple of counter names, instead
> of tracking which pipelines want data from each individual counter".
>
>         There are two type potential relationship between counters. One is
> counters from the same pollster, another one is counters from different
> pollsters. I don't think we need consider counters from notification event
> (am I right?).
>

We will want all of the data published by ceilometer to be processed in the
same way, so the notification data will need to go through the pipelines,
too.


>
> 1. Counters from the same pollster
>         For counters from the same pollster, one example is counters from
> NetPollster, which include packets/bytes for each vNIC. The CW publisher
> need to calculate the overall packets/bytes from all vNIC, or some
> publisher need to calculate the average packet size of each vNIC.
>         I considered 3 solutions for it.
>         The first solution is to add a sequence_id field to counter, and
> all counters from the same NetPollster's emission have the same
> sequence_id. A transformer is used to cache all related counter till all
> related counter is received, e.g. a new sequence_id is received.
>  However, the difficulties happens to the last round counters, since we
> will not receive any new counter, thus no idea if all the related counters
> received. Also we may need maintain the "sequence_id" if we will support
> multi-thread pollster.
>
>         The second solution is similar to first solution, but using
> timestamp instead of the sequence_id. If the timestamp with very near, they
> are assumed from same emission. It has the issues as first solution, also
> this method is not that accurate and may have corner case.
>
>         The third solution, also the solution in current patch, is to pass
> the counters as an iterator to pipelines, so that pipeline will always get
> these related counters as an iterator. One tricky thing here is, possibly
> some pipeline cares only part of these counters (e.g. one publisher only
> want to get the sent packets from all vNIC), thus a filter is needed to
> filter out un-needed counters. This is basically the logic of
> _setup_counters_mapping().
>         With this solution, the pipeline interface is in fact counter
> array, instead of counter, and I think that's the major concern from Doug.
> Originally I thought it's ok since the pollster emits counter array,
> originally it's the main loop that decouples the array and now it's the
> pipeline decouples them. But I'm not quite sure.
>

My main concern was actually that there was a mapping between a tuple of
counter names and the pipeline(s) that wanted those counters, and that the
lookup key for the pipeline was calculated by looking at the incoming
counters and building a similar array. The logic for doing all of that
wasn't clear, and the *purpose* for doing it wasn't clear, and I wasn't
sure what assumptions were being made about the relationship between the
counter objects as they were being processed.

Aggregation of data between counters is an implementation detail of the
transformer or publisher that needs the aggregation, so I don't think we
want to handle it by grouping data in our main loop. For one thing, we
can't predict all of the ways the different transformers will want to
aggregate data (by instance, by project, by something like the public IP
address of a load balanced service, etc.). A more general solution would be
to give the pipeline a signal that there is no more data coming -- a
flush() call, or something similar -- and let the transformer handle
caching data internally until it sees the end of the data.


>
> 2. Counters from different pollster
>         I don't think we need consider relationship among counters from
> different pollster, thus current patch does not support it. Or I'd prefer
> the customer of the publisher , like management software, cares for it,
> since there will be no difference.
>         But if we do want to support it, the first solution above be more
> promising, although maintain the sequence_id among different pollster will
> be tricky.
>

> Any idea/suggestion?
>
> Thanks
> --jyh
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130107/c298b701/attachment.html>