Open Stack

On Tue, 12 Apr 2016, Nadya Shakhat wrote:

>    I'd like to discuss one question with you. Perhaps, you remember that
> in Liberty we decided to get rid of transformers on polling agents [1]. I'd
> like to describe several issues we are facing now because of this decision.
> 1. pipeline.yaml inconsistency.

The original idea with centralizing the transformers to just the
notification agents was to allow a few different things, only one of
which has happened:

* Make the pollster code less complex with few dependencies,
   easing deployment options (this has happened), maintenance
   and custom pollsters.

   With the transformers in the pollsters they must maintain a
   considerable amount of state that makes effective use of eventlet
   (or whatever the chosen concurrent solution is) more difficult.

   The ideal pollster is just something that spits out a dumb piece
   of identified data every interval. And nothing else.

* Make it far easier to use and conceptualize the use of pollsters
   outside of the ceilometer environment as simple data collectors.
   In that context transformation would occur only close to the data
   consumption not at the data production.

   This, following the good practice of services doing one thing
   well.

* Migrate away from the pipeline.yaml that conflated sources and
   sinks to a model that is good both for computers and humans:

   * sources over here
   * sinks over here

That these other things haven't happened means we're in an awkward
situation.

Are the options the following?

* Do what you suggest and pull transformers back into the pollsters.
   Basically revert the change. I think this is the wrong long term
   solution but might be the best option if there's nobody to do the
   other options.

* Implement a pollster.yaml for use by the pollsters and consider
   pipeline.yaml as the canonical file for the notification agents as
   there's where the actual _pipelines_ are. Somewhere in there kill
   interval as a concept on pipeline side.

   This of course doesn't address the messaging complexity. I admit
   that I don't understand all the issues there but it often feels
   like we are doing that aspect of things completely wrong, so I
   would hope that before we change things there we consider all the
   options.

What else?

One probably crazy idea: What about figuring out the desired end-meters
of common transformations and making them into dedicated pollsters?
Encapsulating that transformation not at the level of the polling
manager but at the individual pollster.

-- 
Chris Dent               (╯°□°)╯︵┻━┻            http://anticdent.org/
freenode: cdent                                         tw: @anticdent