[openstack-dev] [ceilometer] polling agent configuration speculation
chdent at redhat.com
Mon Jun 8 13:04:42 UTC 2015
(Posting to the mailing list rather than writing a spec or making
code because I think it is important to get some input and feedback
before going off on something wild. Below I'm talking about
speculative plans and seeking feedback, not reporting decisions
about the future. Some of this discussion is intentionally naive
about how things are because that's not really relevant, what's
relevant is how things should be or could be.
tl;dr: I want to make the configuration of the pollsters more explicit
and not conflate and overlap the entry_points.txt and pipeline.yaml
in confusing and inefficient ways.
* entry_points.txt should define what measurements are possible, not
what measurements are loaded
* something new should define what measurements are loaded and
polled (and their intervals) (sources in pipeline.yaml speak)
* pipeline.yaml should define transformations and publishers
Would people like something like this?)
The longer version:
Several of the outcomes of the Liberty Design Summit were related to
making changes to the agents which gather or hear measurements and
events. Some of these changes have pending specs:
* Ceilometer Collection Agents Split
Splitting the collection agents into their own repo to allow
use and evolution separate from the rest of Ceilometer.
* Adding Meta-Data Caching Spec
Adding metadata caching to the compute agent so the Nova-API is
less assaulted than it currently is.
* Declarative notification handling
Be able to hear and transform a notification to an event without
having to write code.
Reviewing these and other specs and doing some review of the code
points out that we have an opportunity to make some architectural and
user interface improvements (while still maintain existing
functionality). For example:
The current ceilometer polling agent has an interesting start up
1 It determines which namespaces it is operating in ('compute',
2 Using entry_points defined in setup.cfg it initializes all the
polling extensions and all the discovery extensions (independent
of sources defined in pipeline.yaml)
3 Every source in pipeline.yaml is given a list of pollsters that
match the meters defined by the source, creating long running
tasks to do the polling.
4 Each task does resource discovery and partitioning coordination.
5 measurements/samples are gathered and are transformed and published
according the sink rules in pipeline.yaml
A couple things about this seem less than ideal:
* 2 means we load redundant stuff unless we edit entry_points.txt.
We do not want to encourage this sort of behavior. entry_points is
not configuration. We should configure elsewhere to declare "I
care about things X (including the option of "all things")" and
then load the tools to do so, on demand.
* Two things are happening in the same context in step 5 and that
seems quite limiting with regard to opportunities for effective
maintenance and optimizing.
My intuition (which often needs to sanity checked, thus my posting
here) tells me there are some things we could change:
* Separate polling and publishing/transforming into separate
* Extract the definition of sources to be polled from pipeline.yaml
to its own file and use that to be the authority of which
extensions are loaded for polling and discovery.
What do people think?
 This is really the core of my concern and the main part I want
to see change.
Chris Dent tw:@anticdent freenode:cdent
More information about the OpenStack-dev