Open Stack

(Posting to the mailing list rather than writing a spec or making
code because I think it is important to get some input and feedback
before going off on something wild. Below I'm talking about
speculative plans and seeking feedback, not reporting decisions
about the future. Some of this discussion is intentionally naive
about how things are because that's not really relevant, what's
relevant is how things should be or could be.

tl;dr: I want to make the configuration of the pollsters more explicit
and not conflate and overlap the entry_points.txt and pipeline.yaml
in confusing and inefficient ways.

* entry_points.txt should define what measurements are possible, not
   what measurements are loaded
* something new should define what measurements are loaded and
   polled (and their intervals) (sources in pipeline.yaml speak)
* pipeline.yaml should define transformations and publishers

Would people like something like this?)

The longer version:

Several of the outcomes of the Liberty Design Summit were related to
making changes to the agents which gather or hear measurements and
events. Some of these changes have pending specs:

* Ceilometer Collection Agents Split
   https://review.openstack.org/#/c/186964/

   Splitting the collection agents into their own repo to allow
   use and evolution separate from the rest of Ceilometer.

* Adding Meta-Data Caching Spec
   https://review.openstack.org/#/c/185084/

   Adding metadata caching to the compute agent so the Nova-API is
   less assaulted than it currently is.

* Declarative notification handling
   https://review.openstack.org/#/c/178399/

   Be able to hear and transform a notification to an event without
   having to write code.

Reviewing these and other specs and doing some review of the code
points out that we have an opportunity to make some architectural and
user interface improvements (while still maintain existing
functionality). For example:

The current ceilometer polling agent has an interesting start up
process:

1 It determines which namespaces it is operating in ('compute',
   'central', 'ipmi').
2 Using entry_points defined in setup.cfg it initializes all the
   polling extensions and all the discovery extensions (independent
   of sources defined in pipeline.yaml)
3 Every source in pipeline.yaml is given a list of pollsters that
   match the meters defined by the source, creating long running
   tasks to do the polling.
4 Each task does resource discovery and partitioning coordination.
5 measurements/samples are gathered and are transformed and published
   according the sink rules in pipeline.yaml

A couple things about this seem less than ideal:

* 2 means we load redundant stuff unless we edit entry_points.txt.
   We do not want to encourage this sort of behavior. entry_points is
   not configuration[1]. We should configure elsewhere to declare "I
   care about things X (including the option of "all things")" and
   then load the tools to do so, on demand.

* Two things are happening in the same context in step 5 and that
   seems quite limiting with regard to opportunities for effective
   maintenance and optimizing.

My intuition (which often needs to sanity checked, thus my posting
here) tells me there are some things we could change:

* Separate polling and publishing/transforming into separate
   workers/processes.

* Extract the definition of sources to be polled from pipeline.yaml
   to its own file and use that to be the authority of which
   extensions are loaded for polling and discovery.

What do people think?

[1] This is really the core of my concern and the main part I want
to see change.
-- 
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent