Open Stack

On Thu, 2013-06-13 at 13:35 +0800, Zhongyue Luo wrote:
> Hi, all
> 
> 
> I'm facing problems with sample config file generation on some
> projects.
> 
> 
> The doc web team is working on finding a way to get all options for
> all projects (or as needed) to create more up-to-date and accurate doc
> config tables.  They are planning to make the process as generic as
> possible to work with the source for any project so as to generate all
> config options, so that we can then have the options list converted to
> docbook format and update the docs site.
> 
> 
> Currently, listing of all register options can be done in Nova and
> Cinder using the tools below
> https://github.com/openstack/nova/blob/master/tools/conf/generate_sample.sh
> 
> https://github.com/openstack/cinder/blob/master/tools/conf/generate_sample.sh
> 
> 
> 
> So our plan is to move the "generate_sample.sh" to oslo with an
> additional docbook format ouput module which will enable:
> ./generate_sample.sh /opt/stack/nova
> ./generate_sample.sh /opt/stack/quantum
> 
> ./generate_sample.sh /opt/stack/keystone
> 
> 
> 
> The module which traverses the target file handed by
> "generate_sample.sh" is
> https://github.com/openstack/oslo-incubator/blob/master/openstack/common/config/generator.py
> 
> 
> 
> The "generate_sample.sh" looks for files containing "Opt(" which means
> that file has a high chance of defining and registering an option
> inside. For instance,
> 
> 
> resource_tracker_opts = [
> 
> 
>     cfg.IntOpt('reserved_host_disk_mb', default=0,
> 
> 
>                help='Amount of disk in MB to reserve for the host'),
> 
> 
>     cfg.IntOpt('reserved_host_memory_mb', default=512,
> 
> 
>                help='Amount of memory in MB to reserve for the host'),
> 
> 
>     cfg.StrOpt('compute_stats_class',
> 
> 
>                default='nova.compute.stats.Stats',
> 
> 
>                help='Class that will manage stats for the local
> compute host')
> 
> 
> ]
> 
> 
> CONF = cfg.CONF
> 
> 
> CONF.register_opts(resource_tracker_opts)
> 
> 
> 
> 
> So if a file is passed to "generator.py", a module object is created
> by importing the target file and all attributes in that module
> object which are oslo.config.cfg.Opt sub-classes or a list of
> oslo.config.cfg.Opt sub-classes are printed out in the sample config
> file.
> 
> 
> Therefore the assumption was that an option will be defined and
> registered in the same module.
> 
> 
> However, some projects have options defined and registered on
> different modules. For instance,
> https://github.com/openstack/keystone/blob/master/keystone/common/config.py
> 
> 
> Here, the options are defined but instead of directly registering
> them, it provides a function which registers the options. And the
> actual registration happens in,
> https://github.com/openstack/keystone/blob/master/keystone/config.py#L24
> 
> 
> 
> Another pattern which is hard to detect the registered options is,
> https://github.com/openstack/glance/blob/master/glance/cmd/scrubber.py#L54
> 
> 
> 
> This line is not only registering a option inside a function but the
> option is also declared inside cfg.register_opt() which is makes it
> even harder to detect the option.
> 
> 
> How can we solve this problem of not being able to detect options?

Well, first thing I'd say is that I wouldn't worry about handling every
possible way of declaring cfg config options.

We can have a convention for how to register config options and a tool
that supports projects using that convention.

I think we have that now, but keystone chooses not to follow that
convention ... e.g.

  https://review.openstack.org/4547

I'm pretty sure part of my argument for those keystone changes was that
the sample config file generator wouldn't work with keystone as it
stands now.

So, IMHO - any project which doesn't follow the convention should just
come up with its own custom approach to automating config documentation.


>  I have two suggestions.
> 
> 
> First method is to change how generate.py works.
> Rake all modules in project which imports oslo.config.cfg and create a
> module object of it to inspect the cfg.CONF object inside.
> We can cache all the options which were already seen and print out
> newly discovered options.

Ok, so we'd actually look in cfg.CONF for registered options rather than
just looking at *declared* options.

I don't think we should rely on everything being visible through
cfg.CONF. There can be good reasons not to use the global object. Also,
the cfgfilter thing in oslo-incubator would mean that not all options
are visible through cfg.CONF.

> The downside of this solution is that we have to call a private
> function in ConfigOpts called _all_opt_infos() to look at all options
> registered.
> https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L1822
> 
> We would have to create a public api version of it.

I think it would be useful to have an API which would return the Opt
object associated with a given config option name:

  def lookup(self, name, group=None):
      return self._get_opt_info(opt_name, group)['opt']


and then we can just do:

  def print_help(conf, group=None):
      d = conf if group is None else conf[group]
      for key in d.keys():
          if isinstance(conf.GroupAttr, d[key]):
              print_help(conf, group=key)
          else:
              print key, conf.lookup(key, group).help

> The second option is to make it a convention that all options should
> be declared in global view and also registered in the same module
> where they were defined.

Hmm, why would that help?

Doesn't the current tool just require that you declare options at
top-level module scope i.e.

  my_opt = cfg.StrOpt(...)
  my_opts = [cfg.StrOpt(...), cfg.IntOpt(...)

but the problem with keystone is it is dynamically creating the opt
objects at runtime.

> However it would be difficult to write a hacking.py routine to check
> this and there will be significant amount of code we have to change.
> 
> 
> So some questions,
> 1) Is it worth documenting all the options in a project?

Yes.

> 2) Should there be a convention of declaring and registering options?

There is a convention, but keystone doesn't follow it.

Also, I think the important thing for the sample generator should be the
convention for declaring options, not registering options.

>  Or just have generate.py handle things on its own.
> 3) Any objections of adding a public version of _all_opt_infos()?

No objection to something like the lookup() above, but we already have a
way of iterating over registered options and groups.

Cheers,
Mark.