Open Stack

On Thu, Jun 13, 2013 at 2:09 PM, Mark McLoughlin <markmc at redhat.com> wrote:

> On Thu, 2013-06-13 at 13:35 +0800, Zhongyue Luo wrote:
> > Hi, all
> >
> >
> > I'm facing problems with sample config file generation on some
> > projects.
> >
> >
> > The doc web team is working on finding a way to get all options for
> > all projects (or as needed) to create more up-to-date and accurate doc
> > config tables.  They are planning to make the process as generic as
> > possible to work with the source for any project so as to generate all
> > config options, so that we can then have the options list converted to
> > docbook format and update the docs site.
> >
> >
> > Currently, listing of all register options can be done in Nova and
> > Cinder using the tools below
> >
> https://github.com/openstack/nova/blob/master/tools/conf/generate_sample.sh
> >
> >
> https://github.com/openstack/cinder/blob/master/tools/conf/generate_sample.sh
> >
> >
> >
> > So our plan is to move the "generate_sample.sh" to oslo with an
> > additional docbook format ouput module which will enable:
> > ./generate_sample.sh /opt/stack/nova
> > ./generate_sample.sh /opt/stack/quantum
> >
> > ./generate_sample.sh /opt/stack/keystone
> >
> >
> >
> > The module which traverses the target file handed by
> > "generate_sample.sh" is
> >
> https://github.com/openstack/oslo-incubator/blob/master/openstack/common/config/generator.py
> >
> >
> >
> > The "generate_sample.sh" looks for files containing "Opt(" which means
> > that file has a high chance of defining and registering an option
> > inside. For instance,
> >
> >
> > resource_tracker_opts = [
> >
> >
> >     cfg.IntOpt('reserved_host_disk_mb', default=0,
> >
> >
> >                help='Amount of disk in MB to reserve for the host'),
> >
> >
> >     cfg.IntOpt('reserved_host_memory_mb', default=512,
> >
> >
> >                help='Amount of memory in MB to reserve for the host'),
> >
> >
> >     cfg.StrOpt('compute_stats_class',
> >
> >
> >                default='nova.compute.stats.Stats',
> >
> >
> >                help='Class that will manage stats for the local
> > compute host')
> >
> >
> > ]
> >
> >
> > CONF = cfg.CONF
> >
> >
> > CONF.register_opts(resource_tracker_opts)
> >
> >
> >
> >
> > So if a file is passed to "generator.py", a module object is created
> > by importing the target file and all attributes in that module
> > object which are oslo.config.cfg.Opt sub-classes or a list of
> > oslo.config.cfg.Opt sub-classes are printed out in the sample config
> > file.
> >
> >
> > Therefore the assumption was that an option will be defined and
> > registered in the same module.
> >
> >
> > However, some projects have options defined and registered on
> > different modules. For instance,
> >
> https://github.com/openstack/keystone/blob/master/keystone/common/config.py
> >
> >
> > Here, the options are defined but instead of directly registering
> > them, it provides a function which registers the options. And the
> > actual registration happens in,
> > https://github.com/openstack/keystone/blob/master/keystone/config.py#L24
> >
> >
> >
> > Another pattern which is hard to detect the registered options is,
> >
> https://github.com/openstack/glance/blob/master/glance/cmd/scrubber.py#L54
> >
> >
> >
> > This line is not only registering a option inside a function but the
> > option is also declared inside cfg.register_opt() which is makes it
> > even harder to detect the option.
> >
> >
> > How can we solve this problem of not being able to detect options?
>
> Well, first thing I'd say is that I wouldn't worry about handling every
> possible way of declaring cfg config options.
>
> We can have a convention for how to register config options and a tool
> that supports projects using that convention.
>
> I think we have that now, but keystone chooses not to follow that
> convention ... e.g.
>
>   https://review.openstack.org/4547
>
> I'm pretty sure part of my argument for those keystone changes was that
> the sample config file generator wouldn't work with keystone as it
> stands now.
>
> So, IMHO - any project which doesn't follow the convention should just
> come up with its own custom approach to automating config documentation.
>
>
>
Agree, since most of the projects follow the convention of declaring
options, we should first support those projects with sample conf file
generation.


> >  I have two suggestions.
> >
> >
> > First method is to change how generate.py works.
> > Rake all modules in project which imports oslo.config.cfg and create a
> > module object of it to inspect the cfg.CONF object inside.
> > We can cache all the options which were already seen and print out
> > newly discovered options.
>
> Ok, so we'd actually look in cfg.CONF for registered options rather than
> just looking at *declared* options.
>
> I don't think we should rely on everything being visible through
> cfg.CONF. There can be good reasons not to use the global object. Also,
> the cfgfilter thing in oslo-incubator would mean that not all options
> are visible through cfg.CONF.
>

Agree


>
> > The downside of this solution is that we have to call a private
> > function in ConfigOpts called _all_opt_infos() to look at all options
> > registered.
> >
> https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L1822
> >
> > We would have to create a public api version of it.
>
> I think it would be useful to have an API which would return the Opt
> object associated with a given config option name:
>
>   def lookup(self, name, group=None):
>       return self._get_opt_info(opt_name, group)['opt']
>
>
> and then we can just do:
>
>   def print_help(conf, group=None):
>       d = conf if group is None else conf[group]
>       for key in d.keys():
>           if isinstance(conf.GroupAttr, d[key]):
>               print_help(conf, group=key)
>           else:
>               print key, conf.lookup(key, group).help
>

But don't we already have the Opt object retrieved from the module object?
I'm not sure I understand this part.


>
> > The second option is to make it a convention that all options should
> > be declared in global view and also registered in the same module
> > where they were defined.
>
> Hmm, why would that help?


> Doesn't the current tool just require that you declare options at
> top-level module scope i.e.
>
>   my_opt = cfg.StrOpt(...)
>   my_opts = [cfg.StrOpt(...), cfg.IntOpt(...)
>
> but the problem with keystone is it is dynamically creating the opt
> objects at runtime.
>

If we were to use the CONF object to list all options then an option would
be visible where it is registered. If some options are registered in a
different module from where they were declared, then we would have
inconsistency between the option location info in the sample file and the
actual location declared in the code.

But if we only use option declarations following the convention, this won't
be a problem.


>
> > However it would be difficult to write a hacking.py routine to check
> > this and there will be significant amount of code we have to change.
> >
> >
> > So some questions,
> > 1) Is it worth documenting all the options in a project?
>
> Yes.
>
> > 2) Should there be a convention of declaring and registering options?
>
> There is a convention, but keystone doesn't follow it.
>
> Also, I think the important thing for the sample generator should be the
> convention for declaring options, not registering options.
>

Agree


>
> >  Or just have generate.py handle things on its own.
> > 3) Any objections of adding a public version of _all_opt_infos()?
>
> No objection to something like the lookup() above, but we already have a
> way of iterating over registered options and groups.
>

cfg.CONF.items() is what you're referring to, right?


>
> Cheers,
> Mark.
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
*Intel SSG/STOD/DCST/CIT*
880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,
China
+862161166500
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130613/9d1a53a1/attachment.html>