<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 13, 2013 at 2:09 PM, Mark McLoughlin <span dir="ltr"><<a href="mailto:markmc@redhat.com" target="_blank">markmc@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class=""><div class="h5">On Thu, 2013-06-13 at 13:35 +0800, Zhongyue Luo wrote:<br>
> Hi, all<br>
><br>
><br>
> I'm facing problems with sample config file generation on some<br>
> projects.<br>
><br>
><br>
> The doc web team is working on finding a way to get all options for<br>
> all projects (or as needed) to create more up-to-date and accurate doc<br>
> config tables. They are planning to make the process as generic as<br>
> possible to work with the source for any project so as to generate all<br>
> config options, so that we can then have the options list converted to<br>
> docbook format and update the docs site.<br>
><br>
><br>
> Currently, listing of all register options can be done in Nova and<br>
> Cinder using the tools below<br>
> <a href="https://github.com/openstack/nova/blob/master/tools/conf/generate_sample.sh" target="_blank">https://github.com/openstack/nova/blob/master/tools/conf/generate_sample.sh</a><br>
><br>
> <a href="https://github.com/openstack/cinder/blob/master/tools/conf/generate_sample.sh" target="_blank">https://github.com/openstack/cinder/blob/master/tools/conf/generate_sample.sh</a><br>
><br>
><br>
><br>
> So our plan is to move the "generate_sample.sh" to oslo with an<br>
> additional docbook format ouput module which will enable:<br>
> ./generate_sample.sh /opt/stack/nova<br>
> ./generate_sample.sh /opt/stack/quantum<br>
><br>
> ./generate_sample.sh /opt/stack/keystone<br>
><br>
><br>
><br>
> The module which traverses the target file handed by<br>
> "generate_sample.sh" is<br>
> <a href="https://github.com/openstack/oslo-incubator/blob/master/openstack/common/config/generator.py" target="_blank">https://github.com/openstack/oslo-incubator/blob/master/openstack/common/config/generator.py</a><br>
><br>
><br>
><br>
> The "generate_sample.sh" looks for files containing "Opt(" which means<br>
> that file has a high chance of defining and registering an option<br>
> inside. For instance,<br>
><br>
><br>
> resource_tracker_opts = [<br>
><br>
><br>
> cfg.IntOpt('reserved_host_disk_mb', default=0,<br>
><br>
><br>
> help='Amount of disk in MB to reserve for the host'),<br>
><br>
><br>
> cfg.IntOpt('reserved_host_memory_mb', default=512,<br>
><br>
><br>
> help='Amount of memory in MB to reserve for the host'),<br>
><br>
><br>
> cfg.StrOpt('compute_stats_class',<br>
><br>
><br>
> default='nova.compute.stats.Stats',<br>
><br>
><br>
> help='Class that will manage stats for the local<br>
> compute host')<br>
><br>
><br>
> ]<br>
><br>
><br>
> CONF = cfg.CONF<br>
><br>
><br>
> CONF.register_opts(resource_tracker_opts)<br>
><br>
><br>
><br>
><br>
> So if a file is passed to "generator.py", a module object is created<br>
> by importing the target file and all attributes in that module<br>
> object which are oslo.config.cfg.Opt sub-classes or a list of<br>
> oslo.config.cfg.Opt sub-classes are printed out in the sample config<br>
> file.<br>
><br>
><br>
> Therefore the assumption was that an option will be defined and<br>
> registered in the same module.<br>
><br>
><br>
> However, some projects have options defined and registered on<br>
> different modules. For instance,<br>
> <a href="https://github.com/openstack/keystone/blob/master/keystone/common/config.py" target="_blank">https://github.com/openstack/keystone/blob/master/keystone/common/config.py</a><br>
><br>
><br>
> Here, the options are defined but instead of directly registering<br>
> them, it provides a function which registers the options. And the<br>
> actual registration happens in,<br>
> <a href="https://github.com/openstack/keystone/blob/master/keystone/config.py#L24" target="_blank">https://github.com/openstack/keystone/blob/master/keystone/config.py#L24</a><br>
><br>
><br>
><br>
> Another pattern which is hard to detect the registered options is,<br>
> <a href="https://github.com/openstack/glance/blob/master/glance/cmd/scrubber.py#L54" target="_blank">https://github.com/openstack/glance/blob/master/glance/cmd/scrubber.py#L54</a><br>
><br>
><br>
><br>
> This line is not only registering a option inside a function but the<br>
> option is also declared inside cfg.register_opt() which is makes it<br>
> even harder to detect the option.<br>
><br>
><br>
> How can we solve this problem of not being able to detect options?<br>
<br>
</div></div>Well, first thing I'd say is that I wouldn't worry about handling every<br>
possible way of declaring cfg config options.<br>
<br>
We can have a convention for how to register config options and a tool<br>
that supports projects using that convention.<br>
<br>
I think we have that now, but keystone chooses not to follow that<br>
convention ... e.g.<br>
<br>
<a href="https://review.openstack.org/4547" target="_blank">https://review.openstack.org/4547</a><br>
<br>
I'm pretty sure part of my argument for those keystone changes was that<br>
the sample config file generator wouldn't work with keystone as it<br>
stands now.<br>
<br>
So, IMHO - any project which doesn't follow the convention should just<br>
come up with its own custom approach to automating config documentation.<br>
<div class="im"><br>
<br></div></blockquote><div><br></div><div style>Agree, since most of the projects follow the convention of declaring options, we should first support those projects with sample conf file generation.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im">
> I have two suggestions.<br>
><br>
><br>
> First method is to change how generate.py works.<br>
> Rake all modules in project which imports oslo.config.cfg and create a<br>
> module object of it to inspect the cfg.CONF object inside.<br>
> We can cache all the options which were already seen and print out<br>
> newly discovered options.<br>
<br>
</div>Ok, so we'd actually look in cfg.CONF for registered options rather than<br>
just looking at *declared* options.<br>
<br>
I don't think we should rely on everything being visible through<br>
cfg.CONF. There can be good reasons not to use the global object. Also,<br>
the cfgfilter thing in oslo-incubator would mean that not all options<br>
are visible through cfg.CONF.<br></blockquote><div><br></div><div style>Agree</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> The downside of this solution is that we have to call a private<br>
> function in ConfigOpts called _all_opt_infos() to look at all options<br>
> registered.<br>
> <a href="https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L1822" target="_blank">https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L1822</a><br>
><br>
> We would have to create a public api version of it.<br>
<br>
</div>I think it would be useful to have an API which would return the Opt<br>
object associated with a given config option name:<br>
<br>
def lookup(self, name, group=None):<br>
return self._get_opt_info(opt_name, group)['opt']<br>
<br>
<br>
and then we can just do:<br>
<br>
def print_help(conf, group=None):<br>
d = conf if group is None else conf[group]<br>
for key in d.keys():<br>
if isinstance(conf.GroupAttr, d[key]):<br>
print_help(conf, group=key)<br>
else:<br>
print key, conf.lookup(key, group).help<br></blockquote><div><br></div><div style>But don't we already have the Opt object retrieved from the module object?</div><div style>I'm not sure I understand this part.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> The second option is to make it a convention that all options should<br>
> be declared in global view and also registered in the same module<br>
> where they were defined.<br>
<br>
</div>Hmm, why would that help? </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Doesn't the current tool just require that you declare options at<br>
top-level module scope i.e.<br>
<br>
my_opt = cfg.StrOpt(...)<br>
my_opts = [cfg.StrOpt(...), cfg.IntOpt(...)<br>
<br>
but the problem with keystone is it is dynamically creating the opt<br>
objects at runtime.<br></blockquote><div><br></div><div>If we were to use the CONF object to list all options then an option would be visible where it is registered. If some options are registered in a different module from where they were declared, then we would have inconsistency between the option location info in the sample file and the actual location declared in the code.</div>
<div><br></div><div>But if we only use option declarations following the convention, this won't be a problem.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> However it would be difficult to write a hacking.py routine to check<br>
> this and there will be significant amount of code we have to change.<br>
><br>
><br>
> So some questions,<br>
> 1) Is it worth documenting all the options in a project?<br>
<br>
</div>Yes.<br>
<div class="im"><br>
> 2) Should there be a convention of declaring and registering options?<br>
<br>
</div>There is a convention, but keystone doesn't follow it.<br>
<br>
Also, I think the important thing for the sample generator should be the<br>
convention for declaring options, not registering options.<br></blockquote><div><br></div><div style>Agree</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div class="im"><br>
> Or just have generate.py handle things on its own.<br>
> 3) Any objections of adding a public version of _all_opt_infos()?<br>
<br>
</div>No objection to something like the lookup() above, but we already have a<br>
way of iterating over registered options and groups.<br></blockquote><div><br></div><div style>cfg.CONF.items() is what you're referring to, right?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Cheers,<br>
Mark.<br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><b>Intel SSG/STOD/DCST/CIT</b></div>
<div>880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,
China<br></div>
<div>+862161166500</div></div>
</div></div>