<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 13, 2013 at 12:35 AM, Zhongyue Luo <span dir="ltr"><<a href="mailto:zhongyue.nah@intel.com" target="_blank">zhongyue.nah@intel.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi, all<div><br></div><div>I'm facing problems with sample config file generation on some projects.</div>
<div><br></div><div>The <span style="font-family:arial,sans-serif;font-size:13px">doc web team is working on finding a way to get all options for all projects (or as needed) to create more up-to-date and accurate doc config tables. </span><span style="font-family:arial,sans-serif;font-size:13px"> </span><span style="font-family:arial,sans-serif;font-size:13px">They are planning to make the process as generic as possible to work with the source for any project so as to generate all config options, so that we can then have the options list converted to docbook format and update the docs site.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><font face="arial, sans-serif">Currently, listing of all register options can be done in Nova and Cinder using the tools below</font></div>
<div><a href="https://github.com/openstack/nova/blob/master/tools/conf/generate_sample.sh" target="_blank">https://github.com/openstack/nova/blob/master/tools/conf/generate_sample.sh</a><font face="arial, sans-serif"><br>
</font></div>
<div><a href="https://github.com/openstack/cinder/blob/master/tools/conf/generate_sample.sh" target="_blank">https://github.com/openstack/cinder/blob/master/tools/conf/generate_sample.sh</a><font face="arial, sans-serif"><br>
</font></div>
<div><div><br></div><div>So our plan is to move the "generate_sample.sh" to oslo with an additional docbook format ouput module which will enable:</div><div>./generate_sample.sh /opt/stack/nova</div>
<div>./generate_sample.sh /opt/stack/quantum<br></div><div>./generate_sample.sh /opt/stack/keystone<br></div><div><br></div><div>The module which traverses the target file handed by "generate_sample.sh" is</div>
<div><a href="https://github.com/openstack/oslo-incubator/blob/master/openstack/common/config/generator.py" target="_blank">https://github.com/openstack/oslo-incubator/blob/master/openstack/common/config/generator.py</a><br>
</div><div>
<br></div><div>The "generate_sample.sh" looks for files containing "Opt(" which means that file has a high chance of defining and registering an option inside. For instance,</div><div><br></div>
<div><pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px;font-size:12px;font-family:Consolas,'Liberation Mono',Courier,monospace;color:rgb(51,51,51);line-height:18px"><div style="margin:0px;padding:0px 0px 0px 10px;border:0px">
<pre style="margin-top:0px;margin-bottom:0px;padding:0px;border:0px;font-family:Consolas,'Liberation Mono',Courier,monospace"><div style="margin:0px;padding:0px 0px 0px 10px;border:0px"><span style="margin:0px;padding:0px;border:0px">resource_tracker_opts</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px">[</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"> <span style="margin:0px;padding:0px;border:0px">cfg</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">IntOpt</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px;color:rgb(221,17,68)">'reserved_host_disk_mb'</span><span style="margin:0px;padding:0px;border:0px">,</span> <span style="margin:0px;padding:0px;border:0px">default</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,153,153)">0</span><span style="margin:0px;padding:0px;border:0px">,</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"> <span style="margin:0px;padding:0px;border:0px">help</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span><span style="margin:0px;padding:0px;border:0px;color:rgb(221,17,68)">'Amount of disk in MB to reserve for the host'</span><span style="margin:0px;padding:0px;border:0px">),</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"> <span style="margin:0px;padding:0px;border:0px">cfg</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">IntOpt</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px;color:rgb(221,17,68)">'reserved_host_memory_mb'</span><span style="margin:0px;padding:0px;border:0px">,</span> <span style="margin:0px;padding:0px;border:0px">default</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span><span style="margin:0px;padding:0px;border:0px;color:rgb(0,153,153)">512</span><span style="margin:0px;padding:0px;border:0px">,</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"> <span style="margin:0px;padding:0px;border:0px">help</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span><span style="margin:0px;padding:0px;border:0px;color:rgb(221,17,68)">'Amount of memory in MB to reserve for the host'</span><span style="margin:0px;padding:0px;border:0px">),</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"> <span style="margin:0px;padding:0px;border:0px">cfg</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">StrOpt</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px;color:rgb(221,17,68)">'compute_stats_class'</span><span style="margin:0px;padding:0px;border:0px">,</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"> <span style="margin:0px;padding:0px;border:0px">default</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span><span style="margin:0px;padding:0px;border:0px;color:rgb(221,17,68)">'nova.compute.stats.Stats'</span><span style="margin:0px;padding:0px;border:0px">,</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"> <span style="margin:0px;padding:0px;border:0px">help</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span><span style="margin:0px;padding:0px;border:0px;color:rgb(221,17,68)">'Class that will manage stats for the local compute host'</span><span style="margin:0px;padding:0px;border:0px">)</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"><span style="margin:0px;padding:0px;border:0px">]</span></div><div style="margin:0px;padding:0px 0px 0px 10px;border:0px">
<br></div><div style="margin:0px;padding:0px 0px 0px 10px;border:0px"><span style="margin:0px;padding:0px;border:0px">CONF</span> <span style="margin:0px;padding:0px;border:0px;font-weight:bold">=</span> <span style="margin:0px;padding:0px;border:0px">cfg</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">CONF</span></div>
<div style="margin:0px;padding:0px 0px 0px 10px;border:0px"><span style="margin:0px;padding:0px;border:0px">CONF</span><span style="margin:0px;padding:0px;border:0px;font-weight:bold">.</span><span style="margin:0px;padding:0px;border:0px">register_opts</span><span style="margin:0px;padding:0px;border:0px">(</span><span style="margin:0px;padding:0px;border:0px">resource_tracker_opts</span><span style="margin:0px;padding:0px;border:0px">)</span></div>
</pre></div></pre></div><div><br></div><div>So if a file is passed to "generator.py", a module object is created by importing the target file and all attributes in that module object which are oslo.config.cfg.Opt sub-classes or a list of oslo.config.cfg.Opt sub-classes are printed out in the sample config file.</div>
<div><br></div><div>Therefore the assumption was that an option will be defined and registered in the same module.</div><div><br></div><div>However, some projects have options defined and registered on different modules. For instance,</div>
<div><a href="https://github.com/openstack/keystone/blob/master/keystone/common/config.py" target="_blank">https://github.com/openstack/keystone/blob/master/keystone/common/config.py</a></div><div><br></div><div>Here, the options are defined but instead of directly registering them, it provides a function which registers the options. And the actual registration happens in,</div>
<div><a href="https://github.com/openstack/keystone/blob/master/keystone/config.py#L24" target="_blank">https://github.com/openstack/keystone/blob/master/keystone/config.py#L24</a><br></div><div><br></div><div>Another pattern which is hard to detect the registered options is,</div>
<div><a href="https://github.com/openstack/glance/blob/master/glance/cmd/scrubber.py#L54" target="_blank">https://github.com/openstack/glance/blob/master/glance/cmd/scrubber.py#L54</a><br></div><div><br></div><div>This line is not only registering a option inside a function but the option is also declared inside cfg.register_opt() which is makes it even harder to detect the option.</div>
<div><br></div><div>How can we solve this problem of not being able to detect options? I have two suggestions.</div><div><br></div><div>First method is to change how generate.py works.</div><div>
Rake all modules in project which imports oslo.config.cfg and create a module object of it to inspect the cfg.CONF object inside.</div><div>We can cache all the options which were already seen and print out newly discovered options.</div>
<div>The downside of this solution is that we have to call a private function in ConfigOpts called _all_opt_infos() to look at all options registered.</div><div><a href="https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L1822" target="_blank">https://github.com/openstack/oslo.config/blob/master/oslo/config/cfg.py#L1822</a><br>
</div><div>We would have to create a public api version of it.<br></div><div><br></div><div>The second option is to make it a convention that all options should be declared in global view and also registered in the same module where they were defined.</div>
<div>However it would be difficult to write a hacking.py routine to check this and there will be significant amount of code we have to change.</div>
<div><br></div><div>So some questions,</div><div>1) Is it worth documenting all the options in a project?</div></div></div></blockquote><div><br></div><div style>Yes, absolutely. Consistency is worthwhile also, though we will certainly take what we have, make a configuration guide of it, and iterate.</div>
<div style> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>2) Should there be a convention of declaring and registering options? Or just have generate.py handle things on its own.</div>
</div></div></blockquote><div><br></div><div style>I'd like a convention -- you didn't mention swift or glance in your examples. How close or far are these projects? I know both have sample configuration files but I don't know how they're made.</div>
<div style><br></div><div style>Thanks for the efforts here. </div><div style><br></div><div style>Anne</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div>
<div>3) Any objections of adding a public version of _all_opt_infos()?</div><span class="HOEnZb"><font color="#888888">
<div><br></div>-- <br><div dir="ltr"><div><b>Intel SSG/STOD/DCST/CIT</b></div>
<div>880 Zixing Road, Zizhu Science Park, Minhang District, 200241, Shanghai,
China<br></div>
<div><a href="tel:%2B862161166500" value="+862161166500" target="_blank">+862161166500</a></div></div>
</font></span></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>