<div dir="ltr"><div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div><div><div><br></div></div></div></div></div></div></div></div></div></div></div><div class="gmail_quote">On Tue, May 10, 2016 at 1:47 PM, Doug Hellmann <span dir="ltr"><<a href="mailto:doug@doughellmann.com" target="_blank">doug@doughellmann.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Sorry, I missed the reply-all button.<br>
<br>
Doug<br>
<br>
Excerpts from Doug Hellmann's message of 2016-05-10 13:45:19 -0400:<br>
> Excerpts from Andreas Jaeger's message of 2016-05-10 19:03:12 +0200:<br>
<div><div class="h5">> > On 05/10/2016 06:36 PM, Ronald Bradford wrote:<br>
> > > Andreas,<br>
> > ><br>
> > > I would advocate for NOT moving flagmappings files to individual project<br>
> > > repositories. Having worked out recently how configuration tables are<br>
> > > generated and was able to generate them myself I considered how to<br>
> > > improve the process in relation to using the existing oslo.config<br>
> > > generation tools.<br>
> ><br>
> > the idea here is to move this to the creation of the options. So,<br>
> > instead of a flagmappings file, it would be part of each configuration -<br>
> > and thus whenever a new change gets added, the config table value would<br>
> > be added as well.<br>
> ><br>
> > ><br>
> > > The primary reason for not moving the flagmappings file is you are then<br>
> > > dependent on the projects review cycle (and processes) to get these<br>
> > > changes approved. This includes the load on the gate, and applicable +2<br>
> > > by cores (which on major projects have their own priorities for<br>
> > > features) etc. I would not like to see documentation generation data<br>
> > > dependent on all applicable projects.<br>
> ><br>
> > I agree - that's why it should be change itself that adds these.<br>
> ><br>
> > > I see the bigger issue is the current process for generating the config<br>
> > > guides including the frequency and accuracy during the cycle.<br>
> > ><br>
> > > In a nutshell, it's a huge pull process, where at some time somebody<br>
> > > runs the tools which pulls in all projects, and dependencies and uses a<br>
> > > doc specific version to identify all the config options, merge with<br>
> > > flagmappings and produce configuration tables.<br>
> ><br>
> > And the way forward would be to have oslo.config generate this<br>
> > automatically for us.<br>
> ><br>
> > Let's see what Doug and KATO come up with.<br>
> ><br>
> > > Ideally, a more optimal means is to create a push process, that is, if<br>
> > > any configuration settings changes within a project during a commit,<br>
> > > this data change is pushed, so that the documentation can be updated<br>
> > > accordingly when applicable. Nice in theory, in practice it's more<br>
> > > complex. Some high level thoughts on the process.<br>
> > ><br>
> > > Some projects now can generate a sample configuration file (tox<br>
> > > -egenconfig). In Oslo we are hoping to increase this usage, and use the<br>
> ><br>
> > And we just want to get rid of this process, Doug worked on a way to<br>
> > include the sample configs as part of documentation builds.<br>
> ><br>
> > > developer docs as a place to store these files within a project repo.<br>
> > > Leveraging this existing functionality, if we could detect a change in<br>
> > > the sample configuration file during a review commit, then we have the<br>
> > > notification process (keystone already has a bot that does this) that is<br>
> > > the basic information trigger of a new change needed in docs for config.<br>
> > > Now, what is that change of configuration is more complex, and it would<br>
> > > seem a logical necessity to use a neutral format (e.g. yaml) to record<br>
> > > the parsed configuration options, and enable this format to be saved<br>
> > > somewhere/somehow. This becomes both the input to the oslo.config<br>
> > > generator (i.e. we effectively take current functionality and split it<br>
> > > into two parts, the parsing of configuration options into yaml format,<br>
> > > and the generation of sample config files, and developer docs sphinx<br>
> > > extensions using the yaml format. The yaml data also forms the basis of<br>
> > > data for the configuration guide.<br>
> > > One may argue why break up the oslo.config work because it's working,<br>
> > > well because the inputs are used to generate multiple forms of<br>
> > > documentation and at present only some forms are being used.<br>
> > ><br>
> > > I am certainly not familiar with the infra work for bots, and how within<br>
> > > a gate to create, propose and commit work of a produced yaml file, or<br>
> > > change in yaml file.<br>
> > ><br>
> > > It does seem like this could be a lot of work, but it's the foundation<br>
> > > of converging the work the documentation team does with tools that exist<br>
> > > for the development side, as well as reducing the tool complexity that<br>
> > > exists in doc tools now.<br>
> ><br>
> ><br>
> > thanks for the comments. Let's see what Doug and KATO will work out.<br>
> ><br>
> > I would rather use oslo.config to generate RST tables to include than<br>
> > the current process. Using oslo.config the same way that projects run<br>
> > tox -egenconfig today, I hope that Config Ref generation becomes easier,<br>
> ><br>
> > Andreas<br>
><br>
</div></div>> One of the unfortunate thing about the summits is that although we can<br>
> have lots of face to face conversations, many of them happen in<br>
> parallel and so we're not all present for them. At least 3 separate<br>
> sessions had parts of conversations that related to this, for example.<br>
><br>
> Based on my randomly serendipitous participation in one of the infra<br>
> sessions [1], for example, I think we may not want a bot proposing changes<br>
> to the docs repo any more, because infra is trying to cut back on the<br>
> number of those because each one requires access to the CI systems with<br>
> credentials on them.<br>
><br>
> The design requirements for producing the config reference, as I<br>
> understand them are:<br>
><br>
> 1. Someone building the config reference should not need to check out<br>
> all of the source for OpenStack and install it in order for the build<br>
> to work.<br>
><br>
> 2. The config reference maintainers should not have to ever manually<br>
> "pull" the config info from other projects (as is being done now).<br>
><br>
> 3. It should be possible to build the config reference offline after<br>
> downloading all of its parts (this supports the distro build case<br>
> where they don't like the build grabbing files from the internet<br>
> during the build).<br>
><br>
> 4. Updates to the projects included in the config reference should be<br>
> made automatically when configuration-related changes merge in<br>
> projects.<br>
><br>
> 5. No proposal bot.<br>
><br>
> I have two approaches to suggest, there may be more.<br>
><br>
> A. Single guide<br>
><br>
> Add a step to the post-merge queue for projects, to be run after a<br>
> patch is approved and is merged into the repo, to build a data file<br>
> and publish it to a static site. The data file contents are TBD, but<br>
> would at the very least include all of the information we can extract<br>
> about a configuration option, and would probably need to include the<br>
> values currently stored in the flagmappings file.<br>
><br>
> Update the config reference guide build to pull the data file(s) from<br>
> that static site and use them to build the reference. This needs to<br>
> happen in a way that an average person running the build from source<br>
> doesn't have to know to download the files separately, but the distro<br>
> build jobs *can* download the files separately and disable the<br>
> download in the build process.<br>
><br>
> Add another step to the post-merge queue to trigger the config<br>
> reference build job, so that after patches merge in a project the<br>
> guide is automatically updated. We would end up building the guide a<br>
> lot, potentially when nothing changed at all, but we can optimize<br>
> that case away later.<br>
><br>
> B. Separate guide per project.<br>
><br>
> Do away with the current single guide in favor of a collection of<br>
> project-specific guides, similar to what we do with release notes and<br>
> API references.<br>
><br>
> Manage the content of the guide in the same git repo as the project<br>
> source, and trigger a build when something merges (again, just like<br>
> with release notes).<br>
><br>
> Publish a list of the guides from some central repo, and the content<br>
> of each individual guide from the project repo hosting it.<br>
><br>
> Option A has the benefit of centralizing the configuration info, as we<br>
> do now. It is a bit more complex to implement, but we have a similar<br>
> pattern for a lot of cases so it shouldn't be too bad. The changes to<br>
> oslo.config to produce the data file should be straightforward, although<br>
> we still need to solve the flagmappings question.<br>
><br>
> Option B has the benefit of simplicity (we have a lot of similar doc<br>
> build jobs already) at the expense of spreading the work out into a lot<br>
> of repositories. OTOH, that expense also distributes the contribution<br>
> load to the teams that own the code, so maybe that's a benefit not<br>
> a drawback. It would mean potentially duplicating some information,<br>
> like the introduction and docs for library options, but it places all of<br>
> the information relevant to configuring a given project in one location,<br>
> which may be beneficial to readers. It would eliminate (or reduce?) the<br>
> need for flagmappings because the existing oslo.config sphinx extension<br>
> can manage sets of options coming from libraries, and can also be made<br>
> to manage oslo.config option groups, in a way that gives basically the<br>
> same features (at least as I understand the way flagmappings is used<br>
> now). It also has the benefit of giving project teams a place to put<br>
> narrative configuration instructions, like what the glance team has in<br>
> their developer documentation already.<br>
><br></blockquote><div><br></div><div>I don't think it's possible to eliminate the flagmappings file.</div><div><span style="background-color:rgb(255,255,255)"><span style="color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px"> The *.flagmappings file format presently has a 1 to many relationship with a configuration option and the group(s) this may belong to</span><br></span></div><div><span style="color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px;background-color:rgb(255,255,255)"><br></span></div><div><div id="magicdomid57" class="" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px"><span class="" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)">And example of an option with multiple groups is:</span></div><div id="magicdomid58" class="" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px"><span class="" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"> </span></div><div id="magicdomid59" class="" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px"><code style="margin:0px;padding:0px"><span class="" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)">$ grep AGENT/quitting_rpc_timeout tools/autogenerate-config-flagmappings/neutron.flagmappings</span></code></div><div id="magicdomid60" class="" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px"><code style="margin:0px;padding:0px"><span class="" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)">AGENT/quitting_rpc_timeout openvswitch_agent linuxbridge_agent</span></code></div><div id="magicdomid61" class="" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px"><span style="background-color:rgb(255,255,255)"><br style="margin:0px;padding:0px"></span></div><div id="magicdomid62" class="" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px"><span class="" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)">There are presently 42 options that have multiple groups (14 of them have more than 2).</span></div><div id="magicdomid63" class="" style="margin:0px;padding:0px;color:rgb(0,0,0);font-family:'Helvetica Neue',Arial,sans-serif;font-size:12px;line-height:16px"><br style="margin:0px;padding:0px"></div></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
> I have, so far, been thinking only of ways to implement Option A, but<br>
> after the success of the install guide session at the summit (where we<br>
> agreed to encourage having multiple installation guides for different<br>
> purposes/audiences), I'm now actually leaning in support of Option B.<br>
><br>
> I'm curious to know what other folks think, though, and I'll be happy to<br>
> try to help implement either option.<br>
><br>
> Doug<br>
><br>
> [1] <a href="https://etherpad.openstack.org/p/newton-infra-proposal-jobs" rel="noreferrer" target="_blank">https://etherpad.openstack.org/p/newton-infra-proposal-jobs</a><br>
</blockquote></div><br></div></div>