Open Stack

> On May 10, 2016, at 3:49 PM, Ronald Bradford <me at ronaldbradford.com> wrote:
> 
> 
> On Tue, May 10, 2016 at 1:47 PM, Doug Hellmann <doug at doughellmann.com <mailto:doug at doughellmann.com>> wrote:
> Sorry, I missed the reply-all button.
> 
> Doug
> 
> Excerpts from Doug Hellmann's message of 2016-05-10 13:45:19 -0400:
> > Excerpts from Andreas Jaeger's message of 2016-05-10 19:03:12 +0200:
> > > On 05/10/2016 06:36 PM, Ronald Bradford wrote:
> > > > Andreas,
> > > >
> > > > I would advocate for NOT moving flagmappings files to individual project
> > > > repositories.   Having worked out recently how configuration tables are
> > > > generated and was able to generate them myself I considered how to
> > > > improve the process in relation to using the existing oslo.config
> > > > generation tools.
> > >
> > > the idea here is to move this to the creation of the options. So,
> > > instead of a flagmappings file, it would be part of each configuration -
> > > and thus whenever a new change gets added, the config table value would
> > > be added as well.
> > >
> > > >
> > > > The primary reason for not moving the flagmappings file is you are then
> > > > dependent on the projects review cycle (and processes) to get these
> > > > changes approved.  This includes the load on the gate, and applicable +2
> > > > by cores (which on major projects have their own priorities for
> > > > features) etc.  I would not like to see documentation generation data
> > > > dependent on all applicable projects.
> > >
> > > I agree - that's why it should be change itself that adds these.
> > >
> > > > I see the bigger issue is the current process for generating the config
> > > > guides including the frequency and accuracy during the cycle.
> > > >
> > > > In a nutshell, it's a huge pull process, where at some time somebody
> > > > runs the tools which pulls in all projects, and dependencies and uses a
> > > > doc specific version to identify all the config options, merge with
> > > > flagmappings and produce configuration tables.
> > >
> > > And the way forward would be to have oslo.config generate this
> > > automatically for us.
> > >
> > > Let's see what Doug and KATO come up with.
> > >
> > > > Ideally, a more optimal means is to create a push process, that is, if
> > > > any configuration settings changes within a project during a commit,
> > > > this data change is pushed, so that the documentation can be updated
> > > > accordingly when applicable.  Nice in theory, in practice it's more
> > > > complex. Some high level thoughts on the process.
> > > >
> > > > Some projects now can generate a sample configuration file (tox
> > > > -egenconfig).  In Oslo we are hoping to increase this usage, and use the
> > >
> > > And we just want to get rid of this process, Doug worked on a way to
> > > include the sample configs as part of documentation builds.
> > >
> > > > developer docs as a place to store these files within a project repo.
> > > > Leveraging this existing functionality, if we could detect a change in
> > > > the sample configuration file during a review commit, then we have the
> > > > notification process (keystone already has a bot that does this) that is
> > > > the basic information trigger of a new change needed in docs for config.
> > > > Now, what is that change of configuration is more complex, and it would
> > > > seem a logical necessity to use a neutral format (e.g. yaml) to record
> > > > the parsed configuration options, and enable this format to be saved
> > > > somewhere/somehow.  This becomes both the input to the oslo.config
> > > > generator (i.e. we effectively take current functionality and split it
> > > > into two parts, the parsing of configuration options into yaml format,
> > > > and the generation of sample config files, and developer docs sphinx
> > > > extensions using the yaml format. The yaml data also forms the basis of
> > > > data for the configuration guide.
> > > > One may argue why break up the oslo.config work because it's working,
> > > > well because the inputs are used to generate multiple forms of
> > > > documentation and at present only some forms are being used.
> > > >
> > > > I am certainly not familiar with the infra work for bots, and how within
> > > > a gate to create, propose and commit work of a produced yaml file, or
> > > > change in yaml file.
> > > >
> > > > It does seem like this could be a lot of work, but it's the foundation
> > > > of converging the work the documentation team does with tools that exist
> > > > for the development side, as well as reducing the tool complexity that
> > > > exists in doc tools now.
> > >
> > >
> > > thanks for the comments. Let's see what Doug and KATO will work out.
> > >
> > > I would rather use oslo.config to generate RST tables to include than
> > > the current process. Using oslo.config the same way that projects run
> > > tox -egenconfig today, I hope that Config Ref generation becomes easier,
> > >
> > > Andreas
> >
> > One of the unfortunate thing about the summits is that although we can
> > have lots of face to face conversations, many of them happen in
> > parallel and so we're not all present for them. At least 3 separate
> > sessions had parts of conversations that related to this, for example.
> >
> > Based on my randomly serendipitous participation in one of the infra
> > sessions [1], for example, I think we may not want a bot proposing changes
> > to the docs repo any more, because infra is trying to cut back on the
> > number of those because each one requires access to the CI systems with
> > credentials on them.
> >
> > The design requirements for producing the config reference, as I
> > understand them are:
> >
> > 1. Someone building the config reference should not need to check out
> >    all of the source for OpenStack and install it in order for the build
> >    to work.
> >
> > 2. The config reference maintainers should not have to ever manually
> >    "pull" the config info from other projects (as is being done now).
> >
> > 3. It should be possible to build the config reference offline after
> >    downloading all of its parts (this supports the distro build case
> >    where they don't like the build grabbing files from the internet
> >    during the build).
> >
> > 4. Updates to the projects included in the config reference should be
> >    made automatically when configuration-related changes merge in
> >    projects.
> >
> > 5. No proposal bot.
> >
> > I have two approaches to suggest, there may be more.
> >
> > A. Single guide
> >
> >    Add a step to the post-merge queue for projects, to be run after a
> >    patch is approved and is merged into the repo, to build a data file
> >    and publish it to a static site. The data file contents are TBD, but
> >    would at the very least include all of the information we can extract
> >    about a configuration option, and would probably need to include the
> >    values currently stored in the flagmappings file.
> >
> >    Update the config reference guide build to pull the data file(s) from
> >    that static site and use them to build the reference. This needs to
> >    happen in a way that an average person running the build from source
> >    doesn't have to know to download the files separately, but the distro
> >    build jobs *can* download the files separately and disable the
> >    download in the build process.
> >
> >    Add another step to the post-merge queue to trigger the config
> >    reference build job, so that after patches merge in a project the
> >    guide is automatically updated. We would end up building the guide a
> >    lot, potentially when nothing changed at all, but we can optimize
> >    that case away later.
> >
> > B. Separate guide per project.
> >
> >    Do away with the current single guide in favor of a collection of
> >    project-specific guides, similar to what we do with release notes and
> >    API references.
> >
> >    Manage the content of the guide in the same git repo as the project
> >    source, and trigger a build when something merges (again, just like
> >    with release notes).
> >
> >    Publish a list of the guides from some central repo, and the content
> >    of each individual guide from the project repo hosting it.
> >
> > Option A has the benefit of centralizing the configuration info, as we
> > do now. It is a bit more complex to implement, but we have a similar
> > pattern for a lot of cases so it shouldn't be too bad. The changes to
> > oslo.config to produce the data file should be straightforward, although
> > we still need to solve the flagmappings question.
> >
> > Option B has the benefit of simplicity (we have a lot of similar doc
> > build jobs already) at the expense of spreading the work out into a lot
> > of repositories. OTOH, that expense also distributes the contribution
> > load to the teams that own the code, so maybe that's a benefit not
> > a drawback.  It would mean potentially duplicating some information,
> > like the introduction and docs for library options, but it places all of
> > the information relevant to configuring a given project in one location,
> > which may be beneficial to readers.  It would eliminate (or reduce?) the
> > need for flagmappings because the existing oslo.config sphinx extension
> > can manage sets of options coming from libraries, and can also be made
> > to manage oslo.config option groups, in a way that gives basically the
> > same features (at least as I understand the way flagmappings is used
> > now). It also has the benefit of giving project teams a place to put
> > narrative configuration instructions, like what the glance team has in
> > their developer documentation already.
> >
> 
> I don't think it's possible to eliminate the flagmappings file.
>  The *.flagmappings file format presently has a 1 to many relationship with a configuration option and the group(s) this may belong to
> 
> And example of an option with multiple groups is:
>     
> $ grep AGENT/quitting_rpc_timeout tools/autogenerate-config-flagmappings/neutron.flagmappings
> AGENT/quitting_rpc_timeout openvswitch_agent linuxbridge_agent
> 
> There are presently 42 options that have multiple groups (14 of them have more than 2).

It’s not clear that we need to continue to separate those groups, or if we do need to that we have to do it using the same mechanism. My point was that we could add features to oslo.config to support that sort of “doc grouping” feature directly, and eliminate generating the files (assuming option B) or generate the files from the code (assuming option A).

Doug

> 
>  
> > I have, so far, been thinking only of ways to implement Option A, but
> > after the success of the install guide session at the summit (where we
> > agreed to encourage having multiple installation guides for different
> > purposes/audiences), I'm now actually leaning in support of Option B.
> >
> > I'm curious to know what other folks think, though, and I'll be happy to
> > try to help implement either option.
> >
> > Doug
> >
> > [1] https://etherpad.openstack.org/p/newton-infra-proposal-jobs <https://etherpad.openstack.org/p/newton-infra-proposal-jobs>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-docs/attachments/20160510/8920e5ca/attachment-0001.html>