Open Stack

Sorry, I missed the reply-all button.

Doug

Excerpts from Doug Hellmann's message of 2016-05-10 13:45:19 -0400:
> Excerpts from Andreas Jaeger's message of 2016-05-10 19:03:12 +0200:
> > On 05/10/2016 06:36 PM, Ronald Bradford wrote:
> > > Andreas,
> > > 
> > > I would advocate for NOT moving flagmappings files to individual project
> > > repositories.   Having worked out recently how configuration tables are
> > > generated and was able to generate them myself I considered how to
> > > improve the process in relation to using the existing oslo.config
> > > generation tools.
> > 
> > the idea here is to move this to the creation of the options. So,
> > instead of a flagmappings file, it would be part of each configuration -
> > and thus whenever a new change gets added, the config table value would
> > be added as well.
> > 
> > > 
> > > The primary reason for not moving the flagmappings file is you are then
> > > dependent on the projects review cycle (and processes) to get these
> > > changes approved.  This includes the load on the gate, and applicable +2
> > > by cores (which on major projects have their own priorities for
> > > features) etc.  I would not like to see documentation generation data
> > > dependent on all applicable projects.
> > 
> > I agree - that's why it should be change itself that adds these.
> > 
> > > I see the bigger issue is the current process for generating the config
> > > guides including the frequency and accuracy during the cycle.
> > > 
> > > In a nutshell, it's a huge pull process, where at some time somebody
> > > runs the tools which pulls in all projects, and dependencies and uses a
> > > doc specific version to identify all the config options, merge with
> > > flagmappings and produce configuration tables.
> > 
> > And the way forward would be to have oslo.config generate this
> > automatically for us.
> > 
> > Let's see what Doug and KATO come up with.
> > 
> > > Ideally, a more optimal means is to create a push process, that is, if
> > > any configuration settings changes within a project during a commit,
> > > this data change is pushed, so that the documentation can be updated
> > > accordingly when applicable.  Nice in theory, in practice it's more
> > > complex. Some high level thoughts on the process.
> > > 
> > > Some projects now can generate a sample configuration file (tox
> > > -egenconfig).  In Oslo we are hoping to increase this usage, and use the
> > 
> > And we just want to get rid of this process, Doug worked on a way to
> > include the sample configs as part of documentation builds.
> > 
> > > developer docs as a place to store these files within a project repo.
> > > Leveraging this existing functionality, if we could detect a change in
> > > the sample configuration file during a review commit, then we have the
> > > notification process (keystone already has a bot that does this) that is
> > > the basic information trigger of a new change needed in docs for config.
> > > Now, what is that change of configuration is more complex, and it would
> > > seem a logical necessity to use a neutral format (e.g. yaml) to record
> > > the parsed configuration options, and enable this format to be saved
> > > somewhere/somehow.  This becomes both the input to the oslo.config
> > > generator (i.e. we effectively take current functionality and split it
> > > into two parts, the parsing of configuration options into yaml format,
> > > and the generation of sample config files, and developer docs sphinx
> > > extensions using the yaml format. The yaml data also forms the basis of
> > > data for the configuration guide.   
> > > One may argue why break up the oslo.config work because it's working,
> > > well because the inputs are used to generate multiple forms of
> > > documentation and at present only some forms are being used.
> > > 
> > > I am certainly not familiar with the infra work for bots, and how within
> > > a gate to create, propose and commit work of a produced yaml file, or
> > > change in yaml file.
> > > 
> > > It does seem like this could be a lot of work, but it's the foundation
> > > of converging the work the documentation team does with tools that exist
> > > for the development side, as well as reducing the tool complexity that
> > > exists in doc tools now.
> > 
> > 
> > thanks for the comments. Let's see what Doug and KATO will work out.
> > 
> > I would rather use oslo.config to generate RST tables to include than
> > the current process. Using oslo.config the same way that projects run
> > tox -egenconfig today, I hope that Config Ref generation becomes easier,
> > 
> > Andreas
> 
> One of the unfortunate thing about the summits is that although we can
> have lots of face to face conversations, many of them happen in
> parallel and so we're not all present for them. At least 3 separate
> sessions had parts of conversations that related to this, for example.
> 
> Based on my randomly serendipitous participation in one of the infra
> sessions [1], for example, I think we may not want a bot proposing changes
> to the docs repo any more, because infra is trying to cut back on the
> number of those because each one requires access to the CI systems with
> credentials on them.
> 
> The design requirements for producing the config reference, as I
> understand them are:
> 
> 1. Someone building the config reference should not need to check out
>    all of the source for OpenStack and install it in order for the build
>    to work.
> 
> 2. The config reference maintainers should not have to ever manually
>    "pull" the config info from other projects (as is being done now).
> 
> 3. It should be possible to build the config reference offline after
>    downloading all of its parts (this supports the distro build case
>    where they don't like the build grabbing files from the internet
>    during the build).
> 
> 4. Updates to the projects included in the config reference should be
>    made automatically when configuration-related changes merge in
>    projects.
> 
> 5. No proposal bot.
> 
> I have two approaches to suggest, there may be more.
> 
> A. Single guide
> 
>    Add a step to the post-merge queue for projects, to be run after a
>    patch is approved and is merged into the repo, to build a data file
>    and publish it to a static site. The data file contents are TBD, but
>    would at the very least include all of the information we can extract
>    about a configuration option, and would probably need to include the
>    values currently stored in the flagmappings file.
> 
>    Update the config reference guide build to pull the data file(s) from
>    that static site and use them to build the reference. This needs to
>    happen in a way that an average person running the build from source
>    doesn't have to know to download the files separately, but the distro
>    build jobs *can* download the files separately and disable the
>    download in the build process.
> 
>    Add another step to the post-merge queue to trigger the config
>    reference build job, so that after patches merge in a project the
>    guide is automatically updated. We would end up building the guide a
>    lot, potentially when nothing changed at all, but we can optimize
>    that case away later.
> 
> B. Separate guide per project.
> 
>    Do away with the current single guide in favor of a collection of
>    project-specific guides, similar to what we do with release notes and
>    API references.
> 
>    Manage the content of the guide in the same git repo as the project
>    source, and trigger a build when something merges (again, just like
>    with release notes).
> 
>    Publish a list of the guides from some central repo, and the content
>    of each individual guide from the project repo hosting it.
> 
> Option A has the benefit of centralizing the configuration info, as we
> do now. It is a bit more complex to implement, but we have a similar
> pattern for a lot of cases so it shouldn't be too bad. The changes to
> oslo.config to produce the data file should be straightforward, although
> we still need to solve the flagmappings question.
> 
> Option B has the benefit of simplicity (we have a lot of similar doc
> build jobs already) at the expense of spreading the work out into a lot
> of repositories. OTOH, that expense also distributes the contribution
> load to the teams that own the code, so maybe that's a benefit not
> a drawback.  It would mean potentially duplicating some information,
> like the introduction and docs for library options, but it places all of
> the information relevant to configuring a given project in one location,
> which may be beneficial to readers.  It would eliminate (or reduce?) the
> need for flagmappings because the existing oslo.config sphinx extension
> can manage sets of options coming from libraries, and can also be made
> to manage oslo.config option groups, in a way that gives basically the
> same features (at least as I understand the way flagmappings is used
> now). It also has the benefit of giving project teams a place to put
> narrative configuration instructions, like what the glance team has in
> their developer documentation already.
> 
> I have, so far, been thinking only of ways to implement Option A, but
> after the success of the install guide session at the summit (where we
> agreed to encourage having multiple installation guides for different
> purposes/audiences), I'm now actually leaning in support of Option B.
> 
> I'm curious to know what other folks think, though, and I'll be happy to
> try to help implement either option.
> 
> Doug
> 
> [1] https://etherpad.openstack.org/p/newton-infra-proposal-jobs