Open Stack

On Tue, May 10, 2016 at 1:47 PM, Doug Hellmann <doug at doughellmann.com>
wrote:

> Sorry, I missed the reply-all button.
>
> Doug
>
> Excerpts from Doug Hellmann's message of 2016-05-10 13:45:19 -0400:
> > Excerpts from Andreas Jaeger's message of 2016-05-10 19:03:12 +0200:
> > > On 05/10/2016 06:36 PM, Ronald Bradford wrote:
> > > > Andreas,
> > > >
> > > > I would advocate for NOT moving flagmappings files to individual
> project
> > > > repositories.   Having worked out recently how configuration tables
> are
> > > > generated and was able to generate them myself I considered how to
> > > > improve the process in relation to using the existing oslo.config
> > > > generation tools.
> > >
> > > the idea here is to move this to the creation of the options. So,
> > > instead of a flagmappings file, it would be part of each configuration
> -
> > > and thus whenever a new change gets added, the config table value would
> > > be added as well.
> > >
> > > >
> > > > The primary reason for not moving the flagmappings file is you are
> then
> > > > dependent on the projects review cycle (and processes) to get these
> > > > changes approved.  This includes the load on the gate, and
> applicable +2
> > > > by cores (which on major projects have their own priorities for
> > > > features) etc.  I would not like to see documentation generation data
> > > > dependent on all applicable projects.
> > >
> > > I agree - that's why it should be change itself that adds these.
> > >
> > > > I see the bigger issue is the current process for generating the
> config
> > > > guides including the frequency and accuracy during the cycle.
> > > >
> > > > In a nutshell, it's a huge pull process, where at some time somebody
> > > > runs the tools which pulls in all projects, and dependencies and
> uses a
> > > > doc specific version to identify all the config options, merge with
> > > > flagmappings and produce configuration tables.
> > >
> > > And the way forward would be to have oslo.config generate this
> > > automatically for us.
> > >
> > > Let's see what Doug and KATO come up with.
> > >
> > > > Ideally, a more optimal means is to create a push process, that is,
> if
> > > > any configuration settings changes within a project during a commit,
> > > > this data change is pushed, so that the documentation can be updated
> > > > accordingly when applicable.  Nice in theory, in practice it's more
> > > > complex. Some high level thoughts on the process.
> > > >
> > > > Some projects now can generate a sample configuration file (tox
> > > > -egenconfig).  In Oslo we are hoping to increase this usage, and use
> the
> > >
> > > And we just want to get rid of this process, Doug worked on a way to
> > > include the sample configs as part of documentation builds.
> > >
> > > > developer docs as a place to store these files within a project repo.
> > > > Leveraging this existing functionality, if we could detect a change
> in
> > > > the sample configuration file during a review commit, then we have
> the
> > > > notification process (keystone already has a bot that does this)
> that is
> > > > the basic information trigger of a new change needed in docs for
> config.
> > > > Now, what is that change of configuration is more complex, and it
> would
> > > > seem a logical necessity to use a neutral format (e.g. yaml) to
> record
> > > > the parsed configuration options, and enable this format to be saved
> > > > somewhere/somehow.  This becomes both the input to the oslo.config
> > > > generator (i.e. we effectively take current functionality and split
> it
> > > > into two parts, the parsing of configuration options into yaml
> format,
> > > > and the generation of sample config files, and developer docs sphinx
> > > > extensions using the yaml format. The yaml data also forms the basis
> of
> > > > data for the configuration guide.
> > > > One may argue why break up the oslo.config work because it's working,
> > > > well because the inputs are used to generate multiple forms of
> > > > documentation and at present only some forms are being used.
> > > >
> > > > I am certainly not familiar with the infra work for bots, and how
> within
> > > > a gate to create, propose and commit work of a produced yaml file, or
> > > > change in yaml file.
> > > >
> > > > It does seem like this could be a lot of work, but it's the
> foundation
> > > > of converging the work the documentation team does with tools that
> exist
> > > > for the development side, as well as reducing the tool complexity
> that
> > > > exists in doc tools now.
> > >
> > >
> > > thanks for the comments. Let's see what Doug and KATO will work out.
> > >
> > > I would rather use oslo.config to generate RST tables to include than
> > > the current process. Using oslo.config the same way that projects run
> > > tox -egenconfig today, I hope that Config Ref generation becomes
> easier,
> > >
> > > Andreas
> >
> > One of the unfortunate thing about the summits is that although we can
> > have lots of face to face conversations, many of them happen in
> > parallel and so we're not all present for them. At least 3 separate
> > sessions had parts of conversations that related to this, for example.
> >
> > Based on my randomly serendipitous participation in one of the infra
> > sessions [1], for example, I think we may not want a bot proposing
> changes
> > to the docs repo any more, because infra is trying to cut back on the
> > number of those because each one requires access to the CI systems with
> > credentials on them.
> >
> > The design requirements for producing the config reference, as I
> > understand them are:
> >
> > 1. Someone building the config reference should not need to check out
> >    all of the source for OpenStack and install it in order for the build
> >    to work.
> >
> > 2. The config reference maintainers should not have to ever manually
> >    "pull" the config info from other projects (as is being done now).
> >
> > 3. It should be possible to build the config reference offline after
> >    downloading all of its parts (this supports the distro build case
> >    where they don't like the build grabbing files from the internet
> >    during the build).
> >
> > 4. Updates to the projects included in the config reference should be
> >    made automatically when configuration-related changes merge in
> >    projects.
> >
> > 5. No proposal bot.
> >
> > I have two approaches to suggest, there may be more.
> >
> > A. Single guide
> >
> >    Add a step to the post-merge queue for projects, to be run after a
> >    patch is approved and is merged into the repo, to build a data file
> >    and publish it to a static site. The data file contents are TBD, but
> >    would at the very least include all of the information we can extract
> >    about a configuration option, and would probably need to include the
> >    values currently stored in the flagmappings file.
> >
> >    Update the config reference guide build to pull the data file(s) from
> >    that static site and use them to build the reference. This needs to
> >    happen in a way that an average person running the build from source
> >    doesn't have to know to download the files separately, but the distro
> >    build jobs *can* download the files separately and disable the
> >    download in the build process.
> >
> >    Add another step to the post-merge queue to trigger the config
> >    reference build job, so that after patches merge in a project the
> >    guide is automatically updated. We would end up building the guide a
> >    lot, potentially when nothing changed at all, but we can optimize
> >    that case away later.
> >
> > B. Separate guide per project.
> >
> >    Do away with the current single guide in favor of a collection of
> >    project-specific guides, similar to what we do with release notes and
> >    API references.
> >
> >    Manage the content of the guide in the same git repo as the project
> >    source, and trigger a build when something merges (again, just like
> >    with release notes).
> >
> >    Publish a list of the guides from some central repo, and the content
> >    of each individual guide from the project repo hosting it.
> >
> > Option A has the benefit of centralizing the configuration info, as we
> > do now. It is a bit more complex to implement, but we have a similar
> > pattern for a lot of cases so it shouldn't be too bad. The changes to
> > oslo.config to produce the data file should be straightforward, although
> > we still need to solve the flagmappings question.
> >
> > Option B has the benefit of simplicity (we have a lot of similar doc
> > build jobs already) at the expense of spreading the work out into a lot
> > of repositories. OTOH, that expense also distributes the contribution
> > load to the teams that own the code, so maybe that's a benefit not
> > a drawback.  It would mean potentially duplicating some information,
> > like the introduction and docs for library options, but it places all of
> > the information relevant to configuring a given project in one location,
> > which may be beneficial to readers.  It would eliminate (or reduce?) the
> > need for flagmappings because the existing oslo.config sphinx extension
> > can manage sets of options coming from libraries, and can also be made
> > to manage oslo.config option groups, in a way that gives basically the
> > same features (at least as I understand the way flagmappings is used
> > now). It also has the benefit of giving project teams a place to put
> > narrative configuration instructions, like what the glance team has in
> > their developer documentation already.
> >
>

I don't think it's possible to eliminate the flagmappings file.
 The *.flagmappings file format presently has a 1 to many relationship with
a configuration option and the group(s) this may belong to

And example of an option with multiple groups is:

$ grep AGENT/quitting_rpc_timeout
tools/autogenerate-config-flagmappings/neutron.flagmappings
AGENT/quitting_rpc_timeout openvswitch_agent linuxbridge_agent

There are presently 42 options that have multiple groups (14 of them have
more than 2).



> > I have, so far, been thinking only of ways to implement Option A, but
> > after the success of the install guide session at the summit (where we
> > agreed to encourage having multiple installation guides for different
> > purposes/audiences), I'm now actually leaning in support of Option B.
> >
> > I'm curious to know what other folks think, though, and I'll be happy to
> > try to help implement either option.
> >
> > Doug
> >
> > [1] https://etherpad.openstack.org/p/newton-infra-proposal-jobs
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-docs/attachments/20160510/d1a95258/attachment-0001.html>