Open Stack

Excerpts from sfinucan's message of 2017-07-07 15:58:10 +0100:
> tl;dr: pbr's 'build_spinx' derivative is broken again, and we want to just
> remove the feature at this point. However, this is going to necessitate some
> mechanical changes for most projects with docs and this mail serves as a heads
> up and request for input before we proceed.
> 
> --
> 
> Since pretty much the beginning, pbr has provided a custom 'build_sphinx'
> setuptools command [1], which can be run like so:
> 
>     python setup.py build_sphinx
> 
> This is described in the pbr docs [2] and is is based on Sphinx's own
> 'build_sphinx' command [3], which is described in the Sphinx docs [4].
> 
> Historically, this custom version has provided a couple of features that the
> upstream Sphinx extension didn't. These are outlined below. However, this
> derivative has resulted in an inordinately large number of bugs for what should
> be a simple feature (#1702872, #1691129, #1681983, #1674795, #1379998,
> #1496882, ...). This is due to the lack of extension points in the upstream
> code and the way that we've been forced to extend the command (we break almost
> every time Sphinx change their code).
> 
> The latest of these bugs, #1702872, breaks a couple of things when using Sphinx
> > 1.6:
> 
> - We no longer provide confoverrides when using Sphinx 1.6+, which means you
>   may see "node class 'xxx' is already registered' errors, and the project
>   name and version used in the docs may be incorrect [5]
> - The '[build_sphinx] builders' in 'setup.cfg' option is no longer read, so
>   only 'html' docs will be built (no man pages)
> - The apidoc tool won't be automatically run
> 
> Fixing these issues is possible in the short-term and won't make the thing any
> more maintainable going forward. Given the amount of issues this feature has
> caused and the fact that Sphinx has since gained many of the features we
> provided itself, we'd like to just drop the whole thing now. Some of the extra
> this feature provide are no longer necessary, but the migration paths for the
> ones that are are not great and will require mechanical changes to projects.
> 
> Here are the features that the plugin provides, along with the current
> migration plans:
> 
> - Automatically runs sphinx-apidoc [6] before building docs
> 
>   It seems that adding the 'sphinx.ext.apidoc' extension to the 'extensions'
>   list in 'doc/source/conf.py' will ensure this continues to happen
>   automatically. However, I need to check this and will get back when I have.
> 
> - Automatically runs 'autodoc', which seems to do something similar to 'apidoc'
>   but in a different way (I don't know what, tbh)
> 
>   Drop this feature entirely as 'apidoc' appears to do something very similar.
> 
> - Automatically sets project name and version using pbr's machinery
> 
>   Try to set this from 'openstackdocstheme'. If this isn't possible, folks will
>   need to add some some lines to 'conf.py' like keystoneauth does [7]

I'm inclined to just say these values should be set explicitly. We can
simplify it (Monty had an idea for a 1-liner that would set all of the
variables with one function call into pbr), but the project name doesn't
change so there's no real sense in doing a lot of fancy automation for
that one.

> - Supports multiple builders via the '[build_sphinx] builders' option in
>   'setup.cfg'
> 
>   This is natively supported since Sphinx 1.6. You should replace
>   '[build_sphinx] builders' with '[build_sphinx] builder' (singular) now

That would only work if we continue to invoke Sphinx using its
setuptools plugin in the gate. Is that what we want to do? We also
discussed changing the CI interface to build docs to use the "tox -e
docs" command like contributors generally run locally.

> - Historically, supported turning build warning into build falures via the
>   '[build_sphinx] warnerrors' option in 'setup.cfg'
> 
>   This is no longer supported by pbr, as Sphinx now provides a '[build_sphinx]
>   warning-is-error' option. I wrote a load of patches to fix this recently
>   using the topic 'sphinx15', but if I missed your project you need to remove
>   '[pbr] warnerrors' from 'setup.cfg' and add '[build_sphinx]
>   warning-is-error'. You should do this now.

This is also one of the doc-migration steps. Some projects that are
hitting errors caused by the missing config overrides are going to need
to wait to turn the flag on until after we figure out how to work around
those issues.

> Does anyone see any issues with this strategy going forward? If not, we're
> going to start making changes to projects using the 'sphinx16' topic and would
> appreciate reviews on these as they come in. We're not going to get to every
> project, so if you can make the necessary changes to your own project then
> please do.

Let me know if you want some help setting up a burndown chart like
https://doughellmann.com/doc-migration/. Having a tracking etherpad so
folks know whether a repo has a volunteer to do the work will help cut
down on duplicate effort, too.

Doug

> 
> Cheers,
> Stephen
> irc:stephenfin (formerly sfinucan)
> 
> - 
> 
> [1] https://github.com/openstack-dev/pbr/blob/master/pbr/builddoc.py
> [2] https://docs.openstack.org/pbr/latest/user/using.html#build-sphinx
> [3] https://github.com/sphinx-doc/sphinx/blob/master/sphinx/setup_command.py
> [4] http://www.sphinx-doc.org/en/stable/setuptools.html
> [5] http://paste.openstack.org/show/614730/
> [5] www.sphinx-doc.org/en/stable/man/sphinx-apidoc.html
> [7] https://github.com/openstack/keystoneauth/blob/master/doc/source/conf.py#L6
> 5-L72
>