Open Stack

Hey everybody,

For several weeks we've been working on implementing updates to the 
Sphinx documentation jobs to use the new PTI. [0][1][2][3][4]

tl;dr
* we're no longer using tox for building sphinx docs
* if your doc build jobs break all of a sudden, ping us in #openstack-infra
* there is some optional cleanup you can now do if you want

The command that is run to build docs
-------------------------------------

The command that is run by the Zuul jobs is::

     sphinx-build -b html doc/source doc/build/html

If the project has a setup.cfg file with:

     [build_sphinx]
     warnings-is-error=1

then a -W is added:

     sphinx-build -W -b html doc/source doc/build/html

We are no longer using tox for sphinx jobs
------------------------------------------

There are some interesting variations in how people are doing doc 
requirements out there. We've caught many of them, but even just this 
morning found two new ways in which projects had managed to be different.

The long and short of it is that the place we *want* to be is for there 
to be a file, doc/requirements.txt, that contains requirements necessary 
for building sphinx docs. That same interface can be used by Javascript, 
Go, Rust, C++, Ruby, Ansible, Puppet, whatever really - it's not python 
specific.

To facilitate moving to that file, the jobs look for a 
doc/requirements.txt file and use it if they find it. If they don't find 
it, they look for a test-requirements.txt file (since that's what should 
have been driving the old version of the PTI anyway)

If your doc build jobs break all of a sudden
---------------------------------------------

Cases in which you might not be handled and will need a patch to your 
project:

* You did not have your doc requirements expressed in test-requirements.txt.

Some projects have used a [doc] extra in their setup.cfg file. Others 
just have requirements listed on the [docs] and [venv] environments in 
their tox.ini file.

The simple fix is simply to push up a patch putting your doc 
requirements into doc/requirements.txt

* Your docs depend on the pbr autodoc feature.

My bad. TOTALLY missed this one when we were prepping. There is a patch 
[5] up that should fix/workaround this issue that we'll land as soon as 
we verifty it fixes the pbr autodoc using projects.

* Your tools/tox_install.sh is doing something weird and we didn't find 
it and fix it before hand.

These are more case-by-case - so definitely come find us.

There is some optional cleanup you can now do if you want
---------------------------------------------------------

Now that the job update is live, it's no longer necessary for doc 
requirements to be listed in the normal test requirements (turns out you 
don't need to install sphinx to run your unittests)

* If you have a doc/requirements.txt already that we added recently, we 
will have put a reference to it in the [venv] tox.ini. You can remove it 
from there - or honestly from any tox venv that isn't [docs]

* If you have a normal test-requirements.txt file that has both types, 
you can move your documentation requirements to doc/requirements.txt and 
remove them from test-requirements.txt

* If you have distro-package requirements that are needed for your docs 
to build, you can (and should) add them to bindep.txt and mark them with 
the 'doc' profile. We also fall back to finding things in the 'test' 
profile - but being more explicit is more better.

* If you have a 'docs' environment for tox:

   ** Reminder: the tox 'docs' environment is a developer convenience. 
It is not used in building anything in the gate. **

If you are a project that follows constraints, update your docs env to 
look something like:

     [docs]
     deps =
 
-c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt}
       -r{toxinidir}/requirements.txt
       -r{toxinidir}/doc/requirements.txt
     commands = sphinx-build -b html doc/source doc/build/html

The constraints file and requirements.txt file entries are important to 
ensure that you're getting constraints applied. If you are not a project 
that follows the upper-constraints system, you can omit both the 
constraints reference and the requirements.txt reference:

     [docs]
     deps = -r{toxinidir}/doc/requirements.txt
     commands = sphinx-build -b html doc/source doc/build/html

Thanks!
Monty

[0] https://review.openstack.org/#/c/508694/
[1] 
https://governance.openstack.org/tc/reference/project-testing-interface.html#documentation
[2] 
https://governance.openstack.org/tc/reference/pti/python.html#documentation
[3] 
https://governance.openstack.org/tc/reference/pti/golang.html#documentation
[4] 
https://governance.openstack.org/tc/reference/pti/javascript.html#documentation
[5] https://review.openstack.org/#/c/528796