Open Stack

Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:
> Hey everybody,
> 
> Oh goodie, I can hear you say, let's definitely spend some time 
> bikeshedding about specific command invocations related to building docs 
> and tarballs!!!
> 
> tl;dr I want to change the PTI for docs and tarball building to be less 
> OpenStack-specific
> 
> The Problem
> ===========
> 
> As we work on Zuul v3, there are a set of job definitions that are 
> rather fundamental that can totally be directly shared between Zuul 
> installations whether those Zuuls are working with OpenStack content or 
> not. As an example "tox -epy27" is a fairly standard thing, so a Zuul 
> job called "tox-py27" has no qualities specific to OpenStack and could 
> realistically be used by anyone who uses tox to manage their project.
> 
> Docs and Tarballs builds for us, however, are the following:
> 
> tox -evenv -- python setup.py sdist
> tox -evenv -- python setup.py build_sphinx
> 
> Neither of those are things that are likely to work outside of 
> OpenStack. (The 'venv' tox environment is not a default tox thing)
> 
> I'm going to argue neither of them are actually providing us with much 
> value.
> 
> Tarball Creation
> ================
> 
> Tarball creation is super simple. setup_requires is already handled out 
> of band of everything else. Go clone nova into a completely empty system 
> and run python setup.py sdist ... and it works. (actually, nova is big. 
> use something smaller like gertty ...)
> 
>     docker run -it --rm python bash -c 'git clone \
>       https://git.openstack.org/openstack/gertty && cd gertty \
>       && python setup.py sdist'
> 
> There is not much value in that tox wrapper - and it's not like it's 
> making it EASIER to run the command. In fact, it's more typing.
> 
> I propose we change the PTI from:
> 
>    tox -evenv python setup.py sdist
> 
> to:
> 
>    python setup.py sdist
> 
> and then change the gate jobs to use the non-tox form of the command.
> 
> I'd also like to further change it to be explicit that we also build 
> wheels. So the ACTUAL commands that the project should support are:
> 
>    python setup.py sdist
>    python setup.py bdist_wheel
> 
> All of our projects support this already, so this should be a no-op.
> 
> Notes:
> 
> * Python projects that need to build C extensions might need their pip 
> requirements (and bindep requirements) installed in order to run 
> bdist_wheel. We do not support that broadly at the moment ANYWAY - so 
> I'd like to leave that as an outlier and handle it when we need to 
> handle it.
> 
> * It's *possible* that somewhere we have a repo that has somehow done 
> something that would cause python setup.py sdist or python setup.py 
> bdist_wheel to not work without pip requirements installed. I believe we 
> should consider that a bug and fix it in the project if we find such a 
> thing - but since we use pbr in all of the OpenStack projects, I find it 
> extremely unlikely.
> 
> Governance patch submitted: https://review.openstack.org/508693
> 
> Sphinx Documentation
> ====================
> 
> Doc builds are more complex - but I think there is a high amount of 
> value in changing how we invoke them for a few reasons.
> 
> a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
> b) we decided to use sphinx for go and javascript - but we invoke sphinx 
> differently for each of those (since they naturally don't have tox), 
> meaning we can't just have a "build-sphinx-docs" job and even share it 
> with ourselves.
> c) readthedocs.org is an excellent Open Source site that builds and 
> hosts sphinx docs for projects. They have an interface for docs 
> requirements documented and defined that we can align. By aligning, 
> projects can use migrate between docs.o.o and readthedocs.org and still 
> have a consistent experience.
> 
> The PTI I'd like to propose for this is more complex, so I'd like to 
> describe it in terms of:
> 
> - OpenStack organizational requirements
> - helper sugar for developers with per-language recommendations
> 
> I believe the result can be a single in-tree doc PTI that applies to 
> python, go and javascript - and a single Zuul job that applies to all of 
> our projects AND non-OpenStack projects as well.
> 
> Organizational Requirements
> ---------------------------
> 
> Following readthedocs.org logic we can actually support a wider range of 
> schemes technically, but there is human value in having consistency on 
> these topics across our OpenStack repos.
> 
> * docs live in doc/source
> * Python requirements needed by Sphinx to build the docs live in 
> doc/requirements.txt
> 
> If the project is python:
> 
> * doc/requirements.txt can assume the project will have been installed
> * The following should be set in setup.cfg:
> 
>    [build_sphinx]
>    source-dir = doc/source
>    build-dir = doc/build
> 
> Doing the above allows the following commands to work cleanly in 
> automation no matter what the language is:
> 
>    [ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
>    sphinx-build -b doc/source doc/build

I suspect you mean "sphinx-build -b html doc/source doc/build" (the
"html" arg was missing).

We should also include a PDF build, since it is useful for
users who do not have good access to docs.o.o (especially when
governments block the site).

We should also formalize how we are going to handle translations
for both HTML and PDF. I think some of that is implemented, but I'm
not sure if it's actually documented anywhere.

Ian started a separate thread about this [1] that didn't see much
in the way of response because everyone was involved in the zuul
migration work.

[1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122461.html

> 
> No additional commands should be required.
> 
> The setup.cfg stanza allows:
> 
>    python setup.py build_sphinx
> 
> to continue to work. (also, everyone already has one)
> 
> Helper sugar for developers
> ---------------------------
> 
> The following are recommended ways to provide in-tree developer convenience:
> 
> * tox -edocs  # recommended for Python projects
> * make docs  # recommended for golang projects
> * npm run document  # recommended for javascript projects
> 
> These will not be used by Zuul - they are there purely for local 
> developer convenience.
> 
> Rolling it Out
> --------------
> 
> First, the governance patch:
> 
>    https://review.openstack.org/508694
> 
> I've also written a patch for pbr to add support for finding a 
> doc/requirements.txt file, if one exists, and adding the contents to a 
> "docs" extra:
> 
>    https://review.openstack.org/#/c/492620
> 
> That will let people do this in their tox.ini files:
> 
>    [testenv:docs]
>    extras = docs
>    commands=
>      python setup.py build_sphinx
> 
> And a WIP job for Zuul v3 written to provide a non-tox based build:
> 
>    https://review.openstack.org/#/c/492709/
> 
> (that job can be improved a little if we land the pbr docs extras patch)
>