Open Stack

On 09/30/2017 10:40 AM, Doug Hellmann wrote:
> Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:
>> Hey everybody,
>>
>> Oh goodie, I can hear you say, let's definitely spend some time
>> bikeshedding about specific command invocations related to building docs
>> and tarballs!!!
>>
>> tl;dr I want to change the PTI for docs and tarball building to be less
>> OpenStack-specific
>>
>> The Problem
>> ===========
>>
>> As we work on Zuul v3, there are a set of job definitions that are
>> rather fundamental that can totally be directly shared between Zuul
>> installations whether those Zuuls are working with OpenStack content or
>> not. As an example "tox -epy27" is a fairly standard thing, so a Zuul
>> job called "tox-py27" has no qualities specific to OpenStack and could
>> realistically be used by anyone who uses tox to manage their project.
>>
>> Docs and Tarballs builds for us, however, are the following:
>>
>> tox -evenv -- python setup.py sdist
>> tox -evenv -- python setup.py build_sphinx
>>
>> Neither of those are things that are likely to work outside of
>> OpenStack. (The 'venv' tox environment is not a default tox thing)
>>
>> I'm going to argue neither of them are actually providing us with much
>> value.
>>
>> Tarball Creation
>> ================
>>
>> Tarball creation is super simple. setup_requires is already handled out
>> of band of everything else. Go clone nova into a completely empty system
>> and run python setup.py sdist ... and it works. (actually, nova is big.
>> use something smaller like gertty ...)
>>
>>      docker run -it --rm python bash -c 'git clone \
>>        https://git.openstack.org/openstack/gertty && cd gertty \
>>        && python setup.py sdist'
>>
>> There is not much value in that tox wrapper - and it's not like it's
>> making it EASIER to run the command. In fact, it's more typing.
>>
>> I propose we change the PTI from:
>>
>>     tox -evenv python setup.py sdist
>>
>> to:
>>
>>     python setup.py sdist
>>
>> and then change the gate jobs to use the non-tox form of the command.
>>
>> I'd also like to further change it to be explicit that we also build
>> wheels. So the ACTUAL commands that the project should support are:
>>
>>     python setup.py sdist
>>     python setup.py bdist_wheel
>>
>> All of our projects support this already, so this should be a no-op.
>>
>> Notes:
>>
>> * Python projects that need to build C extensions might need their pip
>> requirements (and bindep requirements) installed in order to run
>> bdist_wheel. We do not support that broadly at the moment ANYWAY - so
>> I'd like to leave that as an outlier and handle it when we need to
>> handle it.
>>
>> * It's *possible* that somewhere we have a repo that has somehow done
>> something that would cause python setup.py sdist or python setup.py
>> bdist_wheel to not work without pip requirements installed. I believe we
>> should consider that a bug and fix it in the project if we find such a
>> thing - but since we use pbr in all of the OpenStack projects, I find it
>> extremely unlikely.
>>
>> Governance patch submitted: https://review.openstack.org/508693
>>
>> Sphinx Documentation
>> ====================
>>
>> Doc builds are more complex - but I think there is a high amount of
>> value in changing how we invoke them for a few reasons.
>>
>> a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
>> b) we decided to use sphinx for go and javascript - but we invoke sphinx
>> differently for each of those (since they naturally don't have tox),
>> meaning we can't just have a "build-sphinx-docs" job and even share it
>> with ourselves.
>> c) readthedocs.org is an excellent Open Source site that builds and
>> hosts sphinx docs for projects. They have an interface for docs
>> requirements documented and defined that we can align. By aligning,
>> projects can use migrate between docs.o.o and readthedocs.org and still
>> have a consistent experience.
>>
>> The PTI I'd like to propose for this is more complex, so I'd like to
>> describe it in terms of:
>>
>> - OpenStack organizational requirements
>> - helper sugar for developers with per-language recommendations
>>
>> I believe the result can be a single in-tree doc PTI that applies to
>> python, go and javascript - and a single Zuul job that applies to all of
>> our projects AND non-OpenStack projects as well.
>>
>> Organizational Requirements
>> ---------------------------
>>
>> Following readthedocs.org logic we can actually support a wider range of
>> schemes technically, but there is human value in having consistency on
>> these topics across our OpenStack repos.
>>
>> * docs live in doc/source
>> * Python requirements needed by Sphinx to build the docs live in
>> doc/requirements.txt
>>
>> If the project is python:
>>
>> * doc/requirements.txt can assume the project will have been installed
>> * The following should be set in setup.cfg:
>>
>>     [build_sphinx]
>>     source-dir = doc/source
>>     build-dir = doc/build
>>
>> Doing the above allows the following commands to work cleanly in
>> automation no matter what the language is:
>>
>>     [ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
>>     sphinx-build -b doc/source doc/build
> 
> I suspect you mean "sphinx-build -b html doc/source doc/build" (the
> "html" arg was missing).

Yes! Please amend all code examples with the things to make them 
actually work. :)

> We should also include a PDF build, since it is useful for
> users who do not have good access to docs.o.o (especially when
> governments block the site).

YES

> We should also formalize how we are going to handle translations
> for both HTML and PDF. I think some of that is implemented, but I'm
> not sure if it's actually documented anywhere.

We've got translation handling connected to sphinx builds integrated 
into the releasenotes job. I was just saying to AJaeger that I would 
love to see that extracted into a Sphinx plugin perhaps - but whatever 
it is, I could not agree more.

It also occurs to me (just was poking at the v3 releasenotes job updates 
mnaser did) - that we should perhaps:

a) add reno releasenotes to the PTI for all languages
b) similar to doc/requirements.txt add support for 
releasenotes/requirements.txt - so that we could share much of the same 
logic between releasenotes and docs jobs.

and ...

c) probably add 'use openstackdocstheme" to the docs PTI.

> Ian started a separate thread about this [1] that didn't see much
> in the way of response because everyone was involved in the zuul
> migration work.
> 
> [1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122461.html

Awesome. I'll respond over there too.
>>
>> No additional commands should be required.
>>
>> The setup.cfg stanza allows:
>>
>>     python setup.py build_sphinx
>>
>> to continue to work. (also, everyone already has one)
>>
>> Helper sugar for developers
>> ---------------------------
>>
>> The following are recommended ways to provide in-tree developer convenience:
>>
>> * tox -edocs  # recommended for Python projects
>> * make docs  # recommended for golang projects
>> * npm run document  # recommended for javascript projects
>>
>> These will not be used by Zuul - they are there purely for local
>> developer convenience.
>>
>> Rolling it Out
>> --------------
>>
>> First, the governance patch:
>>
>>     https://review.openstack.org/508694
>>
>> I've also written a patch for pbr to add support for finding a
>> doc/requirements.txt file, if one exists, and adding the contents to a
>> "docs" extra:
>>
>>     https://review.openstack.org/#/c/492620
>>
>> That will let people do this in their tox.ini files:
>>
>>     [testenv:docs]
>>     extras = docs
>>     commands=
>>       python setup.py build_sphinx
>>
>> And a WIP job for Zuul v3 written to provide a non-tox based build:
>>
>>     https://review.openstack.org/#/c/492709/
>>
>> (that job can be improved a little if we land the pbr docs extras patch)
>>
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>