Open Stack

Excerpts from Monty Taylor's message of 2017-10-05 08:45:52 -0500:
> On 09/30/2017 10:40 AM, Doug Hellmann wrote:
> > Excerpts from Monty Taylor's message of 2017-09-30 10:20:08 -0500:
> >> Hey everybody,
> >>
> >> Oh goodie, I can hear you say, let's definitely spend some time
> >> bikeshedding about specific command invocations related to building docs
> >> and tarballs!!!
> >>
> >> tl;dr I want to change the PTI for docs and tarball building to be less
> >> OpenStack-specific
> >>
> >> The Problem
> >> ===========
> >>
> >> As we work on Zuul v3, there are a set of job definitions that are
> >> rather fundamental that can totally be directly shared between Zuul
> >> installations whether those Zuuls are working with OpenStack content or
> >> not. As an example "tox -epy27" is a fairly standard thing, so a Zuul
> >> job called "tox-py27" has no qualities specific to OpenStack and could
> >> realistically be used by anyone who uses tox to manage their project.
> >>
> >> Docs and Tarballs builds for us, however, are the following:
> >>
> >> tox -evenv -- python setup.py sdist
> >> tox -evenv -- python setup.py build_sphinx
> >>
> >> Neither of those are things that are likely to work outside of
> >> OpenStack. (The 'venv' tox environment is not a default tox thing)
> >>
> >> I'm going to argue neither of them are actually providing us with much
> >> value.
> >>
> >> Tarball Creation
> >> ================
> >>
> >> Tarball creation is super simple. setup_requires is already handled out
> >> of band of everything else. Go clone nova into a completely empty system
> >> and run python setup.py sdist ... and it works. (actually, nova is big.
> >> use something smaller like gertty ...)
> >>
> >>      docker run -it --rm python bash -c 'git clone \
> >>        https://git.openstack.org/openstack/gertty && cd gertty \
> >>        && python setup.py sdist'
> >>
> >> There is not much value in that tox wrapper - and it's not like it's
> >> making it EASIER to run the command. In fact, it's more typing.
> >>
> >> I propose we change the PTI from:
> >>
> >>     tox -evenv python setup.py sdist
> >>
> >> to:
> >>
> >>     python setup.py sdist
> >>
> >> and then change the gate jobs to use the non-tox form of the command.
> >>
> >> I'd also like to further change it to be explicit that we also build
> >> wheels. So the ACTUAL commands that the project should support are:
> >>
> >>     python setup.py sdist
> >>     python setup.py bdist_wheel
> >>
> >> All of our projects support this already, so this should be a no-op.
> >>
> >> Notes:
> >>
> >> * Python projects that need to build C extensions might need their pip
> >> requirements (and bindep requirements) installed in order to run
> >> bdist_wheel. We do not support that broadly at the moment ANYWAY - so
> >> I'd like to leave that as an outlier and handle it when we need to
> >> handle it.
> >>
> >> * It's *possible* that somewhere we have a repo that has somehow done
> >> something that would cause python setup.py sdist or python setup.py
> >> bdist_wheel to not work without pip requirements installed. I believe we
> >> should consider that a bug and fix it in the project if we find such a
> >> thing - but since we use pbr in all of the OpenStack projects, I find it
> >> extremely unlikely.
> >>
> >> Governance patch submitted: https://review.openstack.org/508693
> >>
> >> Sphinx Documentation
> >> ====================
> >>
> >> Doc builds are more complex - but I think there is a high amount of
> >> value in changing how we invoke them for a few reasons.
> >>
> >> a) nobody uses 'tox -evenv -- python setup.py build_sphinx' but us
> >> b) we decided to use sphinx for go and javascript - but we invoke sphinx
> >> differently for each of those (since they naturally don't have tox),
> >> meaning we can't just have a "build-sphinx-docs" job and even share it
> >> with ourselves.
> >> c) readthedocs.org is an excellent Open Source site that builds and
> >> hosts sphinx docs for projects. They have an interface for docs
> >> requirements documented and defined that we can align. By aligning,
> >> projects can use migrate between docs.o.o and readthedocs.org and still
> >> have a consistent experience.
> >>
> >> The PTI I'd like to propose for this is more complex, so I'd like to
> >> describe it in terms of:
> >>
> >> - OpenStack organizational requirements
> >> - helper sugar for developers with per-language recommendations
> >>
> >> I believe the result can be a single in-tree doc PTI that applies to
> >> python, go and javascript - and a single Zuul job that applies to all of
> >> our projects AND non-OpenStack projects as well.
> >>
> >> Organizational Requirements
> >> ---------------------------
> >>
> >> Following readthedocs.org logic we can actually support a wider range of
> >> schemes technically, but there is human value in having consistency on
> >> these topics across our OpenStack repos.
> >>
> >> * docs live in doc/source
> >> * Python requirements needed by Sphinx to build the docs live in
> >> doc/requirements.txt
> >>
> >> If the project is python:
> >>
> >> * doc/requirements.txt can assume the project will have been installed
> >> * The following should be set in setup.cfg:
> >>
> >>     [build_sphinx]
> >>     source-dir = doc/source
> >>     build-dir = doc/build
> >>
> >> Doing the above allows the following commands to work cleanly in
> >> automation no matter what the language is:
> >>
> >>     [ -f doc/requirements.txt ] && pip install -rdoc/requirements.txt
> >>     sphinx-build -b doc/source doc/build
> > 
> > I suspect you mean "sphinx-build -b html doc/source doc/build" (the
> > "html" arg was missing).
> 
> Yes! Please amend all code examples with the things to make them 
> actually work. :)
> 
> > We should also include a PDF build, since it is useful for
> > users who do not have good access to docs.o.o (especially when
> > governments block the site).
> 
> YES
> 
> > We should also formalize how we are going to handle translations
> > for both HTML and PDF. I think some of that is implemented, but I'm
> > not sure if it's actually documented anywhere.
> 
> We've got translation handling connected to sphinx builds integrated 
> into the releasenotes job. I was just saying to AJaeger that I would 
> love to see that extracted into a Sphinx plugin perhaps - but whatever 
> it is, I could not agree more.
> 
> It also occurs to me (just was poking at the v3 releasenotes job updates 
> mnaser did) - that we should perhaps:
> 
> a) add reno releasenotes to the PTI for all languages
> b) similar to doc/requirements.txt add support for 
> releasenotes/requirements.txt - so that we could share much of the same 
> logic between releasenotes and docs jobs.
> 
> and ...
> 
> c) probably add 'use openstackdocstheme" to the docs PTI.

Those additions all make sense. Reno is most useful where we have stable
branches, but it does address merge conflict issues on a single branch,
too. And building the release notes with sphinx would work the same way
as building the docs, just using different paths as input arguments.

> 
> > Ian started a separate thread about this [1] that didn't see much
> > in the way of response because everyone was involved in the zuul
> > migration work.
> > 
> > [1] http://lists.openstack.org/pipermail/openstack-dev/2017-September/122461.html
> 
> Awesome. I'll respond over there too.
> >>
> >> No additional commands should be required.
> >>
> >> The setup.cfg stanza allows:
> >>
> >>     python setup.py build_sphinx
> >>
> >> to continue to work. (also, everyone already has one)
> >>
> >> Helper sugar for developers
> >> ---------------------------
> >>
> >> The following are recommended ways to provide in-tree developer convenience:
> >>
> >> * tox -edocs  # recommended for Python projects
> >> * make docs  # recommended for golang projects
> >> * npm run document  # recommended for javascript projects
> >>
> >> These will not be used by Zuul - they are there purely for local
> >> developer convenience.
> >>
> >> Rolling it Out
> >> --------------
> >>
> >> First, the governance patch:
> >>
> >>     https://review.openstack.org/508694
> >>
> >> I've also written a patch for pbr to add support for finding a
> >> doc/requirements.txt file, if one exists, and adding the contents to a
> >> "docs" extra:
> >>
> >>     https://review.openstack.org/#/c/492620
> >>
> >> That will let people do this in their tox.ini files:
> >>
> >>     [testenv:docs]
> >>     extras = docs
> >>     commands=
> >>       python setup.py build_sphinx
> >>
> >> And a WIP job for Zuul v3 written to provide a non-tox based build:
> >>
> >>     https://review.openstack.org/#/c/492709/
> >>
> >> (that job can be improved a little if we land the pbr docs extras patch)
> >>
> > 
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
>