Open Stack

Hi folks,
    This is a heads-up to describe 3 sets of changes you'll start seeing
starting next week.

1) lower-constraints.txt handling
   TL;DR: Make sure projects do not specify a constraint file in install_command
2) Switch to the new canonical constraints URL on master
   TR;DR: Make sure you use https://releases.openstack.org/constraints/upper/master
3) Switch to the new canonical constraints URL on stable branches
   TR;DR: Make sure you use https://releases.openstack.org/constraints/upper/$series

These will be generated from a member of the requirements team[1], and
will be on the gerrit topic constraints-updates.  We'll start next week
to give y'all a few days to digest this email

Now for slightly more details:

1) lower-constraints.txt handling

If a projects has an install_command that includes a constraint file
*and* the also have a lower-constraints test env what may happen is
something like:

given a tox.ini that contains:
---
[testenv]
install_command = pip install -U -c upper-constraints.txt {opts} {deps}
deps = 
   -r requirements.txt
   -r test-requirements.txt
[testenv:lower-constraints]
deps = 
   -c lower-constrtaints.txt
   -r requirements.txt
   -r test-requirements.txt
---


pyXX:
virtualenv -p $python_version .tox/$testenv_name
.tox/$testenv_name/bin/pip install -U -c upper-constratints.txt -r requirements.txt -rtest-requirements.txt
.tox/$testenv_name/bin/pip install .

lower-constraints:
virtualenv -p $python_version .tox/$testenv_name
.tox/$testenv_name/bin/pip install -U -c upper-constratints.txt -c lower-constrtaints.txt -r requirements.txt -rtest-requirements.txt
.tox/$testenv_name/bin/pip install .

pip will constrain a library to the first item it encounters, which
pretty much means that the lower-constraints testenv is still testing
against upper-constraints.txt[3]

The fix is to move all constraints into deps rather than
install_command.  This impacts ~40 projects and will be fixed by hand,
rather than a tool so we can preserve each projects style for tox.ini.

There's a pretty good chance that this change will break the
lower-constraints testenv,  project teams are encouraged to help
identify the incorrect values in lower-constraints, the requirements
team will pick the current value in upper-constraints which almost
certainly raises a value well past the minimum.
https://review.opendev.org/#/c/601188/ is the change we did for keystone
way back when

Some projects have upper-constraints.txt in install_command but
explicitly override it in lower-constratints.txt this works and wont be
changed at this point but ideally all projects will use the same format
to reduce the impact in switching between projects.


2) Switch to the new canonical constraints URL on master

Way back at the first Denver PTG thr requirements, release and
infrastructure teams discussed avoiding using gitweb urls for
constraints urls.  This was discussed[4] in more detail after the
liberty EOL broke consumers still wanting to use tox as:
https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/liberty
'vanished'  The idea was re-raised[5] in Feb. this year where we devised
a plan to create URLs that abstract the git hosting and allow us to
switch between branches and tags at the right times in the OpenStack
life-cycle.

Other, more dynamic, ideas were discussed but over complicate things and
make OpenStack slightly harder to understand or create more work.

for example we could:
a. Add an install_script that looks up $metadata and grab the right constraints
   We're trying avoid scripts like this as it very rapidly gets us to
   the point where we're doing "managed copy-and-paste" between projects
   *or* projects install scripts diverge
b. Add to tox itself the ability to derive the URL from $metadata
   This is doable but frankly requires more time than we have right now.

So what we have is:

redirect 301 /constraints/upper/master http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=master
redirect 301 /constraints/upper/train http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=master
redirect 301 /constraints/upper/stein http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/stein
redirect 301 /constraints/upper/rocky http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/rocky
redirect 301 /constraints/upper/queens http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/queens
redirect 301 /constraints/upper/pike http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/pike
redirect 301 /constraints/upper/ocata http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/ocata
redirect 301 /constraints/upper/newton http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=stable/newton
redirect 301 /constraints/upper/mitaka http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=mitaka-eol
redirect 301 /constraints/upper/liberty http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=liberty-eol
redirect 301 /constraints/upper/kilo http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=kilo-eol
redirect 301 /constraints/upper/juno http://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt?h=juno-eol

[Yes I know we need to use opendev that's tomorrows problem]

so that means that:
http{s,}://releases.openstack.org/constraints/upper/master
http{s,}://releases.openstack.org/constraints/upper/$series

will always do the right thing based on the state of each OpenStack
release.

So we're going to generate changes that point to the right place.

For master we have 2 options:
a. Use .../train

If we use train in the constraints URLs now things will work and when we
branch for 'U' we'd need to update master to point to the right series
name.  This is very doable.  The release team already generates a change
on master at this point in time to update reno.  We also need to
update the new stable branch so that .gitreview points to the new
branch.

b. Use .../master

If we use 'master' in the constratints URLs master will always work and
when we branch for 'U' we'd need to update the stable/train branch to
point to train.  The release team already does this today along with the
.gitreview change.

So the bottom line is that using master (over train) is *slighty* less
work for the release team and *slightly* less work for each project team.
as such that's what we're going to do.

There are several changes in the works that set the constraints url to
either
https://opendev.org/openstack/requirements/raw/branch/master/upper-constraints.txt
or
https://releases.openstack.org/constraints/upper/train

These will work and will not break anything when the project
branches, however for the sake of OpenStack wide consistency, and
avoiding merge conflicts project teams are discouraged from merging
them, or if the change also fixes other references to git.openstack.org
just have the tox.ini updates removed.

Once tox.ini is done the requirements team will look for additional
places where the 'old style' urls are used and follow-up but right now
there's a lot of noise to sift through


3) Switch to the new canonical constraints URL on stable branches

Hopefully with all the background from 2 above this should be pretty
self-explanatory.  This is however much lower priority and probably wont
start until the bulk of 1 and 2 is complete

Yours Tony.

[1] https://review.opendev.org/#/admin/groups/131,members [2]
[2] Probably me actually ;P
[3] There is some nuance here but this is the general rule
[4] In the thread that starts with: http://lists.openstack.org/pipermail/openstack-dev/2017-September/122333.html
[5] In the thread that starts with: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002682.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190522/bb0a14e4/attachment-0001.sig>