Hello Jeremy,

Thanks for the reply. Others are attempting to fix here https://review.opendev.org/c/openstack/magnum/+/767228 Not sure why only magnum is affected by this. I'll point them here.

Thanks, Spyros

On Tue, Dec 8, 2020 at 6:13 PM Jeremy Stanley fungi@yuggoth.org wrote:

On 2020-12-08 13:36:09 +0100 (+0100), Spyros Trigazis wrote:

...

openstack-tox-lower-constraints fails for bashate and coverage. (Maybe more, I bumped bashate and it failed for coverage. I don;t want to waste more resources on our CI) eg https://review.opendev.org/c/openstack/magnum/+/765881 https://review.opendev.org/c/openstack/magnum/+/765979

Do we miss something?

Pip 20.3.0, released 8 days ago, turned on a new and much more thorough dependency resolver. Earlier versions of pip did not try particularly hard to make sure the dependencies claimed by packages were all satisfied. Virtualenv 20.2.2 released yesterday and increased the version of pip it's vendoring to a version which uses the new solver as well. These changes mean that latent version conflicts are now being correctly identified as bugs, and these jobs will do a far better job of actually confirming the declared versions of dependencies are able to be tested.

One thing which looks really weird and completely contradictory to me is that your lower-constraints job on change 765881 is applying both upper and lower constraints lists to the pip install command. Maybe the lower constraints list is expected to override the earlier upper constraints, but is that really going to represent a compatible set? That aside, trying to reproduce locally I run into yet a third error:

Could not find a version that satisfies the requirement
    warlock!=1.3.0,<2,>=1.0.1 (from python-glanceclient)

And indeed, python-glanceclient insists warlock 1.3.0 should be skipped, while magnum's lower-constraints.txt says you must install warlock==1.3.0 so that's a clear contradiction as well.

My recommendation is to work on reproducing this locally first and play a bit of whack-a-mole with the entries in your lower-constraints.txt to find versions of things which will actually be coinstallable with current versions of pip. You don't need to run the full tox testenv, just try installing your constrainted deps into a venv with upgraded pip like so:

python3.8 -m venv foo
foo/bin/pip install -U pip
foo/bin/pip install -c lower-constraints.txt \
    -r test-requirements.txt -r requirements.txt

You'll also likely want to delete and recreate the venv each time you try, since pip will now also try to take the requirements of already installed packages into account, and that might further change the behavior you see. Hope that helps! -- Jeremy Stanley

On Tue, 8 Dec 2020 at 18:24, Jeremy Stanley fungi@yuggoth.org wrote:

On 2020-12-08 13:36:09 +0100 (+0100), Spyros Trigazis wrote:

...

openstack-tox-lower-constraints fails for bashate and coverage. (Maybe more, I bumped bashate and it failed for coverage. I don;t want to waste more resources on our CI) eg https://review.opendev.org/c/openstack/magnum/+/765881 https://review.opendev.org/c/openstack/magnum/+/765979

Do we miss something?

Pip 20.3.0, released 8 days ago, turned on a new and much more thorough dependency resolver. Earlier versions of pip did not try particularly hard to make sure the dependencies claimed by packages were all satisfied. Virtualenv 20.2.2 released yesterday and increased the version of pip it's vendoring to a version which uses the new solver as well. These changes mean that latent version conflicts are now being correctly identified as bugs, and these jobs will do a far better job of actually confirming the declared versions of dependencies are able to be tested.

One thing which looks really weird and completely contradictory to me is that your lower-constraints job on change 765881 is applying both upper and lower constraints lists to the pip install command. Maybe the lower constraints list is expected to override the earlier upper constraints, but is that really going to represent a compatible set? That aside, trying to reproduce locally I run into yet a third error:

Could not find a version that satisfies the requirement
    warlock!=1.3.0,<2,>=1.0.1 (from python-glanceclient)

And indeed, python-glanceclient insists warlock 1.3.0 should be skipped, while magnum's lower-constraints.txt says you must install warlock==1.3.0 so that's a clear contradiction as well.

My recommendation is to work on reproducing this locally first and play a bit of whack-a-mole with the entries in your lower-constraints.txt to find versions of things which will actually be coinstallable with current versions of pip. You don't need to run the full tox testenv, just try installing your constrainted deps into a venv with upgraded pip like so:

python3.8 -m venv foo
foo/bin/pip install -U pip
foo/bin/pip install -c lower-constraints.txt \
    -r test-requirements.txt -r requirements.txt

You'll also likely want to delete and recreate the venv each time you try, since pip will now also try to take the requirements of already installed packages into account, and that might further change the behavior you see. Hope that helps! -- Jeremy Stanley

Hi Jeremy,

Sorry for hijacking this thread, but I am dealing with the same sort of issues in blazar. The lower-constraints job is failing [1], which has broken our gate. However, I cannot reproduce the issue locally: `tox -e lower-constraints` works fine, and so do your recommended commands with the foo venv. I've tried on multiple operating systems, including Ubuntu 20.04 (with Python 3.8 and pip 20.3.3).

So I have been playing whack-a-mole [2] from Zuul logs based on whichever package fails to install, but it's a very slow process. Do you know what I could be missing for reproducing this locally?

Thanks, Pierre

[1] https://zuul.opendev.org/t/openstack/build/34897587cf954016b8027670117213f6 [2] https://review.opendev.org/c/openstack/blazar/+/767593/

On Mon, 4 Jan 2021 at 19:18, Jeremy Stanley fungi@yuggoth.org wrote:

On 2021-01-04 18:41:06 +0100 (+0100), Pierre Riteau wrote: [...]

...

I cannot reproduce the issue locally: `tox -e lower-constraints` works fine, and so do your recommended commands with the foo venv. I've tried on multiple operating systems, including Ubuntu 20.04 (with Python 3.8 and pip 20.3.3).

[...]

...

Do you know what I could be missing for reproducing this locally?

[...]

At this point it may be easier to reproduce with the latest version of virtualenv (20.2.2 at time of writing) since it started pulling in pip 20.3.1 on its own. Just upgrade virtualenv and try your tox command again and see if things are any different for you. It reproduces exactly on my workstation with latest tox/virtualenv just running `tox -e lower-constraints` in a current blazar repo master branch checkout.

As for the doubled-constraints I mentioned in the earlier post, looks like blazar is hitting that as well. To correct it you need to move your upper-constraints addition into the default testenv deps list rather than adding it to the default install_command string. Right now it's getting inherited in testenv:lower-constraints too which can't be a good thing. Compare the approach in blazar with that of nova:

https://opendev.org/openstack/blazar/src/commit/cb7c142a890e84a2b3171395832d...

https://opendev.org/openstack/nova/src/commit/b0f241e5425c99866223bae4b404a4...

Changing it in my local blazar checkout gets farther, though it looks like the old version of markupsafe you're trying to use may not work with latest versions of setuptools (ImportError: cannot import name 'Feature' from 'setuptools'). Hope that helps! -- Jeremy Stanley

Hi Jeremy,

Thank you very much, this helped a lot.

Combining the install_command fix with the latest virtualenv, I managed to reproduce the job failure locally and updated outdated lower constraints. Somehow my patch was working locally while still failing in Zuul with "oslo-db 4.40.0 depends on alembic>=0.9.6", but after bumping alembic it now succeeds.

Cheers, Pierre