[nova] openstack-tox-lower-constraints broken
Hello all,
$subject, I've raised the following bug:
openstack-tox-lower-constraints failing due to unmet dependency on decorator==4.0.0 https://launchpad.net/bugs/1890123
I'm trying to resolve this below but I honestly feel like I'm going around in circles:
https://review.opendev.org/#/q/topic:bug/1890123
If anyone has any tooling and/or recommendations for resolving issues like this I'd appreciate it!
Cheers,
On 8/3/20 7:55 AM, Lee Yarwood wrote:
Hello all,
$subject, I've raised the following bug:
openstack-tox-lower-constraints failing due to unmet dependency on decorator==4.0.0 https://launchpad.net/bugs/1890123
I'm trying to resolve this below but I honestly feel like I'm going around in circles:
https://review.opendev.org/#/q/topic:bug/1890123
If anyone has any tooling and/or recommendations for resolving issues like this I'd appreciate it!
Cheers,
This appears to be broken for everyone. I initially saw the decorator thing with Cinder, but after looking closer realized it's not that package.
The root issue (or at least one level closer to the root issue, that seems to be causing the decorator failure) is that the lower-constraints are not actually being enforced. Even though the logs should it is passing "-c [path to lower-constraints.txt]". So even though things should be constrained to a lower version, presumably a version that works with a different version of decorator, pip is still installing a newer package than what the constraints should allow.
There was a pip release on the 28th. Things don't look like they started failing until the 31st for us though, so either that is not it, or there was just a delay before our nodes started picking up the newer version.
I tested locally, and at least with version 19.3.1, I am getting the correctly constrained packages installed.
Still looking, but thought I would share in case that info triggers any ideas for anyone else.
Sean
The root issue (or at least one level closer to the root issue, that seems to be causing the decorator failure) is that the lower-constraints are not actually being enforced. Even though the logs should it is passing "-c [path to lower-constraints.txt]". So even though things should be constrained to a lower version, presumably a version that works with a different version of decorator, pip is still installing a newer package than what the constraints should allow.
There was a pip release on the 28th. Things don't look like they started failing until the 31st for us though, so either that is not it, or there was just a delay before our nodes started picking up the newer version.
I tested locally, and at least with version 19.3.1, I am getting the correctly constrained packages installed.
Still looking, but thought I would share in case that info triggers any ideas for anyone else.
I upgraded my pip and rebuilt the venv. The new pip has some good warnings emitted about some incompatible conflicts, so that part is good, but it did not change the package installation behavior. I am still able to get the correctly constrained packages installed locally on Fedora 32. So at least so far, it doesn't appear to be a pip issue.
On 03/08, Sean McGinnis wrote:
On 8/3/20 7:55 AM, Lee Yarwood wrote:
Hello all,
$subject, I've raised the following bug:
openstack-tox-lower-constraints failing due to unmet dependency on decorator==4.0.0 https://launchpad.net/bugs/1890123
I'm trying to resolve this below but I honestly feel like I'm going around in circles:
https://review.opendev.org/#/q/topic:bug/1890123
If anyone has any tooling and/or recommendations for resolving issues like this I'd appreciate it!
Cheers,
This appears to be broken for everyone. I initially saw the decorator thing with Cinder, but after looking closer realized it's not that package.
The root issue (or at least one level closer to the root issue, that seems to be causing the decorator failure) is that the lower-constraints are not actually being enforced. Even though the logs should it is passing "-c [path to lower-constraints.txt]". So even though things should be constrained to a lower version, presumably a version that works with a different version of decorator, pip is still installing a newer package than what the constraints should allow.
There was a pip release on the 28th. Things don't look like they started failing until the 31st for us though, so either that is not it, or there was just a delay before our nodes started picking up the newer version.
I tested locally, and at least with version 19.3.1, I am getting the correctly constrained packages installed.
Still looking, but thought I would share in case that info triggers any ideas for anyone else.
Sean
Hi,
Looking at one of my patches I see that the right version of dogpile.cache==0.6.5 is being installed [1], but then at another step we download [2] and install [3] version 1.0.1, and we can see that pip is actually complaining that we have incompatibilities [4].
As far as I can see this is because in that pip install we requested to wipe existing installed packages [6] and we are not passing any constraints in that call.
I don't know why or where we are doing that though.
Cheers, Gorka.
[1]: https://zuul.opendev.org/t/openstack/build/49f226f8efb94c088cb2b22c46565d97/... [2]: https://zuul.opendev.org/t/openstack/build/49f226f8efb94c088cb2b22c46565d97/... [3]: https://zuul.opendev.org/t/openstack/build/49f226f8efb94c088cb2b22c46565d97/... [4]: https://zuul.opendev.org/t/openstack/build/49f226f8efb94c088cb2b22c46565d97/... [5]: https://zuul.opendev.org/t/openstack/build/49f226f8efb94c088cb2b22c46565d97/...
On 2020-08-04 14:26:06 +0200 (+0200), Gorka Eguileor wrote: [...]
Looking at one of my patches I see that the right version of dogpile.cache==0.6.5 is being installed [1], but then at another step we download [2] and install [3] version 1.0.1, and we can see that pip is actually complaining that we have incompatibilities [4].
As far as I can see this is because in that pip install we requested to wipe existing installed packages [6] and we are not passing any constraints in that call.
I don't know why or where we are doing that though.
[...]
Yes, I started digging into this yesterday too. It's affecting all tox jobs, not just lower-constraints jobs (upper-constraints is close enough to unconstrained that this isn't immediately apparent for master branch jobs, but the divergence becomes obvious in stable branch jobs and it's breaking lots of them). It seems this started roughly a week ago.
I don't think we're explicitly doing it, this seems to be a behavior baked into tox itself. Most projects are currently applying constraints via the deps parameter in their tox.ini, and tox appears to invoke pip twice: once to install your deps, and then a second time to install the project being tested. The latter phase does not use the deps parameter, and so no constraints get applied.
We might be able to work around this by going back to overriding install_command and putting the -c option there instead, but I haven't had an opportunity to test that theory yet. If anyone else has time to pursue this line of investigation, I'd be curious to hear whether it helps.
On Tue, 2020-08-04 at 12:39 +0000, Jeremy Stanley wrote:
On 2020-08-04 14:26:06 +0200 (+0200), Gorka Eguileor wrote: [...]
Looking at one of my patches I see that the right version of dogpile.cache==0.6.5 is being installed [1], but then at another step we download [2] and install [3] version 1.0.1, and we can see that pip is actually complaining that we have incompatibilities [4].
As far as I can see this is because in that pip install we requested to wipe existing installed packages [6] and we are not passing any constraints in that call.
I don't know why or where we are doing that though.
[...]
Yes, I started digging into this yesterday too. It's affecting all tox jobs, not just lower-constraints jobs (upper-constraints is close enough to unconstrained that this isn't immediately apparent for master branch jobs, but the divergence becomes obvious in stable branch jobs and it's breaking lots of them). It seems this started roughly a week ago.
I don't think we're explicitly doing it, this seems to be a behavior baked into tox itself. Most projects are currently applying constraints via the deps parameter in their tox.ini, and tox appears to invoke pip twice: once to install your deps, and then a second time to install the project being tested. The latter phase does not use the deps parameter, and so no constraints get applied.
We might be able to work around this by going back to overriding install_command and putting the -c option there instead,
right so stephen asked me to remove that override in one of my recent patches to os-vif that is under view since he made the comment the command we were using was more or less the same as the default we currently set teh -c in deps. so if i understand the workaound correclty we woudl add -c {env:CONSTRAINTS_OPT} to install_command so "install_command = pip install -U {opts} {packages} -c {env:CONSTRAINTS_OPT}" in our case and then for the lower contriats jobs in stead of
deps = -c{toxinidir}/lower-constraints.txt -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -r{toxinidir}/doc/requirements.txt
we would do setenv = CONSTRAINTS_OPT=-c{toxinidir}/lower-constraints.txt deps = -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -r{toxinidir}/doc/requirements.txt
that way we can keep the same install command for both but use the correct constrint file.
but I haven't had an opportunity to test that theory yet. If anyone else has time to pursue this line of investigation, I'd be curious to hear whether it helps.
On 2020-08-04 14:11:03 +0100 (+0100), Sean Mooney wrote: [...]
so if i understand the workaound correclty we woudl add -c {env:CONSTRAINTS_OPT} to install_command so "install_command = pip install -U {opts} {packages} -c {env:CONSTRAINTS_OPT}" in our case and then for the lower contriats jobs in stead of
deps = -c{toxinidir}/lower-constraints.txt -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -r{toxinidir}/doc/requirements.txt
we would do
setenv = CONSTRAINTS_OPT=-c{toxinidir}/lower-constraints.txt deps = -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -r{toxinidir}/doc/requirements.txt
that way we can keep the same install command for both but use the correct constrint file.
[...]
Yep, Sean McGinnis is trying a variant of that in https://review.opendev.org/744698 now to see if it alters tox's behavior like we expect.
Hi all!
After a very interesting and enlightening discussion with Sean and Clark on IRC (thanks!), we were able to test and verify that the issue is related to the latest released version of virtualenv, v2.0.29, that embeds pip 2.20, apparently the real offender here. I submitted a bug to virtualenv [1] for that, the fix is included in pip 2.20.1. The bump in virtualenv is already up [2] and merged and a new version has been released, v2.0.30 [3], that should solve this issue.
[1] https://github.com/pypa/virtualenv/issues/1914 [2] https://github.com/pypa/virtualenv/pull/1915 [3] https://pypi.org/project/virtualenv/20.0.30/
A si biri,
Riccardo
On Tue, Aug 4, 2020 at 3:26 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2020-08-04 14:11:03 +0100 (+0100), Sean Mooney wrote: [...]
so if i understand the workaound correclty we woudl add -c {env:CONSTRAINTS_OPT} to install_command so "install_command = pip install -U {opts} {packages} -c {env:CONSTRAINTS_OPT}" in our case and then for the lower contriats jobs in stead of
deps = -c{toxinidir}/lower-constraints.txt -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -r{toxinidir}/doc/requirements.txt
we would do
setenv = CONSTRAINTS_OPT=-c{toxinidir}/lower-constraints.txt deps = -r{toxinidir}/requirements.txt -r{toxinidir}/test-requirements.txt -r{toxinidir}/doc/requirements.txt
that way we can keep the same install command for both but use the correct constrint file.
[...]
Yep, Sean McGinnis is trying a variant of that in https://review.opendev.org/744698 now to see if it alters tox's behavior like we expect. -- Jeremy Stanley
On 2020-08-04 19:09:33 +0200 (+0200), Riccardo Pittau wrote:
After a very interesting and enlightening discussion with Sean and Clark on IRC (thanks!), we were able to test and verify that the issue is related to the latest released version of virtualenv, v2.0.29, that embeds pip 2.20, apparently the real offender here.
[...]
That was indeed confusing. Until I skimmed virtualenv's changelog it hadn't dawned on me that all the problem libraries had a "." in their names.
On 2020-08-04 17:32:00 +0000 (+0000), Jeremy Stanley wrote:
On 2020-08-04 19:09:33 +0200 (+0200), Riccardo Pittau wrote:
After a very interesting and enlightening discussion with Sean and Clark on IRC (thanks!), we were able to test and verify that the issue is related to the latest released version of virtualenv, v2.0.29, that embeds pip 2.20, apparently the real offender here.
[...]
That was indeed confusing. Until I skimmed virtualenv's changelog it hadn't dawned on me that all the problem libraries had a "." in their names.
Er, pip's changelog I meant, of course.
participants (6)
-
Gorka Eguileor
-
Jeremy Stanley
-
Lee Yarwood
-
Riccardo Pittau
-
Sean McGinnis
-
Sean Mooney