[all][dev] Beware how fun the new pip can be
Hiya, Folks! Sharing what I have just learnt about the new pip's solver. pip install PROJECT no longer guarantees to install the latest version of PROJECT (or, well, giving you the ERROR that it cannot do it because something something :-) ). In fact, it will install the latest version *matching other constraints* and do it *silently*. Like it was recently only with Python version (i.e. py3-only would not get installed on py2 - that is cool) but now it moved into any-package territory. As an example, I can give you [1] where we are experimenting with getting some extracurricular package into our containers, notably fluent-logger. The only dep of fluent-logger is msgpack but the latest msgpack (as in upper constraints: 1.0.2, or any 1.x for that matter) is not compatible. However, the pin was introduced in fluent-logger in its 0.9.5 release (0.9.6 is the latest). Guess what pip does? Here is what it does: INFO:kolla.common.utils.openstack-base:Collecting fluent-logger INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/1a/f5/e6c30ec7... (12 kB) INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/d5/cb/19d83856... (12 kB) INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/d1/d4/f9b3493f... (12 kB) And that's it. Pip is happy, you got your "latest" version. In previous pip one would get the latest version AND a warning. Now just pip's view on what the "latest" version is. I am glad we have upper-constraints which save the day here (forcing the ERROR) but beware of this "in the wild". [1] https://review.opendev.org/c/openstack/kolla/+/759855 -yoctozepto
On Wed, Jan 13, 2021 at 1:37 PM Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
Hiya, Folks!
Sharing what I have just learnt about the new pip's solver. pip install PROJECT no longer guarantees to install the latest version of PROJECT (or, well, giving you the ERROR that it cannot do it because something something :-) ). In fact, it will install the latest version *matching other constraints* and do it *silently*. Like it was recently only with Python version (i.e. py3-only would not get installed on py2 - that is cool) but now it moved into any-package territory.
As an example, I can give you [1] where we are experimenting with getting some extracurricular package into our containers, notably fluent-logger. The only dep of fluent-logger is msgpack but the latest msgpack (as in upper constraints: 1.0.2, or any 1.x for that matter) is not compatible. However, the pin was introduced in fluent-logger in its 0.9.5 release (0.9.6 is the latest). Guess what pip does? Here is what it does:
INFO:kolla.common.utils.openstack-base:Collecting fluent-logger INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/1a/f5/e6c30ec7... (12 kB) INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/d5/cb/19d83856... (12 kB) INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/d1/d4/f9b3493f... (12 kB)
And that's it. Pip is happy, you got your "latest" version. In previous pip one would get the latest version AND a warning. Now just pip's view on what the "latest" version is.
I am glad we have upper-constraints which save the day here (forcing the ERROR) but beware of this "in the wild".
[1] https://review.opendev.org/c/openstack/kolla/+/759855
-yoctozepto
Really big change, no surprise it's full of bugs. I had a situation with an infinite loop of 'Requirement already satisfied' just yesterday. Can only suggest to file issues https://github.com/pypa/pip/issues and fall back to 20.2, i think virtualenv==20.2.1 is the latest that comes with 20.2 pip
A quick addendum after discussion with Clark (to make it easier to digest for everyone). The actors from PyPI: msgpack - the latest is 1.0.2 and that is what is in the upper-constraints that are being used fluent-logger - the latest is 0.9.6 and this is being installed *unconstrained* fluent-logger sets one dep: msgpack<1.0.0 since fluent-logger 0.9.5 The result: 1) old pip msgpack==1.0.2 fluent-logger==0.9.6 and a WARNING that fluent-logger 0.9.6 wants msgpack<1.0.0 2) new pip msgpack==1.0.2 fluent-logger==0.9.4 and no WARNINGs, no ERRORs, no anything, just happy silent "I got you your package, so what if it is not the latest, I am the smart one here" i.e. controlling *dependencies* controls *dependants* And don't get me wrong, pip did what it advertised - took a list of constraints and found a solution. The outtake is simple: beware! :-) -yoctozepto On Wed, Jan 13, 2021 at 8:36 PM Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
Hiya, Folks!
Sharing what I have just learnt about the new pip's solver. pip install PROJECT no longer guarantees to install the latest version of PROJECT (or, well, giving you the ERROR that it cannot do it because something something :-) ). In fact, it will install the latest version *matching other constraints* and do it *silently*. Like it was recently only with Python version (i.e. py3-only would not get installed on py2 - that is cool) but now it moved into any-package territory.
As an example, I can give you [1] where we are experimenting with getting some extracurricular package into our containers, notably fluent-logger. The only dep of fluent-logger is msgpack but the latest msgpack (as in upper constraints: 1.0.2, or any 1.x for that matter) is not compatible. However, the pin was introduced in fluent-logger in its 0.9.5 release (0.9.6 is the latest). Guess what pip does? Here is what it does:
INFO:kolla.common.utils.openstack-base:Collecting fluent-logger INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/1a/f5/e6c30ec7... (12 kB) INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/d5/cb/19d83856... (12 kB) INFO:kolla.common.utils.openstack-base: Downloading http://mirror-int.dfw.rax.opendev.org:8080/pypifiles/packages/d1/d4/f9b3493f... (12 kB)
And that's it. Pip is happy, you got your "latest" version. In previous pip one would get the latest version AND a warning. Now just pip's view on what the "latest" version is.
I am glad we have upper-constraints which save the day here (forcing the ERROR) but beware of this "in the wild".
[1] https://review.opendev.org/c/openstack/kolla/+/759855
-yoctozepto
On 2021-01-13 20:36:06 +0100 (+0100), Radosław Piliszek wrote: [...]
As an example, I can give you [1] where we are experimenting with getting some extracurricular package into our containers, notably fluent-logger. The only dep of fluent-logger is msgpack but the latest msgpack (as in upper constraints: 1.0.2, or any 1.x for that matter) is not compatible. However, the pin was introduced in fluent-logger in its 0.9.5 release (0.9.6 is the latest). [...]
So just to clarify, your concern is that because you've tried to install newer msgpack, pip is selecting an older version of fluent-logger which doesn't declare an incompatibility with that newer version of msgpack. This seems technically correct. I'm willing to bet if you insisted on installing fluent-logger>0.9.5 you would get the behavior you're expecting. The underlying problem is that the package ecosystem has long based dependency versioning choices on side effect behaviors of pip's (lack of coherent) dep resolution. From the user side of things, if you want to install more than one package explicitly, you need to start specifying how new you want those packages to be. However surprising it is, pip seems to be working as intended here. -- Jeremy Stanley
On Wed, Jan 13, 2021 at 9:22 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2021-01-13 20:36:06 +0100 (+0100), Radosław Piliszek wrote: [...]
As an example, I can give you [1] where we are experimenting with getting some extracurricular package into our containers, notably fluent-logger. The only dep of fluent-logger is msgpack but the latest msgpack (as in upper constraints: 1.0.2, or any 1.x for that matter) is not compatible. However, the pin was introduced in fluent-logger in its 0.9.5 release (0.9.6 is the latest). [...]
So just to clarify, your concern is that because you've tried to install newer msgpack, pip is selecting an older version of fluent-logger which doesn't declare an incompatibility with that newer version of msgpack. This seems technically correct. I'm willing to bet if you insisted on installing fluent-logger>0.9.5 you would get the behavior you're expecting.
The underlying problem is that the package ecosystem has long based dependency versioning choices on side effect behaviors of pip's (lack of coherent) dep resolution. From the user side of things, if you want to install more than one package explicitly, you need to start specifying how new you want those packages to be.
However surprising it is, pip seems to be working as intended here.
Yes, it does! See my addendum as well. I will recap once more that I am not saying pip is doing anything wrong. Just BEWARE because you are most likely used to a different behaviour, just like me. Trying to use two conflicting constraints will make pip ERROR out and this is great now. I like new pip for this reason. But, as you mention, the ecosystem is not prepared. -yoctozepto
participants (3)
-
Andrii Ostapenko
-
Jeremy Stanley
-
Radosław Piliszek