[infra] Issue with the new pip
Hi! It seems that the new pip is breaking on infra's package indices. Several jobs crash with: """ ValueError: HTML doctype missing or incorrect. Expected <!DOCTYPE html>. If you believe this error to be incorrect, try passing the command line option --use-deprecated=html5lib and please leave a comment on the pip issue at https://github.com/pypa/pip/issues/10825. [0m [31m """ Example from kolla: https://zuul.opendev.org/t/openstack/build/e1bab19dd7c1487e93afeb67284a695e Example from manila: https://zuul.opendev.org/t/openstack/build/984ed9f10f0845ceba59b573e25bbb87 -yoctozepto
On 2022-01-30 20:01:27 +0100 (+0100), Radosław Piliszek wrote:
It seems that the new pip is breaking on infra's package indices. [...]
Yes, that's due to the pip 22.0 release earlier today. The main issue we're dealing with is https://github.com/pypa/pip/issues/10825 and have some changes up (one of which has already merged) under topic:pip-22 to hopefully address it. Unfortunately change 826969 isn't going to take effect until the wheel cache indices are regenerated by the openstack/requirements periodic pipeline jobs. -- Jeremy Stanley
On 2022-01-30 19:07:45 +0000 (+0000), Jeremy Stanley wrote:
On 2022-01-30 20:01:27 +0100 (+0100), Radosław Piliszek wrote:
It seems that the new pip is breaking on infra's package indices. [...]
Yes, that's due to the pip 22.0 release earlier today. The main issue we're dealing with is https://github.com/pypa/pip/issues/10825 and have some changes up (one of which has already merged) under topic:pip-22 to hopefully address it. Unfortunately change 826969 isn't going to take effect until the wheel cache indices are regenerated by the openstack/requirements periodic pipeline jobs.
We've merged a couple of changes now to fix up our wheel cache indices to be tolerable to pip 22.0's new HTML5 parser, performing some manual deployment steps to speed things along so we wouldn't need to wait for periodic job runs tomorrow, and so any failures related to that (with tracebacks from pip complaining about a missing DOCTYPE declaration) can be safely rechecked. Also pip 22.0.2 was released mere moments ago adding a fallback to the old HTML parser if the new one encounters errors. Be forewarned, this may only be the tip of the iceberg. For example I've already had to fix some of OpenDev's service tests which ran on Python 3.6 (because it's the default python3 for Ubuntu 20.04 LTS) to use a version-specific get-pip.py URL, because pip 22.0 also dropped support for Python 3.6. -- Jeremy Stanley
Thanks, fungi, for handling this. :-) -yoctozepto On Mon, 31 Jan 2022 at 00:21, Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2022-01-30 19:07:45 +0000 (+0000), Jeremy Stanley wrote:
On 2022-01-30 20:01:27 +0100 (+0100), Radosław Piliszek wrote:
It seems that the new pip is breaking on infra's package indices. [...]
Yes, that's due to the pip 22.0 release earlier today. The main issue we're dealing with is https://github.com/pypa/pip/issues/10825 and have some changes up (one of which has already merged) under topic:pip-22 to hopefully address it. Unfortunately change 826969 isn't going to take effect until the wheel cache indices are regenerated by the openstack/requirements periodic pipeline jobs.
We've merged a couple of changes now to fix up our wheel cache indices to be tolerable to pip 22.0's new HTML5 parser, performing some manual deployment steps to speed things along so we wouldn't need to wait for periodic job runs tomorrow, and so any failures related to that (with tracebacks from pip complaining about a missing DOCTYPE declaration) can be safely rechecked. Also pip 22.0.2 was released mere moments ago adding a fallback to the old HTML parser if the new one encounters errors.
Be forewarned, this may only be the tip of the iceberg. For example I've already had to fix some of OpenDev's service tests which ran on Python 3.6 (because it's the default python3 for Ubuntu 20.04 LTS) to use a version-specific get-pip.py URL, because pip 22.0 also dropped support for Python 3.6. -- Jeremy Stanley
On Sun, Jan 30, 2022 at 8:09 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
Yes, that's due to the pip 22.0 release earlier today. The main issue we're dealing with is https://github.com/pypa/pip/issues/10825 and have some changes up (one of which has already merged) under topic:pip-22 to hopefully address it. Unfortunately change 826969 isn't going to take effect until the wheel cache indices are regenerated by the openstack/requirements periodic pipeline jobs. I also have an error due to the new pip version on two jobs, the first one[1] in ussuri and the second one[2] in train. I saw a patch[3] was merged yesterday to support python 3.6 and use the right get-pip, I will try a recheck.
[1] https://zuul.opendev.org/t/openstack/build/a37704d48ef34401bdad0418ad19fda4 [2] https://zuul.opendev.org/t/openstack/build/4e3ee8a01b4e4d4a81db52beb34610b9 [3] https://review.opendev.org/c/opendev/system-config/+/826968/1/playbooks/role...
On 2022-02-01 12:22:06 +0100 (+0100), Daniel Mats Niklas Bengtsson wrote:
I also have an error due to the new pip version on two jobs, the first one[1] in ussuri and the second one[2] in train. I saw a patch[3] was merged yesterday to support python 3.6 and use the right get-pip, I will try a recheck.
[1] https://zuul.opendev.org/t/openstack/build/a37704d48ef34401bdad0418ad19fda4 [2] https://zuul.opendev.org/t/openstack/build/4e3ee8a01b4e4d4a81db52beb34610b9 [3] https://review.opendev.org/c/opendev/system-config/+/826968/1/playbooks/role...
The patch in system-config was for fixing some of the OpenDev Collaboratory's service deployment testing, so wouldn't be much help for the failures you linked. There is work underway to add a similar fix to DevStack: https://review.opendev.org/c/openstack/devstack/+/827155 That will probably have to be backported to earlier branches once it's working, but if you want to assist the QA team with reviewing/improving the fix I'm sure they would appreciate the help. -- Jeremy Stanley
On Tue, Feb 1, 2022 at 3:16 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
That will probably have to be backported to earlier branches once it's working, but if you want to assist the QA team with reviewing/improving the fix I'm sure they would appreciate the help. Yes I will check it. Thanks a lot.
participants (4)
-
Daniel Mats Niklas Bengtsson
-
Daniel Mats Niklas Bengtsson
-
Jeremy Stanley
-
Radosław Piliszek