[all][manila][ironic][swift][bandit] removal of Python standard libraries in Python 3.13
Hi, OpenStack Dalmatian is currently fully in Debian Testing, and in a quite good shape. However, as you may know, Python 3.13 is about to be uploaded to Debian unstable, and we intend to release Debian 13 (aka: Trixie) with it. A number of OpenStack (or related) packages are affected by the removal of Python standard libraries in Python 3.13. We need to fix these ASAP (and not only because of Debian). Here's the list of affected projects, with the name of the bad import that was removed from Python 3.13: manila https://bugs.debian.org/1084543 (pipes) swift https://bugs.debian.org/1084544 (cgi) ironic https://bugs.debian.org/1084566 (crypt) ironic-python-agent https://bugs.debian.org/1084680 (crypt) bandit https://bugs.debian.org/1084700 (crypt, telnetlib) There's also a number of libraries that we don't directly maintain, but that are in our requirements: qpid-proton https://bugs.debian.org/1084552 (cgi) python-wsme https://bugs.debian.org/1084594 (cgi) python-falcon https://bugs.debian.org/1084623 (cgi) python-pysaml2 https://bugs.debian.org/1084670 (cgi) These will need to be fixed. For more details with were the bad import was found, please dive into the individual bug reports. While pipes and crypt have direct replacement (subprocess and for example the pypi module bcrypt), there's a few comments here: https://docs.python.org/3.13/whatsnew/3.13.html While I will for sure attempt to fix some of the above, I cannot be held responsible for all of OpenStack code, and I'd appreciate a lot having some help. Unless I'm being pointed at fixes patches in Gerrit, I'll start filling bugs for the above once I'm done with other tasks (currently, I'm working on validation of OpenStack upgrades in Debian), and hopefully, get all in order before Trixie's freeze in early 2025. I know there's probably going to be more Python 3.13 fixes to come, and I expect some package to fail to build from source (FTBFS) due to some tests failing in 3.13. Then I'll run my usual tempest validation CI, and hopefully, Dalmatian on Trixie will be the best OpenStack ever... :) In the mean time, let's tackle the above. Cheers, Thomas Goirand (zigo)
On Tue, Oct 8, 2024 at 7:59 AM Thomas Goirand <zigo@debian.org> wrote:
Hi,
OpenStack Dalmatian is currently fully in Debian Testing, and in a quite good shape. However, as you may know, Python 3.13 is about to be uploaded to Debian unstable, and we intend to release Debian 13 (aka: Trixie) with it. A number of OpenStack (or related) packages are affected by the removal of Python standard libraries in Python 3.13. We need to fix these ASAP (and not only because of Debian).
Here's the list of affected projects, with the name of the bad import that was removed from Python 3.13:
manila https://bugs.debian.org/1084543 (pipes)
Thank you for sharing this, zigo This was addressed recently on trunk; backporting the change to stable/2024.2 here: https://review.opendev.org/c/openstack/manila/+/931804
swift https://bugs.debian.org/1084544 (cgi) ironic https://bugs.debian.org/1084566 (crypt) ironic-python-agent https://bugs.debian.org/1084680 (crypt) bandit https://bugs.debian.org/1084700 (crypt, telnetlib)
There's also a number of libraries that we don't directly maintain, but that are in our requirements:
qpid-proton https://bugs.debian.org/1084552 (cgi) python-wsme https://bugs.debian.org/1084594 (cgi) python-falcon https://bugs.debian.org/1084623 (cgi) python-pysaml2 https://bugs.debian.org/1084670 (cgi)
These will need to be fixed. For more details with were the bad import was found, please dive into the individual bug reports.
While pipes and crypt have direct replacement (subprocess and for example the pypi module bcrypt), there's a few comments here:
https://docs.python.org/3.13/whatsnew/3.13.html
While I will for sure attempt to fix some of the above, I cannot be held responsible for all of OpenStack code, and I'd appreciate a lot having some help. Unless I'm being pointed at fixes patches in Gerrit, I'll start filling bugs for the above once I'm done with other tasks (currently, I'm working on validation of OpenStack upgrades in Debian), and hopefully, get all in order before Trixie's freeze in early 2025.
I know there's probably going to be more Python 3.13 fixes to come, and I expect some package to fail to build from source (FTBFS) due to some tests failing in 3.13. Then I'll run my usual tempest validation CI, and hopefully, Dalmatian on Trixie will be the best OpenStack ever... :)
In the mean time, let's tackle the above.
Cheers,
Thomas Goirand (zigo)
On 10/8/24 17:16, Goutham Pacha Ravi wrote:
On Tue, Oct 8, 2024 at 7:59 AM Thomas Goirand <zigo@debian.org> wrote:
Hi,
OpenStack Dalmatian is currently fully in Debian Testing, and in a quite good shape. However, as you may know, Python 3.13 is about to be uploaded to Debian unstable, and we intend to release Debian 13 (aka: Trixie) with it. A number of OpenStack (or related) packages are affected by the removal of Python standard libraries in Python 3.13. We need to fix these ASAP (and not only because of Debian).
Here's the list of affected projects, with the name of the bad import that was removed from Python 3.13:
manila https://bugs.debian.org/1084543 (pipes)
Thank you for sharing this, zigo This was addressed recently on trunk; backporting the change to stable/2024.2 here: https://review.opendev.org/c/openstack/manila/+/931804
Thanks a lot. Patch added to the Manila Debian package and uploaded. Cheers, Thomas Goirand (zigo)
While pipes and crypt have direct replacement (subprocess and for example the pypi module bcrypt), there's a few comments here:
BEWARE: crypt is on the radar of Keystone team and one of the biggest issues with it is that one of the replacement candidates is passlib which is unmaintained and that itself depends on crypt. bcrypt is not a replacement for crypt, but rather an alternative hashing algorithm. We went way of getting rid of passlib in Keystone and got prepared to drop crypt support (deprecated at Pike) as such once we start testing for 3.13.
In the mean time, let's tackle the above.
Cheers,
Thomas Goirand (zigo)
Thanks for this overview Artem
Thanks for raising this! I've created a launchpad bug for Ironic and IPA to track the work [1] [1] https://bugs.launchpad.net/ironic/+bug/2083955 Em ter., 8 de out. de 2024 às 12:27, Artem Goncharov < artem.goncharov@gmail.com> escreveu:
While pipes and crypt have direct replacement (subprocess and for example the pypi module bcrypt), there's a few comments here:
BEWARE: crypt is on the radar of Keystone team and one of the biggest issues with it is that one of the replacement candidates is passlib which is unmaintained and that itself depends on crypt. bcrypt is not a replacement for crypt, but rather an alternative hashing algorithm. We went way of getting rid of passlib in Keystone and got prepared to drop crypt support (deprecated at Pike) as such once we start testing for 3.13.
In the mean time, let's tackle the above.
Cheers,
Thomas Goirand (zigo)
Thanks for this overview
Artem
-- *Att[]'s* *Iury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Ironic Core* *Senior Software Engineer at Red Hat Brazil* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory@gmail.com <iurygregory@gmail.com>*
It seems nova is also using the crypt module but is missing from the list https://github.com/openstack/nova/blob/master/nova/virt/disk/api.py#L639-L64... AS pointed by Artem, the serious problem with crypto module is that we don't have a good replacement now. passlib is abandoned and is known to be incompatible with latest bcrypt (which we had to pin in u-c). Probably we need to consider pulling https://pypi.org/project/crypt-r/ or importing the implementation to oslo.utils . On 10/9/24 01:13, Iury Gregory wrote:
Thanks for raising this!
I've created a launchpad bug for Ironic and IPA to track the work [1]
[1] https://bugs.launchpad.net/ironic/+bug/2083955 <https://bugs.launchpad.net/ironic/+bug/2083955>
Em ter., 8 de out. de 2024 às 12:27, Artem Goncharov <artem.goncharov@gmail.com <mailto:artem.goncharov@gmail.com>> escreveu:
> > While pipes and crypt have direct replacement (subprocess and for > example the pypi module bcrypt), there's a few comments here: > > https://docs.python.org/3.13/whatsnew/3.13.html <https://docs.python.org/3.13/whatsnew/3.13.html>
BEWARE: crypt is on the radar of Keystone team and one of the biggest issues with it is that one of the replacement candidates is passlib which is unmaintained and that itself depends on crypt. bcrypt is not a replacement for crypt, but rather an alternative hashing algorithm. We went way of getting rid of passlib in Keystone and got prepared to drop crypt support (deprecated at Pike) as such once we start testing for 3.13.
> > > In the mean time, let's tackle the above. > > Cheers, > > Thomas Goirand (zigo)
Thanks for this overview
Artem
-- /Att[]'s/ /Iury Gregory Melo Ferreira //MSc in Computer Science at UFCG / /Ironic Core/ //Senior Software Engineer at Red Hat Brazil// /Social/:https://www.linkedin.com/in/iurygregory <https://www.linkedin.com/in/iurygregory> /E-mail: iurygregory@gmail.com <mailto:iurygregory@gmail.com>/
On 2024-10-08 16:58:20 +0200 (+0200), Thomas Goirand wrote: [...]
bandit https://bugs.debian.org/1084700 (crypt, telnetlib) [...]
The bandit project is no longer part of OpenStack, and can be found at https://github.com/PyCQA/bandit instead. It's actively maintained and already testing with Python 3.13 for the past 7 months, so presumably has addressed these concerns and just needs a newer version pulled into Debian (sid/trixie have 1.7.8 from March but the latest upstream version is 1.7.10 from September). -- Jeremy Stanley
I have proposed a patch for wsme: https://review.opendev.org/c/x/wsme/+/931843 Given the cgi module is being removed, I think it is safe for wsme to stop parsing cgi.FieldStorage types. Michael On Tue, Oct 8, 2024 at 7:59 AM Thomas Goirand <zigo@debian.org> wrote:
Hi,
OpenStack Dalmatian is currently fully in Debian Testing, and in a quite good shape. However, as you may know, Python 3.13 is about to be uploaded to Debian unstable, and we intend to release Debian 13 (aka: Trixie) with it. A number of OpenStack (or related) packages are affected by the removal of Python standard libraries in Python 3.13. We need to fix these ASAP (and not only because of Debian).
Here's the list of affected projects, with the name of the bad import that was removed from Python 3.13:
manila https://bugs.debian.org/1084543 (pipes) swift https://bugs.debian.org/1084544 (cgi) ironic https://bugs.debian.org/1084566 (crypt) ironic-python-agent https://bugs.debian.org/1084680 (crypt) bandit https://bugs.debian.org/1084700 (crypt, telnetlib)
There's also a number of libraries that we don't directly maintain, but that are in our requirements:
qpid-proton https://bugs.debian.org/1084552 (cgi) python-wsme https://bugs.debian.org/1084594 (cgi) python-falcon https://bugs.debian.org/1084623 (cgi) python-pysaml2 https://bugs.debian.org/1084670 (cgi)
These will need to be fixed. For more details with were the bad import was found, please dive into the individual bug reports.
While pipes and crypt have direct replacement (subprocess and for example the pypi module bcrypt), there's a few comments here:
https://docs.python.org/3.13/whatsnew/3.13.html
While I will for sure attempt to fix some of the above, I cannot be held responsible for all of OpenStack code, and I'd appreciate a lot having some help. Unless I'm being pointed at fixes patches in Gerrit, I'll start filling bugs for the above once I'm done with other tasks (currently, I'm working on validation of OpenStack upgrades in Debian), and hopefully, get all in order before Trixie's freeze in early 2025.
I know there's probably going to be more Python 3.13 fixes to come, and I expect some package to fail to build from source (FTBFS) due to some tests failing in 3.13. Then I'll run my usual tempest validation CI, and hopefully, Dalmatian on Trixie will be the best OpenStack ever... :)
In the mean time, let's tackle the above.
Cheers,
Thomas Goirand (zigo)
participants (7)
-
Artem Goncharov
-
Goutham Pacha Ravi
-
Iury Gregory
-
Jeremy Stanley
-
Michael Johnson
-
Takashi Kajinami
-
Thomas Goirand