cryptography min version (non-rust) through 2024.1
Hi All,
As you probably know, recent versions of cryptography have hard dependencies on rust. Are there any community plans to continue supporting a minimum (non-rust) version of cryptography until a specific release?
The concern I have downstream in Ubuntu is that we need to continue being compatible with cryptography 3.4.8 through openstack 2024.1. This is because all releases through 2024.1 will be backported to the ubuntu 22.04 cloud archives which will use cryptography 3.4.8. Once we get to 2024.2, we will be backporting to 24.04 cloud archives, which will have the new rust-based versions of cryptography.
The current upper-constraint for cryptography is 38.0.2, but the various requirements.txt min versions are much lower (e.g. keystone has cryptography>=2.7). This is likely to lead to patches landing with features that are only in 38.0.2, so it will likely be difficult to enforce min version support. But perhaps a stance toward maintaining compatibility could be established.
Thoughts?
Corey
On Tue, 2023-03-07 at 11:19 -0500, Corey Bryant wrote:
Hi All,
As you probably know, recent versions of cryptography have hard dependencies on rust. Are there any community plans to continue supporting a minimum (non-rust) version of cryptography until a specific release?
i tought we had already raised the min above the version that required rust so not that i am aware of. cryptography>=2.7 is our curret stated minium but we have been testing with a much much newwer version for alont time since we do not test miniums anymore https://github.com/openstack/nova/commit/6caedfd97675940eb3cf07e2f019926dae4...
The concern I have downstream in Ubuntu is that we need to continue being compatible with cryptography 3.4.8 through openstack 2024.1. This is because all releases through 2024.1 will be backported to the ubuntu 22.04 cloud archives which will use cryptography 3.4.8. Once we get to 2024.2, we will be backporting to 24.04 cloud archives, which will have the new rust-based versions of cryptography.
The current upper-constraint for cryptography is 38.0.2, but the various requirements.txt min versions are much lower (e.g. keystone has cryptography>=2.7). This is likely to lead to patches landing with features that are only in 38.0.2, so it will likely be difficult to enforce min version support. But perhaps a stance toward maintaining compatibility could be established.
https://github.com/openstack/governance/blob/584e06b0c186d4355d1d51f2d6df96e... we decided to "Drop Lower Constraints Maintenance" relitivly recently while we have pti guidance for some lanagues rust is not one of them https://github.com/openstack/governance/tree/584e06b0c186d4355d1d51f2d6df96e... and its also not part of the tested runtims https://github.com/openstack/governance/blob/master/reference/runtimes/2023....
so i would proably try to avoid makign any commitment to continuting to supprot non rust based pycryptography release
Thoughts?
Corey
On 2023-03-07 11:19:26 -0500 (-0500), Corey Bryant wrote: [...]
The current upper-constraint for cryptography is 38.0.2, but the various requirements.txt min versions are much lower (e.g. keystone has cryptography>=2.7). This is likely to lead to patches landing with features that are only in 38.0.2, so it will likely be difficult to enforce min version support. But perhaps a stance toward maintaining compatibility could be established.
[...]
While introducing specific tests for this would not be trivial, maybe it's one of those situations where we try to avoid breaking compatibility with older versions and don't reject patches when people find that something has inadvertently started depending on a feature only available in the Rust-based builds?
On Tue, Mar 7, 2023 at 12:30 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2023-03-07 11:19:26 -0500 (-0500), Corey Bryant wrote: [...]
The current upper-constraint for cryptography is 38.0.2, but the various requirements.txt min versions are much lower (e.g. keystone has cryptography>=2.7). This is likely to lead to patches landing with features that are only in 38.0.2, so it will likely be difficult to enforce min version support. But perhaps a stance toward maintaining compatibility could be established.
[...]
While introducing specific tests for this would not be trivial, maybe it's one of those situations where we try to avoid breaking compatibility with older versions and don't reject patches when people find that something has inadvertently started depending on a feature only available in the Rust-based builds? -- Jeremy Stanley
I'd be okay with an approach like this. Would this need to be formally adopted by the TC?
Corey
The concern I have downstream in Ubuntu is that we need to continue being compatible with cryptography 3.4.8 through openstack 2024.1. This is because all releases through 2024.1 will be backported to the ubuntu 22.04 cloud archives which will use cryptography 3.4.8. Once we get to 2024.2, we will be backporting to 24.04 cloud archives, which will have the new rust-based versions of cryptography.
The current upper-constraint for cryptography is 38.0.2, but the various requirements.txt min versions are much lower (e.g. keystone has cryptography>=2.7). This is likely to lead to patches landing with features that are only in 38.0.2, so it will likely be difficult to enforce min version support. But perhaps a stance toward maintaining compatibility could be established.
What is the impetus needed for us to raise the lower-constraint? When do we decide we should do that, generally -- is it just ad-hoc, someone requests it, or is there a more involved process? We certainly don't dictate all versions of required compilers and such in our TC testing docs (although that is implied in distribution-platform) -- so I think there's another piece to it when talking about python dependencies.
I do not think it's wise to commit to supporting older versions of cryptography through 2024.1. In fact, you *must* have a cryptography release that is rust-enabled in order to get OpenSSL 3 support. Not to mention the memory safety benefits from using a rust version. I'm not saying we should force newer cryptography immediately; but it is reason enough to give me significant pause about answering a question about supporting it through two additional releases.
Thanks, Jay Faulkner Ironic PTL TC Member
On 2023-03-07 11:54:43 -0800 (-0800), Jay Faulkner wrote: [...]
I do not think it's wise to commit to supporting older versions of cryptography through 2024.1. In fact, you *must* have a cryptography release that is rust-enabled in order to get OpenSSL 3 support. Not to mention the memory safety benefits from using a rust version. I'm not saying we should force newer cryptography immediately; but it is reason enough to give me significant pause about answering a question about supporting it through two additional releases.
Well, the context here is not "supporting old versions of the PYCA/Cryptography library," it's rather "supporting downstream distributors who backport patches to their stable forks of PYCA/Cryptography." While it may sound like the same thing, there's a subtle difference. Obviously nobody should be running old versions of the library because they'll be missing critical security fixes, but there are stable distributions who take care of backporting security patches for their LTS versions and want newer OpenStack releases to still be usable there.
The PYCA/Cryptography library has been particularly challenging here, since it decided to go all-in on Rust which, while a very exciting and compelling language from a security perspective, is not exactly the most stabilized ecosystem yet and has seen a lot of churn over the past few years leading to Rust-based projects often being entirely unfit for inclusion in stable server distros due to continually requiring newer toolchain versions and replacing build systems.
This isn't just Ubuntu. The latest Debian stable release carries a python3-cryptography based on 3.3.2 from two years ago, older than what Corey's trying to support but still quite new from the perspective of an LTS server distribution. Rocky 9.1 (which I assume is the same as RHEL but I can never find where to look up RHEL package versions) is carrying a python3-cryptography based on 36.0.1 from 2021, so newer than what Corey is trying to support on Ubuntu but not by much (approximately 3 months).
The point is, we can't reasonably test with all these different versions of the library, and that's just one library out of hundreds we're depending on for that matter... but what we can do is say that if people find regressions due to us testing exclusively with newer features of these libraries than are available on platforms we expect our users to deploy on, we'll gladly accept patches to fix that situation.
I expect the TC is going to choose Ubuntu 22.04 LTS as a target platform for at least the OpenStack 2023.2 and 2024.1 coordinated releases, but almost certainly the 2024.2 coordinated release as well since Ubuntu 24.04 LTS won't be officially available before we start that development cycle. That means the first coordinated OpenStack release which would be able to effectively depend on features from a newer python3-cryptography package on Ubuntu is going to be 2025.1. Food for thought.
I expect the TC is going to choose Ubuntu 22.04 LTS as a target platform for at least the OpenStack 2023.2 and 2024.1 coordinated releases, but almost certainly the 2024.2 coordinated release as well since Ubuntu 24.04 LTS won't be officially available before we start that development cycle. That means the first coordinated OpenStack release which would be able to effectively depend on features from a newer python3-cryptography package on Ubuntu is going to be 2025.1. Food for thought.
To be explicit, that testing platform means we require that OpenStack and its dependencies are installable *in a virtualenv*, not using distro python. So while this is a small imitation for this case (we cannot use cryptography that might require newer tooling to build than that ubuntu LTS would), the real motivation behind holding onto a lower-constraint for a longer time is in partnership with stable distros that ship OpenStack, who we don't want to alienate by giving them extra work of using newer releases than they are ready to.
I am OK with this approach; but there is a trade-off: we may be leading OpenStack consumers who don't use distro packaging that it's OK to use an older cryptography which may not have security support. It'd be interesting if we could ensure backwards compatibility through testing while also ensuring that anything installed with pip gets a new enough version... but frankly, I don't know offhand what approach would be best to do that, and I don't have the time to pursue it myself so the status quo wins :D.
Thanks for talking this through, I like when we're explicit about the motivations for what we do!
-- Jay Faulkner Ironic PTL TC Member
On 2023-03-10 10:39:53 -0800 (-0800), Jay Faulkner wrote: [...]
To be explicit, that testing platform means we require that OpenStack and its dependencies are installable *in a virtualenv*, not using distro python.
Well, "sort of." Let's unpack those assertions:
1. "[we're testing] that OpenStack and its dependencies are installable in a virtualenv"
Here I think you mean specifically its Python library dependencies, OpenStack has lots of (in many cases versioned) dependencies on non-Python components of a system and that's a big part of what we're checking by testing on multiple platforms. Where Python libraries are concerned, some may include non-Python "binary" extensions so it's not purely Python in the Python dependencies either. We also install some things like Javascript libraries from places other than the distribution (e.g. NPM) on those platforms.
But we don't only test in virtualenvs/venvs either. Today at least, DevStack/Grenade jobs install a vast majority of those dependencies into the base system environment. That probably won't be the case in the future thanks to increasing adoption of the PEP 668 EXTERNALLY-MANAGED flag, but for now at least we're comingling pip-installed and distro-installed Python libraries in those jobs.
2. "[we're testing] OpenStack and its dependencies [...] not using distro python"
This is definitely not true, but is maybe not what you meant. We definitely test with the exact build of Python interpreter and stdlib that these platforms ship, backported fixes and all. That's also a major reason why we test on multiple platforms, so that we can know our software will work with the Python that's provided on those platforms, which is how many of our users will be running our software too.
So while this is a small imitation for this case (we cannot use cryptography that might require newer tooling to build than that ubuntu LTS would), the real motivation behind holding onto a lower-constraint for a longer time is in partnership with stable distros that ship OpenStack, who we don't want to alienate by giving them extra work of using newer releases than they are ready to.
Yes, more precisely these distros are not likely to backport an entire new Rust toolchain just so that new versions of OpenStack can be installed. Instead, they're going to patch the new versions of OpenStack so that they work with older libraries that don't require an entire new Rust toolchain, because that's the easier of the two options. If we can avoid requiring them to do either of those things, obviously it's even nicer for them. If they're going to do the second thing anyway, we could just say "hey send us those patches and we'll consider them bug fixes, because we want people to be able to easily use OpenStack on stable server platforms."
I am OK with this approach; but there is a trade-off: we may be leading OpenStack consumers who don't use distro packaging that it's OK to use an older cryptography which may not have security support.
I'd argue that we already do this, because our stable branches use frozen-in-time constraints lists specifying obsolete versions of dependencies which are not receiving upstream security support. The intent is that we only use those to test backported fixes without destabilizing our CI jobs, but some of our deployment projects even still build container images or deploy packages from PyPI based on what's in those frozen constraints lists, much to my dismay.
It'd be interesting if we could ensure backwards compatibility through testing while also ensuring that anything installed with pip gets a new enough version... but frankly, I don't know offhand what approach would be best to do that, and I don't have the time to pursue it myself so the status quo wins :D.
[...]
My preference would be to just keep testing with the latest version of PYCA/Cryptography from PyPI in our master branch jobs, and tell people who want to use OpenStack on stable server distributions that we'll accept bug reports and review their fixes if we start doing something that the python3-cryptography package on those distributions doesn't support. That's just one approach though, there are certainly other ways to go about it.
participants (4)
-
Corey Bryant
-
Jay Faulkner
-
Jeremy Stanley
-
Sean Mooney