gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8 [1]. As gmann has noted, doing so would mean nova would no longer be installable on Python 3.6 or 3.7 and there has been a small bit of back and forth on the pros and cons of this. I'm wondering what other people's thoughts on this are. Is this something we should be doing? Should we do it for libraries too or just services? Do we ever want to do this? Thoughts, please!
Stephen
[1] https://review.opendev.org/c/openstack/nova/+/819194/comment/72ecf24f_2bd292...
On Thu, 2021-11-25 at 18:13 +0000, Stephen Finucane wrote:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8 [1]. As gmann has noted, doing so would mean nova would no longer be installable on Python 3.6 or 3.7 and there has been a small bit of back and forth on the pros and cons of this. I'm wondering what other people's thoughts on this are. Is this something we should be doing? Should we do it for libraries too or just services? Do we ever want to do this? Thoughts, please!
so i was advocating for dropping supprot for python below 3.8 this cycle so i woudl be at least in favor of offilly not supprot less then 3.8
in terms of if wew shoudl add python_requries >= 3.8 to project im a littel less certin
i think loss of 3.6 support is effectigly goign to happen via uages of 3.7 and 3.8 only feature anyway.
there are a number of feature in python 3.8 vs 3.7 and 3.6 https://docs.python.org/3/whatsnew/3.8.html https://docs.python.org/3/whatsnew/3.7.html
there are also a number of thing that are removed in 3.9 and 3.10 https://docs.python.org/3/whatsnew/3.7.html#deprecated-python-modules-functi... https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals
https://docs.python.org/3/whatsnew/3.9.html#removed-in-python-39 https://docs.python.org/3/whatsnew/3.10.html#removed
i have not check all of those to determin if we will be impacted by those changes or not but if we look at the end of life of the various python releases 3.6 is nolonger supppred for security updates from (23 Dec 2021) https://endoflife.date/python so i think we shoudl at least increase to 3.7
i woudl prefer to update to 3.8 which would (14 Oct 2024)
that would mean we are testing with a python that is security supproted for the lifetime of the stable yoga branch and byond.
Stephen
[1] https://review.opendev.org/c/openstack/nova/+/819194/comment/72ecf24f_2bd292...
On Thu, Nov 25 2021 at 06:48:37 PM +0000, Sean Mooney smooney@redhat.com wrote:
On Thu, 2021-11-25 at 18:13 +0000, Stephen Finucane wrote:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8 [1]. As gmann has noted, doing so would mean nova would no longer be installable on Python 3.6 or 3.7 and there has been a small bit of back and forth on the pros and cons of this. I'm wondering what other people's thoughts on this are. Is this something we should be doing? Should we do it for libraries too or just services? Do we ever want to do this? Thoughts, please!
so i was advocating for dropping supprot for python below 3.8 this cycle so i woudl be at least in favor of offilly not supprot less then 3.8
in terms of if wew shoudl add python_requries >= 3.8 to project im a littel less certin
i think loss of 3.6 support is effectigly goign to happen via uages of 3.7 and 3.8 only feature anyway.
there are a number of feature in python 3.8 vs 3.7 and 3.6 https://docs.python.org/3/whatsnew/3.8.html https://docs.python.org/3/whatsnew/3.7.html
This is my view too. As soon as we turn off py36 testing somebody unintentionally can propose a patch that uses py38+ only language feature and the gate will not detect it and the reviewers might not detect it either. As soon as we land such a patch the module becomes unusable on system < py3.8 regardless of what we write in our setup.cfg.
Cheers, gibi
there are also a number of thing that are removed in 3.9 and 3.10 https://docs.python.org/3/whatsnew/3.7.html#deprecated-python-modules-functi... https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals
https://docs.python.org/3/whatsnew/3.9.html#removed-in-python-39 https://docs.python.org/3/whatsnew/3.10.html#removed
i have not check all of those to determin if we will be impacted by those changes or not but if we look at the end of life of the various python releases 3.6 is nolonger supppred for security updates from (23 Dec 2021) https://endoflife.date/python so i think we shoudl at least increase to 3.7
i woudl prefer to update to 3.8 which would (14 Oct 2024)
that would mean we are testing with a python that is security supproted for the lifetime of the stable yoga branch and byond.
Stephen
[1] https://review.opendev.org/c/openstack/nova/+/819194/comment/72ecf24f_2bd292...
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote ----
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
Just for some background, we started adding 'python_requires' in 'setup.cfg' when we dropped py2.7 and wanted a hard stop for anyone keep using py2.7 on OpenStack. But in python3 world we do not have to use it for every min version bump as such.
[1] https://governance.openstack.org/tc/reference/runtimes/yoga.html
-gmann
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote ----
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
gmann has been helpfully proposing patches to change the versions of
Python
we're testing against in Yoga. I've suggested that we might want to
bump
'python_requires' in 'setup.cfg' to indicate that we no longer
support any
version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there
will
be no distribution with Py 3.6 to care about?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
Just for some background, we started adding 'python_requires' in
'setup.cfg' when we dropped py2.7 and wanted a hard stop for anyone keep using py2.7 on OpenStack. But in python3 world we do not have to use it for every min version bump as such.
[1] https://governance.openstack.org/tc/reference/runtimes/yoga.html
-gmann
On Fri, Nov 26 2021 at 10:54:26 AM +0100, Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz marcin.juszkiewicz@linaro.org wrote ----
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
gmann has been helpfully proposing patches to change the
versions of Python
we're testing against in Yoga. I've suggested that we might
want to bump
'python_requires' in 'setup.cfg' to indicate that we no longer
support any
version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing
CS8 ->
CS9 migration during Yoga cycle. Can we postpone it to Z when
there will
be no distribution with Py 3.6 to care about?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
I think we can keep py36 support iff we test with py36 on the gate. Which I'm not against personally but there was a TC decision to drop py36 testing.
Cheers, gibi
Just for some background, we started adding 'python_requires' in 'setup.cfg' when we dropped py2.7 and wanted a hard stop for anyone keep using py2.7 on OpenStack. But in python3 world we do not have to use it for every min version bump as such.
[1] https://governance.openstack.org/tc/reference/runtimes/yoga.html
-gmann
On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote:
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote ----
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Just for some background, we started adding 'python_requires' in> 'setup.cfg' when we dropped py2.7 and wanted a hard stop for anyone keep using py2.7 on OpenStack. But in python3 world we do not have to use it for every min version bump as such.
[1] https://governance.openstack.org/tc/reference/runtimes/yoga.html
---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote:
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote ----
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
- Latest Ubuntu LTS - Latest CentOS Stream Major - Latest Debian Stable
[1] https://governance.openstack.org/tc/reference/project-testing-interface.html...
Just for some background, we started adding 'python_requires' in> 'setup.cfg' when we dropped py2.7 and wanted a hard stop for anyone keep using py2.7 on OpenStack. But in python3 world we do not have to use it for every min version bump as such.
[1] https://governance.openstack.org/tc/reference/runtimes/yoga.html
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On 26-11-21 09:37:44, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote:
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote ----
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Cool thanks!
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
I honestly don't know and can't find any docs to point to.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
I appreciate it bloats the support matrix a little but the rest of the thread suggests we need to keep py36 around for now anyway.
Cheers,
---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 09:37:44, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote:
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote ----
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze: > gmann has been helpfully proposing patches to change the > versions of Python we're testing against in Yoga. I've > suggested that we might want to bump 'python_requires' in > 'setup.cfg' to indicate that we no longer support any version > of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Cool thanks!
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
I honestly don't know and can't find any docs to point to.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
This is really good question on upgrade testing we do at upstream and I remember it cameup and discussed a lot during py2.7 drop also that how we are testing the upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did not tested directly but stein and train tested both version so should not be any issue if you upgrade from there (one of FAQ in my blog[1]).
But on distro upgrade testing, as you know we do not test those in upstream neither in grenade where upgrade are done on old node distro only not from old distro version to new distro version with new code. It is not like we do not want to test but if anyone from any distro would like to setup grenade for that and maintain then we are more happy. In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing due to resource bandwidth issue but we will welcome any help here.
[1] https://superuser.openstack.org/articles/openstack-ussuri-is-python3-only-up...
-gmann
I appreciate it bloats the support matrix a little but the rest of the thread suggests we need to keep py36 around for now anyway.
Cheers,
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
---- On Fri, 26 Nov 2021 10:18:16 -0600 Ghanshyam Mann gmann@ghanshyammann.com wrote ----
---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 09:37:44, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote:
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote ---- > W dniu 25.11.2021 o 19:13, Stephen Finucane pisze: > > gmann has been helpfully proposing patches to change the > > versions of Python we're testing against in Yoga. I've > > suggested that we might want to bump 'python_requires' in > > 'setup.cfg' to indicate that we no longer support any version > > of Python before 3.8 > > CentOS Stream 8 has Python 3.6 by default and RDO team is doing > CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z > when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Cool thanks!
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
I honestly don't know and can't find any docs to point to.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
This is really good question on upgrade testing we do at upstream and I remember it cameup and discussed a lot during py2.7 drop also that how we are testing the upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did not tested directly but stein and train tested both version so should not be any issue if you upgrade from there (one of FAQ in my blog[1]).
But on distro upgrade testing, as you know we do not test those in upstream neither in grenade where upgrade are done on old node distro only not from old distro version to new distro version with new code. It is not like we do not want to test but if anyone from any distro would like to setup grenade for that and maintain then we are more happy. In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing due to resource bandwidth issue but we will welcome any help here.
We discussed with amoralej about moving the testing runtime to CentOS stream 8 and py36 or not in TC IRC channel[1].
As we at upstream do not test distro two versions in same release, amoralej agreed to keep CentOS stream 9 if one to choose which is our current testing runtime is. So no change in the direction of current testing runtime and dropping the py3.6 but there is possibility of some trade off here. If any py3.6 breaking changes are happening then it is up to projects goodness, bandwidth, or flexibility about accepting the fix or not or even add a py36 unit test job. As our testing runtime is the minimum things to test and it does not put any max limit of testing, any project can extend their testing as per their bandwidth.
In summary:
(This is what we agreed today in TC channel but as most of the folks are on leave today, I will keep it open until next week so see if any objections from the community and will conclude it accordingly)
* No change in Yoga testing runtime and we move to cs9 and drop py36. * We will not put hard stop on cs8 support and we can: ** Devstack keep supporting cs8 in Yoga ** It can be negotiated with project to add py36 job or fix if any py36 breaking changes are observed by RDO (or any distro interested in py36) but it depends on the project decision and bandwidth.
As next, how we can improve the upgrade testing from distro versions is something we will explore next and see what all we can test to make upgrade easier.
[1] https://meetings.opendev.org/irclogs/%23openstack-tc/%23openstack-tc.2021-11...
-gmann
[1] https://superuser.openstack.org/articles/openstack-ussuri-is-python3-only-up...
-gmann
I appreciate it bloats the support matrix a little but the rest of the thread suggests we need to keep py36 around for now anyway.
Cheers,
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On 26-11-21 11:24:59, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 10:18:16 -0600 Ghanshyam Mann gmann@ghanshyammann.com wrote ----
---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 09:37:44, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote:
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
> ---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < > marcin.juszkiewicz@linaro.org> wrote ---- > > W dniu 25.11.2021 o 19:13, Stephen Finucane pisze: > > > gmann has been helpfully proposing patches to change the > > > versions of Python we're testing against in Yoga. I've > > > suggested that we might want to bump 'python_requires' in > > > 'setup.cfg' to indicate that we no longer support any version > > > of Python before 3.8 > > > > CentOS Stream 8 has Python 3.6 by default and RDO team is doing > > CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z > > when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
> Postponing to Z, you mean dropping the py3.6 tests or bumping it in > in 'setup.cfg' so that no one can install on py3.6 ? > > First one we already did and as per Yoga testing runtime we are > targeting centos9-stream[1] in Yoga itself. > > For making 'python_requires' >=py3.8 in 'setup.cfg', I have no > string opinion on this but I prefer to have flexible here that 'yes > OpenStack is installable in py3.6 but we do not test it anymore from > Yoga onwards so no guarantee'. Our testing runtime main goal is > that we document the version we are testing *at least* which means > it can work on lower or higher versions too but we just do not test > them. >
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Cool thanks!
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
I honestly don't know and can't find any docs to point to.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
This is really good question on upgrade testing we do at upstream and I remember it cameup and discussed a lot during py2.7 drop also that how we are testing the upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did not tested directly but stein and train tested both version so should not be any issue if you upgrade from there (one of FAQ in my blog[1]).
But on distro upgrade testing, as you know we do not test those in upstream neither in grenade where upgrade are done on old node distro only not from old distro version to new distro version with new code. It is not like we do not want to test but if anyone from any distro would like to setup grenade for that and maintain then we are more happy. In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing due to resource bandwidth issue but we will welcome any help here.
We discussed with amoralej about moving the testing runtime to CentOS stream 8 and py36 or not in TC IRC channel[1].
As we at upstream do not test distro two versions in same release, amoralej agreed to keep CentOS stream 9 if one to choose which is our current testing runtime is. So no change in the direction of current testing runtime and dropping the py3.6 but there is possibility of some trade off here. If any py3.6 breaking changes are happening then it is up to projects goodness, bandwidth, or flexibility about accepting the fix or not or even add a py36 unit test job. As our testing runtime is the minimum things to test and it does not put any max limit of testing, any project can extend their testing as per their bandwidth.
In summary:
(This is what we agreed today in TC channel but as most of the folks are on leave today, I will keep it open until next week so see if any objections from the community and will conclude it accordingly)
- No change in Yoga testing runtime and we move to cs9 and drop py36.
- We will not put hard stop on cs8 support and we can:
** Devstack keep supporting cs8 in Yoga ** It can be negotiated with project to add py36 job or fix if any py36 breaking changes are observed by RDO (or any distro interested in py36) but it depends on the project decision and bandwidth.
As next, how we can improve the upgrade testing from distro versions is something we will explore next and see what all we can test to make upgrade easier.
I'm against this, as I said in my setup.cfg >= py38 review for openstack/nova [1] we either list and support runtimes or don't. If RDO and others need CentOS 8 Stream support for a release then lets include it and py36 still for Yoga and make things explicit.
As I've said elsewhere I think the TC really need to adjust their thinking on this topic and allow for one OpenStack release where both the old and new LTS distro release are supported. Ensuring we allow people to actually upgrade in place and later handle the distro upgrade itself.
Cheers,
Lee
[1] https://review.opendev.org/c/openstack/nova/+/819415
Hello,
I’m strongly against dropping py36 support now, unless we’re going to find a solution that works on CentOS Stream 8. RHEL 9 is not out, and probably will not be in months - how do we expect users to use Yoga on production deployments (where they use CentOS Linux/equivalents today)?
Dropping the runtime testing and supporting devstack - and negotiating on a per project basis to support py36 or not - is not a solution. Either Yoga supports py36 as a transition point/release to py38 - or not.
In Kolla - we also did not anticipate (and don’t think it’s a good idea) to support CentOS Stream 9 in Yoga release. With the current decision - we are either forced with supporting CentOS Stream 9 (with no alternatives like Rocky Linux/Alma Linux in place - because RHEL 9 is not out) - or dropping CentOS support completely.
If we pursue CS9 - we also need to support migration from CS8 to CS9 and that’s also a considerable amount of work - which is unplanned.
Best regards, Michal
On 29 Nov 2021, at 10:17, Lee Yarwood lyarwood@redhat.com wrote:
On 26-11-21 11:24:59, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 10:18:16 -0600 Ghanshyam Mann gmann@ghanshyammann.com wrote ----
---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 09:37:44, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood lyarwood@redhat.com wrote ----
On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote: > On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com > wrote: > >> ---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < >> marcin.juszkiewicz@linaro.org> wrote ---- >>> W dniu 25.11.2021 o 19:13, Stephen Finucane pisze: >>>> gmann has been helpfully proposing patches to change the >>>> versions of Python we're testing against in Yoga. I've >>>> suggested that we might want to bump 'python_requires' in >>>> 'setup.cfg' to indicate that we no longer support any version >>>> of Python before 3.8 >>> >>> CentOS Stream 8 has Python 3.6 by default and RDO team is doing >>> CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z >>> when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
> As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and > CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS > version upgrades in the past providing support for both releases in an > OpenStack version to ease the upgrade so I'd like to keep yoga working on > py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
>> Postponing to Z, you mean dropping the py3.6 tests or bumping it in >> in 'setup.cfg' so that no one can install on py3.6 ? >> >> First one we already did and as per Yoga testing runtime we are >> targeting centos9-stream[1] in Yoga itself. >> >> For making 'python_requires' >=py3.8 in 'setup.cfg', I have no >> string opinion on this but I prefer to have flexible here that 'yes >> OpenStack is installable in py3.6 but we do not test it anymore from >> Yoga onwards so no guarantee'. Our testing runtime main goal is >> that we document the version we are testing *at least* which means >> it can work on lower or higher versions too but we just do not test >> them. >> > > May it be possible to keep py3.6 jobs to make sure patches are not > introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Cool thanks!
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
I honestly don't know and can't find any docs to point to.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
This is really good question on upgrade testing we do at upstream and I remember it cameup and discussed a lot during py2.7 drop also that how we are testing the upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did not tested directly but stein and train tested both version so should not be any issue if you upgrade from there (one of FAQ in my blog[1]).
But on distro upgrade testing, as you know we do not test those in upstream neither in grenade where upgrade are done on old node distro only not from old distro version to new distro version with new code. It is not like we do not want to test but if anyone from any distro would like to setup grenade for that and maintain then we are more happy. In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing due to resource bandwidth issue but we will welcome any help here.
We discussed with amoralej about moving the testing runtime to CentOS stream 8 and py36 or not in TC IRC channel[1].
As we at upstream do not test distro two versions in same release, amoralej agreed to keep CentOS stream 9 if one to choose which is our current testing runtime is. So no change in the direction of current testing runtime and dropping the py3.6 but there is possibility of some trade off here. If any py3.6 breaking changes are happening then it is up to projects goodness, bandwidth, or flexibility about accepting the fix or not or even add a py36 unit test job. As our testing runtime is the minimum things to test and it does not put any max limit of testing, any project can extend their testing as per their bandwidth.
In summary:
(This is what we agreed today in TC channel but as most of the folks are on leave today, I will keep it open until next week so see if any objections from the community and will conclude it accordingly)
- No change in Yoga testing runtime and we move to cs9 and drop py36.
- We will not put hard stop on cs8 support and we can:
** Devstack keep supporting cs8 in Yoga ** It can be negotiated with project to add py36 job or fix if any py36 breaking changes are observed by RDO (or any distro interested in py36) but it depends on the project decision and bandwidth.
As next, how we can improve the upgrade testing from distro versions is something we will explore next and see what all we can test to make upgrade easier.
I'm against this, as I said in my setup.cfg >= py38 review for openstack/nova [1] we either list and support runtimes or don't. If RDO and others need CentOS 8 Stream support for a release then lets include it and py36 still for Yoga and make things explicit.
As I've said elsewhere I think the TC really need to adjust their thinking on this topic and allow for one OpenStack release where both the old and new LTS distro release are supported. Ensuring we allow people to actually upgrade in place and later handle the distro upgrade itself.
Cheers,
Lee
[1] https://review.opendev.org/c/openstack/nova/+/819415 https://review.opendev.org/c/openstack/nova/+/819415
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
Hello, I agree with previous statement from Michal.
The upgrade path in for example RDO has been very smooth previously with upgrading to new OpenStack release then switching out the distro version afterwards because they support both during a transition.
Sure they will do that now as well, but if any project decides to break py36 there will be more work for the RDO team and more arguments to not revert changes because py38 would be the “real” supported runtime in OpenStack testing.
The transition period is required to not break the upgrade path.
Best regards Tobias
On 29 Nov 2021, at 11:14, Michał Nasiadka <mnasiadka@gmail.commailto:mnasiadka@gmail.com> wrote:
Hello,
I’m strongly against dropping py36 support now, unless we’re going to find a solution that works on CentOS Stream 8. RHEL 9 is not out, and probably will not be in months - how do we expect users to use Yoga on production deployments (where they use CentOS Linux/equivalents today)?
Dropping the runtime testing and supporting devstack - and negotiating on a per project basis to support py36 or not - is not a solution. Either Yoga supports py36 as a transition point/release to py38 - or not.
In Kolla - we also did not anticipate (and don’t think it’s a good idea) to support CentOS Stream 9 in Yoga release. With the current decision - we are either forced with supporting CentOS Stream 9 (with no alternatives like Rocky Linux/Alma Linux in place - because RHEL 9 is not out) - or dropping CentOS support completely.
If we pursue CS9 - we also need to support migration from CS8 to CS9 and that’s also a considerable amount of work - which is unplanned.
Best regards, Michal
On 29 Nov 2021, at 10:17, Lee Yarwood <lyarwood@redhat.commailto:lyarwood@redhat.com> wrote:
On 26-11-21 11:24:59, Ghanshyam Mann wrote: ---- On Fri, 26 Nov 2021 10:18:16 -0600 Ghanshyam Mann <gmann@ghanshyammann.commailto:gmann@ghanshyammann.com> wrote ---- ---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood <lyarwood@redhat.commailto:lyarwood@redhat.com> wrote ---- On 26-11-21 09:37:44, Ghanshyam Mann wrote: ---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood <lyarwood@redhat.commailto:lyarwood@redhat.com> wrote ---- On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote: On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann <gmann@ghanshyammann.commailto:gmann@ghanshyammann.com> wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.orgmailto:marcin.juszkiewicz@linaro.org> wrote ---- W dniu 25.11.2021 o 19:13, Stephen Finucane pisze: gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Cool thanks!
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
I honestly don't know and can't find any docs to point to.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
This is really good question on upgrade testing we do at upstream and I remember it cameup and discussed a lot during py2.7 drop also that how we are testing the upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did not tested directly but stein and train tested both version so should not be any issue if you upgrade from there (one of FAQ in my blog[1]).
But on distro upgrade testing, as you know we do not test those in upstream neither in grenade where upgrade are done on old node distro only not from old distro version to new distro version with new code. It is not like we do not want to test but if anyone from any distro would like to setup grenade for that and maintain then we are more happy. In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing due to resource bandwidth issue but we will welcome any help here.
We discussed with amoralej about moving the testing runtime to CentOS stream 8 and py36 or not in TC IRC channel[1].
As we at upstream do not test distro two versions in same release, amoralej agreed to keep CentOS stream 9 if one to choose which is our current testing runtime is. So no change in the direction of current testing runtime and dropping the py3.6 but there is possibility of some trade off here. If any py3.6 breaking changes are happening then it is up to projects goodness, bandwidth, or flexibility about accepting the fix or not or even add a py36 unit test job. As our testing runtime is the minimum things to test and it does not put any max limit of testing, any project can extend their testing as per their bandwidth.
In summary:
(This is what we agreed today in TC channel but as most of the folks are on leave today, I will keep it open until next week so see if any objections from the community and will conclude it accordingly)
* No change in Yoga testing runtime and we move to cs9 and drop py36. * We will not put hard stop on cs8 support and we can: ** Devstack keep supporting cs8 in Yoga ** It can be negotiated with project to add py36 job or fix if any py36 breaking changes are observed by RDO (or any distro interested in py36) but it depends on the project decision and bandwidth.
As next, how we can improve the upgrade testing from distro versions is something we will explore next and see what all we can test to make upgrade easier.
I'm against this, as I said in my setup.cfg >= py38 review for openstack/nova [1] we either list and support runtimes or don't. If RDO and others need CentOS 8 Stream support for a release then lets include it and py36 still for Yoga and make things explicit.
As I've said elsewhere I think the TC really need to adjust their thinking on this topic and allow for one OpenStack release where both the old and new LTS distro release are supported. Ensuring we allow people to actually upgrade in place and later handle the distro upgrade itself.
Cheers,
Lee
[1] https://review.opendev.org/c/openstack/nova/+/819415
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
---- On Mon, 29 Nov 2021 04:52:07 -0600 Tobias Urdin tobias.urdin@binero.com wrote ----
Hello,I agree with previous statement from Michal.
The upgrade path in for example RDO has been very smooth previously with upgrading to new OpenStack releasethen switching out the distro version afterwards because they support both during a transition. Sure they will do that now as well, but if any project decides to break py36 there will be more work for the RDO teamand more arguments to not revert changes because py38 would be the “real” supported runtime in OpenStack testing. The transition period is required to not break the upgrade path. Best regardsTobias On 29 Nov 2021, at 11:14, Michał Nasiadka mnasiadka@gmail.com wrote: Hello, I’m strongly against dropping py36 support now, unless we’re going to find a solution that works on CentOS Stream 8.RHEL 9 is not out, and probably will not be in months - how do we expect users to use Yoga on production deployments (where they use CentOS Linux/equivalents today)? Dropping the runtime testing and supporting devstack - and negotiating on a per project basis to support py36 or not - is not a solution.Either Yoga supports py36 as a transition point/release to py38 - or not. In Kolla - we also did not anticipate (and don’t think it’s a good idea) to support CentOS Stream 9 in Yoga release.With the current decision - we are either forced with supporting CentOS Stream 9 (with no alternatives like Rocky Linux/Alma Linux in place - because RHEL 9 is not out) - or dropping CentOS support completely. If we pursue CS9 - we also need to support migration from CS8 to CS9 and that’s also a considerable amount of work - which is unplanned. Best regards,Michal
I agree on all these points on keeping py36 for Yoga can be helpful for most people in migration to the new distro version. I have added it to the TC meeting agenda which is scheduled for Dec 2nd and discuss more on this. Please feel free to join TC meeting, details are below:
https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestio...
-gmann
On 29 Nov 2021, at 10:17, Lee Yarwood lyarwood@redhat.com wrote: On 26-11-21 11:24:59, Ghanshyam Mann wrote: ---- On Fri, 26 Nov 2021 10:18:16 -0600 Ghanshyam Mann gmann@ghanshyammann.com wrote ---- ---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood lyarwood@redhat.com wrote ---- On 26-11-21 09:37:44, Ghanshyam Mann wrote: ---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood lyarwood@redhat.com wrote ---- On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote: On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote ---- W dniu 25.11.2021 o 19:13, Stephen Finucane pisze: gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Cool thanks!
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
I honestly don't know and can't find any docs to point to.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
This is really good question on upgrade testing we do at upstream and I remember it cameup and discussed a lot during py2.7 drop also that how we are testing the upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did not tested directly but stein and train tested both version so should not be any issue if you upgrade from there (one of FAQ in my blog[1]).
But on distro upgrade testing, as you know we do not test those in upstream neither in grenade where upgrade are done on old node distro only not from old distro version to new distro version with new code. It is not like we do not want to test but if anyone from any distro would like to setup grenade for that and maintain then we are more happy. In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing due to resource bandwidth issue but we will welcome any help here.
We discussed with amoralej about moving the testing runtime to CentOS stream 8 and py36 or not in TC IRC channel[1].
As we at upstream do not test distro two versions in same release, amoralej agreed to keep CentOS stream 9 if one to choose which is our current testing runtime is. So no change in the direction of current testing runtime and dropping the py3.6 but there is possibility of some trade off here. If any py3.6 breaking changes are happening then it is up to projects goodness, bandwidth, or flexibility about accepting the fix or not or even add a py36 unit test job. As our testing runtime is the minimum things to test and it does not put any max limit of testing, any project can extend their testing as per their bandwidth.
In summary:
(This is what we agreed today in TC channel but as most of the folks are on leave today, I will keep it open until next week so see if any objections from the community and will conclude it accordingly)
- No change in Yoga testing runtime and we move to cs9 and drop py36.
- We will not put hard stop on cs8 support and we can:
** Devstack keep supporting cs8 in Yoga ** It can be negotiated with project to add py36 job or fix if any py36 breaking changes are observed by RDO (or any distro interested in py36) but it depends on the project decision and bandwidth.
As next, how we can improve the upgrade testing from distro versions is something we will explore next and see what all we can test to make upgrade easier.
I'm against this, as I said in my setup.cfg >= py38 review for openstack/nova [1] we either list and support runtimes or don't. If RDO and others need CentOS 8 Stream support for a release then lets include it and py36 still for Yoga and make things explicit.
As I've said elsewhere I think the TC really need to adjust their thinking on this topic and allow for one OpenStack release where both the old and new LTS distro release are supported. Ensuring we allow people to actually upgrade in place and later handle the distro upgrade itself.
Cheers,
Lee
[1] https://review.opendev.org/c/openstack/nova/+/819415
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
---- On Tue, 30 Nov 2021 13:47:02 -0600 Ghanshyam Mann gmann@ghanshyammann.com wrote ----
---- On Mon, 29 Nov 2021 04:52:07 -0600 Tobias Urdin tobias.urdin@binero.com wrote ----
Hello,I agree with previous statement from Michal.
The upgrade path in for example RDO has been very smooth previously with upgrading to new OpenStack releasethen switching out the distro version afterwards because they support both during a transition. Sure they will do that now as well, but if any project decides to break py36 there will be more work for the RDO teamand more arguments to not revert changes because py38 would be the “real” supported runtime in OpenStack testing. The transition period is required to not break the upgrade path. Best regardsTobias On 29 Nov 2021, at 11:14, Michał Nasiadka mnasiadka@gmail.com wrote: Hello, I’m strongly against dropping py36 support now, unless we’re going to find a solution that works on CentOS Stream 8.RHEL 9 is not out, and probably will not be in months - how do we expect users to use Yoga on production deployments (where they use CentOS Linux/equivalents today)? Dropping the runtime testing and supporting devstack - and negotiating on a per project basis to support py36 or not - is not a solution.Either Yoga supports py36 as a transition point/release to py38 - or not. In Kolla - we also did not anticipate (and don’t think it’s a good idea) to support CentOS Stream 9 in Yoga release.With the current decision - we are either forced with supporting CentOS Stream 9 (with no alternatives like Rocky Linux/Alma Linux in place - because RHEL 9 is not out) - or dropping CentOS support completely. If we pursue CS9 - we also need to support migration from CS8 to CS9 and that’s also a considerable amount of work - which is unplanned. Best regards,Michal
I agree on all these points on keeping py36 for Yoga can be helpful for most people in migration to the new distro version. I have added it to the TC meeting agenda which is scheduled for Dec 2nd and discuss more on this. Please feel free to join TC meeting, details are below:
https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestio...
The Technical Committee considered all the points made in this ML thread and discussed them in today's TC meeting[1]. Testing python 3.6 for one more cycle would not cost much. To have a smooth migration to new distro versions having different default python versions, we are ok to keep the python 3.6 testing for the Yoga cycle and keep CentOS Stream 8 and 9 both in runtime. This does not cost us much in terms of developer-power or infra resources.
Below are the versions we will target for Yoga:
Distro: * Ubuntu 20.04 * CentOS Stream 8 * CentOS Stream 9 * Debian 11
Python: (keeping only lower and highest versions in testing with the assumption that anything working in py3.6 and py3.9 will work in py3.7, py3.8 too) * Python 3.6 * Python 3.9 * We agree to add Python 3.10 as a non-voting job also.
I have pushed the proposal in governance: - https://review.opendev.org/c/openstack/governance/+/820195
as well as in job template: - https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/820286
Thanks to everyone for participating and providing more clarity/feedback from users/operator/distro-provide perspective.
[1] - https://meetings.opendev.org/meetings/tc/2021/tc.2021-12-02-15.02.log.html#l... - https://www.youtube.com/watch?v=PwGYse8j4UY
-gmann
-gmann
On 29 Nov 2021, at 10:17, Lee Yarwood lyarwood@redhat.com wrote: On 26-11-21 11:24:59, Ghanshyam Mann wrote: ---- On Fri, 26 Nov 2021 10:18:16 -0600 Ghanshyam Mann gmann@ghanshyammann.com wrote ---- ---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood lyarwood@redhat.com wrote ---- On 26-11-21 09:37:44, Ghanshyam Mann wrote: ---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood lyarwood@redhat.com wrote ---- On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote: On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz < marcin.juszkiewicz@linaro.org> wrote ---- W dniu 25.11.2021 o 19:13, Stephen Finucane pisze: gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8
CentOS Stream 8 has Python 3.6 by default and RDO team is doing CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z when there will be no distribution with Py 3.6 to care about?
Stupid question that I should know the answer to but does RDO really support RPM based installations anymore? IOW couldn't we just workaround this by providing CS8 py38 based containers during the upgrade?
As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS version upgrades in the past providing support for both releases in an OpenStack version to ease the upgrade so I'd like to keep yoga working on py3.6 included in CS8 and CS9.
If this was the plan why wasn't it made clear to the TC before they dropped CS8 from the Yoga runtimes? Would it even be possible for the TC to add CS8 and py36 back in to the Yoga runtimes?
Postponing to Z, you mean dropping the py3.6 tests or bumping it in in 'setup.cfg' so that no one can install on py3.6 ?
First one we already did and as per Yoga testing runtime we are targeting centos9-stream[1] in Yoga itself.
For making 'python_requires' >=py3.8 in 'setup.cfg', I have no string opinion on this but I prefer to have flexible here that 'yes OpenStack is installable in py3.6 but we do not test it anymore from Yoga onwards so no guarantee'. Our testing runtime main goal is that we document the version we are testing *at least* which means it can work on lower or higher versions too but we just do not test them.
May it be possible to keep py3.6 jobs to make sure patches are not introducing py3.8-only features that would break deployment in CS8?
We should keep CS8 and py36 as supported runtimes if we are keeping the jobs, otherwise this just sets super confusing.
Yeah, I think it create confusion as I can see in this ML thread so agree on keeping 'python_requires' also in sycn with what we test.
Cool thanks!
Now question on going back to centos stream 8 support in Yoga, is it not centos stream 9 is stable released or is it experimental only? If stable then we can keep the latest available version which can be centos stream 9.
I honestly don't know and can't find any docs to point to.
Our project interface testing doc clearly stats 'latest LTS' to consider for testing[1] whenever we are ready. I am not very strongly against of reverting back to centos stream 8 but we should not add two version of same distro in testing which can be a lot of we consider below three distro
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
This is really good question on upgrade testing we do at upstream and I remember it cameup and discussed a lot during py2.7 drop also that how we are testing the upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did not tested directly but stein and train tested both version so should not be any issue if you upgrade from there (one of FAQ in my blog[1]).
But on distro upgrade testing, as you know we do not test those in upstream neither in grenade where upgrade are done on old node distro only not from old distro version to new distro version with new code. It is not like we do not want to test but if anyone from any distro would like to setup grenade for that and maintain then we are more happy. In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing due to resource bandwidth issue but we will welcome any help here.
We discussed with amoralej about moving the testing runtime to CentOS stream 8 and py36 or not in TC IRC channel[1].
As we at upstream do not test distro two versions in same release, amoralej agreed to keep CentOS stream 9 if one to choose which is our current testing runtime is. So no change in the direction of current testing runtime and dropping the py3.6 but there is possibility of some trade off here. If any py3.6 breaking changes are happening then it is up to projects goodness, bandwidth, or flexibility about accepting the fix or not or even add a py36 unit test job. As our testing runtime is the minimum things to test and it does not put any max limit of testing, any project can extend their testing as per their bandwidth.
In summary:
(This is what we agreed today in TC channel but as most of the folks are on leave today, I will keep it open until next week so see if any objections from the community and will conclude it accordingly)
- No change in Yoga testing runtime and we move to cs9 and drop py36.
- We will not put hard stop on cs8 support and we can:
** Devstack keep supporting cs8 in Yoga ** It can be negotiated with project to add py36 job or fix if any py36 breaking changes are observed by RDO (or any distro interested in py36) but it depends on the project decision and bandwidth.
As next, how we can improve the upgrade testing from distro versions is something we will explore next and see what all we can test to make upgrade easier.
I'm against this, as I said in my setup.cfg >= py38 review for openstack/nova [1] we either list and support runtimes or don't. If RDO and others need CentOS 8 Stream support for a release then lets include it and py36 still for Yoga and make things explicit.
As I've said elsewhere I think the TC really need to adjust their thinking on this topic and allow for one OpenStack release where both the old and new LTS distro release are supported. Ensuring we allow people to actually upgrade in place and later handle the distro upgrade itself.
Cheers,
Lee
[1] https://review.opendev.org/c/openstack/nova/+/819415
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On 2021-11-26 16:05:15 +0000 (+0000), Lee Yarwood wrote: [...]
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
I appreciate it bloats the support matrix a little but the rest of the thread suggests we need to keep py36 around for now anyway.
Somehow we manage to get by with testing only one Ubuntu version per OpenStack release.
On Fri, Nov 26, 2021 at 5:26 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-11-26 16:05:15 +0000 (+0000), Lee Yarwood wrote: [...]
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
I appreciate it bloats the support matrix a little but the rest of the thread suggests we need to keep py36 around for now anyway.
Somehow we manage to get by with testing only one Ubuntu version per OpenStack release.
I'm quite sure there is always an overlap, otherwise grenade would not work.
Dmitry
-- Jeremy Stanley
On 2021-11-26 17:45:30 +0100 (+0100), Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 5:26 PM Jeremy Stanley fungi@yuggoth.org wrote:
[...]
Somehow we manage to get by with testing only one Ubuntu version per OpenStack release.
I'm quite sure there is always an overlap, otherwise grenade would not work.
Good point, I hadn't considered grenade. While we don't expressly mention the platform used for upgrade tests as part of the PTI, you are correct that we deploy release N-1 on the default node type for release N-1, then upgrade it in-place to release N and run a battery of tests against it. Technically we do test the new release on the old platform (at least for projects which tests their changes or are otherwise exercised by grenade), even though the PTI says we can drop support for the Python version on that platform.
If someone wants to work on running grenade on centos-8-stream nodes for yoga, we could ensure that's still a viable upgrade path the same way.
On 26-11-21 17:04:07, Jeremy Stanley wrote:
On 2021-11-26 17:45:30 +0100 (+0100), Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 5:26 PM Jeremy Stanley fungi@yuggoth.org wrote:
[...]
Somehow we manage to get by with testing only one Ubuntu version per OpenStack release.
I'm quite sure there is always an overlap, otherwise grenade would not work.
Good point, I hadn't considered grenade. While we don't expressly mention the platform used for upgrade tests as part of the PTI, you are correct that we deploy release N-1 on the default node type for release N-1, then upgrade it in-place to release N and run a battery of tests against it. Technically we do test the new release on the old platform (at least for projects which tests their changes or are otherwise exercised by grenade), even though the PTI says we can drop support for the Python version on that platform.
This is the reason I think we need an overlap release for distro LTS versions before we drop anything. Totally agree that we don't support the underlying distro upgrade but we clearly do support in place OpenStack upgrades and need the latest code to run on the older release.
If someone wants to work on running grenade on centos-8-stream nodes for yoga, we could ensure that's still a viable upgrade path the same way.
Yup happy to look into this today.
On Fri, 2021-11-26 at 17:45 +0100, Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 5:26 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-11-26 16:05:15 +0000 (+0000), Lee Yarwood wrote: [...]
How do we expect operators to upgrade between Xena where CentOS 8 stream is a supported runtime and Yoga where CentOS 9 stream is currently the equivalent supported runtime without supporting both for a single release?
I appreciate it bloats the support matrix a little but the rest of the thread suggests we need to keep py36 around for now anyway.
Somehow we manage to get by with testing only one Ubuntu version per OpenStack release.
I'm quite sure there is always an overlap, otherwise grenade would not work.
in the case fo centso the overlap is py39. centos 9 will have py39+ and there was nonvoting support for py 39 for a few release at this point so yoga,xena and even wallaby should be fucntional with py39.
grenade for the most point does nto test the underlying OS
Dmitry
-- Jeremy Stanley
On Mon, Nov 29, 2021 at 12:26 PM Sean Mooney smooney@redhat.com wrote:
On Fri, 2021-11-26 at 17:45 +0100, Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 5:26 PM Jeremy Stanley fungi@yuggoth.org
wrote:
On 2021-11-26 16:05:15 +0000 (+0000), Lee Yarwood wrote: [...]
How do we expect operators to upgrade between Xena where CentOS 8
stream
is a supported runtime and Yoga where CentOS 9 stream is currently
the
equivalent supported runtime without supporting both for a single release?
I appreciate it bloats the support matrix a little but the rest of
the
thread suggests we need to keep py36 around for now anyway.
Somehow we manage to get by with testing only one Ubuntu version per OpenStack release.
I'm quite sure there is always an overlap, otherwise grenade would not
work. in the case fo centso the overlap is py39. centos 9 will have py39+ and there was nonvoting support for py 39 for a few release at this point so yoga,xena and even wallaby should be fucntional with py39.
I don't think grenade switches Python mid-upgrade.
And as I explain somewhere else in this thread, the non-default Pythons in CentOS 8 are pretty rudimental and don't work for us (neither Bifrost nor Metal3).
Dmitry
grenade for the most point does nto test the underlying OS
Dmitry
-- Jeremy Stanley
Hi all,
Note that this decision will force us to stop supporting Bifrost [1] on CentOS/RHEL completely, unless we find a workaround. While Python 3.8 and 3.9 can be installed, they lack critical modules like python3-dnf or python3-firewalld, which cannot be pip-installed (sigh).
A similar problem in Metal3: we use python3-mod_wsgi, but I guess we can switch to something else in this case.
Dmitry
[1] An upstream installation service for Ironic based on Ansible
On Thu, Nov 25, 2021 at 7:19 PM Stephen Finucane stephenfin@redhat.com wrote:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8 [1]. As gmann has noted, doing so would mean nova would no longer be installable on Python 3.6 or 3.7 and there has been a small bit of back and forth on the pros and cons of this. I'm wondering what other people's thoughts on this are. Is this something we should be doing? Should we do it for libraries too or just services? Do we ever want to do this? Thoughts, please!
Stephen
[1] https://review.opendev.org/c/openstack/nova/+/819194/comment/72ecf24f_2bd292...
On Fri, Nov 26 2021 at 11:47:42 AM +0100, Dmitry Tantsur dtantsur@redhat.com wrote:
Hi all,
Note that this decision will force us to stop supporting Bifrost [1] on CentOS/RHEL completely, unless we find a workaround. While Python 3.8 and 3.9 can be installed, they lack critical modules like python3-dnf or python3-firewalld, which cannot be pip-installed (sigh).
A similar problem in Metal3: we use python3-mod_wsgi, but I guess we can switch to something else in this case.
I'm not sure I got it. Don't OpenStack already supports py38 officially? Based on my understanding of the above it is not the case.
Cheers, gibi
Dmitry
[1] An upstream installation service for Ironic based on Ansible
On Thu, Nov 25, 2021 at 7:19 PM Stephen Finucane stephenfin@redhat.com wrote:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8 [1]. As gmann has noted, doing so would mean nova would no longer be installable on Python 3.6 or 3.7 and there has been a small bit of back and forth on the pros and cons of this. I'm wondering what other people's thoughts on this are. Is this something we should be doing? Should we do it for libraries too or just services? Do we ever want to do this? Thoughts, please!
Stephen
[1] https://review.opendev.org/c/openstack/nova/+/819194/comment/72ecf24f_2bd292...
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Fri, Nov 26, 2021 at 1:28 PM Balazs Gibizer balazs.gibizer@est.tech wrote:
On Fri, Nov 26 2021 at 11:47:42 AM +0100, Dmitry Tantsur dtantsur@redhat.com wrote:
Hi all,
Note that this decision will force us to stop supporting Bifrost [1] on CentOS/RHEL completely, unless we find a workaround. While Python 3.8 and 3.9 can be installed, they lack critical modules like python3-dnf or python3-firewalld, which cannot be pip-installed (sigh).
A similar problem in Metal3: we use python3-mod_wsgi, but I guess we can switch to something else in this case.
I'm not sure I got it. Don't OpenStack already supports py38 officially? Based on my understanding of the above it is not the case.
Now I'm confused as well :)
OpenStack supports 3.8 and 3.9, CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9. Some Python projects may be okay with it, but Ansible requires things that cannot be installed unless provided by OS packages (or built from source). Examples include python3-dnf, python3-libselinux, python3-firewall and presumably python3-mod_wsgi.
Dmitry
Cheers, gibi
Dmitry
[1] An upstream installation service for Ironic based on Ansible
On Thu, Nov 25, 2021 at 7:19 PM Stephen Finucane stephenfin@redhat.com wrote:
gmann has been helpfully proposing patches to change the versions of Python we're testing against in Yoga. I've suggested that we might want to bump 'python_requires' in 'setup.cfg' to indicate that we no longer support any version of Python before 3.8 [1]. As gmann has noted, doing so would mean nova would no longer be installable on Python 3.6 or 3.7 and there has been a small bit of back and forth on the pros and cons of this. I'm wondering what other people's thoughts on this are. Is this something we should be doing? Should we do it for libraries too or just services? Do we ever want to do this? Thoughts, please!
Stephen
[1]
https://review.opendev.org/c/openstack/nova/+/819194/comment/72ecf24f_2bd292...
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Fri, 26 Nov 2021 at 14:31, Dmitry Tantsur dtantsur@redhat.com wrote:
On Fri, Nov 26, 2021 at 1:28 PM Balazs Gibizer balazs.gibizer@est.tech wrote:
On Fri, Nov 26 2021 at 11:47:42 AM +0100, Dmitry Tantsur dtantsur@redhat.com wrote:
Hi all,
Note that this decision will force us to stop supporting Bifrost [1] on CentOS/RHEL completely, unless we find a workaround. While Python 3.8 and 3.9 can be installed, they lack critical modules like python3-dnf or python3-firewalld, which cannot be pip-installed (sigh).
A similar problem in Metal3: we use python3-mod_wsgi, but I guess we can switch to something else in this case.
I'm not sure I got it. Don't OpenStack already supports py38 officially? Based on my understanding of the above it is not the case.
Now I'm confused as well :)
OpenStack supports 3.8 and 3.9, CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9. Some Python projects may be okay with it, but Ansible requires things that cannot be installed unless provided by OS packages (or built from source). Examples include python3-dnf, python3-libselinux, python3-firewall and presumably python3-mod_wsgi.
The question is: is it hard for RDO to provide these deps? If not, it might be the easiest solution. If yes, we (TC) might want to revisit this decision for this cycle.
-yoctozepto
On Fri, Nov 26, 2021 at 3:29 PM Radosław Piliszek < radoslaw.piliszek@gmail.com> wrote:
On Fri, 26 Nov 2021 at 14:31, Dmitry Tantsur dtantsur@redhat.com wrote:
On Fri, Nov 26, 2021 at 1:28 PM Balazs Gibizer balazs.gibizer@est.tech
wrote:
On Fri, Nov 26 2021 at 11:47:42 AM +0100, Dmitry Tantsur dtantsur@redhat.com wrote:
Hi all,
Note that this decision will force us to stop supporting Bifrost [1] on CentOS/RHEL completely, unless we find a workaround. While Python 3.8 and 3.9 can be installed, they lack critical modules like python3-dnf or python3-firewalld, which cannot be pip-installed (sigh).
A similar problem in Metal3: we use python3-mod_wsgi, but I guess we can switch to something else in this case.
I'm not sure I got it. Don't OpenStack already supports py38 officially? Based on my understanding of the above it is not the case.
Now I'm confused as well :)
OpenStack supports 3.8 and 3.9, CentOS/RHEL ships 3.6 and a limited
version of 3.8 and 3.9. Some Python projects may be okay with it, but Ansible requires things that cannot be installed unless provided by OS packages (or built from source). Examples include python3-dnf, python3-libselinux, python3-firewall and presumably python3-mod_wsgi.
The question is: is it hard for RDO to provide these deps? If not, it might be the easiest solution. If yes, we (TC) might want to revisit this decision for this cycle.
Yes, it's hard to provide those deps. The gap is big, in CS8 there are 281 python3- (py3.6) packages vs 36 python39 ones.
Some of them are built with non-python packages, as libselinux, firewalld, libvirt, etc... and would require to fork and fixing packages for coinstalability. Also, there are no warranties that all those versions of packages will work on py39 in CS8.
Alfredo
-yoctozepto
On Fri, 26 Nov 2021 at 18:38, Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Fri, Nov 26, 2021 at 3:29 PM Radosław Piliszek radoslaw.piliszek@gmail.com wrote:
On Fri, 26 Nov 2021 at 14:31, Dmitry Tantsur dtantsur@redhat.com wrote:
On Fri, Nov 26, 2021 at 1:28 PM Balazs Gibizer balazs.gibizer@est.tech wrote:
On Fri, Nov 26 2021 at 11:47:42 AM +0100, Dmitry Tantsur dtantsur@redhat.com wrote:
Hi all,
Note that this decision will force us to stop supporting Bifrost [1] on CentOS/RHEL completely, unless we find a workaround. While Python 3.8 and 3.9 can be installed, they lack critical modules like python3-dnf or python3-firewalld, which cannot be pip-installed (sigh).
A similar problem in Metal3: we use python3-mod_wsgi, but I guess we can switch to something else in this case.
I'm not sure I got it. Don't OpenStack already supports py38 officially? Based on my understanding of the above it is not the case.
Now I'm confused as well :)
OpenStack supports 3.8 and 3.9, CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9. Some Python projects may be okay with it, but Ansible requires things that cannot be installed unless provided by OS packages (or built from source). Examples include python3-dnf, python3-libselinux, python3-firewall and presumably python3-mod_wsgi.
The question is: is it hard for RDO to provide these deps? If not, it might be the easiest solution. If yes, we (TC) might want to revisit this decision for this cycle.
Yes, it's hard to provide those deps. The gap is big, in CS8 there are 281 python3- (py3.6) packages vs 36 python39 ones.
Some of them are built with non-python packages, as libselinux, firewalld, libvirt, etc... and would require to fork and fixing packages for coinstalability. Also, there are no warranties that all those versions of packages will work on py39 in CS8.
Thanks, Alfredo, for shedding more light on it.
I agree with the conclusions made by you and Ghanshyam (in the other branch of this very thread).
-yoctozepto
On Fri, 2021-11-26 at 14:30 +0000, Jeremy Stanley wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
centos 9 will ship 3.9 only at least as of now and rhel 9 will not support 3.6 or 3.8 gaing only 3.9 at least initally it might get 3.10 as a non default at some time
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
Dmitry
-- Jeremy Stanley
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
- https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
-gmann
Dmitry
Jeremy Stanley
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Fri, Nov 26, 2021 at 4:35 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur < dtantsur@redhat.com> wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org
wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us
(and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
I think there is an enormous perception gap between the CentOS team and the rest of the world.
Dmitry
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/runtimes/yoga.rst
-gmann
Dmitry
Jeremy Stanley
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael
O'Neill
On Fri, Nov 26, 2021 at 4:44 PM Dmitry Tantsur dtantsur@redhat.com wrote:
On Fri, Nov 26, 2021 at 4:35 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur < dtantsur@redhat.com> wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org
wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us
(and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
I think there is an enormous perception gap between the CentOS team and the rest of the world.
So, CentOS Stream 9 was released, in the official mirrors and usable since some weeks ago[1][2]. We shouldn't consider it beta or something like that.
As mentioned, support for diskimage-builder has been introduced for CS9 and there are nodepool nodes ready for it. From RDO, we are providing RPMs for master branch content on CentOS Stream 9 [3] and actually we have been doing some tests. Actually, we have recently merged new jobs in puppet-openstack[4].
[1] https://www.centos.org/centos-stream/ http://mirror.stream.centos.org/9-stream/ [2] https://cloud.centos.org/centos/9-stream/x86_64/images/ [3] https://trunk.rdoproject.org/centos9-master/report.html [4] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/793462
Alfredo
Dmitry
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/runtimes/yoga.rst
-gmann
Dmitry
Jeremy Stanley
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs,
Michael O'Neill
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Fri, Nov 26, 2021 at 6:14 PM Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Fri, Nov 26, 2021 at 4:44 PM Dmitry Tantsur dtantsur@redhat.com wrote:
On Fri, Nov 26, 2021 at 4:35 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur < dtantsur@redhat.com> wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org
wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for
us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
I think there is an enormous perception gap between the CentOS team and the rest of the world.
So, CentOS Stream 9 was released, in the official mirrors and usable since some weeks ago[1][2]. We shouldn't consider it beta or something like that.
As mentioned, support for diskimage-builder has been introduced for CS9 and there are nodepool nodes ready for it. From RDO, we are providing RPMs for master branch content on CentOS Stream 9 [3] and actually we have been doing some tests. Actually, we have recently merged new jobs in puppet-openstack[4].
"It's usable since some weeks ago and we even added tests today" is not exactly reassuring :) The PTI uses wording "stable and LTS", which applies to Stream 9 no more than it applies to Fedora.
In the end, what we test with Bifrost is what we will recommend people to deploy it in production on. I do believe people can and should deploy on Stream 8 despite all the FUD around it, but I cannot do it for Stream 9 until RHEL 9 is out.
Dmitry
[1] https://www.centos.org/centos-stream/ http://mirror.stream.centos.org/9-stream/ [2] https://cloud.centos.org/centos/9-stream/x86_64/images/ [3] https://trunk.rdoproject.org/centos9-master/report.html [4] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/793462
Alfredo
Dmitry
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/runtimes/yoga.rst
-gmann
Dmitry
Jeremy Stanley
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs,
Michael O'Neill
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Fri, 2021-11-26 at 19:48 +0100, Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 6:14 PM Alfredo Moralejo Alonso amoralej@redhat.com wrote:
On Fri, Nov 26, 2021 at 4:44 PM Dmitry Tantsur dtantsur@redhat.com wrote:
On Fri, Nov 26, 2021 at 4:35 PM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur < dtantsur@redhat.com> wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org
wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for
us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
I think there is an enormous perception gap between the CentOS team and the rest of the world.
So, CentOS Stream 9 was released, in the official mirrors and usable since some weeks ago[1][2]. We shouldn't consider it beta or something like that.
As mentioned, support for diskimage-builder has been introduced for CS9 and there are nodepool nodes ready for it. From RDO, we are providing RPMs for master branch content on CentOS Stream 9 [3] and actually we have been doing some tests. Actually, we have recently merged new jobs in puppet-openstack[4].
"It's usable since some weeks ago and we even added tests today" is not exactly reassuring :) The PTI uses wording "stable and LTS", which applies to Stream 9 no more than it applies to Fedora.
thats not quite true. yes centos 9 is a roling release but it is more stable then fedroa since packages landing in centos 9 stream have been stablised via fedroa already and any argurments in this regard would also apply to centos 8 stream.
The only reason that centos 8 stream would be more stable then 9 stream is due to less frequent updates as focus moves to 9 stream. 9 stream is effectivly a preview of what whill be rhel 9.
In the end, what we test with Bifrost is what we will recommend people to deploy it in production on. I do believe people can and should deploy on Stream 8 despite all the FUD around it, but I cannot do it for Stream 9 until RHEL 9 is out.
why just because rhel 9.0 is relased does not mean centos 9 is sudennly more stable. Now that centos 9 has been release there shoudl be no more package removalas form centos/rhel so it should have stableised in terms of the minium package set and over time we woudl expect more pacakges to be added. yes centos 8 stream will be supported until the EOL or rhel 8 so people can continue to deploy it. rhel 9 will be released next year, perhaps not before yoga is released but if you are deploying RDO you will not be useing RHEL anyway you will be using centos so stream 9 is the better plathform to use if you plan to continue to upgrade the deploy ment over the next few year as it allow you to avoid the costly OS upgrade when moving to the next openstack relesase.
Dmitry
[1] https://www.centos.org/centos-stream/ http://mirror.stream.centos.org/9-stream/ [2] https://cloud.centos.org/centos/9-stream/x86_64/images/ [3] https://trunk.rdoproject.org/centos9-master/report.html [4] https://review.opendev.org/c/openstack/puppet-openstack-integration/+/793462
Alfredo
Dmitry
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/runtimes/yoga.rst
-gmann
Dmitry
Jeremy Stanley
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs,
Michael O'Neill
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On Mon, Nov 29, 2021 at 12:38 PM Sean Mooney smooney@redhat.com wrote:
On Fri, 2021-11-26 at 19:48 +0100, Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 6:14 PM Alfredo Moralejo Alonso <
amoralej@redhat.com>
wrote:
On Fri, Nov 26, 2021 at 4:44 PM Dmitry Tantsur dtantsur@redhat.com wrote:
On Fri, Nov 26, 2021 at 4:35 PM Ghanshyam Mann <
gmann@ghanshyammann.com>
wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur < dtantsur@redhat.com> wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley <
fungi@yuggoth.org>
wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...] > CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9. [...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement
for
us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos
team
impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we
updated the
centos stream 8 -> 9 and dropped py3.6 testing
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
I think there is an enormous perception gap between the CentOS team
and
the rest of the world.
So, CentOS Stream 9 was released, in the official mirrors and usable
since
some weeks ago[1][2]. We shouldn't consider it beta or something like
that.
As mentioned, support for diskimage-builder has been introduced for CS9 and there are nodepool nodes ready for it. From RDO, we are providing
RPMs
for master branch content on CentOS Stream 9 [3] and actually we have
been
doing some tests. Actually, we have recently merged new jobs in puppet-openstack[4].
"It's usable since some weeks ago and we even added tests today" is not exactly reassuring :) The PTI uses wording "stable and LTS", which
applies
to Stream 9 no more than it applies to Fedora.
thats not quite true. yes centos 9 is a roling release but it is more stable then fedroa since packages landing in centos 9 stream have been stablised via fedroa already and any argurments in this regard would also apply to centos 8 stream.
Stream 8 is part of an already released and maintained RHEL, hence I give it a certain benefit of doubt. More on this below.
The only reason that centos 8 stream would be more stable then 9 stream is due to less frequent updates as focus moves to 9 stream. 9 stream is effectivly a preview of what whill be rhel 9.
In the end, what we test with Bifrost is what we will recommend people to deploy it in production on. I do believe people can and should deploy on Stream 8 despite all the FUD around it, but I cannot do it for Stream 9 until RHEL 9 is out.
why just because rhel 9.0 is relased does not mean centos 9 is sudennly more stable.
Okay, you convinced me, I won't recommend Stream 9 at all :)
Kidding aside, we know that each major branch of RHEL offers a certain degree of compatibility. It's not expected that 8.N+1 breaks a lot of stuff from 8.N, hence it's not expected that Stream between them will break anything (modulo bugs) either. I have no idea what and how gets into Stream 9 now, nor will I risk recommending it for production.
Dmitry
Now that centos 9 has been release there shoudl be no more package removalas form centos/rhel so it should have stableised in terms of the minium package set and over time we woudl expect more pacakges to be added. yes centos 8 stream will be supported until the EOL or rhel 8 so people can continue to deploy it. rhel 9 will be released next year, perhaps not before yoga is released but if you are deploying RDO you will not be useing RHEL anyway you will be using centos so stream 9 is the better plathform to use if you plan to continue to upgrade the deploy ment over the next few year as it allow you to avoid the costly OS upgrade when moving to the next openstack relesase.
Dmitry
[1] https://www.centos.org/centos-stream/ http://mirror.stream.centos.org/9-stream/ [2] https://cloud.centos.org/centos/9-stream/x86_64/images/ [3] https://trunk.rdoproject.org/centos9-master/report.html [4]
https://review.opendev.org/c/openstack/puppet-openstack-integration/+/793462
Alfredo
Dmitry
<
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
-gmann
Dmitry
Jeremy Stanley
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat:
Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs,
Michael O'Neill
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs,
Michael
O'Neill
On Mon, Nov 29, 2021 at 12:50 PM Dmitry Tantsur dtantsur@redhat.com wrote:
On Mon, Nov 29, 2021 at 12:38 PM Sean Mooney smooney@redhat.com wrote:
On Fri, 2021-11-26 at 19:48 +0100, Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 6:14 PM Alfredo Moralejo Alonso <
amoralej@redhat.com>
wrote:
On Fri, Nov 26, 2021 at 4:44 PM Dmitry Tantsur dtantsur@redhat.com wrote:
On Fri, Nov 26, 2021 at 4:35 PM Ghanshyam Mann <
gmann@ghanshyammann.com>
wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur < dtantsur@redhat.com> wrote ---- > > > On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley <
fungi@yuggoth.org>
wrote: > On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: > [...] > > CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9. > [...] > > Is this still true for CentOS Stream 9? The TC decision was to > support that instead of CentOS Stream 8 in Yoga. > > No. But Stream 9 is pretty much beta, so it's not a
replacement for
us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos
team
impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we
updated the
centos stream 8 -> 9 and dropped py3.6 testing
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
I think there is an enormous perception gap between the CentOS team
and
the rest of the world.
So, CentOS Stream 9 was released, in the official mirrors and usable
since
some weeks ago[1][2]. We shouldn't consider it beta or something like
that.
As mentioned, support for diskimage-builder has been introduced for
CS9
and there are nodepool nodes ready for it. From RDO, we are providing
RPMs
for master branch content on CentOS Stream 9 [3] and actually we have
been
doing some tests. Actually, we have recently merged new jobs in puppet-openstack[4].
"It's usable since some weeks ago and we even added tests today" is not exactly reassuring :) The PTI uses wording "stable and LTS", which
applies
to Stream 9 no more than it applies to Fedora.
thats not quite true. yes centos 9 is a roling release but it is more stable then fedroa since packages landing in centos 9 stream have been stablised via fedroa already and any argurments in this regard would also apply to centos 8 stream.
That's correct.
Also, the rolling release nature of CentOS Stream applies to a single major release where Red Hat is committed to provide a set of compatibility rules, upgrade support (ar packaging levels), abi and api compatibility etc... That's not the case for Fedora where is rolling update between major releases including major version rebases, etc...
Stream 8 is part of an already released and maintained RHEL, hence I give it a certain benefit of doubt. More on this below.
The only reason that centos 8 stream would be more stable then 9 stream is due to less frequent updates as focus moves to 9 stream. 9 stream is effectivly a preview of what whill be rhel 9.
In the end, what we test with Bifrost is what we will recommend people
to
deploy it in production on. I do believe people can and should deploy on Stream 8 despite all the FUD around it, but I cannot do it for Stream 9 until RHEL 9 is out.
why just because rhel 9.0 is relased does not mean centos 9 is sudennly more stable.
Okay, you convinced me, I won't recommend Stream 9 at all :)
Kidding aside, we know that each major branch of RHEL offers a certain degree of compatibility. It's not expected that 8.N+1 breaks a lot of stuff from 8.N, hence it's not expected that Stream between them will break anything (modulo bugs) either. I have no idea what and how gets into Stream 9 now, nor will I risk recommending it for production.
I don't understand the doubt here, could you elaborate?
The same compatibility rules applied to RHEL/CentOS Stream 8 are applied to CS/RHEL9. What is in CS9 will end up in RHEL9 at some later point of time. However, instead of pushing updates in minor releases are applied more frequently.
Said this, the concept of "recommended for production" is vague, depends on how organizations work and how they want to manage risks, and each one of us may have our own perception, of course.
Alfredo
Dmitry
Now that centos 9 has been release there shoudl be no more package removalas form centos/rhel so it should have stableised in terms of the minium package set and over time we woudl expect more pacakges to be added. yes centos 8 stream will be supported until the EOL or rhel 8 so people can continue to deploy it. rhel 9 will be released next year, perhaps not before yoga is released but if you are deploying RDO you will not be useing RHEL anyway you will be using centos so stream 9 is the better plathform to use if you plan to continue to upgrade the deploy ment over the next few year as it allow you to avoid the costly OS upgrade when moving to the next openstack relesase.
Dmitry
[1] https://www.centos.org/centos-stream/ http://mirror.stream.centos.org/9-stream/ [2] https://cloud.centos.org/centos/9-stream/x86_64/images/ [3] https://trunk.rdoproject.org/centos9-master/report.html [4]
https://review.opendev.org/c/openstack/puppet-openstack-integration/+/793462
Alfredo
Dmitry
<
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
-gmann
> Dmitry > -- > Jeremy Stanley > > > -- > Red Hat GmbH, https://de.redhat.com/ , Registered seat:
Grasbrunn,
> Commercial register: Amtsgericht Muenchen, HRB 153243, > Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill >
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs,
Michael
O'Neill
-- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
cheers, Zane.
It appears that Centos is no longer a viable platform for production Openstack clusters. Of course we have to continue supporting those who are not yet able to move to another distro, but I think we should be clear-eyed about the fact that these are stopgap measures that only need to be implemented while people work on moving.
Is anyone contemplating continuing to use Centos long-term? If so, I would be interested to hear your reasoning.
-----Original Message----- From: Zane Bitter zbitter@redhat.com Sent: Wednesday, December 1, 2021 10:13 AM To: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] Re: python_requires >= 3.8 during Yoga
Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
cheers, Zane.
On Wed, Dec 1, 2021 at 9:04 AM Braden, Albert abraden@verisign.com wrote:
It appears that Centos is no longer a viable platform for production Openstack clusters. Of course we have to continue supporting those who are not yet able to move to another distro, but I think we should be clear-eyed about the fact that these are stopgap measures that only need to be implemented while people work on moving.
I'm not certain why this is how it's being interpreted. CentOS stream is perfectly viable but it is (as has always been) reliant on packaging. I believe Cloud SIG will still exist and RDO will be publishing packages. In fact CentOS stream is generally better than the previous release cadence that caused all sort of issues when point releases dropped with breaking changes. The difference in stream vs legacy method was the volume of changes over time. Stream you get smaller changes sooner (along with fixes) where as the legacy method you'd only really get updates at point release time and it was always terrible troubleshooting these failures. CentOS Stream 9 will have 3.9 as the default but we need to make sure that we can get it available to the various projects for testing. Right now Puppet OpenStack primarily relies on CentOS for testing (because Ubuntu jobs have been broken for a few years now with no one jumping in to fix). So bumping the minimum version without having a way to deploy for us would directly impact this project.
IMHO the issue with this thread is timing on being able to bump a minimum python version requirement. I personally wouldn't want to bump to something that excludes 3.6 at this time without a really good reason. Are there features we require that will suddenly make developer/operator lives easier (e.g. switching to asyncio or something)? If not, what is the driving factor? I think pushing it off to Z would be best to allow for the initial standup required to have *all* the projects able to support the switch.
Is anyone contemplating continuing to use Centos long-term? If so, I would be interested to hear your reasoning.
-----Original Message----- From: Zane Bitter zbitter@redhat.com Sent: Wednesday, December 1, 2021 10:13 AM To: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] Re: python_requires >= 3.8 during Yoga
Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
cheers, Zane.
On Wed, 2021-12-01 at 09:13 -0700, Alex Schultz wrote:
On Wed, Dec 1, 2021 at 9:04 AM Braden, Albert abraden@verisign.com wrote:
It appears that Centos is no longer a viable platform for production Openstack clusters. Of course we have to continue supporting those who are not yet able to move to another distro, but I think we should be clear-eyed about the fact that these are stopgap measures that only need to be implemented while people work on moving.
I'm not certain why this is how it's being interpreted. CentOS stream is perfectly viable but it is (as has always been) reliant on packaging. I believe Cloud SIG will still exist and RDO will be publishing packages. In fact CentOS stream is generally better than the previous release cadence that caused all sort of issues when point releases dropped with breaking changes. The difference in stream vs legacy method was the volume of changes over time. Stream you get smaller changes sooner (along with fixes) where as the legacy method you'd only really get updates at point release time and it was always terrible troubleshooting these failures. CentOS Stream 9 will have 3.9 as the default but we need to make sure that we can get it available to the various projects for testing. Right now Puppet OpenStack primarily relies on CentOS for testing (because Ubuntu jobs have been broken for a few years now with no one jumping in to fix). So bumping the minimum version without having a way to deploy for us would directly impact this project.
at least internally earlier this year i reached out about rhel9/centos9 supprot specifcly related to py 3.9 support and eventlets. my understandwing was there was already some basic rhel9/centos9 ci happening the puppet moduels since eventlet supprot was tested using a puppet module job.
IMHO the issue with this thread is timing on being able to bump a minimum python version requirement. I personally wouldn't want to bump to something that excludes 3.6 at this time without a really good reason.
there is/was an expecation that rdo woudl backport support for cento 9 and py39 to stable wababy at some point since our downstream osp 17 product will be based on rhel 9 on stable/walaby.
im not directly involved in that packaging effort but i had assumed perhaps incorreectly that redhat would do that work upstream in rdo? is this not the case?
at least form a downstram perspective unless we cahnge direction and decided to build OSP 17 using ubi8 images instead fo ubi9 we will need to do all this packaging work anyway so im surprised there is this much push back to moving to centos 9 and py39.
Are there features we require that will suddenly make developer/operator lives easier (e.g. switching to asyncio or something)?
actuly they are a few nice libary enhanchments but that is not the main driving factor.
many do not want to support deploying the openstack project on python version we do not test. so either we keep 3.6 support and test it in every porject or we drop it and test with 3.8 and 3.9.
If not, what is the driving factor?
i have been pushing for 3.9 testing to become required for all project because centos/rhel 9 will only ship 3.9+ and it will be what our downstream product will be based on and i belive 3.9 is already used on debian 11 and 3.10 i belive is slated for ubuntu 22.04 which will be the next ubuntu lts and will supprot/ship openstack yoga.
so haveing at least 3.9 support in yoga will be imporatn for multiple distros and to help use test that i and proably others suggested raising the minium python version to 3.8 so that we only have to test with 2 version in the gate instead of 3.(3.6 3.8 and 3.9)
continuting to support py36 aslo puts a burden on developers not just on our ci infrastucre as it will becomre harder to test wtih going froward as it stops being shipped in distros and as it exits security support the end of life for python 3.6 is planned for this month https://www.python.org/dev/peps/pep-0494/#lifespan speficically (23 Dec 2021) based on https://endoflife.date/python
i dont think we shoudl be releasing yoga on an end of life python version. if we raise the minium to 3.8 that will be supported until at least 14 Oct 2024 which is after the stabel branch will be retired.
not support python 3.9 and 3.10 for yoga will make packaging openstack very hard for many ditros but since we cant realticaly test 4 verions properly we have to increase the minium verions at some point.
I think pushing it off to Z would be best to allow for the initial standup required to have *all* the projects able to support the switch.
all project where ment to start adding 3.9 support already it has been a non vovting test requiremtn for 2? cycles i belvie since we knew it was gong to be used in debian 11 and centos 9 for well over a year.
i do agree that its unfortunet that centos 9 missed it release target and did not release before the start of the yoga cycle but a better way to adress the upgrade issue i think woudl be to support xena/wallaby on centos 8 and 9 and make yoga 9 only. if we need to supprot yoga on centos 8 because that is not possibel then we coudl keep 3.6 support for one addtional release i guss but that wont be something that we just get for free and it will require use to reinstaet all 3.6 based ci and acnockolage that by the time yoga is released 3.6 will be EOL.
Is anyone contemplating continuing to use Centos long-term? If so, I would be interested to hear your reasoning.
-----Original Message----- From: Zane Bitter zbitter@redhat.com Sent: Wednesday, December 1, 2021 10:13 AM To: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] Re: python_requires >= 3.8 during Yoga
Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
cheers, Zane.
On Wed, Dec 1, 2021 at 10:05 AM Sean Mooney smooney@redhat.com wrote:
On Wed, 2021-12-01 at 09:13 -0700, Alex Schultz wrote:
On Wed, Dec 1, 2021 at 9:04 AM Braden, Albert abraden@verisign.com wrote:
It appears that Centos is no longer a viable platform for production Openstack clusters. Of course we have to continue supporting those who are not yet able to move to another distro, but I think we should be clear-eyed about the fact that these are stopgap measures that only need to be implemented while people work on moving.
I'm not certain why this is how it's being interpreted. CentOS stream is perfectly viable but it is (as has always been) reliant on packaging. I believe Cloud SIG will still exist and RDO will be publishing packages. In fact CentOS stream is generally better than the previous release cadence that caused all sort of issues when point releases dropped with breaking changes. The difference in stream vs legacy method was the volume of changes over time. Stream you get smaller changes sooner (along with fixes) where as the legacy method you'd only really get updates at point release time and it was always terrible troubleshooting these failures. CentOS Stream 9 will have 3.9 as the default but we need to make sure that we can get it available to the various projects for testing. Right now Puppet OpenStack primarily relies on CentOS for testing (because Ubuntu jobs have been broken for a few years now with no one jumping in to fix). So bumping the minimum version without having a way to deploy for us would directly impact this project.
at least internally earlier this year i reached out about rhel9/centos9 supprot specifcly related to py 3.9 support and eventlets. my understandwing was there was already some basic rhel9/centos9 ci happening the puppet moduels since eventlet supprot was tested using a puppet module job.
IMHO the issue with this thread is timing on being able to bump a minimum python version requirement. I personally wouldn't want to bump to something that excludes 3.6 at this time without a really good reason.
there is/was an expecation that rdo woudl backport support for cento 9 and py39 to stable wababy at some point since our downstream osp 17 product will be based on rhel 9 on stable/walaby.
im not directly involved in that packaging effort but i had assumed perhaps incorreectly that redhat would do that work upstream in rdo? is this not the case?
It is, however, because it's not done it is not OK to just do the switch. Really the issue is that you can't make a blanket switch until much of the bootstrapping work has been completed which is why doing the switch in Z makes more sense to allow for this.
at least form a downstram perspective unless we cahnge direction and decided to build OSP 17 using ubi8 images instead fo ubi9 we will need to do all this packaging work anyway so im surprised there is this much push back to moving to centos 9 and py39.
This is the same problem we had with the 7/8 switch where train was ultimately the thing where we had to backport support for py2.7/py3 problems. I would have really preferred to not do this again with 8/9 but here we are. IIRC, we managed to get Ussuri switched to centos8. Due to the delay in 9 availability we weren't able to get it ready in the Xena cycle. We do want to get it done in Yoga but because this thread is about making mandatory, we need to wait until the switch has been completed.
Are there features we require that will suddenly make developer/operator lives easier (e.g. switching to asyncio or something)?
actuly they are a few nice libary enhanchments but that is not the main driving factor.
many do not want to support deploying the openstack project on python version we do not test. so either we keep 3.6 support and test it in every porject or we drop it and test with 3.8 and 3.9.
If not, what is the driving factor?
i have been pushing for 3.9 testing to become required for all project because centos/rhel 9 will only ship 3.9+ and it will be what our downstream product will be based on and i belive 3.9 is already used on debian 11 and 3.10 i belive is slated for ubuntu 22.04 which will be the next ubuntu lts and will supprot/ship openstack yoga.
That's fine to push it as a goal, but I just think the goal needs to be delayed until Z. Is there anything that is broken in 3.9
so haveing at least 3.9 support in yoga will be imporatn for multiple distros and to help use test that i and proably others suggested raising the minium python version to 3.8 so that we only have to test with 2 version in the gate instead of 3.(3.6 3.8 and 3.9)
continuting to support py36 aslo puts a burden on developers not just on our ci infrastucre as it will becomre harder to test wtih going froward as it stops being shipped in distros and as it exits security support the end of life for python 3.6 is planned for this month https://www.python.org/dev/peps/pep-0494/#lifespan speficically (23 Dec 2021) based on https://endoflife.date/python
i dont think we shoudl be releasing yoga on an end of life python version. if we raise the minium to 3.8 that will be supported until at least 14 Oct 2024 which is after the stabel branch will be retired.
So I guess the question is who do we use for language support EOLs? My understanding is that Ubuntu 18.04's default was 3.6 which has standard support until 2023[0] and CentOS Stream 8 will be supported until 2024[1]. Both of which should exceed Yoga's support[2]? With all these arguments about LTS distros, it seems that we should be using their support life cyles for the language versions and not python's[3].
[0] https://wiki.ubuntu.com/Releases [1] https://www.centos.org/cl-vs-cs/ [2] https://releases.openstack.org/ [3] https://devguide.python.org/#status-of-python-branches
not support python 3.9 and 3.10 for yoga will make packaging openstack very hard for many ditros but since we cant realticaly test 4 verions properly we have to increase the minium verions at some point.
I think pushing it off to Z would be best to allow for the initial standup required to have *all* the projects able to support the switch.
all project where ment to start adding 3.9 support already it has been a non vovting test requiremtn for 2? cycles i belvie since we knew it was gong to be used in debian 11 and centos 9 for well over a year.
i do agree that its unfortunet that centos 9 missed it release target and did not release before the start of the yoga cycle but a better way to adress the upgrade issue i think woudl be to support xena/wallaby on centos 8 and 9 and make yoga 9 only. if we need to supprot yoga on centos 8 because that is not possibel then we coudl keep 3.6 support for one addtional release i guss but that wont be something that we just get for free and it will require use to reinstaet all 3.6 based ci and acnockolage that by the time yoga is released 3.6 will be EOL.
Is anyone contemplating continuing to use Centos long-term? If so, I would be interested to hear your reasoning.
-----Original Message----- From: Zane Bitter zbitter@redhat.com Sent: Wednesday, December 1, 2021 10:13 AM To: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] Re: python_requires >= 3.8 during Yoga
Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
cheers, Zane.
On Wed, Dec 1, 2021, at 9:31 AM, Alex Schultz wrote:
On Wed, Dec 1, 2021 at 10:05 AM Sean Mooney smooney@redhat.com wrote:
Apologies if my reduction of thread content was too aggressive.
so haveing at least 3.9 support in yoga will be imporatn for multiple distros and to help use test that i and proably others suggested raising the minium python version to 3.8 so that we only have to test with 2 version in the gate instead of 3.(3.6 3.8 and 3.9)
continuting to support py36 aslo puts a burden on developers not just on our ci infrastucre as it will becomre harder to test wtih going froward as it stops being shipped in distros and as it exits security support the end of life for python 3.6 is planned for this month https://www.python.org/dev/peps/pep-0494/#lifespan speficically (23 Dec 2021) based on https://endoflife.date/python
i dont think we shoudl be releasing yoga on an end of life python version. if we raise the minium to 3.8 that will be supported until at least 14 Oct 2024 which is after the stabel branch will be retired.
So I guess the question is who do we use for language support EOLs? My understanding is that Ubuntu 18.04's default was 3.6 which has standard support until 2023[0] and CentOS Stream 8 will be supported until 2024[1]. Both of which should exceed Yoga's support[2]? With all these arguments about LTS distros, it seems that we should be using their support life cyles for the language versions and not python's[3].
[0] https://wiki.ubuntu.com/Releases [1] https://www.centos.org/cl-vs-cs/ [2] https://releases.openstack.org/ [3] https://devguide.python.org/#status-of-python-branches
I'd like to make a proposal for moving forward. But first let's summarize some facts so that we can evaluate the proposal against the state of the world.
* Python 3.6 reaches EOL soon. * CentOS 9 Stream is released and makes no mention of being in a beta status (https://www.centos.org/centos-stream/) * We have the ability to run tests on CentOS 9 Stream today * It is useful to users to ensure we have an OpenStack release tested against old distro release with old python and new distro release with new python. This enables them to update underlying distro releases independently from upgrading OpenStack. * The various distros are not in sync over which python versions they support.
Given this state it seems reasonable that we continue to test Yoga against python3.6 and newer python. It would also probably be a good idea to explicitly update the PTI and distro support policies to indicate that when adding new distro releases we continue to support the old distro release's python for one cycle. The level of actual testing done on these platforms will depend on who steps up to maintain that testing, but at the very least we'll accept bug fixes for old python and not prevent installation via setup.cfg python_requires settings.
Yes, this means another cycle of python3.6 support which isn't ideal given it's soon EOL, but the intent is to support distros who will continue to support the older python on their platform.
If we want to be bold and try an experiment, we could also reduce testing of the python versions to oldest and newest supported python versions. For example python3.6 and python3.9 during Yoga. Don't worry about python3.7 and python3.8 as it is likely they will continue to function as long as 3.6 and 3.9 are happy. This would reduce the size of the testing matrix. We can always reevaluate if problems for 3.7 and 3.8 occur.
Finally, I think it is important to note that OpenStack is using CentOS as a stand in for RHEL. This worked a lot better when CentOS was an actual copy of RHEL. What we've got today is CentOS stream and it makes no claims about being in beta until RHEL releases (that I can find anyway), and I don't think we should make those assertions ourselves. If there is interest in using a different stand in (Alma or Rocky?) then we should replace CentOS stream with one of those alternatives. Another option may be to assert that Stream only functions as a standin once RHEL has released their corresponding update, but it is my understanding that Stream and RHEL will always have some delta.
Clark
)))
On Wed, Dec 1, 2021 at 11:09 AM Clark Boylan cboylan@sapwetik.org wrote:
On Wed, Dec 1, 2021, at 9:31 AM, Alex Schultz wrote:
On Wed, Dec 1, 2021 at 10:05 AM Sean Mooney smooney@redhat.com wrote:
Apologies if my reduction of thread content was too aggressive.
so haveing at least 3.9 support in yoga will be imporatn for multiple distros and to help use test that i and proably others suggested raising the minium python version to 3.8 so that we only have to test with 2 version in the gate instead of 3.(3.6 3.8 and 3.9)
continuting to support py36 aslo puts a burden on developers not just on our ci infrastucre as it will becomre harder to test wtih going froward as it stops being shipped in distros and as it exits security support the end of life for python 3.6 is planned for this month https://www.python.org/dev/peps/pep-0494/#lifespan speficically (23 Dec 2021) based on https://endoflife.date/python
i dont think we shoudl be releasing yoga on an end of life python version. if we raise the minium to 3.8 that will be supported until at least 14 Oct 2024 which is after the stabel branch will be retired.
So I guess the question is who do we use for language support EOLs? My understanding is that Ubuntu 18.04's default was 3.6 which has standard support until 2023[0] and CentOS Stream 8 will be supported until 2024[1]. Both of which should exceed Yoga's support[2]? With all these arguments about LTS distros, it seems that we should be using their support life cyles for the language versions and not python's[3].
[0] https://wiki.ubuntu.com/Releases [1] https://www.centos.org/cl-vs-cs/ [2] https://releases.openstack.org/ [3] https://devguide.python.org/#status-of-python-branches
For this reply, "LTS Distro" means support/updates > 13 months This means regular Ubuntu and Fedora releases don't qualify. Ubuntu LTS, CentOS Stream * and Debian LTS would qualify. Can we all agree on this definition?
https://ubuntu.com/about/release-cycle https://www.centos.org/cl-vs-cs/#end-of-life https://wiki.debian.org/LTS
I'd like to make a proposal for moving forward. But first let's summarize some facts so that we can evaluate the proposal against the state of the world.
- Python 3.6 reaches EOL soon.
- CentOS 9 Stream is released and makes no mention of being in a beta status (https://www.centos.org/centos-stream/)
- We have the ability to run tests on CentOS 9 Stream today
Yes some OpenStack projects can run on CentOS 9 Stream today but not all can (Puppet OpenStack being one). The assumption that pip install is sufficient for *all* projects is not correct. At least some testing needs to be able to be run on all projects before projects can officially declare <= 3.6 dead. This hopefully will be completed during Yoga but it takes time.
- It is useful to users to ensure we have an OpenStack release tested against old distro release with old python and new distro release with new python. This enables them to update underlying distro releases independently from upgrading OpenStack.
- The various distros are not in sync over which python versions they support.
Given this state it seems reasonable that we continue to test Yoga against python3.6 and newer python. It would also probably be a good idea to explicitly update the PTI and distro support policies to indicate that when adding new distro releases we continue to support the old distro release's python for one cycle. The level of actual testing done on these platforms will depend on who steps up to maintain that testing, but at the very least we'll accept bug fixes for old python and not prevent installation via setup.cfg python_requires settings.
Yes, this means another cycle of python3.6 support which isn't ideal given it's soon EOL, but the intent is to support distros who will continue to support the older python on their platform.
IMHO "another cycle" means continued support in Z not Yoga. I think the ask here was just delay the minimum version until Z so this would be the last cycle with 3.6 support. Jeremy said in a subsequent message that the interpreter comes from an LTS distro which IMHO means the EOL is the distro's EOL not Python's. Now this brings up issues if we can't get 3.6 fixes for specific versions of our dependencies that come in via pypi, but do we have data how often this is actually a problem? Isn't that already a problem for stable branches? Or are we worrying about something that doesn't happen? We need to all agree on where EOL dates come from for dependencies and expectations around them.
LTS Distros are being referenced in this thread but people keep going back to EOL dates that are not aligned with these distros. For the full OpenStack project ecosystem, testing is occurring on Ubuntu, CentOS, and a bit of Debian. IMHO when a release (e.g. Yoga) is released, we should use what is available at the start of cycle and using the EOL dates of the tested distros in determining decisions around version support. If we use this for Yoga, realistically CentOS Stream 8 and Ubuntu 20.04 could be used which would leave us with 3.6/3.8. It seems Debian was the only one with 3.7 (https://wiki.debian.org/Python). Since CentOS Stream 9 was released during the Yoga cycle, we couldn't pick it up as a version until Z but we could then make the 3.8 cutoff then. If Ubuntu releases a new version (the website says they might) during Yoga with 3.9 then we could talk about raising it further in Z. This would allow for lower bounds to be pushed, but requires that one of our qualifying LTS distros in check/gate were already available at the start of a cycle.
What I would like is that people stop pointing to Python's EOL date which is newer but does not actually align with the assumption of an LTS distro. It would only make sense to use that date if we were basing our testing on non-LTS distros which may be able to follow their EOL dates more closely. I keep hammering on the LTS distro because OpenStack is heavily reliant on packages and hardware supported by distros that it is very closely tied to what is available. If OpenStack had no tie to distro hardware support, venvs and pip installing all the things might be sufficient but there are many dependencies which end up not being satisfied this way. You can get away with it up to a point, but it's an important integration point.
If we want to be bold and try an experiment, we could also reduce testing of the python versions to oldest and newest supported python versions. For example python3.6 and python3.9 during Yoga. Don't worry about python3.7 and python3.8 as it is likely they will continue to function as long as 3.6 and 3.9 are happy. This would reduce the size of the testing matrix. We can always reevaluate if problems for 3.7 and 3.8 occur.
The issue here I think is trying to align on a default LTS distro python. CS8,18.04 is 3.6, Debian Buster is 3.7, 20.04 is 3.8 and CS9 will be 3.9. Personally I think 3.6/3.9 is sufficient because for the projects I contribute to this covers my cases. But this would exclude the 3.8 on 20.04. Would this be a problem for Ubuntu's UCA?
Finally, I think it is important to note that OpenStack is using CentOS as a stand in for RHEL. This worked a lot better when CentOS was an actual copy of RHEL. What we've got today is CentOS stream and it makes no claims about being in beta until RHEL releases (that I can find anyway), and I don't think we should make those assertions ourselves. If there is interest in using a different stand in (Alma or Rocky?) then we should replace CentOS stream with one of those alternatives. Another option may be to assert that Stream only functions as a standin once RHEL has released their corresponding update, but it is my understanding that Stream and RHEL will always have some delta.
There has always been a delta. The change now is the delta has been reversed such that we'll get fixes for things before RHEL but this is likely better for OpenStack's use cases where the previous meant long delays to get necessary fixes. I'm uncertain why the RHEL release schedule is being factored in to determine some indication of stability when testing is occurring (https://www.centos.org/cl-vs-cs/#testing).
Thanks, -Alex
Clark
On 2021-12-01 12:46:08 -0700 (-0700), Alex Schultz wrote: [...]
For this reply, "LTS Distro" means support/updates > 13 months This means regular Ubuntu and Fedora releases don't qualify. Ubuntu LTS, CentOS Stream * and Debian LTS would qualify. Can we all agree on this definition?
https://ubuntu.com/about/release-cycle https://www.centos.org/cl-vs-cs/#end-of-life https://wiki.debian.org/LTS
[...]
Technically, Debian on its own would meet this definition, as releases come every ~2.5 years and are generally supported for ~6 months after the next release, making it roughly 3 years of normal support. "LTS" in Debian is something which extends beyond the normal security support for the distribution, but is irrelevant here given the already long support timeframe for Debian releases.
Jeremy said in a subsequent message that the interpreter comes from an LTS distro which IMHO means the EOL is the distro's EOL not Python's. Now this brings up issues if we can't get 3.6 fixes for specific versions of our dependencies that come in via pypi, but do we have data how often this is actually a problem?
You can get some idea by counting the number of times "python_version" appears in global-requirements.txt or upper-constraints.txt in openstack/requirements by the end of a release cycle.
Isn't that already a problem for stable branches?
Not really, since we freeze openstack/requirements when branching, in a crude attempt to mimic testing with versions contemporary with what distros are probably packaging alongside that OpenStack release.
Or are we worrying about something that doesn't happen? We need to all agree on where EOL dates come from for dependencies and expectations around them.
It's definitely a thing which happens. We're running Python 3.6 jobs on OpenStack project master branches now while pinning to older versions of more than a dozen dependencies, and when mainline 3.6 EOL comes that will almost certainly rise sharply as our dependencies start to aggressively drop support for it. Having this happen during a cycle definitely leads to a wider disparity in versions of dependencies used for various jobs, compared to having it happen after we freeze requirements for the release.
LTS Distros are being referenced in this thread but people keep going back to EOL dates that are not aligned with these distros. For the full OpenStack project ecosystem, testing is occurring on Ubuntu, CentOS, and a bit of Debian. IMHO when a release (e.g. Yoga) is released, we should use what is available at the start of cycle and using the EOL dates of the tested distros in determining decisions around version support. If we use this for Yoga, realistically CentOS Stream 8 and Ubuntu 20.04 could be used which would leave us with 3.6/3.8. It seems Debian was the only one with 3.7 (https://wiki.debian.org/Python). Since CentOS Stream 9 was released during the Yoga cycle, we couldn't pick it up as a version until Z but we could then make the 3.8 cutoff then. If Ubuntu releases a new version (the website says they might) during Yoga with 3.9 then we could talk about raising it further in Z. This would allow for lower bounds to be pushed, but requires that one of our qualifying LTS distros in check/gate were already available at the start of a cycle.
Actually, Debian's last stable release (11 a.k.a. bullseye) was available approximately 2 months before we started on OpenStack Yoga and uses Python 3.9 as its default python3, so testing 3.6 and 3.9 would be the preferred lower and upper bounds there.
What I would like is that people stop pointing to Python's EOL date which is newer but does not actually align with the assumption of an LTS distro. It would only make sense to use that date if we were basing our testing on non-LTS distros which may be able to follow their EOL dates more closely.
[...]
The underlying reason it keeps coming up is that, during development, we don't attempt to match the versions of dependencies we test with to the versions those distributions are packaging alongside the Python 3 interpreter. As a result, when our dependencies begin dropping upstream support for Python releases is what impacts us, at least under our current testing model.
On Wed, Dec 1, 2021, at 11:46 AM, Alex Schultz wrote:
)))
On Wed, Dec 1, 2021 at 11:09 AM Clark Boylan cboylan@sapwetik.org wrote:
On Wed, Dec 1, 2021, at 9:31 AM, Alex Schultz wrote:
On Wed, Dec 1, 2021 at 10:05 AM Sean Mooney smooney@redhat.com wrote:
Apologies if my reduction of thread content was too aggressive.
so haveing at least 3.9 support in yoga will be imporatn for multiple distros and to help use test that i and proably others suggested raising the minium python version to 3.8 so that we only have to test with 2 version in the gate instead of 3.(3.6 3.8 and 3.9)
continuting to support py36 aslo puts a burden on developers not just on our ci infrastucre as it will becomre harder to test wtih going froward as it stops being shipped in distros and as it exits security support the end of life for python 3.6 is planned for this month https://www.python.org/dev/peps/pep-0494/#lifespan speficically (23 Dec 2021) based on https://endoflife.date/python
i dont think we shoudl be releasing yoga on an end of life python version. if we raise the minium to 3.8 that will be supported until at least 14 Oct 2024 which is after the stabel branch will be retired.
So I guess the question is who do we use for language support EOLs? My understanding is that Ubuntu 18.04's default was 3.6 which has standard support until 2023[0] and CentOS Stream 8 will be supported until 2024[1]. Both of which should exceed Yoga's support[2]? With all these arguments about LTS distros, it seems that we should be using their support life cyles for the language versions and not python's[3].
[0] https://wiki.ubuntu.com/Releases [1] https://www.centos.org/cl-vs-cs/ [2] https://releases.openstack.org/ [3] https://devguide.python.org/#status-of-python-branches
For this reply, "LTS Distro" means support/updates > 13 months This means regular Ubuntu and Fedora releases don't qualify. Ubuntu LTS, CentOS Stream * and Debian LTS would qualify. Can we all agree on this definition?
https://ubuntu.com/about/release-cycle https://www.centos.org/cl-vs-cs/#end-of-life https://wiki.debian.org/LTS
I'd like to make a proposal for moving forward. But first let's summarize some facts so that we can evaluate the proposal against the state of the world.
- Python 3.6 reaches EOL soon.
- CentOS 9 Stream is released and makes no mention of being in a beta status (https://www.centos.org/centos-stream/)
- We have the ability to run tests on CentOS 9 Stream today
Yes some OpenStack projects can run on CentOS 9 Stream today but not all can (Puppet OpenStack being one). The assumption that pip install is sufficient for *all* projects is not correct. At least some testing needs to be able to be run on all projects before projects can officially declare <= 3.6 dead. This hopefully will be completed during Yoga but it takes time.
- It is useful to users to ensure we have an OpenStack release tested against old distro release with old python and new distro release with new python. This enables them to update underlying distro releases independently from upgrading OpenStack.
- The various distros are not in sync over which python versions they support.
Given this state it seems reasonable that we continue to test Yoga against python3.6 and newer python. It would also probably be a good idea to explicitly update the PTI and distro support policies to indicate that when adding new distro releases we continue to support the old distro release's python for one cycle. The level of actual testing done on these platforms will depend on who steps up to maintain that testing, but at the very least we'll accept bug fixes for old python and not prevent installation via setup.cfg python_requires settings.
Yes, this means another cycle of python3.6 support which isn't ideal given it's soon EOL, but the intent is to support distros who will continue to support the older python on their platform.
IMHO "another cycle" means continued support in Z not Yoga. I think the ask here was just delay the minimum version until Z so this would be the last cycle with 3.6 support. Jeremy said in a subsequent message that the interpreter comes from an LTS distro which IMHO means the EOL is the distro's EOL not Python's. Now this brings up issues if we can't get 3.6 fixes for specific versions of our dependencies that come in via pypi, but do we have data how often this is actually a problem? Isn't that already a problem for stable branches? Or are we worrying about something that doesn't happen? We need to all agree on where EOL dates come from for dependencies and expectations around them.
The point I was trying to make was more that we should be explicit about when the overlap happens; write that down in our policy so that the next time this comes up the process goes more smoothly. I suppose if we have to delay for an additional cycle that may be the cost of not having considered this ahead of time.
I think Jeremy covered why the Python EOL is important. Our dependencies tend to start dropping like flies when python versions go EOL. I think we can survive for a time, but we shouldn't try to keep them alive for the long term.
LTS Distros are being referenced in this thread but people keep going back to EOL dates that are not aligned with these distros. For the full OpenStack project ecosystem, testing is occurring on Ubuntu, CentOS, and a bit of Debian. IMHO when a release (e.g. Yoga) is released, we should use what is available at the start of cycle and using the EOL dates of the tested distros in determining decisions around version support. If we use this for Yoga, realistically CentOS Stream 8 and Ubuntu 20.04 could be used which would leave us with 3.6/3.8. It seems Debian was the only one with 3.7 (https://wiki.debian.org/Python). Since CentOS Stream 9 was released during the Yoga cycle, we couldn't pick it up as a version until Z but we could then make the 3.8 cutoff then. If Ubuntu releases a new version (the website says they might) during Yoga with 3.9 then we could talk about raising it further in Z. This would allow for lower bounds to be pushed, but requires that one of our qualifying LTS distros in check/gate were already available at the start of a cycle.
What I would like is that people stop pointing to Python's EOL date which is newer but does not actually align with the assumption of an LTS distro. It would only make sense to use that date if we were basing our testing on non-LTS distros which may be able to follow their EOL dates more closely. I keep hammering on the LTS distro because OpenStack is heavily reliant on packages and hardware supported by distros that it is very closely tied to what is available. If OpenStack had no tie to distro hardware support, venvs and pip installing all the things might be sufficient but there are many dependencies which end up not being satisfied this way. You can get away with it up to a point, but it's an important integration point.
If we want to be bold and try an experiment, we could also reduce testing of the python versions to oldest and newest supported python versions. For example python3.6 and python3.9 during Yoga. Don't worry about python3.7 and python3.8 as it is likely they will continue to function as long as 3.6 and 3.9 are happy. This would reduce the size of the testing matrix. We can always reevaluate if problems for 3.7 and 3.8 occur.
The issue here I think is trying to align on a default LTS distro python. CS8,18.04 is 3.6, Debian Buster is 3.7, 20.04 is 3.8 and CS9 will be 3.9. Personally I think 3.6/3.9 is sufficient because for the projects I contribute to this covers my cases. But this would exclude the 3.8 on 20.04. Would this be a problem for Ubuntu's UCA?
Ubuntu 20.04 includes a python 3.9. I don't know how that would impact the UCA or Ubuntu's packaging though. That said I have a strong suspicion that if OpenStack runs on 3.6 and 3.9 that it should just work on 3.8 as well. And as I mentioned if we prove that wrong through experience we can always fix 3.8 and go back to explicitly testing it. I don't think this particular change is super critical, but I know some people were concerned about the number of pythons that would need testing.
Finally, I think it is important to note that OpenStack is using CentOS as a stand in for RHEL. This worked a lot better when CentOS was an actual copy of RHEL. What we've got today is CentOS stream and it makes no claims about being in beta until RHEL releases (that I can find anyway), and I don't think we should make those assertions ourselves. If there is interest in using a different stand in (Alma or Rocky?) then we should replace CentOS stream with one of those alternatives. Another option may be to assert that Stream only functions as a standin once RHEL has released their corresponding update, but it is my understanding that Stream and RHEL will always have some delta.
There has always been a delta. The change now is the delta has been reversed such that we'll get fixes for things before RHEL but this is likely better for OpenStack's use cases where the previous meant long delays to get necessary fixes. I'm uncertain why the RHEL release schedule is being factored in to determine some indication of stability when testing is occurring (https://www.centos.org/cl-vs-cs/#testing).
I'm explicitly saying don't compare to RHEL for stability. CentOS 9 Stream is released as far as I can tell. It isn't labeled a beta release. You can download and run it today. We run CentOS testing because OpenStack asserts that it supports RHEL and historically CentOS was the closest we could get to RHEL. I'm saying we might consider if continuing to use CentOS for that goal is still appropriate. If it is then lets just use 9 Stream today as it is released and ready to go. If it isn't appropriate because it isn't close enough to RHEL, then we should consider an alternative like Alma or Rocky or perhaps they will converge more once RHEL has released. Basically I'm saying lets get away from thinking about this in terms of "is stream ready/beta/etc" and think in terms of "does using stream accomplish the goals of testing to ensure RHEL compatibility".
Thanks, -Alex
Clark
On Wed, Dec 1, 2021 at 4:34 PM Clark Boylan cboylan@sapwetik.org wrote:
<snip>
If we want to be bold and try an experiment, we could also reduce
testing of the python versions to oldest and newest supported python versions. For example python3.6 and python3.9 during Yoga. Don't worry about python3.7 and python3.8 as it is likely they will continue to function as long as 3.6 and 3.9 are happy. This would reduce the size of the testing matrix. We can always reevaluate if problems for 3.7 and 3.8 occur.
The issue here I think is trying to align on a default LTS distro python. CS8,18.04 is 3.6, Debian Buster is 3.7, 20.04 is 3.8 and CS9 will be 3.9. Personally I think 3.6/3.9 is sufficient because for the projects I contribute to this covers my cases. But this would exclude the 3.8 on 20.04. Would this be a problem for Ubuntu's UCA?
Ubuntu 20.04 includes a python 3.9. I don't know how that would impact the UCA or Ubuntu's packaging though. That said I have a strong suspicion that if OpenStack runs on 3.6 and 3.9 that it should just work on 3.8 as well. And as I mentioned if we prove that wrong through experience we can always fix 3.8 and go back to explicitly testing it. I don't think this particular change is super critical, but I know some people were concerned about the number of pythons that would need testing.
Ubuntu will be supporting python3.8 (on 20.04) and python3.10 (on 22.04) for Yoga by default. Once python 3.10.1 is released, it will be backported to 20.04, so I was hoping to add non-voting py310 unit tests upstream. Tthoughts? We did that in the past, I forget what release.
Python3.9 testing is probably good enough coverage for Python3.8. Generally testing with the min and max python versions seems sensible, although the gap feels like it's getting wider than it was in the past, likely due to python release cadence.
Corey
On Thu, 2021-12-02 at 08:25 -0500, Corey Bryant wrote:
On Wed, Dec 1, 2021 at 4:34 PM Clark Boylan cboylan@sapwetik.org wrote:
<snip>
If we want to be bold and try an experiment, we could also reduce
testing of the python versions to oldest and newest supported python versions. For example python3.6 and python3.9 during Yoga. Don't worry about python3.7 and python3.8 as it is likely they will continue to function as long as 3.6 and 3.9 are happy. This would reduce the size of the testing matrix. We can always reevaluate if problems for 3.7 and 3.8 occur.
The issue here I think is trying to align on a default LTS distro python. CS8,18.04 is 3.6, Debian Buster is 3.7, 20.04 is 3.8 and CS9 will be 3.9. Personally I think 3.6/3.9 is sufficient because for the projects I contribute to this covers my cases. But this would exclude the 3.8 on 20.04. Would this be a problem for Ubuntu's UCA?
Ubuntu 20.04 includes a python 3.9. I don't know how that would impact the UCA or Ubuntu's packaging though. That said I have a strong suspicion that if OpenStack runs on 3.6 and 3.9 that it should just work on 3.8 as well. And as I mentioned if we prove that wrong through experience we can always fix 3.8 and go back to explicitly testing it. I don't think this particular change is super critical, but I know some people were concerned about the number of pythons that would need testing.
Ubuntu will be supporting python3.8 (on 20.04) and python3.10 (on 22.04) for Yoga by default. Once python 3.10.1 is released, it will be backported to 20.04, so I was hoping to add non-voting py310 unit tests upstream. Tthoughts? We did that in the past, I forget what release.
i think that likely would be a good idea. i know eventlets are currently adding inital 3.10 support and have had to adapt to some changes sicne py3.6 by increasing the min version of some of there deps like dns python getting a head up early and ensuring we can get our upper-constraits in order i think woudl help make that smother.
https://github.com/eventlet/eventlet/pull/715
Python3.9 testing is probably good enough coverage for Python3.8. Generally testing with the min and max python versions seems sensible, although the gap feels like it's getting wider than it was in the past, likely due to python release cadence.
ya it is wider then in the past for yoga we will effectlyve need to support 3.6, 3.7, 3.8, and 3.9 during development and then 3.10 once ubuntu 22.04 release.
long term supporting 5 python version i dont think is sutainable so if we can reduce that to 3.8 3.9 and 3.10 for Z i think that is what we should aim for. we might event want to aim for 3.9 and 3.10 only if the distro support matrix makes sense. suse is perhaps the only distro that may not supprot 3.9+ ? and they nolonger ship an openstack distro so perhaps we coudl limit to 3.9+ in z
Corey
---- On Thu, 02 Dec 2021 07:25:16 -0600 Corey Bryant corey.bryant@canonical.com wrote ----
On Wed, Dec 1, 2021 at 4:34 PM Clark Boylan cboylan@sapwetik.org wrote:
<snip> >> >> If we want to be bold and try an experiment, we could also reduce testing of the python versions to oldest and newest supported python versions. For example python3.6 and python3.9 during Yoga. Don't worry about python3.7 and python3.8 as it is likely they will continue to function as long as 3.6 and 3.9 are happy. This would reduce the size of the testing matrix. We can always reevaluate if problems for 3.7 and 3.8 occur. >> > > The issue here I think is trying to align on a default LTS distro > python. CS8,18.04 is 3.6, Debian Buster is 3.7, 20.04 is 3.8 and CS9 > will be 3.9. Personally I think 3.6/3.9 is sufficient because for the > projects I contribute to this covers my cases. But this would exclude > the 3.8 on 20.04. Would this be a problem for Ubuntu's UCA?
Ubuntu 20.04 includes a python 3.9. I don't know how that would impact the UCA or Ubuntu's packaging though. That said I have a strong suspicion that if OpenStack runs on 3.6 and 3.9 that it should just work on 3.8 as well. And as I mentioned if we prove that wrong through experience we can always fix 3.8 and go back to explicitly testing it. I don't think this particular change is super critical, but I know some people were concerned about the number of pythons that would need testing.
Ubuntu will be supporting python3.8 (on 20.04) and python3.10 (on 22.04) for Yoga by default. Once python 3.10.1 is released, it will be backported to 20.04, so I was hoping to add non-voting py310 unit tests upstream. Tthoughts? We did that in the past, I forget what release.
Yeah, adding 3.10 as non-voting is a good idea like we did for py3.9.
Anyways TC is going to discuss these in today's TC meeting, please join us there to conclude things.
-gmann
Python3.9 testing is probably good enough coverage for Python3.8. Generally testing with the min and max python versions seems sensible, although the gap feels like it's getting wider than it was in the past, likely due to python release cadence.
Corey
On 2021-12-02 08:25:16 -0500 (-0500), Corey Bryant wrote: [...]
Python3.9 testing is probably good enough coverage for Python3.8. Generally testing with the min and max python versions seems sensible, although the gap feels like it's getting wider than it was in the past, likely due to python release cadence.
It seems like we're primarily talking about unit tests here. There will still be a fair amount of direct 3.8 coverage as DevStack/Tempest/Grenade jobs use the default python3 on Focal unless we tell them not to. Same goes for docs builds, release jobs, et cetera.
On 2021-12-01 10:31:58 -0700 (-0700), Alex Schultz wrote: [...]
So I guess the question is who do we use for language support EOLs? My understanding is that Ubuntu 18.04's default was 3.6 which has standard support until 2023[0] and CentOS Stream 8 will be supported until 2024[1]. Both of which should exceed Yoga's support[2]? With all these arguments about LTS distros, it seems that we should be using their support life cyles for the language versions and not python's[3].
[...]
The answer is that it's not clear. We use the packaged Python 3 interpreter and standard library from specific LTS distros, but we consume additional Python 3 library dependencies from PyPI rather than from packages in those distros contemporary with the interpreters they've packaged. The result is that we're stuck relying on when those dependencies decide to drop support for what they consider to be "old" interpreter versions.
At one time we tried to rely on distro packaged dependencies, but discovered that breaks down as soon as projects want to use libraries which aren't yet packaged in those distributions. It's sort of Catch-22 in that the distros look to us to determine when they should be adding or updating dependencies, so if we look to them for which dependencies/versions are available then we can quickly get deadlocked.
Obviously Redhat will continue to support and use Openstack on Centos, and Redhat employees are a substantial part of the Openstack community. Without the contributions from Redhat employees, Openstack would struggle to survive. However, I don’t think that use by a single company is enough to justify the effort required to support a distro, even when that single company is a major contributor. I hope and believe that Redhat would continue their contributions to the Openstack community if we continued to support Openstack on RHEL and dropped support for Centos.
I'm curious whether anyone who does not work for Redhat is planning to continue using Openstack on Centos long-term, and if so, I'd like to hear your reasoning.
-----Original Message----- From: Alex Schultz aschultz@redhat.com Sent: Wednesday, December 1, 2021 11:13 AM To: Braden, Albert abraden@verisign.com Cc: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] Re: Re: python_requires >= 3.8 during Yoga
Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On Wed, Dec 1, 2021 at 9:04 AM Braden, Albert abraden@verisign.com wrote:
It appears that Centos is no longer a viable platform for production Openstack clusters. Of course we have to continue supporting those who are not yet able to move to another distro, but I think we should be clear-eyed about the fact that these are stopgap measures that only need to be implemented while people work on moving.
I'm not certain why this is how it's being interpreted. CentOS stream is perfectly viable but it is (as has always been) reliant on packaging. I believe Cloud SIG will still exist and RDO will be publishing packages. In fact CentOS stream is generally better than the previous release cadence that caused all sort of issues when point releases dropped with breaking changes. The difference in stream vs legacy method was the volume of changes over time. Stream you get smaller changes sooner (along with fixes) where as the legacy method you'd only really get updates at point release time and it was always terrible troubleshooting these failures. CentOS Stream 9 will have 3.9 as the default but we need to make sure that we can get it available to the various projects for testing. Right now Puppet OpenStack primarily relies on CentOS for testing (because Ubuntu jobs have been broken for a few years now with no one jumping in to fix). So bumping the minimum version without having a way to deploy for us would directly impact this project.
IMHO the issue with this thread is timing on being able to bump a minimum python version requirement. I personally wouldn't want to bump to something that excludes 3.6 at this time without a really good reason. Are there features we require that will suddenly make developer/operator lives easier (e.g. switching to asyncio or something)? If not, what is the driving factor? I think pushing it off to Z would be best to allow for the initial standup required to have *all* the projects able to support the switch.
Is anyone contemplating continuing to use Centos long-term? If so, I would be interested to hear your reasoning.
-----Original Message----- From: Zane Bitter zbitter@redhat.com Sent: Wednesday, December 1, 2021 10:13 AM To: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] Re: python_requires >= 3.8 during Yoga
Caution: This email originated from outside the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
cheers, Zane.
On 2021-12-01 18:06:47 +0000 (+0000), Braden, Albert wrote: [...]
I hope and believe that Redhat would continue their contributions to the Openstack community if we continued to support Openstack on RHEL and dropped support for Centos.
[...]
Just to clear up some misconceptions, what we mean when we say "support" (from the OpenStack community perspective) is that we test it upstream:
https://governance.openstack.org/tc/resolutions/20170620-volunteer-support.h...
In that sense, OpenStack does not officially "support" RHEL because we lack the ability to test it due to licensing challenges. We've asked on multiple occasions, and been told that running RHEL in arbitrary public cloud environments driven by automation is not feasible to license (due to a combination of legalities and logistics), and is still outside the scope of the "free for developer use" RHEL license program available today. This is why we traditionally tested on CentOS as a stand-in for RHEL, rather than using actual RHEL.
Informally we "support" RHEL in the sense that we'd like users to be able to run our software on the platform, but we rely on others to test that's possible since we can't guarantee it ourselves.
---- On Wed, 01 Dec 2021 09:13:02 -0600 Zane Bitter zbitter@redhat.com wrote ----
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
Those are good points. We were discussing with the RDO team on the start of Yoga cycle itself whether to move to CentOS Stream9 or not.
Yes, there is a clear confusion on what is stable or not in CentOS Stream distro. Like other distro has, is there any document we as TC can refer to in the future on checking if the CentOS Stream *X* version is stable/LTS or not? I cannot find it for CentOS Stream 8 also, is that stable or LTS?
If we have such official documentation or announcement in RDO community then we can avoid such situations in future where we get to know the distro version stability well before trying it in OpenStack testing?
- https://centos.org/download/
-gmann
cheers, Zane.
On Wed, Dec 1, 2021 at 9:30 AM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Wed, 01 Dec 2021 09:13:02 -0600 Zane Bitter zbitter@redhat.com wrote ----
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
Those are good points. We were discussing with the RDO team on the start of Yoga cycle itself whether to move to CentOS Stream9 or not.
Yes, there is a clear confusion on what is stable or not in CentOS Stream distro. Like other distro has, is there any document we as TC can refer to in the future on checking if the CentOS Stream *X* version is stable/LTS or not? I cannot find it for CentOS Stream 8 also, is that stable or LTS?
I guess what's the definition of LTS? Support > 1 year?
https://www.centos.org/cl-vs-cs/
CentOS Stream 8 is to be supported till 2024 and 9 is estimated until 2027. That seems pretty LTS to me.
Additionally what is the definition of stable in this case? My understanding is that there still won't be major version bumps to core software similar to how CentOS used to operate. So it'd be a similar stability as previously existed in the non stream 7/8. What changes is that fixes will be available sooner, but it would have been the same that would have been released at a previous point release in the old method.
If we have such official documentation or announcement in RDO community then we can avoid such situations in future where we get to know the distro version stability well before trying it in OpenStack testing?
-gmann
cheers, Zane.
On Wed, Dec 1, 2021 at 5:48 PM Alex Schultz aschultz@redhat.com wrote:
On Wed, Dec 1, 2021 at 9:30 AM Ghanshyam Mann gmann@ghanshyammann.com wrote:
---- On Wed, 01 Dec 2021 09:13:02 -0600 Zane Bitter zbitter@redhat.com
wrote ----
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur <
dtantsur@redhat.com> wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley <
fungi@yuggoth.org> wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement
for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos
team impression was CentOS stream 9 is released and that is
what we should update In OpenStack testing. And then only we
updated the centos stream 8 -> 9 and dropped py3.6 testing
https://review.opendev.org/c/openstack/governance/+/815851/3..6/reference/ru...
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only
distro
being treated differently.
The only difference is there is no plan for a CentOS-branded release
to
define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so:
RHEL9
is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK
with
the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a
LTS
distro to a beta one, that was dropped on us in the middle of a
release
cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
Those are good points. We were discussing with the RDO team on the start of Yoga cycle itself whether to move to CentOS Stream9 or not.
Yes, there is a clear confusion on what is stable or not in CentOS
Stream distro.
Like other distro has, is there any document we as TC can refer to in
the future on
checking if the CentOS Stream *X* version is stable/LTS or not? I cannot
find it
for CentOS Stream 8 also, is that stable or LTS?
I guess what's the definition of LTS? Support > 1 year?
https://www.centos.org/cl-vs-cs/
CentOS Stream 8 is to be supported till 2024 and 9 is estimated until 2027. That seems pretty LTS to me.
Actually, there are no special LTS releases in CentOS. In Red Hat family distros, Fedora would be the non-LTS distro while CentOS is the LTS one.
Additionally what is the definition of stable in this case? My understanding is that there still won't be major version bumps to core software similar to how CentOS used to operate. So it'd be a similar stability as previously existed in the non stream 7/8. What changes is that fixes will be available sooner, but it would have been the same that would have been released at a previous point release in the old method.
Exactly, so as mentioned several times in this thread the main difference between CentOS Stream and CentOS Linux is the fact that updates are not done in big batches (minor releases) but in more frequent and small package updates in a continuous delivery approach [1]. Stability should not be very different to what we had with CentOS Linux.
[1] https://blog.centos.org/2020/12/centos-stream-is-continuous-delivery/
If we have such official documentation or announcement in RDO community then we can avoid such situations in future where we get to know the
distro
version stability well before trying it in OpenStack testing?
-gmann
cheers, Zane.
On 1/12/21 11:28, Ghanshyam Mann wrote:
---- On Wed, 01 Dec 2021 09:13:02 -0600 Zane Bitter zbitter@redhat.com wrote ----
On 26/11/21 10:29, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 09:20:39 -0600 Dmitry Tantsur dtantsur@redhat.com wrote ----
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote: On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
I think here is the confusion. In TC, after checking with centos team impression was CentOS stream 9 is released and that is what we should update In OpenStack testing. And then only we updated the centos stream 8 -> 9 and dropped py3.6 testing
The guidelines the TC have set are not that something exists, but that it is a stable LTS release. Debian sid, Ubuntu 20.10, Fedora rawhide, and OpenSUSE Tumbleweed all exist, but nobody mistakes them for stable LTS releases. It's not clear to me why CentOS Stream is the only distro being treated differently.
The only difference is there is no plan for a CentOS-branded release to define a point in time where Stream 9 becomes an LTS release. However, other parties do have such plans, but pointedly have not done so: RHEL9 is in beta; Rocky Linux, Alma Linux, and Oracle Linux are all yet to release a version based on Stream 9.
Presumably RDO folks were consulted about this decision and were OK with the time frame. However, there are other users out there, and from a Metal³ perspective this is a giant PITA, requiring us to move from a LTS distro to a beta one, that was dropped on us in the middle of a release cycle in flagrant violation of the TC's own guidelines that the stable distibutions must be chosen from among those available at the *beginning* of the release cycle (which CentOS Stream 9 was not).
Those are good points. We were discussing with the RDO team on the start of Yoga cycle itself whether to move to CentOS Stream9 or not.
As of today CentOS haven't announced the launch yet[1]. AIUI, it wasn't even possible for folks to start building packages against it before October 20.[2] The Yoga cycle had already started by at least October 11. So unless we're actively looking for ways to bend the rules to make users' lives harder, I think this is an easy call for Y.
It's great if the RDO team are happy, but what I'm pointing out is that RDO is not the only consumer of OpenStack components in the EL ecosystem.
Yes, there is a clear confusion on what is stable or not in CentOS Stream distro.
CentOS stream is the upstream for RHEL. That means that once RHEL is released, any patches to it must follow Red Hat's backward compatibility policies for RHEL.
What policies apply to CentOS Stream commits when RHEL is still in beta? TBH I don't know. But I'd expect them to be... you know... beta policies.
A beta is the opposite of stable long-term support.
Like other distro has, is there any document we as TC can refer to in the future on checking if the CentOS Stream *X* version is stable/LTS or not? I cannot find it for CentOS Stream 8 also, is that stable or LTS?
When RHEL 9 is released you will definitely hear about it.
If you don't want to tie it to a commercial distro, wait for Rocky Linux 9 (which is essentially equivalent to the old CentOS) to be released. I expect that will be later, not sooner, than RHEL though.
If we have such official documentation or announcement in RDO community then we can avoid such situations in future where we get to know the distro version stability well before trying it in OpenStack testing?
I would have thought that waiting for a blog post from CentOS would be the absolute minimum, even if you don't agree that we should wait for *somebody* to build a long-term supported release from it before calling it an LTS release.
cheers, Zane.
[1] https://lists.centos.org/pipermail/centos-promo/2021-December/003599.html [2] https://lists.centos.org/pipermail/centos-devel/2021-October/077383.html
On 2021-11-26 16:20:39 +0100 (+0100), Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us (and we don't have nodes in nodepool with it even yet?).
It was added to our nodepool over 3 weeks ago when https://review.opendev.org/816465 merged, and I see it in-use per https://zuul.opendev.org/t/openstack/nodes right now.
On Fri, Nov 26, 2021 at 4:59 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-11-26 16:20:39 +0100 (+0100), Dmitry Tantsur wrote:
On Fri, Nov 26, 2021 at 3:35 PM Jeremy Stanley fungi@yuggoth.org
wrote:
On 2021-11-26 14:29:53 +0100 (+0100), Dmitry Tantsur wrote: [...]
CentOS/RHEL ships 3.6 and a limited version of 3.8 and 3.9.
[...]
Is this still true for CentOS Stream 9? The TC decision was to support that instead of CentOS Stream 8 in Yoga.
No. But Stream 9 is pretty much beta, so it's not a replacement for us
(and
we don't have nodes in nodepool with it even yet?).
It was added to our nodepool over 3 weeks ago when https://review.opendev.org/816465 merged, and I see it in-use per https://zuul.opendev.org/t/openstack/nodes right now.
Maybe I misunderstand something, but I cannot find any stream-9 nodes in use now, and https://review.opendev.org/c/openstack/bifrost/+/819058/ failed with "The nodeset "centos-9-stream" was not found" just 2 days ago.
Dmitry
-- Jeremy Stanley
On 2021-11-26 17:18:51 +0100 (+0100), Dmitry Tantsur wrote: [...]
Maybe I misunderstand something, but I cannot find any stream-9 nodes in use now,
Ahh, yeah whether they appear in the nodes list will depend on if any builds are in progress for them. Today there's not a lot going on, so I likely just got lucky when I looked earlier.
and https://review.opendev.org/c/openstack/bifrost/+/819058/ failed with "The nodeset "centos-9-stream" was not found" just 2 days ago.
https://review.opendev.org/793462 is an example of a change which was passing builds on centos-9-stream nodes weeks ago (albeit non-voting). There is currently no nodeset named centos-9-stream, but there is a node label. Adding the nodeset definition soon would probably be good to avoid further confusion.
participants (17)
-
Alex Schultz
-
Alfredo Moralejo Alonso
-
Balazs Gibizer
-
Braden, Albert
-
Clark Boylan
-
Corey Bryant
-
Dmitry Tantsur
-
Ghanshyam Mann
-
Jeremy Stanley
-
Lee Yarwood
-
Marcin Juszkiewicz
-
Michał Nasiadka
-
Radosław Piliszek
-
Sean Mooney
-
Stephen Finucane
-
Tobias Urdin
-
Zane Bitter