Open Stack

On Wed, Dec 1, 2021, at 11:46 AM, Alex Schultz wrote:
> )))
>
> On Wed, Dec 1, 2021 at 11:09 AM Clark Boylan <cboylan at sapwetik.org> wrote:
>>
>> On Wed, Dec 1, 2021, at 9:31 AM, Alex Schultz wrote:
>> > On Wed, Dec 1, 2021 at 10:05 AM Sean Mooney <smooney at redhat.com> wrote:
>> >
>>
>> Apologies if my reduction of thread content was too aggressive.
>>
>> >
>> >> so haveing at least 3.9  support in yoga will be imporatn for multiple distros and to help use test that i and proably others
>> >> suggested raising the minium python version to 3.8 so that we only have to test with 2 version in the gate instead of 3.(3.6 3.8 and 3.9)
>> >>
>> >> continuting to support py36 aslo puts a burden on developers not just on our ci infrastucre as it will becomre harder to test wtih going froward
>> >> as it stops being shipped in distros and as it exits security support
>> >> the end of life for python 3.6 is planned for this month https://www.python.org/dev/peps/pep-0494/#lifespan
>> >> speficically (23 Dec 2021) based on https://endoflife.date/python
>> >>
>> >> i dont think we shoudl be releasing yoga on an end of life python version.
>> >> if we raise the minium to 3.8 that will be supported until at least 14 Oct 2024
>> >> which is after the stabel branch will be retired.
>> >>
>> >
>> > So I guess the question is who do we use for language support EOLs?
>> > My understanding is that Ubuntu 18.04's default was 3.6 which has
>> > standard support until 2023[0] and CentOS Stream 8 will be supported
>> > until 2024[1].  Both of which should exceed Yoga's support[2]?  With
>> > all these arguments about LTS distros, it seems that we should be
>> > using their support life cyles for the language versions and not
>> > python's[3].
>> >
>> > [0] https://wiki.ubuntu.com/Releases
>> > [1] https://www.centos.org/cl-vs-cs/
>> > [2] https://releases.openstack.org/
>> > [3] https://devguide.python.org/#status-of-python-branches
>> >
>
> For this reply, "LTS Distro" means support/updates > 13 months  This
> means regular Ubuntu and Fedora releases don't qualify. Ubuntu LTS,
> CentOS Stream * and Debian LTS would qualify.  Can we all agree on
> this definition?
>
> https://ubuntu.com/about/release-cycle
> https://www.centos.org/cl-vs-cs/#end-of-life
> https://wiki.debian.org/LTS
>
>>
>> I'd like to make a proposal for moving forward. But first let's summarize some facts so that we can evaluate the proposal against the state of the world.
>>
>> * Python 3.6 reaches EOL soon.
>> * CentOS 9 Stream is released and makes no mention of being in a beta status (https://www.centos.org/centos-stream/)
>> * We have the ability to run tests on CentOS 9 Stream today
>
> Yes some OpenStack projects can run on CentOS 9 Stream today but not
> all can (Puppet OpenStack being one).  The assumption that pip install
> is sufficient for *all* projects is not correct. At least some testing
> needs to be able to be run on all projects before projects can
> officially declare <= 3.6 dead.  This hopefully will be completed
> during Yoga but it takes time.
>
>> * It is useful to users to ensure we have an OpenStack release tested against old distro release with old python and new distro release with new python. This enables them to update underlying distro releases independently from upgrading OpenStack.
>> * The various distros are not in sync over which python versions they support.
>>
>> Given this state it seems reasonable that we continue to test Yoga against python3.6 and newer python. It would also probably be a good idea to explicitly update the PTI and distro support policies to indicate that when adding new distro releases we continue to support the old distro release's python for one cycle. The level of actual testing done on these platforms will depend on who steps up to maintain that testing, but at the very least we'll accept bug fixes for old python and not prevent installation via setup.cfg python_requires settings.
>>
>> Yes, this means another cycle of python3.6 support which isn't ideal given it's soon EOL, but the intent is to support distros who will continue to support the older python on their platform.
>
> IMHO "another cycle" means continued support in Z not Yoga.  I think
> the ask here was just delay the minimum version until Z so this would
> be the last cycle with 3.6 support.  Jeremy said in a subsequent
> message that the interpreter comes from an LTS distro which IMHO means
> the EOL is the distro's EOL not Python's. Now this brings up issues if
> we can't get 3.6 fixes for specific versions of our dependencies that
> come in via pypi, but do we have data how often this is actually a
> problem? Isn't that already a problem for stable branches? Or are we
> worrying about something that doesn't happen?  We need to all agree on
> where EOL dates come from for dependencies and expectations around
> them.

The point I was trying to make was more that we should be explicit about when the overlap happens; write that down in our policy so that the next time this comes up the process goes more smoothly. I suppose if we have to delay for an additional cycle that may be the cost of not having considered this ahead of time.

I think Jeremy covered why the Python EOL is important. Our dependencies tend to start dropping like flies when python versions go EOL. I think we can survive for a time,  but we shouldn't try to keep them alive for the long term.

>
> LTS Distros are being referenced in this thread but people keep going
> back to EOL dates that are not aligned with these distros.  For the
> full OpenStack project ecosystem, testing is occurring on Ubuntu,
> CentOS, and a bit of Debian.  IMHO when a release (e.g. Yoga) is
> released, we should use what is available at the start of cycle and
> using the EOL dates of the tested distros in determining decisions
> around version support.  If we use this for Yoga, realistically CentOS
> Stream 8 and Ubuntu 20.04 could be used which would leave us with
> 3.6/3.8.  It seems Debian was the only one with 3.7
> (https://wiki.debian.org/Python).  Since CentOS Stream 9 was released
> during the Yoga cycle, we couldn't pick it up as a version until Z but
> we could then make the 3.8 cutoff then.  If Ubuntu releases a new
> version (the website says they might) during Yoga with 3.9 then we
> could talk about raising it further in Z.  This would allow for lower
> bounds to be pushed, but requires that one of our qualifying LTS
> distros in check/gate were already available at the start of a cycle.
>
> What I would like is that people stop pointing to Python's EOL date
> which is newer but does not actually align with the assumption of an
> LTS distro.  It would only make sense to use that date if we were
> basing our testing on non-LTS distros which may be able to follow
> their EOL dates more closely.  I keep hammering on the LTS distro
> because OpenStack is heavily reliant on packages and hardware
> supported by distros that it is very closely tied to what is
> available.  If OpenStack had no tie to distro hardware support, venvs
> and pip installing all the things might be sufficient but there are
> many dependencies which end up not being satisfied this way.  You can
> get away with it up to a point, but it's an important integration
> point.
>
>>
>> If we want to be bold and try an experiment, we could also reduce testing of the python versions to oldest and newest supported python versions. For example python3.6 and python3.9 during Yoga. Don't worry about python3.7 and python3.8 as it is likely they will continue to function as long as 3.6 and 3.9 are happy. This would reduce the size of the testing matrix. We can always reevaluate if problems for 3.7 and 3.8 occur.
>>
>
> The issue here I think is trying to align on a default LTS distro
> python. CS8,18.04 is 3.6, Debian Buster is 3.7, 20.04 is 3.8 and CS9
> will be 3.9.  Personally I think 3.6/3.9 is sufficient because for the
> projects I contribute to this covers my cases. But this would exclude
> the 3.8 on 20.04.  Would this be a problem for Ubuntu's UCA?

Ubuntu 20.04 includes a python 3.9. I don't know how that would impact the UCA or Ubuntu's packaging though. That said I have a strong suspicion that if OpenStack runs on 3.6 and 3.9 that it should just work on 3.8 as well. And as I mentioned if we prove that wrong through experience we can always fix 3.8 and go back to explicitly testing it. I don't think this particular change is super critical, but I know some people were concerned about the number of pythons that would need testing.

>
>> Finally, I think it is important to note that OpenStack is using CentOS as a stand in for RHEL. This worked a lot better when CentOS was an actual copy of RHEL. What we've got today is CentOS stream and it makes no claims about being in beta until RHEL releases (that I can find anyway), and I don't think we should make those assertions ourselves. If there is interest in using a different stand in (Alma or Rocky?) then we should replace CentOS stream with one of those alternatives. Another option may be to assert that Stream only functions as a standin once RHEL has released their corresponding update, but it is my understanding that Stream and RHEL will always have some delta.
>
> There has always been a delta. The change now is the delta has been
> reversed such that we'll get fixes for things before RHEL but this is
> likely better for OpenStack's use cases where the previous meant long
> delays to get necessary fixes.  I'm uncertain why the RHEL release
> schedule is being factored in to determine some indication of
> stability when testing is occurring
> (https://www.centos.org/cl-vs-cs/#testing).

I'm explicitly saying don't compare to RHEL for stability. CentOS 9 Stream is released as far as I can tell. It isn't labeled a beta release. You can download and run it today. We run CentOS testing because OpenStack asserts that it supports RHEL and historically CentOS was the closest we could get to RHEL. I'm saying we might consider if continuing to use CentOS for that goal is still appropriate. If it is then lets just use 9 Stream today as it is released and ready to go. If it isn't appropriate because it isn't close enough to RHEL, then we should consider an alternative like Alma or Rocky or perhaps they will converge more once RHEL has released. Basically I'm saying lets get away from thinking about this in terms of "is stream ready/beta/etc" and think in terms of "does using stream accomplish the goals of testing to ensure RHEL compatibility".

>
> Thanks,
> -Alex
>
>>
>> Clark
>>