Open Stack

On Thu, 2023-10-12 at 11:35 +0200, Dmitriy Rabotyagov wrote:
> > It isn't one year.
> 
> We could drop things in April 2023 (for 2023.2, which we didn't do),
actully we did.

we drop supprot for py38 and hten re-added it because some project were not
ready to drop it. i stongly belive that was a mistake.

to be clear the 2023.1 release was intened to be the final release to suport python 3.8
and ubuntu 20.04 https://github.com/openstack/governance/blob/master/reference/runtimes/2023.1.rst
it was the release where ubuntu 20.04 was supproted for one addtional release for smooth upgrades.

when the 2023.2 runtime was selected we removed support for ubuntu 20.04 because all user of 2023.2 are
expected to have upgraded to ubuntu 22.04 prior to upgrading openstack to 2023.2

https://github.com/openstack/governance/commit/6ecb9c7210b152bc4ad36d1e830f4ed6008c0198 is the orgianl 2023.2
testing runtime and then in may  python 3.8 was readded
https://github.com/openstack/governance/commit/05ac02c2acd6117092c628fb5b04a17c640c415e
i really dont think that the correct decsion espically after we had already started removing support for py38

> or we can drop them in April 2024 (for 2024.2), which is roughly 1
> year? I think I was talking more about ability to strategically plan
> such removal so that they were not matching non-SLURP releases.
if we dont drop supprot in 2024.1 then i think we must drop it in 2024.2
i dont think it would be reasonable to ask porject to continue to supprot it and i honestly dont think it
is a reasonbale ask for 2024.1 or 2023.2 i was very unhappy with teh late readdtion of it in may after it was
finally removed.
> 
> > Python 3.8 will be EOL'd by its maintainers in ~October 2024. We have seen that when python releases become EOL'd
> > many libraries stop supporting that release. This potentially leads to gaps in security and general bugfixing.
> 
> Well, while I do agree with concerns here, I think it's missing some
> important notes:
> 1. Py3.8. is going to be supported by Ubuntu 20.04 until April 2025 at
> worst, and then there's also ESM till 2030. I bet RHEL has smth like
> that as well. So security backports for python packages and python
> itself is not really our pain.
we do not use packages form the disto in our testing outside the standar libary
and intepreter so it is a problem for the libs we use from pypi
> 2. We still can test things quite consistently, since we have all our
> dependencies maintained in upper-constraints.. Basically due to U-C I
> can build and install Ocata as of today with Py2.7 (I'm not saying
> it's good or bad, just stating the fact)
not entirly. in ci or a contienr yes but some of our deps will not compile
on newer operating systems. i.e. to run nova's pep8 tox env on wallaby i need
to create an vm/container to do that because it wont actully work on a modern
operating system. the system packages that we leverage are too new and cause issues.

> . Also the proposal is quite
> clear to minimize testing to bare minimal, i.e. unit tests, so I don't
> see how it's a deal breaker here. And it's in fact an operator's
> responsibility to ensure their environments are secure enough.
> 
> > If 2023.2 had dropped python3.8:
> > 
> > * Run 2023.1 on python3.8
> >  * Convert 2023.1 deployment to python3.10
> >  * Upgrade 2023.1 to 2023.2 or 2024.1
> 
> Yes, that is right. Important thing here is that it would be expected.
> Like I know in advance (with a year in my pocket), that in order to
> upgrade to 2024.1 or to 2023.2 I need to upgrade python first, and
> that is totally fine. So such thing can be planned, tested and
> executed in 2 separate steps.
you have already been given that notice as we notified everyone that ubuntu 20.04
was last supported in 2023.1 an all othe supported distos in that release ship python 3.9+
debian 11 and centos 9 stream both already use python 3.9, ubuntu 22.04 ships 3.10

so this is not a issue for the removal of python 3.8
i dont know of any suppproted distribution where openstack 2023.2+ was shiped on py38
so keeping supprot in 2023.2 or 2024.1 seam academic to me. do we know of any operator
or deployment that would actully deploy this way?

> 
> > If 2024.1 drops python3.8 support:
> > 
> >  * Run 2023.1 on python3.8
> >  * Optionally upgrade deployment to 2023.2
> >  * Convert 2023.x to python3.10
> >  * Upgrade to 2024.1
this was not intended to be a supproted upgrade path
on yoga and zed the only supported runtime that had python 3.8 was ubuntu 20.04
https://github.com/openstack/governance/blob/master/reference/runtimes/yoga.rst
https://github.com/openstack/governance/blob/master/reference/runtimes/zed.rst
so when you upgraded to 2023.1 ( the first slurp release) you did that on 20.04
then for smooth upgrade we supported ubuntu 20.04 and 22.04 in antelop(2023.1)
so the expecation was if your on ubuntu using 20.04 and 2023.1 you would upgrade
to 22.04 before doing any futher openstack upgrades.

ill note that canonical has supported openstack on ubuntu 22.04 since yoga
https://ubuntu.com/openstack/docs/supported-versions so they expected people to upgrade
form 20.04 to 22.04 after upgrading to yoga on 18.04. we dropped supprot for it well after
when any operator should have still be using it if the installed openstack form packages.

what is also relevant to this conversation is openstack 2024.1 will be supported on ubuntu 22.04
and 24.04 by canonical for smoth upgrade and we will additionally suprot 2024.2 on 22.04 for smoth upgrade.

so to go form 2023.1 -> 2023.2 or 2024.1 the upgrade path is
oepnstack 2023.1 to oepnstack 2023.1 on ubuntu 22.04 and python 3.10
then and only then can you upgrade to 2023.2 or 2024.1

note that nova does not support the libvirt/qemu shiped on ubuntu 20.04 in 2023.2
we declared our next min version in wallaby and deferd the bump twice as it was orgianlly planned for z
so all users of the the 2023.1 slrup release were notified that this was going to happen well before we did it.

> 
> And here it's not precisely correct. As with that, upgrade to 2023.2
> is not _really_ optional.
it is optional ubuntu 20.04 and python 3.8 was not intended to be supprot
in 2023.2 which is why you are ment to do the os upgrade and python upgrade
on 2023.1 or eariler.
>  And such upgrade (having temporary version
> in between) from operational perspective is really a nightmare:
> * Firstly it would take a 3x more time get the upgrade done (as you're
> in fact doing 3 upgrades)
> * Secondly you need to somehow explain the users that the API versions
> (or microversions) you get in between is not smth you should get used
> to, because it they will really soon
> * Should you also upload new images for magnum/amphora/trove and
> failover everything for this intermediate upgrade?
> * Way more time you need to spend on testing such upgrade, as there's
> another release now that needs to be handled.
> * Last, but not least, each upgrade still brings some disturbance
> (like restart of heat-engine might got stack borked), and now you need
> to do that twice
> 
> So eventually, with all that, you come to the same flow as you
> described if 2023.2 had dropped python3.8, meaning you should upgrade
> your 2023.1 deployment to python3.10 and then proceed to 2024.1.
yes that is the expected flow. upgrade to 3.9+ on 2023.1 before proceeding
to 2024.1
> Except:
> 1. You relied on TC's decision that things will not be removed on
> non-SLURP so invested your time into 2024.1 upgrade preparation(and
> maybe development of features for 2024.1). Now you need to scrap all
> your upgrade (or contribution) plans and plan python upgrade
> internally instead, as this becomes basically a pre-requirement.
i would consieer anyoen that takes that perspecte to have miss understood
the intent of our supproted runtimes and tc decisions.

im sorry but form my perspective we did not agree to supproting python 3.8
until 2024.1 when we were selecting the the runtime for 2023.1 or
agreeing on the new slurp process. form my perspective it was clear to me
that 2023.1 was intened to be the last release to supprot eith ubuntu 20.04
or python 3.8
> 2. Python upgrade for operations is not only about OpenStack,
> actually. As you also need to adopt your tooling, scripts and plenty
> of stuff to work against the new Python version.
> 3. If you've already notified end-users to expect new features that
> 2024.1 will bring to them in 6 month - you have to disappoint them
> now, since upgrades won't happen in that timeline, because you need to
> deal with other things.
form my perspecitve we notified them that 2023.1 would be the last release
to supprot python 3.8 and then revers couse on that in 2023.2 for what i
condider to be invalid reasons. yes i know our ci broke when we start
enforce a min version but i think we shoudl have fixed the project
that coudl not drop 38 not readded it.
> 
> 
> > I don't see a strong relationship with SLURP or non-SLURP
> 
> In the motivation part for SLURP there's that thing:
> "Upgrades will be supported between “SLURP” releases, in addition to
> between adjacent major releases (as they are today)."
> IMO, with removing py38 now we're breaking that.
and in my view py38 was deprecated for removal in 2023.1 so there is no
breakage.
>  If you're on 2023.1
> and running py3.8 you can't just jump to 2024.1, which means that this
> upgrade path is not _really_ supported. Or we really need to clarify
> what we mean under "supported upgrades". I see how I can be wrong in
> that assumption and that there're arguments against this vision. But I
> assume this perspective has a right to exist as well.
> 
> Again. I really don't care about py3.8 as we are not running it since
> upgrade to 2023.1 as switched to py3.10 right after it.
> I'm more fighting for consistency between our decisions and actions
> that we take afterwards, so that any openstack user or operator could
> rely on decisions that were taken and promoted, so that OpenStack was
> perceived as a solid platform.
i think retoactivly adding platform and python version that were previously
agreed to be drop undermines that platfrom and our ablity to maintain it.
to be clear as a contibutoer and core review i felt underminded by the tc
when python 3.8 was readded in may. and i felt like we sent the wrong message
to operators because we said somethign was supproted when none of the testing runtim
operationg systems supproted a unifed deployment of openstack on that versoin of python.
> 
> чт, 12 окт. 2023 г. в 09:42, Sławek Kapłoński <skaplons at redhat.com>:
> > 
> > Hi,
> > 
> > Dnia czwartek, 12 października 2023 01:23:16 CEST Clark Boylan pisze:
> > > On Wed, Oct 11, 2023, at 2:05 PM, Dmitriy Rabotyagov wrote:
> > > > > 
> > > > > 
> > > > > It is more than an exaggeration: it just isn't realistic to support anything else in my opinion. You'll be
> > > > > stuck supporting ancient everything if you don't take a position like this.
> > > > 
> > > > Then I'd say that SLURP approach should be cancelled as I'm not sure
> > > > any more what real-users situation it is solving then.
> > > > And if 1 year is a too long period which makes everything "ancient".
> > > 
> > > It isn't one year. Python 3.8 has been out since October 2019, was part of the Ubuntu Focal release in April 2020,
> > > was first supported in OpenStack Victoria (October 2020), and if kept as part of the 2024.1 release would be kept
> > > alive in OpenStack until approximately October 2025.
> > > 
> > > Python 3.8 will be EOL'd by its maintainers in ~October 2024. We have seen that when python releases become EOL'd
> > > many libraries stop supporting that release. This potentially leads to gaps in security and general bugfixing. I
> > > think giving users a clear signal of when they should update runtimes before that runtime becomes problematic is a
> > > good thing.
> > > 
> > > > 
> > > > Because as I said, removal of py3.8 had been raised at beginning of
> > > > 2023.2 (and I bet everyone recall the mess we got into because of
> > > > that), when we _really_ could drop it, but just by handling that in a
> > > > better way. But now in non-SLURP I feel pretty much wrong in doing so.
> > > 
> > > This is the part I do not understand. What is the functional difference for operators going through an upgrade
> > > process if we drop it in the .1 release vs the .2 release? Let's draw up the resulting upgrade paths.
> > > 
> > > If 2023.2 had dropped python3.8:
> > > 
> > >   * Run 2023.1 on python3.8
> > >   * Convert 2023.1 deployment to python3.10
> > >   * Upgrade 2023.1 to 2023.2 or 2024.1
> > > 
> > > If 2024.1 drops python3.8 support:
> > > 
> > >   * Run 2023.1 on python3.8
> > >   * Optionally upgrade deployment to 2023.2
> > >   * Convert 2023.x to python3.10
> > >   * Upgrade to 2024.1
> > > 
> > > There isn't a huge difference here. And in both cases users can still skip an OpenStack version in the upgrade
> > > process if they choose to do so. The only real difference is if we want people to continue running 2024.1 on
> > > python3.8 before converting to probably python3.11 or python3.12 at that point. I don't think that is the sort of
> > > signal we want to give because deployments in that situation are very likely to run into struggles with library
> > > updates.
> > 
> > That's exactly my understanding. I don't see big difference there too.
> > 
> > > 
> > > > 
> > > > And again, I am not protecting py3.8 specifically, but I am standing
> > > > for expectations that we set by introducing SLURPs, which IMO was one
> > > > of the biggest deals for operators lately.
> > > 
> > > 
> > 
> > 
> > --
> > Slawek Kaplonski
> > Principal Software Engineer
> > Red Hat
>