Open Stack

 ---- On Tue, 30 Nov 2021 13:47:02 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  ---- On Mon, 29 Nov 2021 04:52:07 -0600 Tobias Urdin <tobias.urdin at binero.com> wrote ----
 >  > Hello,I agree with previous statement from Michal.
 >  > 
 >  > The upgrade path in for example RDO has been very smooth previously with upgrading to new OpenStack releasethen switching out the distro version afterwards because they support both during a transition.
 >  > Sure they will do that now as well, but if any project decides to break py36 there will be more work for the RDO teamand more arguments to not revert changes because py38 would be the “real” supported runtime in OpenStack testing.
 >  > The transition period is required to not break the upgrade path.
 >  > Best regardsTobias
 >  > On 29 Nov 2021, at 11:14, Michał Nasiadka <mnasiadka at gmail.com> wrote:
 >  > Hello,
 >  > I’m strongly against dropping py36 support now, unless we’re going to find a solution that works on CentOS Stream 8.RHEL 9 is not out, and probably will not be in months - how do we expect users to use Yoga on production deployments (where they use CentOS Linux/equivalents today)?
 >  > Dropping the runtime testing and supporting devstack - and negotiating on a per project basis to support py36 or not - is not a solution.Either Yoga supports py36 as a transition point/release to py38 - or not.
 >  > In Kolla - we also did not anticipate (and don’t think it’s a good idea) to support CentOS Stream 9 in Yoga release.With the current decision - we are either forced with supporting CentOS Stream 9 (with no alternatives like Rocky Linux/Alma Linux in place - because RHEL 9 is not out) - or dropping CentOS support completely.
 >  > If we pursue CS9 - we also need to support migration from CS8 to CS9 and that’s also a considerable amount of work - which is unplanned.
 >  > Best regards,Michal
 > 
 > I agree on all these points on keeping py36 for Yoga can be helpful for most people in migration to the new distro version.
 > I have added it to the TC meeting agenda which is scheduled for Dec 2nd and discuss more on this. Please feel free
 > to join TC meeting, details are below:
 > 
 > https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Agenda_Suggestions

The Technical Committee considered all the points made in this ML thread and
discussed them in today's TC meeting[1]. Testing python 3.6 for one more cycle
would not cost much. To have a smooth migration to new distro versions having
different default python versions, we are ok to keep the python 3.6 testing
for the Yoga cycle and keep CentOS Stream 8 and 9 both in runtime. This does not
cost us much in terms of developer-power or infra resources. 

Below are the versions we will target for Yoga:

Distro:
* Ubuntu 20.04
* CentOS Stream 8
* CentOS Stream 9
* Debian 11

Python: (keeping only lower and highest versions in testing with the assumption that anything working
in py3.6 and py3.9 will work in py3.7, py3.8 too)
* Python 3.6
* Python 3.9
* We agree to add Python 3.10 as a non-voting job also.

I have pushed the proposal in governance:
- https://review.opendev.org/c/openstack/governance/+/820195

as well as in job template:
- https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/820286

Thanks to everyone for participating and providing more clarity/feedback from
users/operator/distro-provide perspective.

[1]
- https://meetings.opendev.org/meetings/tc/2021/tc.2021-12-02-15.02.log.html#l-33
- https://www.youtube.com/watch?v=PwGYse8j4UY 


-gmann

> 
 > -gmann
 > 
 >  > 
 >  > On 29 Nov 2021, at 10:17, Lee Yarwood <lyarwood at redhat.com> wrote:
 >  > On 26-11-21 11:24:59, Ghanshyam Mann wrote:
 >  > ---- On Fri, 26 Nov 2021 10:18:16 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  > ---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood <lyarwood at redhat.com> wrote ----
 >  > On 26-11-21 09:37:44, Ghanshyam Mann wrote:
 >  > ---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood <lyarwood at redhat.com> wrote ----
 >  > On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote:
 >  > On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann <gmann at ghanshyammann.com>
 >  > wrote:
 >  > 
 >  > ---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz <
 >  > marcin.juszkiewicz at linaro.org> wrote ----
 >  > W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
 >  > gmann has been helpfully proposing patches to change the
 >  > versions of Python we're testing against in Yoga. I've
 >  > suggested that we might want to bump 'python_requires' in
 >  > 'setup.cfg' to indicate that we no longer support any version
 >  > of Python before 3.8
 >  > 
 >  > CentOS Stream 8 has Python 3.6 by default and RDO team is doing
 >  > CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z
 >  > when there will be no distribution with Py 3.6 to care about?
 >  > 
 >  > Stupid question that I should know the answer to but does RDO really
 >  > support RPM based installations anymore? IOW couldn't we just workaround
 >  > this by providing CS8 py38 based containers during the upgrade?
 >  > 
 >  > As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and
 >  > CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS
 >  > version upgrades in the past providing support for both releases in an
 >  > OpenStack version to ease the upgrade so I'd like to keep yoga working on
 >  > py3.6 included in CS8 and CS9.
 >  > 
 >  > If this was the plan why wasn't it made clear to the TC before they
 >  > dropped CS8 from the Yoga runtimes? Would it even be possible for the TC
 >  > to add CS8 and py36 back in to the Yoga runtimes?
 >  > 
 >  > Postponing to Z, you mean dropping the py3.6 tests or bumping it in
 >  > in 'setup.cfg' so that no one can install on py3.6 ?
 >  > 
 >  > First one we already did and as per Yoga testing runtime we are
 >  > targeting centos9-stream[1] in Yoga itself.
 >  > 
 >  > For making 'python_requires' >=py3.8 in 'setup.cfg', I have no
 >  > string opinion on this but I prefer to have flexible here that 'yes
 >  > OpenStack is installable in py3.6 but we do not test it anymore from
 >  > Yoga onwards so no guarantee'.  Our testing runtime main goal is
 >  > that we document the version we are testing *at least* which means
 >  > it can work on lower or higher versions too but we just do not test
 >  > them.
 >  > 
 >  > 
 >  > May it be possible to keep py3.6 jobs to make sure patches are not
 >  > introducing py3.8-only features that would break deployment in CS8?
 >  > 
 >  > We should keep CS8 and py36 as supported runtimes if we are keeping the
 >  > jobs, otherwise this just sets super confusing.
 >  > 
 >  > Yeah, I think it create confusion as I can see in this ML thread so
 >  > agree on keeping  'python_requires'  also in sycn with what we test.
 >  > 
 >  > Cool thanks!
 >  > 
 >  > Now question on going back to centos stream 8 support in Yoga, is it
 >  > not centos stream 9 is stable released or is it experimental only? If
 >  > stable then we can keep the latest available version which can be
 >  > centos stream 9.
 >  > 
 >  > I honestly don't know and can't find any docs to point to.
 >  > 
 >  > Our project interface testing doc clearly stats 'latest LTS' to
 >  > consider for testing[1] whenever we are ready. I am not very strongly
 >  > against of reverting back to centos stream 8 but we should not add two
 >  > version of same distro in testing which can be a lot of we consider
 >  > below three distro
 >  > 
 >  > How do we expect operators to upgrade between Xena where CentOS 8 stream
 >  > is a supported runtime and Yoga where CentOS 9 stream is currently the
 >  > equivalent supported runtime without supporting both for a single
 >  > release?
 >  > 
 >  > This is really good question on upgrade testing we do at upstream and I remember
 >  > it cameup and discussed a lot during py2.7 drop also that how we are testing the
 >  > upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did
 >  > not tested directly but stein and train tested both version so should not be any issue
 >  > if you upgrade from there (one of FAQ in my blog[1]).
 >  > 
 >  > But on distro upgrade testing, as you know we do not test those in upstream neither
 >  > in grenade where upgrade are done on old node distro only not from old distro version to
 >  > new distro version with new code.  It is not like we do not want to test but if anyone
 >  > from any distro would like to setup grenade for that and maintain then we are more happy.
 >  > In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing
 >  > due to resource bandwidth issue but we will welcome any help here.
 >  > 
 >  > We discussed with amoralej about moving the testing runtime to CentOS
 >  > stream 8 and py36 or not in TC IRC channel[1].
 >  > 
 >  > As we at upstream do not test distro two versions in same release,
 >  > amoralej agreed to keep CentOS stream 9 if one to choose which is our
 >  > current testing runtime is. So no change in the direction of current
 >  > testing runtime and dropping the py3.6 but there is possibility of
 >  > some trade off here. If any py3.6 breaking changes are happening then
 >  > it is up to projects goodness, bandwidth, or flexibility about
 >  > accepting the fix or not or even add a py36 unit test job. As our
 >  > testing runtime is the minimum things to test and it does not put any
 >  > max limit of testing, any project can extend their testing as per
 >  > their bandwidth.  
 >  > 
 >  > In summary:
 >  > 
 >  > (This is what we agreed today in TC channel but as most of the folks
 >  > are on leave today, I will keep it open until next week so see if any
 >  > objections from the community and will conclude it accordingly)
 >  > 
 >  > * No change in Yoga testing runtime and we move to cs9 and drop py36.
 >  > * We will not put hard stop on cs8 support and we can:
 >  > ** Devstack keep supporting cs8 in Yoga
 >  > ** It can be negotiated with project to add py36 job or fix if any
 >  >   py36 breaking changes are observed by RDO (or any distro interested in
 >  >   py36) but it depends on the project decision and bandwidth.  
 >  > 
 >  > As next, how we can improve the upgrade testing from distro versions
 >  > is something we will explore next and see what all we can test to make
 >  > upgrade easier.
 >  > 
 >  > I'm against this, as I said in my setup.cfg >= py38 review for
 >  > openstack/nova [1] we either list and support runtimes or don't. If RDO
 >  > and others need CentOS 8 Stream support for a release then lets include
 >  > it and py36 still for Yoga and make things explicit.
 >  > 
 >  > As I've said elsewhere I think the TC really need to adjust their
 >  > thinking on this topic and allow for one OpenStack release where both
 >  > the old and new LTS distro release are supported. Ensuring we allow
 >  > people to actually upgrade in place and later handle the distro upgrade
 >  > itself.
 >  > 
 >  > Cheers,
 >  > 
 >  > Lee
 >  > 
 >  > [1] https://review.opendev.org/c/openstack/nova/+/819415
 >  > 
 >  > -- 
 >  > Lee Yarwood                 A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76
 >  > 
 >  > 
 > 
 >