Open Stack

Hello,
I agree with previous statement from Michal.

The upgrade path in for example RDO has been very smooth previously with upgrading to new OpenStack release
then switching out the distro version afterwards because they support both during a transition.

Sure they will do that now as well, but if any project decides to break py36 there will be more work for the RDO team
and more arguments to not revert changes because py38 would be the “real” supported runtime in OpenStack testing.

The transition period is required to not break the upgrade path.

Best regards
Tobias

On 29 Nov 2021, at 11:14, Michał Nasiadka <mnasiadka at gmail.com<mailto:mnasiadka at gmail.com>> wrote:

Hello,

I’m strongly against dropping py36 support now, unless we’re going to find a solution that works on CentOS Stream 8.
RHEL 9 is not out, and probably will not be in months - how do we expect users to use Yoga on production deployments (where they use CentOS Linux/equivalents today)?

Dropping the runtime testing and supporting devstack - and negotiating on a per project basis to support py36 or not - is not a solution.
Either Yoga supports py36 as a transition point/release to py38 - or not.

In Kolla - we also did not anticipate (and don’t think it’s a good idea) to support CentOS Stream 9 in Yoga release.
With the current decision - we are either forced with supporting CentOS Stream 9 (with no alternatives like Rocky Linux/Alma Linux in place - because RHEL 9 is not out) - or dropping CentOS support completely.

If we pursue CS9 - we also need to support migration from CS8 to CS9 and that’s also a considerable amount of work - which is unplanned.

Best regards,
Michal

On 29 Nov 2021, at 10:17, Lee Yarwood <lyarwood at redhat.com<mailto:lyarwood at redhat.com>> wrote:

On 26-11-21 11:24:59, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 10:18:16 -0600 Ghanshyam Mann <gmann at ghanshyammann.com<mailto:gmann at ghanshyammann.com>> wrote ----
---- On Fri, 26 Nov 2021 10:05:15 -0600 Lee Yarwood <lyarwood at redhat.com<mailto:lyarwood at redhat.com>> wrote ----
On 26-11-21 09:37:44, Ghanshyam Mann wrote:
---- On Fri, 26 Nov 2021 06:29:42 -0600 Lee Yarwood <lyarwood at redhat.com<mailto:lyarwood at redhat.com>> wrote ----
On 26-11-21 10:54:26, Alfredo Moralejo Alonso wrote:
On Thu, Nov 25, 2021 at 10:23 PM Ghanshyam Mann <gmann at ghanshyammann.com<mailto:gmann at ghanshyammann.com>>
wrote:

---- On Thu, 25 Nov 2021 13:58:28 -0600 Marcin Juszkiewicz <
marcin.juszkiewicz at linaro.org<mailto:marcin.juszkiewicz at linaro.org>> wrote ----
W dniu 25.11.2021 o 19:13, Stephen Finucane pisze:
gmann has been helpfully proposing patches to change the
versions of Python we're testing against in Yoga. I've
suggested that we might want to bump 'python_requires' in
'setup.cfg' to indicate that we no longer support any version
of Python before 3.8

CentOS Stream 8 has Python 3.6 by default and RDO team is doing
CS8 -> CS9 migration during Yoga cycle. Can we postpone it to Z
when there will be no distribution with Py 3.6 to care about?

Stupid question that I should know the answer to but does RDO really
support RPM based installations anymore? IOW couldn't we just workaround
this by providing CS8 py38 based containers during the upgrade?

As Marcin posted, the plan in RDO is to support both CentOS Stream 8 and
CentOS Stream 9 in Yoga. This is how we have managed previous major CentOS
version upgrades in the past providing support for both releases in an
OpenStack version to ease the upgrade so I'd like to keep yoga working on
py3.6 included in CS8 and CS9.

If this was the plan why wasn't it made clear to the TC before they
dropped CS8 from the Yoga runtimes? Would it even be possible for the TC
to add CS8 and py36 back in to the Yoga runtimes?

Postponing to Z, you mean dropping the py3.6 tests or bumping it in
in 'setup.cfg' so that no one can install on py3.6 ?

First one we already did and as per Yoga testing runtime we are
targeting centos9-stream[1] in Yoga itself.

For making 'python_requires' >=py3.8 in 'setup.cfg', I have no
string opinion on this but I prefer to have flexible here that 'yes
OpenStack is installable in py3.6 but we do not test it anymore from
Yoga onwards so no guarantee'.  Our testing runtime main goal is
that we document the version we are testing *at least* which means
it can work on lower or higher versions too but we just do not test
them.


May it be possible to keep py3.6 jobs to make sure patches are not
introducing py3.8-only features that would break deployment in CS8?

We should keep CS8 and py36 as supported runtimes if we are keeping the
jobs, otherwise this just sets super confusing.

Yeah, I think it create confusion as I can see in this ML thread so
agree on keeping  'python_requires'  also in sycn with what we test.

Cool thanks!

Now question on going back to centos stream 8 support in Yoga, is it
not centos stream 9 is stable released or is it experimental only? If
stable then we can keep the latest available version which can be
centos stream 9.

I honestly don't know and can't find any docs to point to.

Our project interface testing doc clearly stats 'latest LTS' to
consider for testing[1] whenever we are ready. I am not very strongly
against of reverting back to centos stream 8 but we should not add two
version of same distro in testing which can be a lot of we consider
below three distro

How do we expect operators to upgrade between Xena where CentOS 8 stream
is a supported runtime and Yoga where CentOS 9 stream is currently the
equivalent supported runtime without supporting both for a single
release?

This is really good question on upgrade testing we do at upstream and I remember
it cameup and discussed a lot during py2.7 drop also that how we are testing the
upgrade from py2.7 to py3. Can we do in grenade? But that we answered as we did
not tested directly but stein and train tested both version so should not be any issue
if you upgrade from there (one of FAQ in my blog[1]).

But on distro upgrade testing, as you know we do not test those in upstream neither
in grenade where upgrade are done on old node distro only not from old distro version to
new distro version with new code.  It is not like we do not want to test but if anyone
from any distro would like to setup grenade for that and maintain then we are more happy.
In summary, yes we cannot guarantee distro upgrade testing from OpenStack upstream testing
due to resource bandwidth issue but we will welcome any help here.

We discussed with amoralej about moving the testing runtime to CentOS
stream 8 and py36 or not in TC IRC channel[1].

As we at upstream do not test distro two versions in same release,
amoralej agreed to keep CentOS stream 9 if one to choose which is our
current testing runtime is. So no change in the direction of current
testing runtime and dropping the py3.6 but there is possibility of
some trade off here. If any py3.6 breaking changes are happening then
it is up to projects goodness, bandwidth, or flexibility about
accepting the fix or not or even add a py36 unit test job. As our
testing runtime is the minimum things to test and it does not put any
max limit of testing, any project can extend their testing as per
their bandwidth.

In summary:

(This is what we agreed today in TC channel but as most of the folks
are on leave today, I will keep it open until next week so see if any
objections from the community and will conclude it accordingly)

* No change in Yoga testing runtime and we move to cs9 and drop py36.
* We will not put hard stop on cs8 support and we can:
** Devstack keep supporting cs8 in Yoga
** It can be negotiated with project to add py36 job or fix if any
  py36 breaking changes are observed by RDO (or any distro interested in
  py36) but it depends on the project decision and bandwidth.

As next, how we can improve the upgrade testing from distro versions
is something we will explore next and see what all we can test to make
upgrade easier.

I'm against this, as I said in my setup.cfg >= py38 review for
openstack/nova [1] we either list and support runtimes or don't. If RDO
and others need CentOS 8 Stream support for a release then lets include
it and py36 still for Yoga and make things explicit.

As I've said elsewhere I think the TC really need to adjust their
thinking on this topic and allow for one OpenStack release where both
the old and new LTS distro release are supported. Ensuring we allow
people to actually upgrade in place and later handle the distro upgrade
itself.

Cheers,

Lee

[1] https://review.opendev.org/c/openstack/nova/+/819415

--
Lee Yarwood                 A5D1 9385 88CB 7E5F BE64  6618 BCA6 6E33 F672 2D76


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20211129/309cc36e/attachment-0001.htm>