Open Stack

Hi, 

Thanks for starting this thread. As a stable maintainer let me also share
my thoughts:

- It is really sad to see, that important CVE fixes haven't arrived to
  old stable branches, that is clearly a sign that stable maintenance is
  not in a good shape on EM branches
- I also see that stable maintenance is not in a good shape on
  'maintained' branches either (2023.1, zed and yoga) for most of the
  projects, so that is also a visible problem
- so, I understand that teams want to focus more on their maintained
  branches

Still, I don't feel that eliminating the 'failed experiment' with
Extended Maintenance process would solve our issue. Though I agree that,
if we now EOL all our EM branches, that would really call some attention
to some vendors, operators, etc. (Would it help? Would companies step up
to spend more resources on upstream maintenance? Good question.)


Now let me also share some general thoughts about the topics people
brought up in this thread (please don't read those which does not
interest you o:)):

1) who should maintain EM branches?

Well, it was stressed out several times that maintainers can be different
then the project core team. Though how I understood this is not like that
there is a 'stable maintainer' group for 'maintained' branches and a
completely different 'extended maintainer' group for branches in
'extended maintenance', rather that it's not necessary the *same* core
team that develops the master branch. As a trivial way of working is
that A maintainer from X company is maybe interested back to let's say
Wallaby branch, so that is what branches they maintain, where they
primarily backport bug fixes, review backports, fixes CI; B maintainer
from Y company then does the same but till Xena branch; etc. (Of course,
the best is if time to time they can help out each other).
I know this is quite idealistic... I believe that maintainers have an
*employer* with an interest to keep XY branch as maintained as possible
and they act accordingly as much as possible (idealistic, too).

2) what tests to keep?

I understand that keeping old CI jobs functional is cumbersome and
resource needy. On the other hand, *every* vendor and companies benefit
from this, as downstream CI usually not that good as the upstream one OR
very expensive (resource, maintenance, fixing the uncaught bugs, etc).
I also tend to agree the opinion that only keeping unit tests and
functional tests doesn't give enough confidence in quality. Devstack
based tempest tests need to be kept, though I agree that teams can
somewhat reduce their job count, maybe with rationalising and dropping
somewhat redundant, expensive, time and resource consuming test jobs
(like different kind of grenade jobs, non-voting jobs, special case jobs,
etc), at least when they start to fail and there's no one to fix them.

3) are EM branches 'fully maintained'?

The original idea was that it isn't. EM just means that maintainers can
propose backports and can review them. It doesn't mean that every bugfix
will be backported and reviewed (unfortunately the same can be seen in
'maintained' branches as well).
Though I also share the views of those, who say that the fact that
important CVE fixes don't arrive to even "younger" EM branches is a
signal that those branches are doomed and probably should be EOL'd.
(Though some of you noted that the recent CVEs have cross project
dependencies, thus not that trivial to backport them.)

4) real count of EM branches

According to releases.openstack.org, it's 7, yes. In reality it's rather
5 or less. For some projects it's even just 2 or 3. Because it can be
different for every project.
Yes, we could have EOL'd waaay earlier rocky (for every project) and
stein branches (rocky is like 98% EOL'd, waiting for only some of the
PTLs to approve their project's EOL transition patch; Stein is really
rotten, but will be next as soon as rocky is done; but for example
stable/train gate for Nova, Neutron, etc, is not blocked, tests can pass,
even though there are not many patches that get merged).

5) is EM a 'failed experiment'?

Somewhat yes, somewhat no. There have been many tested bug fixes that
landed on those branches over the years, so companies could have benefit
from them. But of course, it's still not equal to consider those branches
as 'maintained', so yes that could rise some misconception. And as you
also said, not getting CVE fixes landed shows that those branches are far
from being 'maintained'.
Anyway, personally, I would not end this 'experiment' (as I'm probably
too optimistic :)), but I see that stable maintenance is a problem.

I'm curious about where this thread will lead and happy to see that this
got a forum topic on the Vancouver PTG schedule (thanks Tony! :)).
Unfortunately, I cannot participate as I could not travel there, but I'm
eagerly waiting for the thoughts (and maybe resolution) of the in-person
discussion!

Thanks for your time reading this long monologue o:)

Cheers,

Előd Illés
irc: elodilles @ #openstack-stable / #openstack-release




From: Ghanshyam Mann <gmann at ghanshyammann.com>
Sent: Thursday, June 8, 2023 5:37 PM
To: Brian Rosmaita <rosmaita.fossdev at gmail.com>
Cc: openstack-discuss <openstack-discuss at lists.openstack.org>
Subject: Re: [cinder][all][tc][ops][stable] EOL EM branches 
 
---- On Thu, 08 Jun 2023 06:39:28 -0700  Brian Rosmaita  wrote --- 
 > On 6/7/23 1:46 PM, Ghanshyam Mann wrote:
 > >   ---- On Wed, 07 Jun 2023 10:38:03 -0700  Jay Faulkner  wrote ---
 > >   >
 > >   > On Wed, Jun 7, 2023 at 10:23 AM Ghanshyam Mann gmann at ghanshyammann.com> wrote:
 > >   >
 > >   >  ---- On Tue, 06 Jun 2023 12:48:43 -0700  Jeremy Stanley  wrote ---
 > >   >  > On 2023-06-06 12:17:23 -0700 (-0700), Ghanshyam Mann wrote:
 > [snip]
 > >   > I agree. If I see the main overhead in EM maintenance is keeping testing green.
 > >   > it is not easy to keep 11 branches (including Em, supported stable and master)
 > >   > testing up to date. My point is if we remove all the integration testing (can keep pep8
 > >   > and unit tests) at the time the branch move to EM will solve the problem that the upstream
 > >   > community faces to maintain EM branches.
 > >   >
 > >   >
 > >   > This, IMO, is akin to retiring the branches. How could I, as a developer, patch an older version of a branch against a vulnerability of the style of the recent Cinder one, where the impact is felt cross-project, and you clearly need a working dev environment (such as devstack).
 > >   > If, as you propose, we stopped doing any integration testing on branches older than 18 months, we would be de-facto retiring the integration testing infrastructure, which shares a huge amount of DNA with our dev tooling infrastructure.
 > > 
 > > It is not the same as retiring but if we see it can still run unit/functional tests and changes have been
 > > tested till supported and stable so we did testing of those fixes at some level. And there cannot be the
 > > case where I apply the fix directly to the EM branch.
 > 
 > I agree with Jay on this.  IMO, what keeps devstack functional in the EM 
 > branches is that it's needed to run tempest tests.  If we rely on 
 > unit/functional tests only, that motivation goes away.
 > 
 > Further, as Jay points out, a working OpenStack deployment requires a 
 > harmonization of multiple components beyond the individual projects' 
 > unit/functional tests.  For example, this (invalid) bug:
 >    https://bugs.launchpad.net/cinder/+bug/2020382
 > was reported after backporting a patch that had gone through the normal 
 > backport process upstream from master through stable/xena without 
 > skipping any branches; the xena patch applied cleanly and I'm pretty 
 > sure it passed unit and functional tests (I didn't run them myself). 
 > The issue did not occur until the code was actually used by cinder 
 > interacting with a real nova.
 > 
 > So relying on unit and functional tests only is not adequate.  When I 
 > approve a backport, I'm supposed to be fairly confident that the change 
 > is low-risk and will not cause regressions.  Clean tempest jobs give me 
 > some useful evidence when making that assessment.  A patch that passes 
 > CI is not guaranteed to be OK, but if it causes a CI failure, we know 
 > it's not OK.
 > 
 > > In our current doc also, we have the minimum testing expectation and I am just saying to reduce the testing
 > > at the time branch moved to EM instead of waiting for the gate to break and getting frustrated while backporting.
 > > 
 > > EM as we meant since starting it not upstream maintained/guaranteed things so leaving testing expectation at
 > > downstream is no bug change than what current policy is.
 > 
 > That's correct, but as I think has been mentioned elsewhere in this 
 > thread, this has not proved to be workable.  The stable cores on the 
 > project teams take their work seriously, and even though the docs say 
 > that EM branches should be treated caveat emptor, we still feel that our 
 > approval should mean something.  So even though the docs say there's no 
 > guarantee on EM branches, nobody wants to have their name show up as 
 > approving a patch that caused a regression, even in an EM branch.
 > 
 > Further (and I don't think I'm speaking only for myself here), I don't 
 > like the idea of other people merging unvetted stuff into our codebase. 
 > But that hasn't become an issue, because as Jeremy pointed out earlier, 
 > no one outside the project teams has showed up to take ownership of EM 
 > branches.  Though (to get back to my real point here) if such people did 
 > show up, I would expect them to also maintain the full CI, including 
 > tempest integration tests, for the reasons I mentioned earlier.  So I'm 
 > against the idea unit/functional testing is adequate for EM branches.

I do not disagree with Jay and you on more and more testing, but I am saying
reducing testing (which is what the original idea was) is one of the tradeoffs
between keeping extended branches available for fixes for a long time and
upstream maintenance costs.  We are clearly at the stage where the upstream
community cannot maintain them with proper testing. Either we have to remove
the idea of EM or try any new idea that can add more cost in upstream maintenance.

I still do not find it very odd that we do not guarantee the EM backport fixes testing
but at the same time make sure they are tested all the way from master to supported
stable branches backporting. Leave the complete testing to the downstream consumers
to test properly before applying the fixes.

 > 
 > > 
 > > -gmann
 > 
 > (By the way, I am not implying that gmann is in favor of poor QA.  He's 
 > articulated clearly what the current docs say about EM branches.  But 
 > he's also been heroically responsible for keeping a lot of the EM 
 > integration gates functional!)


Apart from maintaining, pinning tempest/plugins version also takes a lot of time.
Now I am starting the pinning tempest/plugins for recent EM stable/xena and
it requires a large amount of time to test/pin the compatible version of tempest
and plugins on stable/xena.

-gmann

 > 
 > >   >
 > >   > I don't know what the answer is; but this as a middle ground seems like the worst of all worlds: the branches still exist, and we will not have the tools to (manually, not just CI) test meaningful changes on them.
 > >   > Just a thought!
 > >   > -Jay FaulknerIronic PTL
 > > 
 > 
 > 
 >