Open Stack

 ---- On Tue, 06 Jun 2023 10:32:32 -0700  Jeremy Stanley  wrote --- 
 > On 2023-06-06 20:01:45 +0530 (+0530), Rajat Dhasmana wrote:
 > [...]
 > > The discussion started because of the CVE fixes where we
 > > backported to active stable branches i.e. Yoga, Zed and 2023.1 but
 > > there were no backports to further EM stable branches like Xena,
 > > Wallaby ... all the way to Train.
 > [...]
 > 
 > The idea behind EM branches was that downstream distributions who
 > need to backport these changes for their own customers would push
 > the patches to the upstream EM branches so that they didn't all have
 > to redo the same work and could benefit from each other's knowledge
 > and experience around backports. If that's not happening for
 > critical security patches, then I agree that the goal of the EM
 > model has failed.
 > 
 > Taking OSSA-2023-003 (CVE-2023-2088) as the most recent example,
 > the advisory and patches for maintained branches were published
 > four weeks ago. Fixes for stable/xena were developed along with the
 > other backports because the branch had only just transitioned to EM
 > a week or two prior. Of the four deliverables which were patched as
 > part of that advisory, only Nova provided a patch for anything
 > older, and that was just to stable/wallaby (and possibly added as a
 > mistake or due to miscommunication between the various groups
 > involved).
 > 
 > In the four weeks since, I haven't seen anyone outside the core
 > review teams for Cinder, Glance or Nova supply changes for
 > additional backports, even though I expect the downstream
 > distributions patched versions contemporary with some branches still
 > in EM. It could be that this vulnerability is a poor example,
 > because a lot of deployments use RBD by default and it wasn't
 > affected, but the situation with the two other advisories earlier in
 > the year wasn't much different where backports were concerned.
 > 
 > > 1) We have less review bandwidth even for active stable branches
 > > (Yoga, Zed and 2023.1)
 > 
 > The intent for EM was that reviewing branches no longer under normal
 > maintenance could be delegated to other members of the community. Of
 > course, that was the idea with stable maintenance as well. Perhaps
 > it would be more accurate to restate this as there are no volunteers
 > to review the changes (it shouldn't be the core review team's
 > obligation either way)?
 > 
 > > 2) No one, apart from the project team, does backport of critical
 > > fixes implying that those branches aren't used much for
 > > collaboration
 > 
 > This is the best rationale for scrapping the whole EM idea, in my
 > opinion. There's a possibility that if the core reviewers weren't
 > pushing backports someone else would have done so eventually, but I
 > think we have plenty of evidence now to indicate that doesn't really
 > happen in practice.
 > 
 > > 3) Will save gate resources for some periodic jobs and the patches
 > > proposed
 > 
 > EM branches don't have to run the same jobs as maintained stable
 > branches (or really any jobs at all), but that does still at a
 > minimum need the attention of someone interested in removing the
 > unwanted jobs.
 > 
 > > 4) Save project team's time to fix gate issues
 > [...]
 > 
 > Similar to the earlier points, the project team shouldn't feel
 > obligated to fix testing issues on EM branches. If testing breaks,
 > it should be up to the volunteers proposing and reviewing changes to
 > do that. If they don't, nothing will merge. If they don't care
 > enough to keep it possible to merge things, then sure that's
 > basically back to item #1 again.

This is true but I think this is the point where things are becoming difficult. Even
we do not need to but we as community developers keep fixing the EM gate, at
least I can tell from my QA experience for this. We should stop at some line but
in reality, we end up doing it.

IMO, we should do some policies and testing changes which can help to understand
the EM clearly and spend only the required time on those. A few of the ideas are:

1. Reduce the number of EM count, currently we have 7 EM branches. we should make
it a little less say 4 or 5 as max any time. And after the max count limit (say 4) we will make
the last one EOL. Having 4 EM branches means 2 years of extended support which is good
enough.

2. Completely remove all the integration test jobs even if they are passing at the time branch
is moving to EM state. This way project team will get less frustrated by seeing the gate failure
on backport. This will simply ask them to backport it and make it available for downstream
consumers who will test it properly before use. Until we keep the integration testing 
because it was passing today make more work for future maintenance. 

-gmann


 > -- 
 > Jeremy Stanley
 >