Open Stack

 ---- On Thu, 08 Jun 2023 06:39:28 -0700  Brian Rosmaita  wrote --- 
 > On 6/7/23 1:46 PM, Ghanshyam Mann wrote:
 > >   ---- On Wed, 07 Jun 2023 10:38:03 -0700  Jay Faulkner  wrote ---
 > >   >
 > >   > On Wed, Jun 7, 2023 at 10:23 AM Ghanshyam Mann gmann at ghanshyammann.com> wrote:
 > >   >
 > >   >  ---- On Tue, 06 Jun 2023 12:48:43 -0700  Jeremy Stanley  wrote ---
 > >   >  > On 2023-06-06 12:17:23 -0700 (-0700), Ghanshyam Mann wrote:
 > [snip]
 > >   > I agree. If I see the main overhead in EM maintenance is keeping testing green.
 > >   > it is not easy to keep 11 branches (including Em, supported stable and master)
 > >   > testing up to date. My point is if we remove all the integration testing (can keep pep8
 > >   > and unit tests) at the time the branch move to EM will solve the problem that the upstream
 > >   > community faces to maintain EM branches.
 > >   >
 > >   >
 > >   > This, IMO, is akin to retiring the branches. How could I, as a developer, patch an older version of a branch against a vulnerability of the style of the recent Cinder one, where the impact is felt cross-project, and you clearly need a working dev environment (such as devstack).
 > >   > If, as you propose, we stopped doing any integration testing on branches older than 18 months, we would be de-facto retiring the integration testing infrastructure, which shares a huge amount of DNA with our dev tooling infrastructure.
 > > 
 > > It is not the same as retiring but if we see it can still run unit/functional tests and changes have been
 > > tested till supported and stable so we did testing of those fixes at some level. And there cannot be the
 > > case where I apply the fix directly to the EM branch.
 > 
 > I agree with Jay on this.  IMO, what keeps devstack functional in the EM 
 > branches is that it's needed to run tempest tests.  If we rely on 
 > unit/functional tests only, that motivation goes away.
 > 
 > Further, as Jay points out, a working OpenStack deployment requires a 
 > harmonization of multiple components beyond the individual projects' 
 > unit/functional tests.  For example, this (invalid) bug:
 >    https://bugs.launchpad.net/cinder/+bug/2020382
 > was reported after backporting a patch that had gone through the normal 
 > backport process upstream from master through stable/xena without 
 > skipping any branches; the xena patch applied cleanly and I'm pretty 
 > sure it passed unit and functional tests (I didn't run them myself). 
 > The issue did not occur until the code was actually used by cinder 
 > interacting with a real nova.
 > 
 > So relying on unit and functional tests only is not adequate.  When I 
 > approve a backport, I'm supposed to be fairly confident that the change 
 > is low-risk and will not cause regressions.  Clean tempest jobs give me 
 > some useful evidence when making that assessment.  A patch that passes 
 > CI is not guaranteed to be OK, but if it causes a CI failure, we know 
 > it's not OK.
 > 
 > > In our current doc also, we have the minimum testing expectation and I am just saying to reduce the testing
 > > at the time branch moved to EM instead of waiting for the gate to break and getting frustrated while backporting.
 > > 
 > > EM as we meant since starting it not upstream maintained/guaranteed things so leaving testing expectation at
 > > downstream is no bug change than what current policy is.
 > 
 > That's correct, but as I think has been mentioned elsewhere in this 
 > thread, this has not proved to be workable.  The stable cores on the 
 > project teams take their work seriously, and even though the docs say 
 > that EM branches should be treated caveat emptor, we still feel that our 
 > approval should mean something.  So even though the docs say there's no 
 > guarantee on EM branches, nobody wants to have their name show up as 
 > approving a patch that caused a regression, even in an EM branch.
 > 
 > Further (and I don't think I'm speaking only for myself here), I don't 
 > like the idea of other people merging unvetted stuff into our codebase. 
 > But that hasn't become an issue, because as Jeremy pointed out earlier, 
 > no one outside the project teams has showed up to take ownership of EM 
 > branches.  Though (to get back to my real point here) if such people did 
 > show up, I would expect them to also maintain the full CI, including 
 > tempest integration tests, for the reasons I mentioned earlier.  So I'm 
 > against the idea unit/functional testing is adequate for EM branches.

I do not disagree with Jay and you on more and more testing, but I am saying
reducing testing (which is what the original idea was) is one of the tradeoffs
between keeping extended branches available for fixes for a long time and
upstream maintenance costs.  We are clearly at the stage where the upstream
community cannot maintain them with proper testing. Either we have to remove
the idea of EM or try any new idea that can add more cost in upstream maintenance.

I still do not find it very odd that we do not guarantee the EM backport fixes testing
but at the same time make sure they are tested all the way from master to supported
stable branches backporting. Leave the complete testing to the downstream consumers
to test properly before applying the fixes.

 > 
 > > 
 > > -gmann
 > 
 > (By the way, I am not implying that gmann is in favor of poor QA.  He's 
 > articulated clearly what the current docs say about EM branches.  But 
 > he's also been heroically responsible for keeping a lot of the EM 
 > integration gates functional!)


Apart from maintaining, pinning tempest/plugins version also takes a lot of time.
Now I am starting the pinning tempest/plugins for recent EM stable/xena and
it requires a large amount of time to test/pin the compatible version of tempest
and plugins on stable/xena.

-gmann

 > 
 > >   >
 > >   > I don't know what the answer is; but this as a middle ground seems like the worst of all worlds: the branches still exist, and we will not have the tools to (manually, not just CI) test meaningful changes on them.
 > >   > Just a thought!
 > >   > -Jay FaulknerIronic PTL
 > > 
 > 
 > 
 >