Open Stack

On 6/7/23 1:46 PM, Ghanshyam Mann wrote:
>   ---- On Wed, 07 Jun 2023 10:38:03 -0700  Jay Faulkner  wrote ---
>   >
>   > On Wed, Jun 7, 2023 at 10:23 AM Ghanshyam Mann gmann at ghanshyammann.com> wrote:
>   >
>   >  ---- On Tue, 06 Jun 2023 12:48:43 -0700  Jeremy Stanley  wrote ---
>   >  > On 2023-06-06 12:17:23 -0700 (-0700), Ghanshyam Mann wrote:
[snip]
>   > I agree. If I see the main overhead in EM maintenance is keeping testing green.
>   > it is not easy to keep 11 branches (including Em, supported stable and master)
>   > testing up to date. My point is if we remove all the integration testing (can keep pep8
>   > and unit tests) at the time the branch move to EM will solve the problem that the upstream
>   > community faces to maintain EM branches.
>   >
>   >
>   > This, IMO, is akin to retiring the branches. How could I, as a developer, patch an older version of a branch against a vulnerability of the style of the recent Cinder one, where the impact is felt cross-project, and you clearly need a working dev environment (such as devstack).
>   > If, as you propose, we stopped doing any integration testing on branches older than 18 months, we would be de-facto retiring the integration testing infrastructure, which shares a huge amount of DNA with our dev tooling infrastructure.
> 
> It is not the same as retiring but if we see it can still run unit/functional tests and changes have been
> tested till supported and stable so we did testing of those fixes at some level. And there cannot be the
> case where I apply the fix directly to the EM branch.

I agree with Jay on this.  IMO, what keeps devstack functional in the EM 
branches is that it's needed to run tempest tests.  If we rely on 
unit/functional tests only, that motivation goes away.

Further, as Jay points out, a working OpenStack deployment requires a 
harmonization of multiple components beyond the individual projects' 
unit/functional tests.  For example, this (invalid) bug:
   https://bugs.launchpad.net/cinder/+bug/2020382
was reported after backporting a patch that had gone through the normal 
backport process upstream from master through stable/xena without 
skipping any branches; the xena patch applied cleanly and I'm pretty 
sure it passed unit and functional tests (I didn't run them myself). 
The issue did not occur until the code was actually used by cinder 
interacting with a real nova.

So relying on unit and functional tests only is not adequate.  When I 
approve a backport, I'm supposed to be fairly confident that the change 
is low-risk and will not cause regressions.  Clean tempest jobs give me 
some useful evidence when making that assessment.  A patch that passes 
CI is not guaranteed to be OK, but if it causes a CI failure, we know 
it's not OK.

> In our current doc also, we have the minimum testing expectation and I am just saying to reduce the testing
> at the time branch moved to EM instead of waiting for the gate to break and getting frustrated while backporting.
> 
> EM as we meant since starting it not upstream maintained/guaranteed things so leaving testing expectation at
> downstream is no bug change than what current policy is.

That's correct, but as I think has been mentioned elsewhere in this 
thread, this has not proved to be workable.  The stable cores on the 
project teams take their work seriously, and even though the docs say 
that EM branches should be treated caveat emptor, we still feel that our 
approval should mean something.  So even though the docs say there's no 
guarantee on EM branches, nobody wants to have their name show up as 
approving a patch that caused a regression, even in an EM branch.

Further (and I don't think I'm speaking only for myself here), I don't 
like the idea of other people merging unvetted stuff into our codebase. 
But that hasn't become an issue, because as Jeremy pointed out earlier, 
no one outside the project teams has showed up to take ownership of EM 
branches.  Though (to get back to my real point here) if such people did 
show up, I would expect them to also maintain the full CI, including 
tempest integration tests, for the reasons I mentioned earlier.  So I'm 
against the idea unit/functional testing is adequate for EM branches.

> 
> -gmann

(By the way, I am not implying that gmann is in favor of poor QA.  He's 
articulated clearly what the current docs say about EM branches.  But 
he's also been heroically responsible for keeping a lot of the EM 
integration gates functional!)

>   >
>   > I don't know what the answer is; but this as a middle ground seems like the worst of all worlds: the branches still exist, and we will not have the tools to (manually, not just CI) test meaningful changes on them.
>   > Just a thought!
>   > -Jay FaulknerIronic PTL
>