Open Stack

On Wed, Jun 17, 2020, at 11:17 AM, Ghanshyam Mann wrote:
> Hello Everyone,
> 
> As you know devstack (so does grenade) got broken due to uwsgi new 
> release, the master branch is fixed
> and stable branches are in progress[1]. But It is hard to maintain or 
> fix the EM stable for those issues. Especially
> the greande job which depends on the source branch (previous branch of 
> one where the job is running).
> For example, for stein grenade job, we need to keep rocky branch 
> working and fix if failure.
> 
> Till now, we have not removed the grenade testing from any of the EM 
> stable branches because they
> were running fine till now but with uwsgi issues, those are failing and 
> need more work to fix. This triggers
> the discussion of grenade testing on EM stable branches. 
> 
> Usual policy for grenade testing is to keep the job running from the 
> 'oldest supported stable +1' branch.
> For example, if stein is the oldest supported stable (in the old stable 
> definition) then run grenade from train onwards.
> But with the Extended Maintainance model, defining 'oldest supported 
> stable' is not clear whether it is the oldest
> non-EM(stein) or oldest EM stable(ocata).
> 
> To make it clear, we discussed this in the QA channel and come up with 
> the below proposal.
> 
> * 'oldest' is the oldest non-EM. In current time, it is stein.
> * With the above 'oldest' definition, we will:
> ** Make grenade jobs as n-v on all EM stable branches (which is till 
> stable/rocky as of today) + on stable/stein also because that is 
> 'oldest' as of today. 
> ** Keep supporting and running grenade job on 'oldest+1' which is 
> stable/train onwards as of today.
> 
> NOTE: we will make n-v when they start failing and anyone can volunteer 
> to fix them and change back to voting.
> elod expressed interest to work on current failure.

Another important note for if/when there are grenade failures again: fixes to devstack and grenade that affect the grenade job need to be proposed in a "bottom up" fashion. The normal stable backport procedures are the wrong process because grenade uses the previous branch to test the upgrade to the current branch. This means we have to fix the previous branch first.

Instead of backporting we need to "fowardport" which we can do using depends-on between branches to ensure the entire series across all branches functions.

Calling this out because it is a departure from normal operations, but upgrade testing and grenade make it necessary.

> 
> If no objection to the above proposal, I will document it on the 
> grenade documents to follow it whenever we see EM failing and need more 
> work.
> In Tempest, we already have the EM stable testing policy documented 
> which is to support those till they run fine[2].
> 
> [1] 
> http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015496.html
> [2] 
> https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html
> [3] 
> http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-06-17.log.html#t2020-06-17T14:12:42
> 
> -gmann
> 
>