[all][tc][stable][qa] Grenade testing for Extended Maintenance stable
Hello Everyone, As you know devstack (so does grenade) got broken due to uwsgi new release, the master branch is fixed and stable branches are in progress[1]. But It is hard to maintain or fix the EM stable for those issues. Especially the greande job which depends on the source branch (previous branch of one where the job is running). For example, for stein grenade job, we need to keep rocky branch working and fix if failure. Till now, we have not removed the grenade testing from any of the EM stable branches because they were running fine till now but with uwsgi issues, those are failing and need more work to fix. This triggers the discussion of grenade testing on EM stable branches. Usual policy for grenade testing is to keep the job running from the 'oldest supported stable +1' branch. For example, if stein is the oldest supported stable (in the old stable definition) then run grenade from train onwards. But with the Extended Maintainance model, defining 'oldest supported stable' is not clear whether it is the oldest non-EM(stein) or oldest EM stable(ocata). To make it clear, we discussed this in the QA channel and come up with the below proposal. * 'oldest' is the oldest non-EM. In current time, it is stein. * With the above 'oldest' definition, we will: ** Make grenade jobs as n-v on all EM stable branches (which is till stable/rocky as of today) + on stable/stein also because that is 'oldest' as of today. ** Keep supporting and running grenade job on 'oldest+1' which is stable/train onwards as of today. NOTE: we will make n-v when they start failing and anyone can volunteer to fix them and change back to voting. elod expressed interest to work on current failure. If no objection to the above proposal, I will document it on the grenade documents to follow it whenever we see EM failing and need more work. In Tempest, we already have the EM stable testing policy documented which is to support those till they run fine[2]. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015496.html [2] https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html [3] http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-... -gmann
[...]
Usual policy for grenade testing is to keep the job running from the 'oldest supported stable +1' branch. For example, if stein is the oldest supported stable (in the old stable definition) then run grenade from train onwards. But with the Extended Maintainance model, defining 'oldest supported stable' is not clear whether it is the oldest non-EM(stein) or oldest EM stable(ocata).
To make it clear, we discussed this in the QA channel and come up with the below proposal.
* 'oldest' is the oldest non-EM. In current time, it is stein. * With the above 'oldest' definition, we will: ** Make grenade jobs as n-v on all EM stable branches (which is till stable/rocky as of today) + on stable/stein also because that is 'oldest' as of today. ** Keep supporting and running grenade job on 'oldest+1' which is stable/train onwards as of today. [...]
The way to phrase this consistent with our branch status terminology is: We only perform upgrade testing between the current source contents of adjacent Maintained or Development status branches (not Extended Maintenance or Unmaintained status branches, nor specific tags such as End Of Life versions). This means that the branch *prior* to any branch Grenade tests must be in a Maintained status, and so we do not perform upgrade testing on the oldest Maintained status branch. -- Jeremy Stanley
On Wed, Jun 17, 2020, at 11:17 AM, Ghanshyam Mann wrote:
Hello Everyone,
As you know devstack (so does grenade) got broken due to uwsgi new release, the master branch is fixed and stable branches are in progress[1]. But It is hard to maintain or fix the EM stable for those issues. Especially the greande job which depends on the source branch (previous branch of one where the job is running). For example, for stein grenade job, we need to keep rocky branch working and fix if failure.
Till now, we have not removed the grenade testing from any of the EM stable branches because they were running fine till now but with uwsgi issues, those are failing and need more work to fix. This triggers the discussion of grenade testing on EM stable branches.
Usual policy for grenade testing is to keep the job running from the 'oldest supported stable +1' branch. For example, if stein is the oldest supported stable (in the old stable definition) then run grenade from train onwards. But with the Extended Maintainance model, defining 'oldest supported stable' is not clear whether it is the oldest non-EM(stein) or oldest EM stable(ocata).
To make it clear, we discussed this in the QA channel and come up with the below proposal.
* 'oldest' is the oldest non-EM. In current time, it is stein. * With the above 'oldest' definition, we will: ** Make grenade jobs as n-v on all EM stable branches (which is till stable/rocky as of today) + on stable/stein also because that is 'oldest' as of today. ** Keep supporting and running grenade job on 'oldest+1' which is stable/train onwards as of today.
NOTE: we will make n-v when they start failing and anyone can volunteer to fix them and change back to voting. elod expressed interest to work on current failure.
Another important note for if/when there are grenade failures again: fixes to devstack and grenade that affect the grenade job need to be proposed in a "bottom up" fashion. The normal stable backport procedures are the wrong process because grenade uses the previous branch to test the upgrade to the current branch. This means we have to fix the previous branch first. Instead of backporting we need to "fowardport" which we can do using depends-on between branches to ensure the entire series across all branches functions. Calling this out because it is a departure from normal operations, but upgrade testing and grenade make it necessary.
If no objection to the above proposal, I will document it on the grenade documents to follow it whenever we see EM failing and need more work. In Tempest, we already have the EM stable testing policy documented which is to support those till they run fine[2].
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015496.html [2] https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html [3] http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-...
-gmann
---- On Wed, 17 Jun 2020 13:17:26 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
As you know devstack (so does grenade) got broken due to uwsgi new release, the master branch is fixed and stable branches are in progress[1]. But It is hard to maintain or fix the EM stable for those issues. Especially the greande job which depends on the source branch (previous branch of one where the job is running). For example, for stein grenade job, we need to keep rocky branch working and fix if failure.
Till now, we have not removed the grenade testing from any of the EM stable branches because they were running fine till now but with uwsgi issues, those are failing and need more work to fix. This triggers the discussion of grenade testing on EM stable branches.
Usual policy for grenade testing is to keep the job running from the 'oldest supported stable +1' branch. For example, if stein is the oldest supported stable (in the old stable definition) then run grenade from train onwards. But with the Extended Maintainance model, defining 'oldest supported stable' is not clear whether it is the oldest non-EM(stein) or oldest EM stable(ocata).
To make it clear, we discussed this in the QA channel and come up with the below proposal.
* 'oldest' is the oldest non-EM. In current time, it is stein. * With the above 'oldest' definition, we will: ** Make grenade jobs as n-v on all EM stable branches (which is till stable/rocky as of today) + on stable/stein also because that is 'oldest' as of today. ** Keep supporting and running grenade job on 'oldest+1' which is stable/train onwards as of today.
NOTE: we will make n-v when they start failing and anyone can volunteer to fix them and change back to voting. elod expressed interest to work on current failure.
If no objection to the above proposal, I will document it on the grenade documents to follow it whenever we see EM failing and need more work. In Tempest, we already have the EM stable testing policy documented which is to support those till they run fine[2].
I have pushed the patches making grenade jobs non-voting on stable/stein[1]. With that patch merge in your project, it will bring back the stable/stein onwards gate green. Once '<= stable/rocky] gate is up you can backport these grenade-n-v patches. Also, I am documenting the process in grenade for future reference. - https://review.opendev.org/#/c/736866/ [1] https://review.opendev.org/#/q/topic:grenade-em-nv+(status:open+OR+status:me...) -gmann
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015496.html [2] https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html [3] http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-...
-gmann
Hi, any updates on the older branches? Specifically, we’re interested in queens. We have a patch [1] that waits for it. [1] https://review.opendev.org/#/c/736129/ Thanks Renat Akhmerov @Nokia On 23 Jun 2020, 23:17 +0700, Ghanshyam Mann <gmann@ghanshyammann.com>, wrote:
---- On Wed, 17 Jun 2020 13:17:26 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
As you know devstack (so does grenade) got broken due to uwsgi new release, the master branch is fixed and stable branches are in progress[1]. But It is hard to maintain or fix the EM stable for those issues. Especially the greande job which depends on the source branch (previous branch of one where the job is running). For example, for stein grenade job, we need to keep rocky branch working and fix if failure.
Till now, we have not removed the grenade testing from any of the EM stable branches because they were running fine till now but with uwsgi issues, those are failing and need more work to fix. This triggers the discussion of grenade testing on EM stable branches.
Usual policy for grenade testing is to keep the job running from the 'oldest supported stable +1' branch. For example, if stein is the oldest supported stable (in the old stable definition) then run grenade from train onwards. But with the Extended Maintainance model, defining 'oldest supported stable' is not clear whether it is the oldest non-EM(stein) or oldest EM stable(ocata).
To make it clear, we discussed this in the QA channel and come up with the below proposal.
* 'oldest' is the oldest non-EM. In current time, it is stein. * With the above 'oldest' definition, we will: ** Make grenade jobs as n-v on all EM stable branches (which is till stable/rocky as of today) + on stable/stein also because that is 'oldest' as of today. ** Keep supporting and running grenade job on 'oldest+1' which is stable/train onwards as of today.
NOTE: we will make n-v when they start failing and anyone can volunteer to fix them and change back to voting. elod expressed interest to work on current failure.
If no objection to the above proposal, I will document it on the grenade documents to follow it whenever we see EM failing and need more work. In Tempest, we already have the EM stable testing policy documented which is to support those till they run fine[2].
I have pushed the patches making grenade jobs non-voting on stable/stein[1]. With that patch merge in your project, it will bring back the stable/stein onwards gate green. Once '<= stable/rocky] gate is up you can backport these grenade-n-v patches.
Also, I am documenting the process in grenade for future reference. - https://review.opendev.org/#/c/736866/
[1] https://review.opendev.org/#/q/topic:grenade-em-nv+(status:open+OR+status:me...)
-gmann
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015496.html [2] https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html [3] http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-...
-gmann
Hi, The backported uwsgi fixes cannot merge yet, as Tempest jobs are failing. (As an ansible post_run role in Tempest started to fail, too, - in Rocky and older branches - after the new base images were introduced.) Gmann prepared a workaround [1], unfortunately it does not work as expected (as far as I understand) and still needs modification in Tempest. Which is a catch 22, as the root of the issue is that Tempest cannot be modified (in Rocky, Queens, ...), as it does not have stable branches, just a master branch. But the stable branches in Extended Maintenance are pinned to an older tag from Tempest and we cannot modify that version of Tempest. In my opinion, one solution could be to cut a (stable?) branch from Tempest that can be used for branches currently in Extended Maintenance (rocky, queens, ...) and fix there the failing role. I know that Tempest is branchless, but can we have an exception somehow, just to be able to fix the Rocky, Queens, Pike branches? [1] https://review.opendev.org/#/q/topic:move-stackviz-role Thanks, Előd On 2020. 06. 25. 12:03, Renat Akhmerov wrote:
Hi, any updates on the older branches? Specifically, we’re interested in queens. We have a patch [1] that waits for it.
[1] https://review.opendev.org/#/c/736129/
Thanks
Renat Akhmerov @Nokia On 23 Jun 2020, 23:17 +0700, Ghanshyam Mann <gmann@ghanshyammann.com>, wrote:
---- On Wed, 17 Jun 2020 13:17:26 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
As you know devstack (so does grenade) got broken due to uwsgi new release, the master branch is fixed and stable branches are in progress[1]. But It is hard to maintain or fix the EM stable for those issues. Especially the greande job which depends on the source branch (previous branch of one where the job is running). For example, for stein grenade job, we need to keep rocky branch working and fix if failure.
Till now, we have not removed the grenade testing from any of the EM stable branches because they were running fine till now but with uwsgi issues, those are failing and need more work to fix. This triggers the discussion of grenade testing on EM stable branches.
Usual policy for grenade testing is to keep the job running from the 'oldest supported stable +1' branch. For example, if stein is the oldest supported stable (in the old stable definition) then run grenade from train onwards. But with the Extended Maintainance model, defining 'oldest supported stable' is not clear whether it is the oldest non-EM(stein) or oldest EM stable(ocata).
To make it clear, we discussed this in the QA channel and come up with the below proposal.
* 'oldest' is the oldest non-EM. In current time, it is stein. * With the above 'oldest' definition, we will: ** Make grenade jobs as n-v on all EM stable branches (which is till stable/rocky as of today) + on stable/stein also because that is 'oldest' as of today. ** Keep supporting and running grenade job on 'oldest+1' which is stable/train onwards as of today.
NOTE: we will make n-v when they start failing and anyone can volunteer to fix them and change back to voting. elod expressed interest to work on current failure.
If no objection to the above proposal, I will document it on the grenade documents to follow it whenever we see EM failing and need more work. In Tempest, we already have the EM stable testing policy documented which is to support those till they run fine[2].
I have pushed the patches making grenade jobs non-voting on stable/stein[1]. With that patch merge in your project, it will bring back the stable/stein onwards gate green. Once '<= stable/rocky] gate is up you can backport these grenade-n-v patches.
Also, I am documenting the process in grenade for future reference. - https://review.opendev.org/#/c/736866/
[1] https://review.opendev.org/#/q/topic:grenade-em-nv+(status:open+OR+status:me...)
-gmann
[1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015496.html [2] https://docs.openstack.org/tempest/latest/stable_branch_support_policy.html [3] http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2020-...
-gmann
participants (5)
-
Clark Boylan
-
Előd Illés
-
Ghanshyam Mann
-
Jeremy Stanley
-
Renat Akhmerov