Open Stack

Hello Everyone,

I am summarizing the discussion QA team had in PTG and action items for quick ref.

Victoria Retrospective:
-----------------------------
We talked about the retrospective of the last cycle, one main challenge we discussed is the slow review in QA
due to fewer contributors. Below are the action items that came up from improving things:
* Action:
- microversion need more work also when you try more combination or latest (gmann).
- Triage bugs in the office hour as 30 min -> update agenda
- Speed up the review.

Office hour schedule:
---------------------------
With daylight ending in Nov, we decided to shift the QA office hour by an hour late which is at 14 UTC.

Move back the single horizon test in tempest tree from tempest-horizon:
-------------------------------------------------------------------------------------------
Tempest and Horizon team decided to bring back the single horizon test to Tempest tree and retire the
tempest-horizon plugins which was too much maintenance for single tests. It is unlikely to happen but if
Tempest team in future plan to remove this test then it needs to be consulted with Horizon team first.
* Action:
- Move test back to Tempest - gmann
- Deprecate and retire tempest-horizon - amotoki
- Check with the release team if any other release needed. (amotoki + gmann)
- modify the 'horizon-dsvm-tempest-plugin' job to direct to tests' new location

Add run_validation as a centralized point for Tempest scenario tests:
-------------------------------------------------------------------------------------
For now, validation resources automation and skip ssh part via run_validation is done in API test only and this
proposal is to extend it to scenario tests also. There is no reason not to do that which will help to run tempest
scenario tests on images where ssh is not allowed. There is a most possible situation where complete tests
need to be skipped but we will see case by case and at least skipping automatically will help tester to avoid
explicitly excluding the scenario tests to the skip list.
*Action:
- Implement it in scenario tests - paras333
- Refactor scenario manager's create_server method to consume common/compute.py - paras333

Patrole stable release:
----------------------------
We are lacking the maintainer in Patrole project which is a big challenge to release a stable version of it. Sonia
and doug (already helping) will try to spend more bandwidth in Patrole. Another
thing to reduce the Patrole tests execution time. We discussed a few options like policy engine to return the
API based on the flag or use fake driver etc but we did not conclude any option yet as they need more investigation.
* Action:
- Keep eyes on gate and how tests are running and based on that we can make a stable release for Wallaby. - gmann
- Keep exploring the options for making patrole lightweight. - gmann, doug


Grenade: extending  the testing for zero downtime upgrade testing:
------------------------------------------------------------------------------------
This is just a thought. There is TC tag 'assert:supports-zero-downtime-upgrade' which is not
used by any of the projects and also we do not have any testing framework which can verify this
tag. We talked about it if we can do such testing in grenade or not. Before moving forward on
discussion and investigation we checked if anyone can volunteer for this work. As of now, no
volunteer.

Use different guest image for gate jobs to run tempest tests:
---------------------------------------------------------------------------
We only test cirros guest image to create the test VM and all proposal from Paras is to try more
images to enhance the different configuration scenarios in upstream testing. This will help
to able to catch more failure/scenario at the upstream gate itself compare to the current situation where
most of them are reported from downstream testing.
* Action:
- Add a job as experimental - paras333

Tempest Cred Manager proposed changes:
-----------------------------------------------------
'primary' and 'alt_promary' credentials in Tempest are hardcoded to non-admin and they are assigned
configured 'tempest_role'. There is no way we can assign a different role to both of these creds. The idea here
is to make 'primary' and 'alt_promary' credentials configurable so that different deployment can configure
these with different roles in various combinations. We will be adding two config option similar to
 'tempest_role' and default to an empty list so that we continue the default behavior with what we have currently.
So that there is no backward-incompatible change instead it is an additional new thing provided. 
* Action:
- Doug needs to propose the patch.

Tempest Scenario manager stable version brainstorming:
-----------------------------------------------------------------------
We talked about a few method changes to make the scenario manager stable. If any method is used among plugins
only and not in Tempest then we do not actually need to move it to Tempest and they can stay on the side of the plugin itself.
We ran out of time here and will continue the brainstorming in office hours or so. 
(this etherpad - https://etherpad.opendev.org/p/tempest-scenario-manager)
* Action: 
- Sonia and Martin will continue working on this.

Wallaby Priority & Planning:
------------------------------------
We did not prioritize the things as such but listed all working items in below etherpad along with Victoria cycle backlogs.
- https://etherpad.opendev.org/p/qa-wallaby-ptg

Detailed discussions are in this Etherpad: https://etherpad.opendev.org/p/qa-wallaby-ptg

-gmann