[qa][form][ptg] QA Summary for Forum & PTG

Ghanshyam Mann gmann at ghanshyammann.com
Mon Nov 25 00:17:59 UTC 2019

Hello Everyone,

I am summarizing the QA PTG and Forum sessions held in Shanghai.

* OpenStack QA – Project Update: 

We gave the updates on what we finished on Train and draft plan for the Ussuri cycle.
due to fewer contributors in QA, Train cycle activities are decreased as compare to Stein.  We tried to maintain the daily QA
activity and finished a few important things. Slides are uploaded on the schedule site.

* Users / Operators adoption of QA tools / plugins :

Etherpad: https://etherpad.openstack.org/p/PVG-forum-qa-ops-user-feedback.
This is another useful session for QA to get feedback as well as information about downstream tooling.
Few tools we talked about:
- Fault injection tests
- https://opendev.org/x/tobiko/
- whitebox testing(Whitebox Tempest Plugin)
- high availability testing: https://review.opendev.org/#/c/443504/
Run patrole jobs in the neutron upstream(Keystone was interested before)
One big concern shared from a few people about a long time to get merged tempest patches. One idea to solve this is to bring critical reviews in Office hours.

It was a small gathering this time for one day for PTG on Wednesday. Even with the small number of developers, we had good discussions on many topics. 

Etherpad: https://etherpad.openstack.org/p/shanghai-ptg-qa

* Train Retrospective:

Retrospective bought up the few key issues where we need improvement. We collected the below action items including bug triage. Untriage QA bugs are increasing day by day.
- need to discuss blacklist plugins and how to notify and remove them if dead - gmann
- start the process of community-goal work in QA - masayuki
- sprint for bug triage with number of volunteers - 
- (chandankumar)Include one bug in each sprint in TripleO CI tempest member
- Traige the new bug and then pick the bug based on priority
- For tripleo Ci team we will track here: https://tree.taiga.io/project/tripleo-ci-board/ - chandankumar

* How to deal with an aging testing stack:

With testtools being not so active, we need to think on the alternate or best suitable options to solve this issue. We discussed the few options which need to be discussed further on ML.
- Can we fork the dependecies of testtools in Temepst or stestr ? 
- As we are removing the py2.7 support in tempest, we can completly ignore/remove the unittest2 things but that is not case for testtools ?
- Similar example done in different project: https://github.com/plone/Products.CMFPlone/issues/1882
- Remove the support of unittest2 from testtools ? py2.7 is going away from everywhere and testools can create tag or something for py2.7 usage ?
- Since Python2 is going EOL on 01st Jan, 2020, so let's create a tag and remove the unitest2 with unitest for python3 release only
- Document the official supported test runner by Tempest. -  Soniya Vyas/Chandan Kumar
- ML to discuss the above options - gmann 

* Remove/migrate the .testr.conf to .stestr:

60 openstack/* repositories have .stestr.conf AND .testr.conf. We don't need to have both files at least. Let's take a look some of them and make a plan to remove if we can.
- http://paste.openstack.org/show/785832/
- If both exist then remove the .testr.conf and Then verify that .stestr conf has the correct test path. If only .testr.conf then migrate to stestr.conf
- We need to figure out the purpose of pbr .testr.conf code before removing. Is this just old codes or necessary?
   - https://opendev.org/openstack/pbr/src/branch/master/pbr/hooks/commands.py#L53
   - https://opendev.org/openstack/pbr/src/branch/master/pbr/testr_command.py

* Moving subunit2html script from os-testr:

Since os-testr runner piece in os-testr project is deprecated but subunit2html project still exists there, it is widely used across the OpenStack ecosystem, Can we move to somewhere else?  I do not find any benefits to move those scripts to other places. We asked chandan to open an issue on stestr to discuss moving to stestr repo. mtreinish replied on this: os-testr meant to be the place in openstack that we could host the ostestr runner wrapper/script subunit2html, generate_subunit, etc. Just because ostestr is deprecated and being removed doesn't mean it's not the proper home for those other tools.

* Separate integrated services tests can be used in TriplO CI:

TriplO CI maintains a separate file to run dependent tests per service. Tempest has dependent services tox and integrated jobs and the same can be used in TriplO CI.
For example:
- blacklist file for networking testing [1].
- tox for networking[2].

* RBAC testing strategy:

This was a cross-project strategy for positive/negative testing for system scope and new defaults in keystone. Keystone has implemented the new defaults and system scope in its policy and added a unit test to cover the new policies.  Nova is implementing the same in Ussuri cycle. As discussed in Denver PTG also, Tempest will implement the new credential for all 9 personas available in keystone.  Slowly migrate the tests start using the new policies. That will be done via a flag switching Tempest to use system scope or new defaults and that flag will be false to keep using the old policies for stable branch testing.

We can use patrole tests or implement new tests in the Tempest plugin and verify the response. Both have the issue of performing the complete operation which is not required always for policy verification.  Running full functional tests is expensive and duplicates existing tests. One solution for that (we talked about it in Denver PTG also) is via some flag like os-profiler by just do the policy check and return the API response with specific return code.
- Tempest to provide the all 9 personas available from keystone. Slowly migrate Tempest existing tests to run with new policies.
- We agreed to have two ways to test the policy:
- Tempest like tests in tempest plugins with the complete operation and verify the things on response, not just policy return code. It depends on the project if they want to implement such tests.
- Unit/Functional tests on the project's side.
- Document the both way so that project can adopt the best suitable one.

* How to remove tempest plugin sanity BLACKLIST:

We have Tempest plugin blacklist. It should be removed in the future if possible. Some of them shouldn't be as a tempest-plugin because they're just neutron studium things which already moved to neutron-tempest-plugin but still exiting in repo also. Some of them are less active.  Remove below plugins from BLACKLIST:
- x/group-based-policy - https://opendev.org/x/group-based-policy 
- openstack/networking-generic-switch needs to be checked (setup.py/cfg?)
- Add the start date in blacklist doc so that we can know how long a plugin is blacklisted. 
- After 60 days: we send the email notification to openstack-discuss, PTL, maitainer and TC to either fix it or remove it from the governance. 

* Python 2.7 drop plan:

We discussed the next steps to drop the py2 from Tempest and other QA tools.
- Will be doing before milestone 2
- Create a new tag for python-2.7 saying it is the last tag and document that the Tempest tag needs Train u-c. 
- Test the Tempest tag with Train u-c, if fail then we will disucss. 
- TripleO and OSA is going to use CentOS 8 for train and master

* Adding New glance tests to Tempest:

We discussed on testing the new glance v2 api and feature. Below are the glance features and agreed points on how to test them.
- Hide old images: Test can be added in Tempest. Hide the image and try to boot the server from the image in scenario tests. 
- Delete barbican secrets from glance images: This test belongs to barbican-tempest-plugin which can be run as part of the barbican gate using an existing job. Running barbican job on glance gate is not required, we can add a new job (multi stores) on glance gate which can run this + other new features tests. 
- Multiple stores: DevStack patch is already up, add a new zuul job to set up multiple stores and run on the glance gate with api and scenario tests. gmann to setup the zuulv3 job for that.

* Tempest volunteers for reviewing patches:

We've noticed that the amount of merged patches in October is less than in September and much less than it was during the summer. This has been brought in feedback sessions also. There is no perfect solution for this. Nowadays QA has less active core developers. We encourage people to bring up the critical or stuck patches in office hours.

* Improving Tempest cleanup:

Tempest cleanup is not so stable and not a perfect design. We have spec up to redesign that but could not get a consensus on that. I am ok to move with resource prefix with UUID.  We should extend the cleanup tool for plugins also.

* Ussuri Priority & Planning:

This was the last session for the PTG which could not happen on Wed due to strict time-up policy of the conference place which I really liked. Time-based working is much needed for IT people :). We met on Thursday morning in coffee area and discussed about priority for Ussuri cycle. QA Ussuri Priority Etherpad[3] has the priority items with the assignee.

[1] https://github.com/openstack/tempest/blob/9cdd5250615bb6ab26a1a9a80743a03cc81b3b4a/tools/tempest-integrated-gate-networking-blacklist.txt
[2] https://github.com/openstack/tempest/blob/9cdd5250615bb6ab26a1a9a80743a03cc81b3b4a/tox.ini#L121
[3] https://etherpad.openstack.org/p/qa-ussuri-priority


More information about the openstack-discuss mailing list