Open Stack

Hello Everyone,

We had a good discussion at QA forum and PTG. I am summarizing those and will start the
separate thread for the few topics which needs more feedback.

Summit: QA Forum sessions:

1. OpenStack QA - Project Update:  Tuesday, April 30, 2:35pm-2:55pm
We gave the updates on what we finished on Stein and draft plan for Train cycle. The good thing to note
is we still have a lot of activity going on in QA. As overall QA projects, we did >3000 reviews and 750 commits.
Video is not up still so I am copying the slide link below.     
Slides: https://docs.google.com/presentation/d/10zupeFZuOlxroAMl29qVJl78nD4_YWHkQxANNVlIjE0/edit?ts=5cc73ae8#slide=id.p1 
 
2. OpenStack QA - Project Onboarding : Wednesday, May 1, 9:00am-9:40am
We did host the QA onboarding sessions but there were only 3 attendees and no new contributor. 
I think it is hard to see any new contributor in summits now so I am thinking whether we should host
the onboarding sessions from next time.
Etherpad: https://etherpad.openstack.org/p/DEN-qa-onboarding

3. Users / Operators adoption of QA tools / plugins : Wednesday, May 1, 10:50am-11:30am3.
As usual, we had more attendees in this session and useful feedback. There are few tooling is being
shared by attendees: 1. Python hardware module for bare metal detailed hardware inspection & anomaly detection
https://github.com/redhat-cip/hardware 2. Workload testing: https://opendev.org/x/tobiko/ 
Another good idea from Doug was plugin feature in openstack-health dashboard. That is something we
discussed in PTG. For more details on this, refer the PTG  " OpenStack-health improvement" section.
Etherpad: https://etherpad.openstack.org/p/Den-forum-qa-ops-user-feedback

QA PTG: 2nd - 3rd May:
We were 3-4 attendee in the room always and others attended per topics. Good discussions and few
good improvement ideas about gate stability and dashboard etc.

1. Topic: Stein Retrospective
We collect good and need improvement things in this session. In term of good things, we
completed the OpenStack gate migration fro Xenial to Bionic, lot of reviews and code.
Doug from AT&T mentioned about to add tempest and patrole to gates and check in their
production deployment process, "Thank you for all of the hard work from the QA team!!!"

Slow reviews are a concern as we have a good number of the incoming request. This is something
we should improve in Train.
Action items:
gmann: start the plan for backlogs especially for review and doc cleanup.
masayukig: plan to have resource leakage check in gate.
ds6901:will work with his team to clean up leaks and submit bugs

2. Topic: Keystone system-scope testing
QA and Keystone team gathered together in this cross-project session about next steps on system
scope testing. We talked on multiple points about how to cover all new roles for system scope and
how to keep the backward compatibility testing for stable branches still testing the without system
scope. We decided to move forward for system_admin as of now and fall back the system_admin
to project scope if there is no system_scope testing flag is true on Tempest side (this will cover the
stable branch testing unaffected).

We agreed :
- To move forward with system admin - https://review.opendev.org/#/c/604909/
 - Add tempest job to test system scope - https://review.opendev.org/#/c/614484/
    - Then add to tempest full - gmann
- Then add testing for system reader
- Investigate more advanced RBAC testing with Patrole - gmann  
Etherpad: https://etherpad.openstack.org/p/keystone-train-ptg-testing-system-scope-in-tempest

3. Topic: new whitebox plugin for tempest:
This is a new idea from artom about testing things outside of Tempest's scope (currently mostly used
to check instance XML for NFV use case tests). Currently, this tool does ssh into VM and fetch the
xml for further verification etc. 

We agreed on point to avoid any duplicate test verification from the Tempest or nova functional tests
This is good to tests from more extra verification by going inside VM like after migration data, CPU pinning etc.
As next step artom to propose the QA spec with details and proposal of this plugin under QA program.

4. Topic: Document the QA process or TODO things for releases, stable branch cut:
Idea is to start a centralized doc page for QA activities and process etc. we want to use
the qa-specs repo to publish the content to doc.openstack.org/qa/. This can be not so
easy and need few tweaks on doc jobs. I will get into the details and then discuss with
infra team. This is a low priority for now.

5. Topic: Plugin sanity check
Current tempest-plugins-sanity job is not stable and so it is n-v. We want to make it voting
by only installing the active plugins. many plugins are failing either they are dead or not so active.
We agreed on:
- make faulty plugins as blacklist with bug/patch link and notify the same on ML every time we detect any failure
- Publish the blakclist on plugins-registry doc.
- After that make this job voting, make the process of fixing and removing the faulty plugin which unblocks
   the tempest gate with author self-approve.
- Make sanity job running on plugins which are dependent on each other. For example, congress-tempest-plugin
   use neutron-tempest-plugin, mistral-tempest-plugin etc so all these plugins should have a sanity job which can
   install and list these plugins tests only not all the plugins.

6. Topic: Planning for Patrole Stable release:
We had a good amount of discussions for Patrole improvements area to release it stable.
Refer the below ML thread for details and further discussions on this topic:
-  http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005870.html 

7. Topic: How to make tempest-full stable  ( Don't fail integrated job when not related test will fail )
Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We discussed
the few ideas to improve it. Refer the below ML thread for details and further discussions on this topic
: http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html 

8. Topic: OpenStack-Health Improvement
Doug from AT&T has few improvement ideas for health dashboard which has been discussed in PTG:
- Test Grouping
    - Define groups
    - Assigned test to groups
    - filter by groups
- Compare 2 runs
= Look into push AQuA report to subunit2SQL as a tool
Action Items:
 - Doug is going to write the spec for plugin approach. All other ideas can be done after we have the plugin
    approach ready. 
    - filter
    - presentation

9. Topic: Stein Backlogs & Train priorities & Planning
We collected the Train items in below mentioned etherpad with the assignee.
Anyone would like to help on any of the item, ping me on IRC or reply here.
Etherpad: https://etherpad.openstack.org/p/qa-train-priority
 
10. Topic: grenade zuulv3 jobs review/discussions
We did not get the chance to review these. Let's continue it after PTG.

Full Detail discussion: https://etherpad.openstack.org/p/qa-train-ptg 

-gmann