Hello Everyone, We had a good discussion at QA forum and PTG. I am summarizing those and will start the separate thread for the few topics which needs more feedback. Summit: QA Forum sessions: 1. OpenStack QA - Project Update: Tuesday, April 30, 2:35pm-2:55pm We gave the updates on what we finished on Stein and draft plan for Train cycle. The good thing to note is we still have a lot of activity going on in QA. As overall QA projects, we did >3000 reviews and 750 commits. Video is not up still so I am copying the slide link below. Slides: https://docs.google.com/presentation/d/10zupeFZuOlxroAMl29qVJl78nD4_YWHkQxAN... 2. OpenStack QA - Project Onboarding : Wednesday, May 1, 9:00am-9:40am We did host the QA onboarding sessions but there were only 3 attendees and no new contributor. I think it is hard to see any new contributor in summits now so I am thinking whether we should host the onboarding sessions from next time. Etherpad: https://etherpad.openstack.org/p/DEN-qa-onboarding 3. Users / Operators adoption of QA tools / plugins : Wednesday, May 1, 10:50am-11:30am3. As usual, we had more attendees in this session and useful feedback. There are few tooling is being shared by attendees: 1. Python hardware module for bare metal detailed hardware inspection & anomaly detection https://github.com/redhat-cip/hardware 2. Workload testing: https://opendev.org/x/tobiko/ Another good idea from Doug was plugin feature in openstack-health dashboard. That is something we discussed in PTG. For more details on this, refer the PTG " OpenStack-health improvement" section. Etherpad: https://etherpad.openstack.org/p/Den-forum-qa-ops-user-feedback QA PTG: 2nd - 3rd May: We were 3-4 attendee in the room always and others attended per topics. Good discussions and few good improvement ideas about gate stability and dashboard etc. 1. Topic: Stein Retrospective We collect good and need improvement things in this session. In term of good things, we completed the OpenStack gate migration fro Xenial to Bionic, lot of reviews and code. Doug from AT&T mentioned about to add tempest and patrole to gates and check in their production deployment process, "Thank you for all of the hard work from the QA team!!!" Slow reviews are a concern as we have a good number of the incoming request. This is something we should improve in Train. Action items: gmann: start the plan for backlogs especially for review and doc cleanup. masayukig: plan to have resource leakage check in gate. ds6901:will work with his team to clean up leaks and submit bugs 2. Topic: Keystone system-scope testing QA and Keystone team gathered together in this cross-project session about next steps on system scope testing. We talked on multiple points about how to cover all new roles for system scope and how to keep the backward compatibility testing for stable branches still testing the without system scope. We decided to move forward for system_admin as of now and fall back the system_admin to project scope if there is no system_scope testing flag is true on Tempest side (this will cover the stable branch testing unaffected). We agreed : - To move forward with system admin - https://review.opendev.org/#/c/604909/ - Add tempest job to test system scope - https://review.opendev.org/#/c/614484/ - Then add to tempest full - gmann - Then add testing for system reader - Investigate more advanced RBAC testing with Patrole - gmann Etherpad: https://etherpad.openstack.org/p/keystone-train-ptg-testing-system-scope-in-... 3. Topic: new whitebox plugin for tempest: This is a new idea from artom about testing things outside of Tempest's scope (currently mostly used to check instance XML for NFV use case tests). Currently, this tool does ssh into VM and fetch the xml for further verification etc. We agreed on point to avoid any duplicate test verification from the Tempest or nova functional tests This is good to tests from more extra verification by going inside VM like after migration data, CPU pinning etc. As next step artom to propose the QA spec with details and proposal of this plugin under QA program. 4. Topic: Document the QA process or TODO things for releases, stable branch cut: Idea is to start a centralized doc page for QA activities and process etc. we want to use the qa-specs repo to publish the content to doc.openstack.org/qa/. This can be not so easy and need few tweaks on doc jobs. I will get into the details and then discuss with infra team. This is a low priority for now. 5. Topic: Plugin sanity check Current tempest-plugins-sanity job is not stable and so it is n-v. We want to make it voting by only installing the active plugins. many plugins are failing either they are dead or not so active. We agreed on: - make faulty plugins as blacklist with bug/patch link and notify the same on ML every time we detect any failure - Publish the blakclist on plugins-registry doc. - After that make this job voting, make the process of fixing and removing the faulty plugin which unblocks the tempest gate with author self-approve. - Make sanity job running on plugins which are dependent on each other. For example, congress-tempest-plugin use neutron-tempest-plugin, mistral-tempest-plugin etc so all these plugins should have a sanity job which can install and list these plugins tests only not all the plugins. 6. Topic: Planning for Patrole Stable release: We had a good amount of discussions for Patrole improvements area to release it stable. Refer the below ML thread for details and further discussions on this topic: - http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005870.html 7. Topic: How to make tempest-full stable ( Don't fail integrated job when not related test will fail ) Current integrated-gate jobs (tempest-full) is not so stable for various bugs specially timeout. We discussed the few ideas to improve it. Refer the below ML thread for details and further discussions on this topic : http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005871.html 8. Topic: OpenStack-Health Improvement Doug from AT&T has few improvement ideas for health dashboard which has been discussed in PTG: - Test Grouping - Define groups - Assigned test to groups - filter by groups - Compare 2 runs = Look into push AQuA report to subunit2SQL as a tool Action Items: - Doug is going to write the spec for plugin approach. All other ideas can be done after we have the plugin approach ready. - filter - presentation 9. Topic: Stein Backlogs & Train priorities & Planning We collected the Train items in below mentioned etherpad with the assignee. Anyone would like to help on any of the item, ping me on IRC or reply here. Etherpad: https://etherpad.openstack.org/p/qa-train-priority 10. Topic: grenade zuulv3 jobs review/discussions We did not get the chance to review these. Let's continue it after PTG. Full Detail discussion: https://etherpad.openstack.org/p/qa-train-ptg -gmann