Open Stack

Hi All,

Thanks for joining Stein QA PTG and making it successful. 
I am writing the QA PTG summary and detailed discussion can be found on main PTG etherpad - https://etherpad.openstack.org/p/qa-stein-ptg
We are continuing the 'owner' for each working item so that we have single point of contact to track those. 


1. QA Help Room
-------------------------------------------------------
QA team was present in Help Room on Monday. We were happy to help few queries  from Octavia multinode job and Kuryr-kubernetes testing part. 
Other than that, there was not much that day except few other random queries. 

2. Rocky Retrospective
---------------------------------------------------------
We discussed the Rocky Retrospective as first thing on Tuesday. We went through 1. what went well and 2. what needs to improve and gather some concrete action
items.

Patrole has good progress in Rocky cycle with code as well as documentation.  Also we were able to fill the compute microversion gap almost all till Rocky. 

Action Items:
- Need to add Tempest CLI documentation and other usefull stuff from tripleo Doc  to Tempest Doc - chandankumar
- Run all tests in tempest-full-parallel job and move it to periodic job pipeline - afazekas
- Need to merge the QA office hour, check with andrea for 17 UTC office hour and if ok then, close that and modify the current office hour from 9 UTC to 8 UTC .  - gmann
- Need to ask chandankumar or manik for bug triage volunteer. - gmann
- Create the low hanging items list and publish for new contributors - gmann

We will be tracking the above AI in our QA office hour to finish them on time.
Owner: gmann
Etherpad link: https://etherpad.openstack.org/p/qa-rocky-retrospective 


3. Stable interfaces from Tempest Plugins
-----------------------------------------------------------
We discussed about having stable interface from Tempest plugins like Tempest so that other plugins can consume those. Service client is good example of those which are required to do cross project testing. For example: congress tempest plugin needs to use mistral service clients for integration testing of congress+mistral [1].  Similarly Patrole need to use neutron tempest plugin service client(for n/n-1/n-2).  

Idea here is to have lib or stable interface in Tempest plugins side like Tempest so that other plugins can use them. We will start with some documentation about use case and benefits and then work with neutron-tempest-plugin team to make their service client expose as stable interface. Once that is done then, we can suggest the same to other plugins.  

Action Items:
    - Need some documentation and guidance with use case and example, benefits for plugins. - felipemonteiro
    - mailing list discussions on making specific plugins stable that are consumed by other plugins - felipemonteiro
    - check with requirement team to add the tempest plugin in g-r and then those can be added on other plugins requirement.txt - gmann
Owner: felipemonteiro
Etherpad link: https://etherpad.openstack.org/p/stable-interfaces-from-tempest-plugins


4. Tempest Plugins CI to cover stable branches & Plugins release and tagging clarification
--------------------------------------------------------------------------------------------------------------
We discussed about how other projects or Plugins can setup the CI to cover the stable branches testing on their master changes. Solution can be simple to define the supported stable branches and run them on master gate (same way Tempest does).  QA team will start the guidelines on this. 
Other part we need to cover is release and tagging guidelines. There were lot of confusion about release of Tempest plugins in Rocky. To make it better, QA team will write guidelines and document the clear process. 

Action Items:
    - move/update documentation on branchless considerations in tempest to somewhere more global so that it covers plugins documentation too - gmann
    - Add tagging and release clarification for plugins. 
    - talk with neutron team about moving in-tree tempest plugins of stadium projects to neutron-tempest-plugin or separate tempest-plugins repositories - slaweq
    - Add config options to disable the plugins load - gmann
Owner: gmann
Etherpad link: https://etherpad.openstack.org/p/tempest-plugins-ci-release-tagging-clarification 


5. Tempest Cleanup Feature 
---------------------------------
Current Tempest CLI for cleanup the test resource is not so good. It does cleanup the resources based on saved_state.json file which save the resources difference before and after Tempest run. This can end up cleaning up the other non-test resources which got created during time period of tempest run. 

There is a QA spec which proposing the different approach for cleanup[2]. After discussing all those approach, we decided to go with resource_prefix. We will bring back the resource_prefix approach (which got removed after deprecation) and modify the  "tempest cleanup" cli  to cleanup resource based on resource_prefix.  Complete discussion can be found in etherpad. As of felipemonteiro is the owner but he will check with nicholashelgeson or AT&T folks to further work on this. 

Action Item:
  - Update spec with idea from 0.0.2 (because it's relatively easy to implement) - get merged - felipemonteiro/nicholashelgeson
  - Add back resource_prefix config option and add back to data_utils.rand_name - felipemonteiro/nicholashelgeson
  - Go through all tempest tests and make sure data_utils.rand_name is used for resources - felipemonteiro/nicholashelgeson
  - Update tempest cleanup - felipemonteiro/nicholashelgeson
  - Update documentation - felipemonteiro/nicholashelgeson
Owner: felipemonteiro
Etherpad links: https://etherpad.openstack.org/p/tempest-cleanup-feature


6. Tempest conf Plugin Discovery process
-------------------------------------------------------------------------
This topic is about generating the Tempest conf from plugins config options too. This idea is more for python-tempestconf not for QA as such. But there were python-tempestconf folks present in QA room and discussed that this is doable in python-tempestconf itself. 

There is nothing from QA side on this so I would like to drop this item from QA tracking. 

Etherpad link: https://etherpad.openstack.org/p/tempest-conf-plugin-discovery-process


7. Proper handling of interface/volume/pci devece attach/detach hotplug/unplug in tempest
-------------------------------------------------------------------------
Tempest tests not handling the hotplug/unplug events properly. The guest does not check for button press event at early boot time.
The hotplug events sent before the kernel fully initialized can be lost. test_stamp_pattern.py could be unskipped, if we would try to ssh vm before hot plug event (volume attach). Also there are API tests which knows nothing about the guest state, therefore it cannot know when the guest is ready for checking the button press. Details problem can be found here [3].

Idea is to perform the ssh validation step before we consider the test server ready to use in test.  

Action Items:
            - Adding ssh validation steps for api/scenrio tests where is required - afazekas
            - making the run_validation default true - afazekas
            - soft reboot , is nova event tells was it soft reboot (check) , is some special register on the machine can tell it ?  - afazekas
Owner: afazekas
Etherpad link:  https://etherpad.openstack.org/p/handling-of-interface-attach-detach-hotplug-unplug


8. Shared network handling
-----------------------------------------------------------------------------
Attila observed few tests failing when using shard network. But looks like the only 100% reproducible issue is test_create_list_show_delete_interfaces_by_fixed_ip[4][5]
There should not be issue for shared network also and as of now we will just fix the failing tests. 

Action Items:
    - Fix the failing tests - afazekas
    - Try to run tempest in parallel with shared network create/delete parallel testS to search for other incidents locally)   - afazekas
Owner: afazekas
Etherpad link: https://etherpad.openstack.org/p/shared-network-handling



9. Planning for Patrole Stable release
----------------------------------------------------------
We continue the Patrole stable release discussion in PTG. We prepared the concrete list of items we need to release it stable and targeted for Stein cycle. 
Along with multi-policy, system scope support, we will check the framework stability also. Documentation is already in better shape.  

TODO items before stable release: 
1. multi-policy
2. system scope support:
3. Better support for generic check policy rules (e.g. user_id:%(user.id))
4. Iterate through all the patrole interface/framework which needs to be used outside of patrole 

Action Items:
    - let's finish the above planned items in Stein.  - felipemonteiro
Owner: felipemonteiro
Etherpad link: https://etherpad.openstack.org/p/planning-for-patrole-stable-release


10. Proposal for New QA project: Harbinger
-----------------------------------------------------------------
OpenStack QA currently lacks a dataplane testing framework that can be easily consumed, configured and maintained. To fill that gap, there is a new project proposal called "Harbinger". Harbinger allows execution of various OpenStack data plane testing frameworks while maintaining a single standardized interface and input format.

Currently it cover Shaker and Yardstick and Kloudbuster  is WIP. This can be useful to consume in Eris (extreme testing idea).  There are few points which need more clarification like Standardization of output, can it cover Control plane testing etc. IMO, this is good project to start and can be consumed in Eris and cross community testing. Author or this project was not present in PTG and felipemonteiro proxy this too. We would like to extend this discussion on ML and with Extreme testing stackholders and also start the QA spec. 

Action Items:
- There are many item we planned as AI but first step will to start the ML and spec. 
Owner: felipemonteiro told to convey the discussion with Harbinger author. Leaving the owner empty as of now. 
Etherpad links: https://etherpad.openstack.org/p/new-qa-project-harbinger


11. Clean up the tempest documentation
-----------------------------------------------------------------
This is always outstanding item :-). We discussed about more improvement in documentation like better doc structure, CLI doc, consuming the Tempest  related docs at central place which is Tempest.  We had list of item to cover with different assignee. 

Action Items:
- Complete all the documentation points written in etherpad. - tosky, gmann, masayukig
Owner: gmann
Etherpad links: https://etherpad.openstack.org/p/clean-up-the-tempest-documentation


12. Consuming all Tempest CLI in gate
-----------------------------------------------------------------
Tempest has many CLI anddue to lack of unite tests[6], there are chance where we can/did break the CLI. Idea is to consume all the CLI on gate job so that we can improve their testing coverage.  Few CLI will be covered in main gate job and other as functional testing. 

Action Items:
    - Continue this patch on zuul v3 - https://review.openstack.org/#/c/355666/ - masayukig
    - add funcional tests and new functionl job - gmann
Owner: masayukig
Etherpad links: https://etherpad.openstack.org/p/consuming-all-tempest-cli-in-gate 

13. Migration from Launchpad to storyboard
-----------------------------------------------------------------
We discussed about possibility of migration to storyboard. Patrole is first QA projects we are trying and based on that we can proceed on other projects. 

Action Items:
 - Wait for BP script from storyboard team and then finish the Patrole projects migration.  - gmann
 - Feedback or request for Storyboard:
    - Create a story for adding some filed to indicative user interest (heat / vote / points ) - afazekas
    - Gerrit automation work to story about automatic assignee etc or adding the Topic field on gerrit. 
    - Have a way to not diverge too much the set of used tags - possible solution: sort proposed tags by popularity 
Owner: gmann
Etherpad links: https://etherpad.openstack.org/p/migration-from-lp-to-sb

14. QA Stein Priority:
--------------------------------------------------------------------
We discussed about the priority items for Stein and listed the items
and owner of each items in below etherpads. 

We will be tracking each item in office hour and try to keep the good progress.

Etherpad link: https://etherpad.openstack.org/p/qa-stein-priority 

Let's work closely as team and have another successful cycle.


[1] http://lists.openstack.org/pipermail/openstack-dev/2018-March/128225.html
[2] https://review.openstack.org/#/c/595277
[3] https://docs.google.com/presentation/d/1Im-iYVzroKwXKP23p12Q5vsUGdk2V26SPpLWF3I5dbA/edit?usp=sharing
[4] http://logs.openstack.org/33/601433/2/check/tempest-full/b4ea6bf/testr_results.html.gz
[5] https://bugs.launchpad.net/tempest/+bug/1790864
[6] https://blueprints.launchpad.net/tempest/+spec/tempest-cli-unit-test-coverage

-gmann