Open Stack

Hi,

** Intention **
Intention is to expand Patrole testing to some service clients that already exist in some Tempest plugins, for core services only.

** Background **
Digging through Neutron testing, it seems like there is currently a lot of test duplication between neutron-tempest-plugin and Tempest [1]. Under some circumstances it seems OK to have redundant testing/parallel testing: "Having potential duplication between testing is not a big deal especially compared to the alternative of removing something which is actually providing value and is actively catching bugs, or blocking incorrect patches from landing" [2].

This leads me to the following question: If API test duplication is OK, what about service client duplication? Patches like [3] and [4]  promote service client duplication with neutron-tempest-plugin. As far as I can tell, Neutron builds out some of its service clients dynamically here: [5]. Which includes segments service client (proposed as an addition to tempest.lib in [4]) here: [6].

This leads to a situation where if we want to offer RBAC testing for these APIs (to validate their policy enforcement), we can't really do so without adding the service client to Tempest, unless we rely on the neutron-tempest-plugin (for example) in Patrole's .zuul.yaml.

** Path Forward **
Option #1: For the core services, most service clients should live in tempest.lib for standardization/governance around documentation and stability for those clients. Service client duplication should try to be minimized as much as possible. API testing related to some service clients, though, should remain in the Tempest plugins.

Option #2: Proceed with service client duplication, either by adding the service client to Tempest (or as yet another alternative, Patrole). This leads to maintenance overhead: have to maintain service clients in the plugins and Tempest itself.

Option #3: Don't offer RBAC testing in Patrole plugin for those APIs.

Thanks,

Felipe

[1] https://bugs.launchpad.net/neutron/+bug/1552960
[2] https://docs.openstack.org/tempest/latest/test_removal.html
[3] https://review.openstack.org/#/c/482395/
[4] https://review.openstack.org/#/c/582340/
[5] http://git.openstack.org/cgit/openstack/neutron-tempest-plugin/tree/neutron_tempest_plugin/services/network/json/network_client.py
[6]  http://git.openstack.org/cgit/openstack/neutron-tempest-plugin/tree/neutron_tempest_plugin/api/test_timestamp.py


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180723/7ea7a5d3/attachment.html>