Open Stack

TL; DR
A few folks are proposing to stop running tests for neutron advanced
services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on
the neutron gate.

Reason: projects like nova are 100% orthogonal to neutron advanced
services. Also, there have been episodes in the past of unreliability of
tests for these services, and it would be good to limit affected projects
considering that more api tests and scenarios are being added.

-----

So far the neutron full job runs tests (api and scenarios) for neutron
"core" functionality as well as neutron "advanced services", which run as
neutron service plugin.

It's highly unlikely, if not impossible, that changes in projects such as
nova, glance or ceilometer can have an impact on the stability of these
services.
On the other hand, instability in these services can trigger gate failures
in unrelated projects as long as tests for these services are run in the
neutron full job in the integrated gate. There have already been several
gate-breaking bugs in lbaas scenario tests are firewall api tests.
Admittedly, advanced services do not have the same level of coverage as
core neutron functionality. Therefore as more tests are being added, there
is an increased possibility of unearthing dormant bugs.

For this reason we are proposing to not run anymore tests for neutron
advanced services in the integrated gate, but keep them running on the
neutron gate.
This means we will have two neutron jobs:
1) check-tempest-dsvm-neutron-full which will run only "core" neutron
functionality
2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full
job is today.

The former will be part of the integrated gate, the latter will be part of
the neutron gate.
Considering that other integrating services should not have an impact on
neutron advanced services, this should not make gate testing asymmetric.

However, there might be exceptions for:
- "orchestration" project like heat which in the future might leverage
capabilities like load balancing
- oslo-* libraries, as changes in them might have an impact on neutron
advanced services, since they consume those libraries

Another good question is whether "extended" tests should be performed as
part of functional or tempest checks. My take on this is that scenario
tests should always be part of tempest. On the other hand I reckon API
tests should exclusively be part of functional tests, but as so far tempest
is running a gazillion of API tests, this is probably a discussion for the
medium/long term.

In order to add this new job there are a few patches under review:
[1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
[3] Are the patches implementing a blueprint which will enable us to
specify for which extensions test should be executed.

Finally, one more note about smoketests. Although we're planning to get rid
of them soon, we still have failures in the pg job because of [4]. For this
reasons smoketests are still running for postgres in the integrated gate.
As load balancing and firewall API tests are part of it, they should be
removed from the smoke test executed on the integrated gate ([5], [6]).
This is a temporary measure until the postgres issue is fixed.

Regards,
Salvatore

[1] https://review.openstack.org/#/c/114933/
[2] https://review.openstack.org/#/c/114932/
[3]
https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
[4] https://bugs.launchpad.net/nova/+bug/1305892
[5] https://review.openstack.org/#/c/115022/
[6] https://review.openstack.org/#/c/115023/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140827/300964a3/attachment.html>