Open Stack

On Tue, Aug 26, 2014 at 4:47 PM, Salvatore Orlando <sorlando at nicira.com>
wrote:

> TL; DR
> A few folks are proposing to stop running tests for neutron advanced
> services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only on
> the neutron gate.
>
> Reason: projects like nova are 100% orthogonal to neutron advanced
> services. Also, there have been episodes in the past of unreliability of
> tests for these services, and it would be good to limit affected projects
> considering that more api tests and scenarios are being added.
>
> -----
>
> So far the neutron full job runs tests (api and scenarios) for neutron
> "core" functionality as well as neutron "advanced services", which run as
> neutron service plugin.
>
> It's highly unlikely, if not impossible, that changes in projects such as
> nova, glance or ceilometer can have an impact on the stability of these
> services.
> On the other hand, instability in these services can trigger gate failures
> in unrelated projects as long as tests for these services are run in the
> neutron full job in the integrated gate. There have already been several
> gate-breaking bugs in lbaas scenario tests are firewall api tests.
> Admittedly, advanced services do not have the same level of coverage as
> core neutron functionality. Therefore as more tests are being added, there
> is an increased possibility of unearthing dormant bugs.
>
>
I support this split but for slightly different reasons.  I am under the
impression that neutron advanced services are not ready for prime time. If
that is correct I don't think we should be gating on things that aren't
ready.



> For this reason we are proposing to not run anymore tests for neutron
> advanced services in the integrated gate, but keep them running on the
> neutron gate.
> This means we will have two neutron jobs:
> 1) check-tempest-dsvm-neutron-full which will run only "core" neutron
> functionality
> 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron full
> job is today.
>

Using my breakdown, the extended job would include experimental neutron
features.


>
> The former will be part of the integrated gate, the latter will be part of
> the neutron gate.
> Considering that other integrating services should not have an impact on
> neutron advanced services, this should not make gate testing asymmetric.
>
> However, there might be exceptions for:
> - "orchestration" project like heat which in the future might leverage
> capabilities like load balancing
> - oslo-* libraries, as changes in them might have an impact on neutron
> advanced services, since they consume those libraries
>

Once another service starts consuming an advanced feature I think it makes
sense to move it to the main neutron-full job. Especially if we assume that
things will only depend on neutron features that are not too experimental.


>
> Another good question is whether "extended" tests should be performed as
> part of functional or tempest checks. My take on this is that scenario
> tests should always be part of tempest. On the other hand I reckon API
> tests should exclusively be part of functional tests, but as so far tempest
> is running a gazillion of API tests, this is probably a discussion for the
> medium/long term.
>
> In order to add this new job there are a few patches under review:
> [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
> [3] Are the patches implementing a blueprint which will enable us to
> specify for which extensions test should be executed.
>
> Finally, one more note about smoketests. Although we're planning to get
> rid of them soon, we still have failures in the pg job because of [4]. For
> this reasons smoketests are still running for postgres in the integrated
> gate. As load balancing and firewall API tests are part of it, they should
> be removed from the smoke test executed on the integrated gate ([5], [6]).
> This is a temporary measure until the postgres issue is fixed.
>

++


>
> Regards,
> Salvatore
>
> [1] https://review.openstack.org/#/c/114933/
> [2] https://review.openstack.org/#/c/114932/
> [3]
> https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
> [4] https://bugs.launchpad.net/nova/+bug/1305892
> [5] https://review.openstack.org/#/c/115022/
> [6] https://review.openstack.org/#/c/115023/
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140903/47a583ec/attachment.html>