Open Stack

On 08/26/2014 07:47 PM, Salvatore Orlando wrote:
> TL; DR
> A few folks are proposing to stop running tests for neutron advanced
> services [ie: (lb|vpn|fw)aas] in the integrated gate, and run them only
> on the neutron gate.
> 
> Reason: projects like nova are 100% orthogonal to neutron advanced
> services. Also, there have been episodes in the past of unreliability of
> tests for these services, and it would be good to limit affected
> projects considering that more api tests and scenarios are being added.
> 
> -----
> 
> So far the neutron full job runs tests (api and scenarios) for neutron
> "core" functionality as well as neutron "advanced services", which run
> as neutron service plugin.
> 
> It's highly unlikely, if not impossible, that changes in projects such
> as nova, glance or ceilometer can have an impact on the stability of
> these services.
> On the other hand, instability in these services can trigger gate
> failures in unrelated projects as long as tests for these services are
> run in the neutron full job in the integrated gate. There have already
> been several gate-breaking bugs in lbaas scenario tests are firewall api
> tests.
> Admittedly, advanced services do not have the same level of coverage as
> core neutron functionality. Therefore as more tests are being added,
> there is an increased possibility of unearthing dormant bugs.
> 
> For this reason we are proposing to not run anymore tests for neutron
> advanced services in the integrated gate, but keep them running on the
> neutron gate.
> This means we will have two neutron jobs:
> 1) check-tempest-dsvm-neutron-full which will run only "core" neutron
> functionality
> 2) check-tempest-dsvm-neutron-full-ext which will be what the neutron
> full job is today.
> 
> The former will be part of the integrated gate, the latter will be part
> of the neutron gate.
> Considering that other integrating services should not have an impact on
> neutron advanced services, this should not make gate testing asymmetric.
> 
> However, there might be exceptions for:
> - "orchestration" project like heat which in the future might leverage
> capabilities like load balancing
> - oslo-* libraries, as changes in them might have an impact on neutron
> advanced services, since they consume those libraries
> 
> Another good question is whether "extended" tests should be performed as
> part of functional or tempest checks. My take on this is that scenario
> tests should always be part of tempest. On the other hand I reckon API
> tests should exclusively be part of functional tests, but as so far
> tempest is running a gazillion of API tests, this is probably a
> discussion for the medium/long term. 
> 
> In order to add this new job there are a few patches under review:
> [1] and [2] Introduces the 'full-ext' job and devstack-gate support for it.
> [3] Are the patches implementing a blueprint which will enable us to
> specify for which extensions test should be executed.
> 
> Finally, one more note about smoketests. Although we're planning to get
> rid of them soon, we still have failures in the pg job because of [4].
> For this reasons smoketests are still running for postgres in the
> integrated gate. As load balancing and firewall API tests are part of
> it, they should be removed from the smoke test executed on the
> integrated gate ([5], [6]). This is a temporary measure until the
> postgres issue is fixed.
> 
> Regards,
> Salvatore
> 
> [1] https://review.openstack.org/#/c/114933/
> [2] https://review.openstack.org/#/c/114932/
> [3] https://review.openstack.org/#/q/status:open+branch:master+topic:bp/branchless-tempest-extensions,n,z
> [4] https://bugs.launchpad.net/nova/+bug/1305892
> [5] https://review.openstack.org/#/c/115022/
> [6] https://review.openstack.org/#/c/115023/

+1

I realistically think that we should think about neutron as 2 things.
The L2/L3 services, and the advanced services. L2/L3 seem appropriate to
ensure are tightly integrated to the rest of OpenStack. The advanced
services really are a different beast (and honestly might be better as a
separate OpenStack service that's not neutron).

	-Sean

-- 
Sean Dague
http://dague.net