Open Stack

On 02/02/2016 10:03 PM, Matthew Treinish wrote:
> On Tue, Feb 02, 2016 at 05:09:47PM -0800, Armando M. wrote:
>> Folks,
>>
>> We have some IPv6 related bugs [1,2,3] that have been lingering for some
>> time now. They have been hurting the gate (e.g. [4] the most recent
>> offending failure) and since it looks like they have been without owners
>> nor a plan of action for some time, I made the hard decision of skipping
>> them [5] ahead of the busy times ahead.
> 
> So TBH I don't think the failure rate for these tests are really at a point
> necessitating a skip:
> 
> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac
> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os
> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os
> 
> (also just a cool side-note, you can see the very obvious performance regression
> caused by the keystonemiddleware release and when we excluded that version in
> requirements)
> 
> Well, test_dualnet_dhcp6_stateless_from_os is kinda there with a ~10% failure
> rate, but the other 2 really aren't. I normally would be -1 on the skip patch
> because of that. We try to save the skips for cases where the bugs are really
> severe and preventing productivity at a large scale. 
> 
> But, in this case these ipv6 tests are kinda of out of place in tempest. Having
> all the permutations of possible ip allocation configurations always seemed a
> bit too heavy handed. These tests are also consistently in the top 10 slowest
> for a run. We really should have trimmed down this set a while ago so we're only
> have a single case in tempest. Neutron should own the other possible
> configurations as an in-tree test.
> 
> Brian Haley has a patch up from Dec. that was trying to clean it up:
> 
> https://review.openstack.org/#/c/239868/
> 
> We probably should revisit that soon, since quite clearly no one is looking at
> these right now.

We definitely shouldn't be running all the IPv6 tests.

But I also think the assumption that the failure rate is low is not a
valid reason to keep a test. Unreliable tests that don't have anyone
looking into them should be deleted. They are providing negative value.
Because people just recheck past them even if their code made the race
worse. So any legitimate issues they are exposing are being ignored.

If the neutron PTL wants tests pulled, we should just do it.

	-Sean

-- 
Sean Dague
http://dague.net