Open Stack

On 3 February 2016 at 04:28, Sean Dague <sean at dague.net> wrote:

> On 02/02/2016 10:03 PM, Matthew Treinish wrote:
> > On Tue, Feb 02, 2016 at 05:09:47PM -0800, Armando M. wrote:
> >> Folks,
> >>
> >> We have some IPv6 related bugs [1,2,3] that have been lingering for some
> >> time now. They have been hurting the gate (e.g. [4] the most recent
> >> offending failure) and since it looks like they have been without owners
> >> nor a plan of action for some time, I made the hard decision of skipping
> >> them [5] ahead of the busy times ahead.
> >
> > So TBH I don't think the failure rate for these tests are really at a
> point
> > necessitating a skip:
> >
> >
> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_multi_prefix_slaac
> >
> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dualnet_dhcp6_stateless_from_os
> >
> http://status.openstack.org/openstack-health/#/test/tempest.scenario.test_network_v6.TestGettingAddress.test_dhcp6_stateless_from_os
> >
> > (also just a cool side-note, you can see the very obvious performance
> regression
> > caused by the keystonemiddleware release and when we excluded that
> version in
> > requirements)
> >
> > Well, test_dualnet_dhcp6_stateless_from_os is kinda there with a ~10%
> failure
> > rate, but the other 2 really aren't. I normally would be -1 on the skip
> patch
> > because of that. We try to save the skips for cases where the bugs are
> really
> > severe and preventing productivity at a large scale.
> >
> > But, in this case these ipv6 tests are kinda of out of place in tempest.
> Having
> > all the permutations of possible ip allocation configurations always
> seemed a
> > bit too heavy handed. These tests are also consistently in the top 10
> slowest
> > for a run. We really should have trimmed down this set a while ago so
> we're only
> > have a single case in tempest. Neutron should own the other possible
> > configurations as an in-tree test.
> >
> > Brian Haley has a patch up from Dec. that was trying to clean it up:
> >
> > https://review.openstack.org/#/c/239868/
> >
> > We probably should revisit that soon, since quite clearly no one is
> looking at
> > these right now.
>
> We definitely shouldn't be running all the IPv6 tests.
>
> But I also think the assumption that the failure rate is low is not a
> valid reason to keep a test. Unreliable tests that don't have anyone
> looking into them should be deleted. They are providing negative value.
> Because people just recheck past them even if their code made the race
> worse. So any legitimate issues they are exposing are being ignored.
>
> If the neutron PTL wants tests pulled, we should just do it.
>
>
Thanks for the support! Having said, I think it's important to make a
judgement call on a case by case basis, because removing tests blindly
might as well backfire.

In this specific instance and all things considered, merging [2] (or even
better [1]) feel like a net gain.

Cheers,
Armando

[1] https://review.openstack.org/#/c/239868/
[2] https://review.openstack.org/#/c/275457/


>         -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160203/276112ed/attachment.html>