[openstack-dev] [infra] reverify/recheck
Sean Dague
sean at dague.net
Tue Feb 18 13:52:07 UTC 2014
We're still working through kinks in the new system, which is why it's
not fully documented yet.
During the 2 weeks of gate wedging in January, we discovered a number of
interesting things. Some review teams are really slow at dealing with
patches with a +2 already on them, and may wait 2 - 6 *weeks* before a
second core reviewer shows up to approve. In that period of time the
dependencies used in the tests probably changed 3 or 4 times. Which
means the test results were invalid.
We also found that once a patch gets a +A people (core and non-core)
blindly reverify on any failure, over and over and over again. I saw one
core developer "reverify 123456789" on a patch.
Or that when trying to land a patch people will recheck half a dozen
times to get one clean run, then get their patch pushed to the gate.
All this blind meatgrinder behavior was putting tons of code into the
gate that could not pass. That, coupled with other race conditions we
were dealing with, put us into a state where we had a 60hr gate queue.
So we're trying out a new system. A change will not move into the gate
unless there is both a +A and Jenkins +1 on the patch.
We also rerun check results on comment if the test results are > 72hrs
old, so there is always a reasonably fresh version of the results. This
also helps detect merge conflicts early.
Also when you +A a patch, if there aren't 24hr fresh test results, we
rerun check first, then if that passes, it gets sent to the gate.
There is still a bit of a sticky part of the flow if it fails in the
gate. Because that means you don't satisfy the +1 jenkins requirement.
Still trying to get the right flow worked out there.
-Sean
On 02/18/2014 07:51 AM, Sergey Lukjanov wrote:
> You already can reverify on any approved change. Jenkins automatically
> runs check pipeline jobs after 3 days.
>
>
> On Tue, Feb 18, 2014 at 4:29 PM, Christopher Yeoh <cbkyeoh at gmail.com
> <mailto:cbkyeoh at gmail.com>> wrote:
>
> On Tue, Feb 18, 2014 at 12:27 AM, Gary Kotton <gkotton at vmware.com
> <mailto:gkotton at vmware.com>> wrote:
>
> Thanks!
> That makes sense. Just odd how a -1 was received.
>
>
> So I think that now a "check" run is automatically done if the
> latest Jenkins run is too old (7 days?)
> This is done because gate failures are so costly. The check run
> failing would return -1 instead of a
> gate failure of -2.
>
>
>
>
> On 2/17/14 3:15 PM, "Akihiro Motoki" <motoki at da.jp.nec.com
> <mailto:motoki at da.jp.nec.com>> wrote:
>
> >Hi Gary,
> >
> >According to zuul layout.yaml [1], "reverify bug #" should
> still work
> >but it seems to work only when verified score from jenkins is -2.
> >https://urldefense.proofpoint.com/v1/url?u=https://github.com/openstack-in
> >fra/config/blob/master/modules/openstack_project/files/zuul/layout.yaml%23
> >L25&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg4
> >5MkPhCZFxPEq8%3D%0A&m=Nf6%2FyHMSSchQO59xIcg6NKX1O2wlkkHHd42bYQim0k0%3D%0A&
> >s=8fc4d7e6970936977dc85989e4e7e9429afb15f5091a768efa4ce236417d9c9b
> >
> >Note that core team can trigger gate jobs by re-approving the
> patch.
> >
> >Thanks,
> >Akihiro
> >
> >(2014/02/17 22:03), Gary Kotton wrote:
> >> Hi,
> >> The "reverify no bug" was removed. But "reverify bug #" used
> to work.
> >>That no longer does. With the constant gate failures how can
> we ensure
> >>that a approved patch does a retry?
> >> Thanks
> >> Gary
> >>
> >> From: Sylvain Bauza <sylvain.bauza at gmail.com
> <mailto:sylvain.bauza at gmail.com>
> >><mailto:sylvain.bauza at gmail.com <mailto:sylvain.bauza at gmail.com>>>
> >> Reply-To: "OpenStack Development Mailing List (not for usage
> >>questions)" <openstack-dev at lists.openstack.org
> <mailto:openstack-dev at lists.openstack.org>
> >><mailto:openstack-dev at lists.openstack.org
> <mailto:openstack-dev at lists.openstack.org>>>
> >> Date: Monday, February 17, 2014 2:53 PM
> >> To: "OpenStack Development Mailing List (not for usage
> questions)"
> >><openstack-dev at lists.openstack.org
> <mailto:openstack-dev at lists.openstack.org>
> >><mailto:openstack-dev at lists.openstack.org
> <mailto:openstack-dev at lists.openstack.org>>>
> >> Subject: Re: [openstack-dev] [infra] reverify/recheck
> >>
> >> Hi Gary,
> >>
> >> That's normal, this command has been removed since Dec'13, see
> >>https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/pip
> >>ermail/openstack-dev/2013-December/021649.html&k=oIvRg1%2BdGAgOoM1BIlLLqw
> >>%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=Nf6%2Fy
> >>HMSSchQO59xIcg6NKX1O2wlkkHHd42bYQim0k0%3D%0A&s=826274ca8ad9562b557322274a
> >>cdacd97482338456820af8c330b76ea7639838
> >>
> >><https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/pi
> >>permail/openstack-dev/2013-December/021649.html&k=oIvRg1%2BdGAgOoM1BIlLLq
> >>w%3D%3D%0A&r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=B8yQ75
> >>uRFa7dyD%2FbgPgO%2FMT2x229MkK1vHWBVAzpFtM%3D%0A&s=d9cbcbde99b7c88c15ca138
> >>499de8f36edd4247ce351e41b241d675b09b79956>
> >>
> >> -Sylvain
> >>
> >>
> >> 2014-02-17 13:00 GMT+01:00 Gary Kotton <gkotton at vmware.com
> <mailto:gkotton at vmware.com>
> >><mailto:gkotton at vmware.com <mailto:gkotton at vmware.com>>>:
> >>
> >> Hi,
> >> It seems that the command 'reverify bug <number>' is not
> working.
> >>Anyone else experienced this lately.
> >> Thanks
> >> Gary
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> <mailto:OpenStack-dev at lists.openstack.org>
> >><mailto:OpenStack-dev at lists.openstack.org
> <mailto:OpenStack-dev at lists.openstack.org>>
> >>
> >>https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi
> >>-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r
> >>=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=Nf6%2FyHMSSchQO59x
> >>Icg6NKX1O2wlkkHHd42bYQim0k0%3D%0A&s=1a2a59024e18b786b5c2c13535b7f3d050363
> >>ff628dc96b8561e12489d30c0bd
> >>
> >><https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cg
> >>i-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&
> >>r=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=B8yQ75uRFa7dyD%2F
> >>bgPgO%2FMT2x229MkK1vHWBVAzpFtM%3D%0A&s=21e7f4ebc24f0a13780981720f9332111d
> >>d603f3c0661229dc90d82bfb5c3122>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> <mailto:OpenStack-dev at lists.openstack.org>
> >>
> >>https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi
> >>-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r
> >>=eH0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=Nf6%2FyHMSSchQO59x
> >>Icg6NKX1O2wlkkHHd42bYQim0k0%3D%0A&s=1a2a59024e18b786b5c2c13535b7f3d050363
> >>ff628dc96b8561e12489d30c0bd
> >>
> >_______________________________________________
> >OpenStack-dev mailing list
> >OpenStack-dev at lists.openstack.org
> <mailto:OpenStack-dev at lists.openstack.org>
> >https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-
> >bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=e
> >H0pxTUZo8NPZyF6hgoMQu%2BfDtysg45MkPhCZFxPEq8%3D%0A&m=Nf6%2FyHMSSchQO59xIcg
> >6NKX1O2wlkkHHd42bYQim0k0%3D%0A&s=1a2a59024e18b786b5c2c13535b7f3d050363ff62
> >8dc96b8561e12489d30c0bd
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> <mailto:OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> <mailto:OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> --
> Sincerely yours,
> Sergey Lukjanov
> Savanna Technical Lead
> Mirantis Inc.
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
--
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 547 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140218/657c58d3/attachment.pgp>
More information about the OpenStack-dev
mailing list