[all][TC] Bare rechecks stats week of 25.07
Hi,
Here are fresh data about bare rechecks.
+--------------------+---------------+--------------+-------------------+ | Team | Bare rechecks | All Rechecks | Bare rechecks [%] | +--------------------+---------------+--------------+-------------------+ | requirements | 2 | 2 | 100.0 | | keystone | 1 | 1 | 100.0 | | OpenStackSDK | 3 | 3 | 100.0 | | tacker | 3 | 3 | 100.0 | | freezer | 1 | 1 | 100.0 | | rally | 1 | 1 | 100.0 | | OpenStack-Helm | 14 | 15 | 93.33 | | cinder | 6 | 7 | 85.71 | | kolla | 20 | 25 | 80.0 | | neutron | 4 | 5 | 80.0 | | Puppet OpenStack | 10 | 13 | 76.92 | | OpenStack Charms | 3 | 6 | 50.0 | | nova | 11 | 22 | 50.0 | | tripleo | 8 | 19 | 42.11 | | ironic | 1 | 3 | 33.33 | | Quality Assurance | 1 | 3 | 33.33 | | octavia | 0 | 1 | 0.0 | | OpenStackAnsible | 0 | 1 | 0.0 | | designate | 0 | 1 | 0.0 | | Release Management | 0 | 1 | 0.0 | | horizon | 0 | 1 | 0.0 | +--------------------+---------------+--------------+-------------------+
Those data should be more accurate as my script now counts only comments from last 7 days, not all comments from patches updated in last 7 days.
Reminder: "bare rechecks" are recheck comments without any reason given. If You need to do recheck for patch due to failed job(s), please first check such failed job and try to identify what was the issue there. Maybe there is already opened bug for that or maybe You can open new one and add it as explanation in the recheck comment. Or maybe it was some infra issue, in such case short explanation in the comment would also be enough.
Hi. This is going to be embarrassing, but in my opinion it's better to be embarrassed and corrected than stay in error.
What is the preferred way of stating the reason for a recheck? I've kind of assumed that it's something like writing:
" recheck: this is broken"
but that doesn't trigger the CI. I've tried looking it up in the Zuul and Opendev docs, but couldn't find anything.
On Thu, Jul 28, 2022 at 1:02 PM Slawek Kaplonski skaplons@redhat.com wrote:
Hi,
Here are fresh data about bare rechecks.
+--------------------+---------------+--------------+-------------------+ | Team | Bare rechecks | All Rechecks | Bare rechecks [%] | +--------------------+---------------+--------------+-------------------+ | requirements | 2 | 2 | 100.0 | | keystone | 1 | 1 | 100.0 | | OpenStackSDK | 3 | 3 | 100.0 | | tacker | 3 | 3 | 100.0 | | freezer | 1 | 1 | 100.0 | | rally | 1 | 1 | 100.0 | | OpenStack-Helm | 14 | 15 | 93.33 | | cinder | 6 | 7 | 85.71 | | kolla | 20 | 25 | 80.0 | | neutron | 4 | 5 | 80.0 | | Puppet OpenStack | 10 | 13 | 76.92 | | OpenStack Charms | 3 | 6 | 50.0 | | nova | 11 | 22 | 50.0 | | tripleo | 8 | 19 | 42.11 | | ironic | 1 | 3 | 33.33 | | Quality Assurance | 1 | 3 | 33.33 | | octavia | 0 | 1 | 0.0 | | OpenStackAnsible | 0 | 1 | 0.0 | | designate | 0 | 1 | 0.0 | | Release Management | 0 | 1 | 0.0 | | horizon | 0 | 1 | 0.0 | +--------------------+---------------+--------------+-------------------+
Those data should be more accurate as my script now counts only comments from last 7 days, not all comments from patches updated in last 7 days.
Reminder: "bare rechecks" are recheck comments without any reason given. If You need to do recheck for patch due to failed job(s), please first check such failed job and try to identify what was the issue there. Maybe there is already opened bug for that or maybe You can open new one and add it as explanation in the recheck comment. Or maybe it was some infra issue, in such case short explanation in the comment would also be enough.
--
Slawek Kaplonski
Principal Software Engineer
Red Hat
On 2022-07-28 16:33:46 +0200 (+0200), Jiri Podivin wrote: [...]
What is the preferred way of stating the reason for a recheck? I've kind of assumed that it's something like writing:
" recheck: this is broken"
but that doesn't trigger the CI. I've tried looking it up in the Zuul and Opendev docs, but couldn't find anything.
[...]
The regular expression for it is configured here:
https://opendev.org/openstack/project-config/src/commit/6b12602/zuul.d/pipel...
What I usually do is "recheck because of nondeterminism in test foo" (or whatever the reason).
Don't prepend anything before the word "recheck" and also follow the word with a space before you add any other text. I think Zuul must assume a required space after the regular expression since that doesn't seem to be encoded in the regex you see there. Make sure the "recheck" appears on the very first line of the comment too. Aside from that, you can add anything you like after it.
Switching in/out of WIP and leaving votes at the same time as the recheck should also be avoided as they seem to cause the expression not to match.
Thanks Jeremy, I'll keep it in mind.
On Thu, Jul 28, 2022 at 5:26 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2022-07-28 16:33:46 +0200 (+0200), Jiri Podivin wrote: [...]
What is the preferred way of stating the reason for a recheck? I've kind
of
assumed that it's something like writing:
" recheck: this is broken"
but that doesn't trigger the CI. I've tried looking it up in the Zuul and Opendev docs, but couldn't find anything.
[...]
The regular expression for it is configured here:
https://opendev.org/openstack/project-config/src/commit/6b12602/zuul.d/pipel...
What I usually do is "recheck because of nondeterminism in test foo" (or whatever the reason).
Don't prepend anything before the word "recheck" and also follow the word with a space before you add any other text. I think Zuul must assume a required space after the regular expression since that doesn't seem to be encoded in the regex you see there. Make sure the "recheck" appears on the very first line of the comment too. Aside from that, you can add anything you like after it.
Switching in/out of WIP and leaving votes at the same time as the recheck should also be avoided as they seem to cause the expression not to match. -- Jeremy Stanley
On 2022-07-28 15:15:34 +0000 (+0000), Jeremy Stanley wrote: [...]
follow the word "recheck" with a space before you add any other text. I think Zuul must assume a required space after the regular expression since that doesn't seem to be encoded in the regex you see there.
[...]
Just to follow up, I misinterpreted the results of an earlier experiment when trying to confirm this. Based on revisiting it today with https://review.opendev.org/852605 it looks like you don't need a space between "recheck" and other text, so "recheck: whatever" also works fine.
On Do, 2022-07-28 at 12:51 +0200, Slawek Kaplonski wrote:
Hi,
Here are fresh data about bare rechecks.
+--------------------+---------------+--------------+-------------------+ | Team | Bare rechecks | All Rechecks | Bare rechecks [%] | +--------------------+---------------+--------------+-------------------+ | requirements | 2 | 2 | 100.0 | | keystone | 1 | 1 | 100.0 | | OpenStackSDK | 3 | 3 | 100.0 | | tacker | 3 | 3 | 100.0 | | freezer | 1 | 1 | 100.0 | | rally | 1 | 1 | 100.0 | | OpenStack-Helm | 14 | 15 | 93.33 | | cinder | 6 | 7 | 85.71 | | kolla | 20 | 25 | 80.0 | | neutron | 4 | 5 | 80.0 | | Puppet OpenStack | 10 | 13 | 76.92 | | OpenStack Charms | 3 | 6 | 50.0 | | nova | 11 | 22 | 50.0 | | tripleo | 8 | 19 | 42.11 | | ironic | 1 | 3 | 33.33 | | Quality Assurance | 1 | 3 | 33.33 | | octavia | 0 | 1 | 0.0 | | OpenStackAnsible | 0 | 1 | 0.0 | | designate | 0 | 1 | 0.0 | | Release Management | 0 | 1 | 0.0 | | horizon | 0 | 1 | 0.0 | +--------------------+---------------+--------------+-------------------+
Those data should be more accurate as my script now counts only comments from last 7 days, not all comments from patches updated in last 7 days.
Reminder: "bare rechecks" are recheck comments without any reason given. If You need to do recheck for patch due to failed job(s), please first check such failed job and try to identify what was the issue there. Maybe there is already opened bug for that or maybe You can open new one and add it as explanation in the recheck comment. Or maybe it was some infra issue, in such case short explanation in the comment would also be enough.
Sorry to resurrect this old thread, but while this is all very interesting, is there any documentation on rechecks in the developer guide? We are unable to find anything beside https://docs.openstack.org/contributors/code-and-documentation/elastic-reche...
which does not really say much about when and how to recheck and when and where to report bugs.
any advice would be greatly appreciated.
On 2022-12-16 12:53:06 +0000 (+0000), Sven Kieske wrote: [...
Sorry to resurrect this old thread, but while this is all very interesting, is there any documentation on rechecks in the developer guide? We are unable to find anything beside https://docs.openstack.org/contributors/code-and-documentation/elastic-reche...
which does not really say much about when and how to recheck and when and where to report bugs.
any advice would be greatly appreciated.
It's covered in this section of the Project Team Guide:
https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-tes...
On Fri, 2022-12-16 at 12:59 +0000, Jeremy Stanley wrote:
On 2022-12-16 12:53:06 +0000 (+0000), Sven Kieske wrote: [...
Sorry to resurrect this old thread, but while this is all very interesting, is there any documentation on rechecks in the developer guide? We are unable to find anything beside https://docs.openstack.org/contributors/code-and-documentation/elastic-reche...
which does not really say much about when and how to recheck and when and where to report bugs.
any advice would be greatly appreciated.
It's covered in this section of the Project Team Guide:
https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-tes...
there is not only documentatoin as noted above but the zuul message links that to your in its comment on every ci failure.
""" Build failed (check pipeline). For information on how to proceed, see https://docs.opendev.org/opendev/infra-manual/latest/developers.html#automat... """
we might also want to add https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-tes... to that message here https://github.com/openstack/project-config/blob/master/zuul.d/pipelines.yam...
i kind fo feel tracking the barerehceck in team calls weekly or on the mailing list is not useful anymore.
the people that pay attention to these already know why we dont want to do bare rehcecks and the cases that still happen are either because they dont care, where in a rush and left a comment seperatly that the script did not see ( i.e. hit send to soon and added a comment after explinging the recheck) or forgot.
we have set the expecation that contibutors shoudl provide a reason before rechecking so other then updating the zuul failure message or adding a zuul job to trigger on bare rechecks and comment saying "you shoudl not do this and here is what you should do instead" im not sure that we are going to make more progress on this.
On 2022-12-16 13:38:48 +0000 (+0000), Sean Mooney wrote: [...]
we might also want to add https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-tes... to that message here https://github.com/openstack/project-config/blob/master/zuul.d/pipelines.yam...
[...]
The challenge with that is that the "openstack" tenant in OpenDev's Zuul deployment is shared by a lot of other non-OpenStack projects who may have their own independent policies about such things, so we try to stay neutral (as well as brief) in those tenant-wide messages and not refer to project-specific documentation there. However, the OpenDev Developer's Guide section we already link to does say a couple of things like "if this is an OpenStack project then..." so I suppose we could expand that section to link to the OpenStack Project Team Guide in addition to the current OpenStack Contributor Guide link.
---- On Fri, 16 Dec 2022 05:54:38 -0800 Jeremy Stanley wrote ---
On 2022-12-16 13:38:48 +0000 (+0000), Sean Mooney wrote: [...]
we might also want to add https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-tes... to that message here https://github.com/openstack/project-config/blob/master/zuul.d/pipelines.yam...
[...]
The challenge with that is that the "openstack" tenant in OpenDev's Zuul deployment is shared by a lot of other non-OpenStack projects
I thought ..openstack/project-config/blob/master/zuul.d/pipelines.yaml is specifc to the OpenStack also top of file mentioned the same - "Shared zuul config specific to the OpenStack Project.."
I also think adding that link to zuul message can be more useful and easy to learn about these bare rechecks. I have noticed in other place also where developers asked about any documentation on bare rechecks.
-gmann
who may have their own independent policies about such things, so we try to stay neutral (as well as brief) in those tenant-wide messages and not refer to project-specific documentation there. However, the OpenDev Developer's Guide section we already link to does say a couple of things like "if this is an OpenStack project then..." so I suppose we could expand that section to link to the OpenStack Project Team Guide in addition to the current OpenStack Contributor Guide link. -- Jeremy Stanley
On 2022-12-16 16:41:49 -0800 (-0800), Ghanshyam Mann wrote: [...]
I thought ..openstack/project-config/blob/master/zuul.d/pipelines.yaml is specifc to the OpenStack also top of file mentioned the same - "Shared zuul config specific to the OpenStack Project.."
That's definitely inaccurate and should get fixed. The "openstack" tenant was originally the only Zuul tenant we had (when Zuul first became multi-tenant capable), and only a few projects have moved out to or started in other tenants since then. For example, StarlingX and Airship repositories all gate in the "openstack" tenant. Alternatively, we could decide to kick all non-OpenStack projects out of that tenant so it can be customized more for OpenStack's use, but would be a significant undertaking if we went that route.
I also think adding that link to zuul message can be more useful and easy to learn about these bare rechecks. I have noticed in other place also where developers asked about any documentation on bare rechecks.
We need to keep the comments from Zuul as brief and to the point as possible. Overloading those comments with more documentation URLs is not good for usability, but making sure the documentation we link to gets people to the right information by having it link to more places is fine.
participants (6)
-
Ghanshyam Mann
-
Jeremy Stanley
-
Jiri Podivin
-
Sean Mooney
-
Slawek Kaplonski
-
Sven Kieske