[Horizon][Designate][Tacker][Tobiko][OSH][Ironic] Flaky jobs and retries
Hello, Zuulv3 has a feature where it will rerun jobs that it thinks failed due to infrastructure problems up to some configured limit. For us the limit is 3 job attempts. The circumstances where Zuul thinks it has infrastructure issues are when pre-run playbooks fail (because these are expected to be stable), and when Ansible returns with an exit code signifying network connectivity issues. Recently I updated Zuul to report the job attempt in job inventories and am using that to report this data into logstash and elasticsearch. Now I can use queries like: 'zuul_attempts:"3" AND filename:"job-output.txt" AND message:"Job console starting"' to see which jobs are flaky and reattempting. I'm writing this email to report on some of what I've found in the hopes that the problems we have can be fixed and hopefully result in better jobs across the board. Designate Horizon Dashboard The nodejs jobs for stable/rocky expect to run against nodejs4 but run on Bionic which has no nodejs 4 packaging. This causes the pre-run to fail as it cannot install the requested package, https://zuul.opendev.org/t/openstack/build/498ebb052e2b4a3393b0939820ee8927/.... Fix in progress here, https://review.opendev.org/#/c/699241/1, to pin to Xenial. Thank you! Horizon projects and dashboards may want to double check they don't have similar problems with nodejs version and node mismatches. Tacker tacker-functional-devstack-multinode-python3 (and possibly other tacker devstack jobs) attempt to download an openwrt image during devstack runs and the server isn't responding. This fails in pre-run, http://zuul.openstack.org/build/27ff514c77724250968d60469923f613/log/control..., and results in the job being retried. I am not aware of a fix for this, but you'll likely need to find a new image host. Let the infra team know if we can help. Tobiko tobiko-devstack-faults-centos-7 is failing because this job runs on CentOS 7 using python2 but Nova needs python3.6 now. This fails in pre-run, https://zuul.opendev.org/t/openstack/build/8071780e7ba748169b447f1d42e069fc/..., and forces the job to be rerun. I'm not sure what the best fix is here. Maybe run your jobs on Ubuntu until CentOS 8 is supported by devstack? OpenStack Helm Ansible has decided that become_user is not valid on include_role tasks. https://opendev.org/openstack/openstack-helm-infra/src/branch/master/roles/d... seems to be the issue. This causes Airship's deckhand functional docker jobs to fail in pre-run, https://zuul.opendev.org/t/openstack/build/7493038ee1744465b2387b44e067d029/..., and the jobs are retried. It is possible this is fallout from Zuul's default ansible version being bumped to 2.8 from 2.7. This also seems to affect OSH's kubernetes keystone auth job, https://zuul.opendev.org/t/openstack/build/68d937b3e5e3449cb6fe2e6947bbf0db/... Ironic The Ironic example is a bit different and runs into some fun Ansible behaviors. Ironic's IPA multinode jobs (possibly others too) are filling the root disk of the test nodes on some clouds, http://paste.openstack.org/show/787641/. Ansible uses /tmp to bootstrap its ssh connectivity and when /tmp is on / and / is full that results in Ansible returning a network failure error code. This then causes Zuul to rerun the job. Unfortunately because Ansible thinks networking is broken we have to do special things in a Zuul cleanup playbook to double check disk usage and that info is only available on the Zuul executors. But I've double checked for you and can confirm this is what is happening. The fix here is to use the ephemeral disk which is mounted on /opt and contains much more disk space. Another option would be to reduce the size of Ironic's fake baremetal images. In any case they are aware of this and we should expect a fix soon. There are likely more cases I haven't found yet, but I wanted to point out this Zuul behavior to people. The intent is that it handle exceptional cases and not paper over failures that happen often or even 100% of the time. We should do our best to fix these problems when they pop up. Thank you to everyone that has already stepped up to help fix some of these! Clark
On Mon, Dec 16, 2019, at 10:20 AM, Clark Boylan wrote:
Hello,
Zuulv3 has a feature where it will rerun jobs that it thinks failed due to infrastructure problems up to some configured limit. For us the limit is 3 job attempts. The circumstances where Zuul thinks it has infrastructure issues are when pre-run playbooks fail (because these are expected to be stable), and when Ansible returns with an exit code signifying network connectivity issues.
Recently I updated Zuul to report the job attempt in job inventories and am using that to report this data into logstash and elasticsearch. Now I can use queries like: 'zuul_attempts:"3" AND filename:"job-output.txt" AND message:"Job console starting"' to see which jobs are flaky and reattempting. I'm writing this email to report on some of what I've found in the hopes that the problems we have can be fixed and hopefully result in better jobs across the board.
snip
Tobiko
tobiko-devstack-faults-centos-7 is failing because this job runs on CentOS 7 using python2 but Nova needs python3.6 now. This fails in pre-run, https://zuul.opendev.org/t/openstack/build/8071780e7ba748169b447f1d42e069fc/..., and forces the job to be rerun. I'm not sure what the best fix is here. Maybe run your jobs on Ubuntu until CentOS 8 is supported by devstack?
Looks like Tobiko was updated to build python3 on CentOS 7 to address this issue. Unfortunately, it looks like glance imports sqlite and the built python did not include sqlite bindings. This results in g-api failing to start and the job still fails with RETRY_LIMIT, https://zuul.opendev.org/t/openstack/build/880ded9fe11c41da9c7622d85746df23/.... Note that this job ran against the proposed fix and was tested pre merge. You don't need to wait for the change to merge to observe these results. Instead Zuul allows you to push the fix up, confirm it works, then merge the change. The fix for the sqlite issue is in progress here, https://review.opendev.org/#/c/699945/2, but I wanted to point out that Zuul helps us to avoid these problems in the first place with its pre merge testing. Finally, CentOS 7 does provide python3.6 packages in the main repository now. I'm not sure if that is sufficient for Tobiko's needs but that may help simplify things here. Clark
On Tue, Dec 17, 2019 at 3:22 AM Clark Boylan <cboylan@sapwetik.org> wrote:
Hello,
Zuulv3 has a feature where it will rerun jobs that it thinks failed due to infrastructure problems up to some configured limit. For us the limit is 3 job attempts. The circumstances where Zuul thinks it has infrastructure issues are when pre-run playbooks fail (because these are expected to be stable), and when Ansible returns with an exit code signifying network connectivity issues.
Recently I updated Zuul to report the job attempt in job inventories and am using that to report this data into logstash and elasticsearch. Now I can use queries like: 'zuul_attempts:"3" AND filename:"job-output.txt" AND message:"Job console starting"' to see which jobs are flaky and reattempting. I'm writing this email to report on some of what I've found in the hopes that the problems we have can be fixed and hopefully result in better jobs across the board.
Designate Horizon Dashboard
The nodejs jobs for stable/rocky expect to run against nodejs4 but run on Bionic which has no nodejs 4 packaging. This causes the pre-run to fail as it cannot install the requested package, https://zuul.opendev.org/t/openstack/build/498ebb052e2b4a3393b0939820ee8927/.... Fix in progress here, https://review.opendev.org/#/c/699241/1, to pin to Xenial. Thank you!
Horizon projects and dashboards may want to double check they don't have similar problems with nodejs version and node mismatches.
"nodejs4-jobs job" in stable/rocky specifies "nodeset: ubuntu-xenial" in openstack-zuul-jobs, but most horizon plugins requires horizon as required-projects so they usually redefine nodejs4 related jobs. I checked about half of horizon plugins and confirm they specify "nodeset: ubuntu-xenial" if they redefine the nodejs4 jobs with openstack/horizon. Perhaps an ideal solution would be that the horizon team split out stuffs used by horizon plugins into a separate project like horizon-lib, but considering the current development resources I don't think it can happen in near future unless someone steps up. Thanks, Akihiro Motoki (irc: amotoki)
Tacker
tacker-functional-devstack-multinode-python3 (and possibly other tacker devstack jobs) attempt to download an openwrt image during devstack runs and the server isn't responding. This fails in pre-run, http://zuul.openstack.org/build/27ff514c77724250968d60469923f613/log/control..., and results in the job being retried. I am not aware of a fix for this, but you'll likely need to find a new image host. Let the infra team know if we can help.
Tobiko
tobiko-devstack-faults-centos-7 is failing because this job runs on CentOS 7 using python2 but Nova needs python3.6 now. This fails in pre-run, https://zuul.opendev.org/t/openstack/build/8071780e7ba748169b447f1d42e069fc/..., and forces the job to be rerun. I'm not sure what the best fix is here. Maybe run your jobs on Ubuntu until CentOS 8 is supported by devstack?
OpenStack Helm
Ansible has decided that become_user is not valid on include_role tasks. https://opendev.org/openstack/openstack-helm-infra/src/branch/master/roles/d... seems to be the issue. This causes Airship's deckhand functional docker jobs to fail in pre-run, https://zuul.opendev.org/t/openstack/build/7493038ee1744465b2387b44e067d029/..., and the jobs are retried. It is possible this is fallout from Zuul's default ansible version being bumped to 2.8 from 2.7.
This also seems to affect OSH's kubernetes keystone auth job, https://zuul.opendev.org/t/openstack/build/68d937b3e5e3449cb6fe2e6947bbf0db/...
Ironic
The Ironic example is a bit different and runs into some fun Ansible behaviors. Ironic's IPA multinode jobs (possibly others too) are filling the root disk of the test nodes on some clouds, http://paste.openstack.org/show/787641/. Ansible uses /tmp to bootstrap its ssh connectivity and when /tmp is on / and / is full that results in Ansible returning a network failure error code. This then causes Zuul to rerun the job. Unfortunately because Ansible thinks networking is broken we have to do special things in a Zuul cleanup playbook to double check disk usage and that info is only available on the Zuul executors. But I've double checked for you and can confirm this is what is happening.
The fix here is to use the ephemeral disk which is mounted on /opt and contains much more disk space. Another option would be to reduce the size of Ironic's fake baremetal images. In any case they are aware of this and we should expect a fix soon.
There are likely more cases I haven't found yet, but I wanted to point out this Zuul behavior to people. The intent is that it handle exceptional cases and not paper over failures that happen often or even 100% of the time. We should do our best to fix these problems when they pop up. Thank you to everyone that has already stepped up to help fix some of these!
Clark
participants (2)
-
Akihiro Motoki
-
Clark Boylan