Open Stack

On Tue, Sep 20, 2022, at 5:01 AM, Amol Kahat wrote:
> Hello All,
>
> Description of problem:
> NODE_FAILURE when running tripleo-ci-centos-9-scenario010-standalone job.
>
> We have been seeing this failure[1] since 09/17. Logs are not present 
> so it's hard to say what is the root cause of this issue.

NODE_FAILURE indicates that Nodepool could not boot any nodes to fulfill the nodeset requested by your job. The reason there are no job logs is that this occurs before any Zuul jobs can run. See below for the information that is available though.

>
> This job uses nodeset: single-centos-9-node-nested-virt - so assumption 
> is that it's the nest-virt nodeset

Fungi ended up debugging and correcting [2] this issue, but I was able to get a good idea of what might be happening using just a phone and no special access. This label is provided by four cloud providers [3][4][5][6] only two of which currently have positive max-servers values [7][8]. We can check the general health of those providers using Grafana [9]. This shows the providers idled for some reason. Finding the specific cause of that idling did require extra privileges, and fungi pasted that info for us [10].

While I agree it is more difficult to say what the root cause is, there is still plenty of information to narrow the problem down and determine what might be going on. Ideally we would publish the Nodepool launcher logs too, then we wouldn't need special access to retrieve the traceback in the paste. Unfortunately, there has been a long standing concern that we might leak cloud credentials if openstacksdk or Nodepool logging do something we don't expect. There is also a Zuul spec to merge Nodepool functionality into Zuul proper [11] which should allow us to report better errors when NODE_FAILURE occurs.

>
> [1] 
> https://zuul.openstack.org/builds?job_name=tripleo-ci-centos-9-scenario010-standalone+&skip=0

[2] https://review.opendev.org/c/openstack/project-config/+/858523
[3] https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl02.opendev.org.yaml#L210
[4] https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl03.opendev.org.yaml#L404
[5] https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl04.opendev.org.yaml#L174
[6] https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl04.opendev.org.yaml#L195
[7] https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl04.opendev.org.yaml#L82
[8] https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl04.opendev.org.yaml#L194
[9] https://grafana.opendev.org/d/2b4dba9e25/nodepool-ovh?orgId=1&from=now-5d&to=now
[10] https://paste.opendev.org/show/816812/
[11] https://zuul-ci.org/docs/zuul/latest/developer/specs/nodepool-in-zuul.html

>
> Thanks,
> -- 
> *Amol Kahat*
> Software Engineer
> *Red Hat India Pvt. Ltd. Pune, India.*
> akahat at redhat.com
> B764 E6F8 F4C1 A1AF 816C  6840 FDD3 BA6C 832D 7715