Well, somehow I accidentally clicked send! \o/ If you can confirm that the provision_state is ERROR, and if you can identify how the machines got there, it would be helpful. If the machines are still in working order in the database, you may need to actually edit the database because we offer no explicit means to force override the state, mainly to help prevent issues sort of exactly like this. I suspect you may be encountering issues if the node is marked in maintenance state. If the power state is None, maintenance is also set automatically. Newer versions of ironic _do_ periodically check nodes and reset that state, but again it is something to check and if there are continued connectivity issues to the BMC then that may not be happening. So: to recap: 1) Verify the node's provision_state is ERROR. If ERROR is coming from Nova, that is a different situation. 2) Ensure the node is not set in maintenance mode[3] 3) You may also need to ensure the ipmi_address/ipmi_username/ipmi_password is also correct for the node that matches what can be accessed on the motherboard. Additionally, you may also want to externally verify that you actually query the IPMI BMCs. If this somehow started down this path due to power management being lost due to the BMC, some BMCs can have some weirdness around IP networking so it is always good just to manually check using ipmitool. One last thing, is target_provision_state set for these nodes? [3]: https://docs.openstack.org/python-ironicclient/latest/cli/osc/v1/index.html#... On Wed, Sep 23, 2020 at 9:20 PM Julia Kreger <juliaashleykreger@gmail.com> wrote:
Greetings Paul,
Obviously, deleting and re-enrolling would be an action of last resort. The only way that I can think you could have gotten the machines into the provision state of ERROR is if they were somehow requested to be un-provisioned.
The state machine diagram[0], refers to the provision state verb as "deleted", but the command line tool command this is undeploy[1].
[0]: https://docs.openstack.org/ironic/latest/_images/states.svg [1]: https://docs.openstack.org/python-ironicclient/latest/cli/osc/v1/index.html#...
On Wed, Sep 23, 2020 at 4:58 PM Paul Browne <pfb29@cam.ac.uk> wrote:
Hello all,
I have a handful of baremetal nodes enrolled in Ironic that use the IPMI hardware type, whose motherboards were recently replaced in a hardware recall by the vendor.
After the replacement, the BMC IPMI-over-LAN feature was accidentally left disabled on the nodes, and future attempts to control them with Ironic has put these nodes into the ERROR provisioning state.
The IPMI-over-LAN feature on the boards has been enabled again as expected, but is there now any easy way to get the BM nodes back out of that ERROR state, without first deleting and re-enrolling them?
-- ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29@cam.ac.uk Tel: 0044-1223-746548 *******************