Open Stack

Well, somehow I accidentally clicked send! \o/

If you can confirm that the provision_state is ERROR, and if you can
identify how the machines got there, it would be helpful. If the
machines are still in working order in the database, you may need to
actually edit the database because we offer no explicit means to force
override the state, mainly to help prevent issues sort of exactly like
this. I suspect you may be encountering issues if the node is marked
in maintenance state. If the power state is None, maintenance is also
set automatically. Newer versions of ironic _do_ periodically check
nodes and reset that state, but again it is something to check and if
there are continued connectivity issues to the BMC then that may not
be happening.

So: to recap:

1) Verify the node's provision_state is ERROR. If ERROR is coming from
Nova, that is a different situation.
2) Ensure the node is not set in maintenance mode[3]
3) You may also need to ensure the
ipmi_address/ipmi_username/ipmi_password is also correct for the node
that matches what can be accessed on the motherboard.

Additionally, you may also want to externally verify that you actually
query the IPMI BMCs. If this somehow started down this path due to
power management being lost due to the BMC, some BMCs can have some
weirdness around IP networking so it is always good just to manually
check using ipmitool.

One last thing, is target_provision_state set for these nodes?

[3]: https://docs.openstack.org/python-ironicclient/latest/cli/osc/v1/index.html#baremetal-node-maintenance-unset

On Wed, Sep 23, 2020 at 9:20 PM Julia Kreger
<juliaashleykreger at gmail.com> wrote:
>
> Greetings Paul,
>
> Obviously, deleting and re-enrolling would be an action of last
> resort. The only way that I can think you could have gotten the
> machines into the provision state of ERROR is if they were somehow
> requested to be un-provisioned.
>
> The state machine diagram[0], refers to the provision state verb as
> "deleted", but the command line tool command this is undeploy[1].
>
>
> [0]: https://docs.openstack.org/ironic/latest/_images/states.svg
> [1]: https://docs.openstack.org/python-ironicclient/latest/cli/osc/v1/index.html#baremetal-node-undeploy
>
>
>
> On Wed, Sep 23, 2020 at 4:58 PM Paul Browne <pfb29 at cam.ac.uk> wrote:
> >
> > Hello all,
> >
> > I have a handful of baremetal nodes enrolled in Ironic that use the IPMI hardware type, whose motherboards were recently replaced in a hardware recall by the vendor.
> >
> > After the replacement, the BMC IPMI-over-LAN feature was accidentally left disabled on the nodes, and future attempts to control them with Ironic has put these nodes into the ERROR provisioning state.
> >
> > The IPMI-over-LAN feature on the boards has been enabled again as expected, but is there now any easy way to get the BM nodes back out of that ERROR state, without first deleting and re-enrolling them?
> >
> > --
> > *******************
> > Paul Browne
> > Research Computing Platforms
> > University Information Services
> > Roger Needham Building
> > JJ Thompson Avenue
> > University of Cambridge
> > Cambridge
> > United Kingdom
> > E-Mail: pfb29 at cam.ac.uk
> > Tel: 0044-1223-746548
> > *******************