[ironic][xena] problems updating redfish_password for existing node

Wade Albright the.wade.albright at gmail.com
Mon Jul 18 22:20:32 UTC 2022


I'm happy to supply some logs, what verbosity level should i use? And
should I just embed the logs in email to the list or upload somewhere?

On Mon, Jul 18, 2022 at 3:14 PM Julia Kreger <juliaashleykreger at gmail.com>
wrote:

> If you could supply some conductor logs, that would be helpful. It
> should be re-authenticating, but obviously we have a larger bug there
> we need to find the root issue behind.
>
> On Mon, Jul 18, 2022 at 3:06 PM Wade Albright
> <the.wade.albright at gmail.com> wrote:
> >
> > I was able to use the patches to update the code, but unfortunately the
> problem is still there for me.
> >
> > I also tried an RPM upgrade to the versions Julia mentioned had the
> fixes, namely Sushy 3.12.1 - Released May 2022 and Ironic 18.2.1 - Released
> in January 2022. But it did not fix the problem.
> >
> > I am able to consistently reproduce the error.
> >  - step 1: change BMC password directly on the node itself
> >  - step 2: update BMC password (redfish_password) in ironic with
> 'openstack baremetal node set <nodename> --driver-info
> redfish_password='newpass'
> >
> > After step 1 there are errors in the logs entries like "Session
> authentication appears to have been lost at some point in time" and
> eventually it puts the node into maintenance mode and marks the power state
> as "none."
> > After step 2 and taking the host back out of maintenance mode, it goes
> through a similar set of log entries puts the node into MM again.
> >
> > After the above steps, a conductor restart fixes the problem and
> operations work normally again. Given this it seems like there is still
> some kind of caching issue.
> >
> > On Sat, Jul 16, 2022 at 6:01 PM Wade Albright <
> the.wade.albright at gmail.com> wrote:
> >>
> >> Hi Julia,
> >>
> >> Thank you so much for the reply! Hopefully this is the issue. I'll try
> out the patches next week and report back. I'll also email you on Monday
> about the versions, that would be very helpful to know.
> >>
> >> Thanks again, really appreciate it.
> >>
> >> Wade
> >>
> >>
> >>
> >> On Sat, Jul 16, 2022 at 4:36 PM Julia Kreger <
> juliaashleykreger at gmail.com> wrote:
> >>>
> >>> Greetings!
> >>>
> >>> I believe you need two patches, one in ironic and one in sushy.
> >>>
> >>> Sushy:
> >>> https://review.opendev.org/c/openstack/sushy/+/832860
> >>>
> >>> Ironic:
> >>> https://review.opendev.org/c/openstack/ironic/+/820588
> >>>
> >>> I think it is variation, and the comment about working after you
> restart the conductor is the big signal to me. I’m on a phone on a bad data
> connection, if you email me on Monday I can see what versions the fixes
> would be in.
> >>>
> >>> For the record, it is a session cache issue, the bug was that the
> service didn’t quite know what to do when auth fails.
> >>>
> >>> -Julia
> >>>
> >>>
> >>> On Fri, Jul 15, 2022 at 2:55 PM Wade Albright <
> the.wade.albright at gmail.com> wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> I'm hitting a problem when trying to update the redfish_password for
> an existing node. I'm curious to know if anyone else has encountered this
> problem. I'm not sure if I'm just doing something wrong or if there is a
> bug. Or if the problem is unique to my setup.
> >>>>
> >>>> I have a node already added into ironic with all the driver details
> set, and things are working fine. I am able to run deployments.
> >>>>
> >>>> Now I need to change the redfish password on the host. So I update
> the password for redfish access on the host, then use an 'openstack
> baremetal node set <node> --driver-info redfish_password=<newpass>' command
> to set the new redfish_password.
> >>>>
> >>>> Once this has been done, deployment no longer works. I see redfish
> authentication errors in the logs and the operation fails. I waited a bit
> to see if there might just be a delay in updating the password, but after
> awhile it still didn't work.
> >>>>
> >>>> I restarted the conductor, and after that things work fine again. So
> it seems like the password is cached or something. Is there a way to force
> the password to update? I even tried removing the redfish credentials and
> re-adding them, but that didn't work either. Only a conductor restart seems
> to make the new password work.
> >>>>
> >>>> We are running Xena, using rpm installation on Oracle Linux 8.5.
> >>>>
> >>>> Thanks in advance for any help with this issue.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20220718/a8bc4590/attachment-0001.htm>


More information about the openstack-discuss mailing list