Open Stack

I was able to use the patches to update the code, but unfortunately the
problem is still there for me.

I also tried an RPM upgrade to the versions Julia mentioned had the fixes,
namely Sushy 3.12.1 - Released May 2022 and Ironic 18.2.1 - Released in
January 2022. But it did not fix the problem.

I am able to consistently reproduce the error.
 - step 1: change BMC password directly on the node itself
 - step 2: update BMC password (redfish_password) in ironic with 'openstack
baremetal node set <nodename> --driver-info redfish_password='newpass'

After step 1 there are errors in the logs entries like "Session
authentication appears to have been lost at some point in time" and
eventually it puts the node into maintenance mode and marks the power state
as "none."
After step 2 and taking the host back out of maintenance mode, it goes
through a similar set of log entries puts the node into MM again.

After the above steps, a conductor restart fixes the problem and operations
work normally again. Given this it seems like there is still some kind of
caching issue.

On Sat, Jul 16, 2022 at 6:01 PM Wade Albright <the.wade.albright at gmail.com>
wrote:

> Hi Julia,
>
> Thank you so much for the reply! Hopefully this is the issue. I'll try out
> the patches next week and report back. I'll also email you on Monday about
> the versions, that would be very helpful to know.
>
> Thanks again, really appreciate it.
>
> Wade
>
>
>
> On Sat, Jul 16, 2022 at 4:36 PM Julia Kreger <juliaashleykreger at gmail.com>
> wrote:
>
>> Greetings!
>>
>> I believe you need two patches, one in ironic and one in sushy.
>>
>> Sushy:
>> https://review.opendev.org/c/openstack/sushy/+/832860
>>
>> Ironic:
>> https://review.opendev.org/c/openstack/ironic/+/820588
>>
>> I think it is variation, and the comment about working after you restart
>> the conductor is the big signal to me. I’m on a phone on a bad data
>> connection, if you email me on Monday I can see what versions the fixes
>> would be in.
>>
>> For the record, it is a session cache issue, the bug was that the service
>> didn’t quite know what to do when auth fails.
>>
>> -Julia
>>
>>
>> On Fri, Jul 15, 2022 at 2:55 PM Wade Albright <
>> the.wade.albright at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I'm hitting a problem when trying to update the redfish_password for an
>>> existing node. I'm curious to know if anyone else has encountered this
>>> problem. I'm not sure if I'm just doing something wrong or if there is a
>>> bug. Or if the problem is unique to my setup.
>>>
>>> I have a node already added into ironic with all the driver details set,
>>> and things are working fine. I am able to run deployments.
>>>
>>> Now I need to change the redfish password on the host. So I update the
>>> password for redfish access on the host, then use an 'openstack baremetal
>>> node set <node> --driver-info redfish_password=<newpass>' command to set
>>> the new redfish_password.
>>>
>>> Once this has been done, deployment no longer works. I see redfish
>>> authentication errors in the logs and the operation fails. I waited a bit
>>> to see if there might just be a delay in updating the password, but after
>>> awhile it still didn't work.
>>>
>>> I restarted the conductor, and after that things work fine again. So it
>>> seems like the password is cached or something. Is there a way to force the
>>> password to update? I even tried removing the redfish credentials and
>>> re-adding them, but that didn't work either. Only a conductor restart seems
>>> to make the new password work.
>>>
>>> We are running Xena, using rpm installation on Oracle Linux 8.5.
>>>
>>> Thanks in advance for any help with this issue.
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20220718/08519a98/attachment-0001.htm>