Hello Magnum users and devs,

I was wondering if any may be able to provide some clues on a mystifying problem we've been seeing with some Magnum-deployed k8s clusters, of older deployment date (v1.17, deployed ~6 months ago) and Keystone trusts used to manage OpenStack cloud resources.

I've attached a copy of the Magnum template for the k8s cluster to give an idea of its initial environment.

The issue we've been seeing is that the Keystone trust generated at Magnum cluster creation time seems no longer be usable to successfully authenticate to the OpenStack APIs, and so operations such as creation of Manila shares used as k8s Persistent Volume Claims then get stuck in Pending state, forever.

The strange thing is that this doesn't happen for newly created Magnum clusters of the same template (in attached file, I use trust credentials for a newly deployed and old cluster; OpenStack API calls for new credentials succeed, whereas old cluster trust fails).

Unfortunately following debug-level Keystone logging for rejected trust auth attempts hasn't led me much further to understanding the root issue; the Magnum generated trust exists and has no expiry, but it still seems to be rejected. Keystone debug logging doesn't seem to give much indication as to *why* it is being rejected or what policy violations may be involved, but perhaps there's details hidden in there I'm not seeing yet.

Does anyone have any clues as to why this may be happening, or any advice on how we may be able to replace or refresh the trust that the Magnum-created k8s cluster is expecting to use in calling OpenStack APIs?

Many thanks, Paul