Hi, I just stumbled over this a couple of days ago. After I upgraded to Antelope a user reported that he is no longer able to see his instances in Horizon. After I figured out what was causing it, I simply changed the role of all users to include member. But now that you mentioned Magnum I wonder if I still have an issue because every new K8s cluster that I deploy suddenly has failed pods. :( Best Regards, Oliver Dmitriy Rabotyagov noonedeadpunk@gmail.com schrieb am 14. Nov. 2023 um 20:05: Hey, Theoretically that's a valid approach, but that won't work for application credentials due to the mentioned bug report in keystone [1]. Also, keep in mind trusts, especially if you're running Magnum. We had to mess up with the keystone database and update the role UUID for application credentials and trusts where _member_ was assigned. And flush cache (memcached in our case) after doing that. [1] https://bugs.launchpad.net/keystone/+bug/2030061 вт, 14 нояб. 2023 г. в 19:45, Christian Stelter refugee@last-refuge.net: Hi! As someone who hasn't yet delved very deeply into keystone and the policies of the individual OpenStack services, I wondered whether setting member as the implied role for _member_ could pick up all those users who didn't manage to rotate their application credentials before switching to Zed Antelope. Is this a valid approach or could it cause problems? Kind regards, Christian

Just to mention, all commands/scripts were made with minimal available time on hands, so shouldn't be considered as a best or production-ready approach but just drafts that were made in a couple of minutes to address the issue. So review them carefully and optimize them where possible.

So we did the following things: * Assigned all users who had a _member_ role with member. This can be done with a script like that [1] * For application credentials, you likely need not only to add "member" role, but also a "reader", since some Nova APIs require to have a reader, and implied roles do not work as mentioned earlier: MySQL> update keystone.application_credential_role set role_id = <role member uuid> where role_id = <role _member_ uuid>; $ for i in $(mysql -sN -e "select application_credential_id from keystone.application_credential_role where role_id = '<role _member_ uuid>'"); do mysql -e "INSERT INTO keystone.application_credential_role(application_credential_id, role_id) VALUES (${i}, '<role reader uuid>')"; done * For trusts: MySQL> update keystone.trust_role set role_id = '<role member uuid>' where role_id = '<role _member_ uuid>';

[1] https://paste.openstack.org/show/btQPY6Hzpi6G0SOM0BDo/

ср, 15 нояб. 2023 г. в 22:11, Michael Knox michael@knox.net.nz:

Hi there,

What is the approach though?

Do we just script a change to move folks from _member_ to member? So no one in _member_?

We have a number of installs, but no magnum.

Cheers Michael

On Wed, Nov 15, 2023 at 10:02 AM Oliver Weinmann oliver.weinmann@me.com wrote:

...

Hi,

I just stumbled over this a couple of days ago. After I upgraded to Antelope a user reported that he is no longer able to see his instances in Horizon. After I figured out what was causing it, I simply changed the role of all users to include member. But now that you mentioned Magnum I wonder if I still have an issue because every new K8s cluster that I deploy suddenly has failed pods. :(

Best Regards, Oliver

Dmitriy Rabotyagov noonedeadpunk@gmail.com schrieb am 14. Nov. 2023 um 20:05:

Hey,

Theoretically that's a valid approach, but that won't work for application credentials due to the mentioned bug report in keystone [1]. Also, keep in mind trusts, especially if you're running Magnum.

We had to mess up with the keystone database and update the role UUID for application credentials and trusts where _member_ was assigned. And flush cache (memcached in our case) after doing that.

[1] https://bugs.launchpad.net/keystone/+bug/2030061

вт, 14 нояб. 2023 г. в 19:45, Christian Stelter refugee@last-refuge.net:

Hi!

As someone who hasn't yet delved very deeply into keystone and the policies of the individual OpenStack services, I wondered whether setting member as the implied role for _member_ could pick up all those users who didn't manage to rotate their application credentials before switching to Zed Antelope.

Is this a valid approach or could it cause problems?

Kind regards,

Christian

Hi, Many thanks for your scripts. I tried to use your script but something is weird. It seems that no user in the admin project has the _member_ role assigned. So I don't understand how magnum should be affected by this. For testing I created a user test in the admin project in the default domain and this is shown as a member of the _member_ role. My OpenStack is connected to an AD domain via LDAP but here I have manually added all users to the member role. Can someone please confirm that the magnum user is not assigned to the _member_ role? Best Regards, Oliver Michael Knox michael@knox.net.nz schrieb am 16. Nov. 2023 um 14:42: Thanks! I will use that as a starting point, appreciate the direction. Cheers. Michael On Thu, Nov 16, 2023 at 8:04 AM Dmitriy Rabotyagov < noonedeadpunk@gmail.com > wrote: Just to mention, all commands/scripts were made with minimal available time on hands, so shouldn't be considered as a best or production-ready approach but just drafts that were made in a couple of minutes to address the issue. So review them carefully and optimize them where possible. So we did the following things: * Assigned all users who had a _member_ role with member. This can be done with a script like that [1] * For application credentials, you likely need not only to add "member" role, but also a "reader", since some Nova APIs require to have a reader, and implied roles do not work as mentioned earlier: MySQL> update keystone.application_credential_role set role_id = <role member uuid> where role_id = <role _member_ uuid>; $ for i in $(mysql -sN -e "select application_credential_id from keystone.application_credential_role where role_id = '<role _member_ uuid>'"); do mysql -e "INSERT INTO keystone.application_credential_role(application_credential_id, role_id) VALUES (${i}, '<role reader uuid>')"; done * For trusts: MySQL> update keystone.trust_role set role_id = '<role member uuid>' where role_id = '<role _member_ uuid>'; [1] https://paste.openstack.org/show/btQPY6Hzpi6G0SOM0BDo/ ср, 15 нояб. 2023 г. в 22:11, Michael Knox < michael@knox.net.nz >: > > Hi there, > > What is the approach though? > > Do we just script a change to move folks from _member_ to member? So no one in _member_? > > We have a number of installs, but no magnum. > > Cheers > Michael > > On Wed, Nov 15, 2023 at 10:02 AM Oliver Weinmann < oliver.weinmann@me.com > wrote: >> >> Hi, >> >> I just stumbled over this a couple of days ago. After I upgraded to Antelope a user reported that he is no longer able to see his instances in Horizon. After I figured out what was causing it, I simply changed the role of all users to include member. But now that you mentioned Magnum I wonder if I still have an issue because every new K8s cluster that I deploy suddenly has failed pods. :( >> >> Best Regards, >> Oliver >> >> Dmitriy Rabotyagov < noonedeadpunk@gmail.com > schrieb am 14. Nov. 2023 um 20:05: >> >> >> Hey, >> >> Theoretically that's a valid approach, but that won't work for >> application credentials due to the mentioned bug report in keystone >> [1]. Also, keep in mind trusts, especially if you're running Magnum. >> >> We had to mess up with the keystone database and update the role UUID >> for application credentials and trusts where _member_ was assigned. >> And flush cache (memcached in our case) after doing that. >> >> [1] https://bugs.launchpad.net/keystone/+bug/2030061 >> >> вт, 14 нояб. 2023 г. в 19:45, Christian Stelter < refugee@last-refuge.net >: >> >> >> Hi! >> >> As someone who hasn't yet delved very deeply into keystone and the >> policies of the individual OpenStack services, I wondered whether >> setting member as the implied role for _member_ could pick up all >> those users who didn't manage to rotate their application credentials >> before switching to Zed Antelope. >> >> Is this a valid approach or could it cause problems? >> >> Kind regards, >> >> Christian >> >>

Thanks. So the issue I'm facing with Magnum is not related to the member role. :) Dmitriy Rabotyagov noonedeadpunk@gmail.com schrieb am 23. Nov. 2023 um 14:34: No, I don't think that magnum user is ever connected to the _member_ role. Regarding magnum - it was affected by trusts, since we had `cluster_user_trust = True` in [trust] section of magnum.conf. So all trusts that were created for users were having _member_ role for users and this was what we had to adjust as well. чт, 23 нояб. 2023 г. в 13:05, Oliver Weinmann oliver.weinmann@me.com: Hi, Many thanks for your scripts. I tried to use your script but something is weird. It seems that no user in the admin project has the _member_ role assigned. So I don't understand how magnum should be affected by this. For testing I created a user test in the admin project in the default domain and this is shown as a member of the _member_ role. My OpenStack is connected to an AD domain via LDAP but here I have manually added all users to the member role. Can someone please confirm that the magnum user is not assigned to the _member_ role? Best Regards, Oliver Michael Knox michael@knox.net.nz schrieb am 16. Nov. 2023 um 14:42: Thanks! I will use that as a starting point, appreciate the direction. Cheers. Michael On Thu, Nov 16, 2023 at 8:04 AM Dmitriy Rabotyagov noonedeadpunk@gmail.com wrote: Just to mention, all commands/scripts were made with minimal available time on hands, so shouldn't be considered as a best or production-ready approach but just drafts that were made in a couple of minutes to address the issue. So review them carefully and optimize them where possible. So we did the following things: * Assigned all users who had a _member_ role with member. This can be done with a script like that [1] * For application credentials, you likely need not only to add "member" role, but also a "reader", since some Nova APIs require to have a reader, and implied roles do not work as mentioned earlier: MySQL> update keystone.application_credential_role set role_id = <role member uuid> where role_id = <role _member_ uuid>; $ for i in $(mysql -sN -e "select application_credential_id from keystone.application_credential_role where role_id = '<role _member_ uuid>'"); do mysql -e "INSERT INTO keystone.application_credential_role(application_credential_id, role_id) VALUES (${i}, '<role reader uuid>')"; done * For trusts: MySQL> update keystone.trust_role set role_id = '<role member uuid>' where role_id = '<role _member_ uuid>'; [1] https://paste.openstack.org/show/btQPY6Hzpi6G0SOM0BDo/ ср, 15 нояб. 2023 г. в 22:11, Michael Knox michael@knox.net.nz: > > Hi there, > > What is the approach though? > > Do we just script a change to move folks from _member_ to member? So no one in _member_? > > We have a number of installs, but no magnum. > > Cheers > Michael > > On Wed, Nov 15, 2023 at 10:02 AM Oliver Weinmann oliver.weinmann@me.com wrote: >> >> Hi, >> >> I just stumbled over this a couple of days ago. After I upgraded to Antelope a user reported that he is no longer able to see his instances in Horizon. After I figured out what was causing it, I simply changed the role of all users to include member. But now that you mentioned Magnum I wonder if I still have an issue because every new K8s cluster that I deploy suddenly has failed pods. :( >> >> Best Regards, >> Oliver >> >> Dmitriy Rabotyagov noonedeadpunk@gmail.com schrieb am 14. Nov. 2023 um 20:05: >> >> >> Hey, >> >> Theoretically that's a valid approach, but that won't work for >> application credentials due to the mentioned bug report in keystone >> [1]. Also, keep in mind trusts, especially if you're running Magnum. >> >> We had to mess up with the keystone database and update the role UUID >> for application credentials and trusts where _member_ was assigned. >> And flush cache (memcached in our case) after doing that. >> >> [1] https://bugs.launchpad.net/keystone/+bug/2030061 >> >> вт, 14 нояб. 2023 г. в 19:45, Christian Stelter refugee@last-refuge.net: >> >> >> Hi! >> >> As someone who hasn't yet delved very deeply into keystone and the >> policies of the individual OpenStack services, I wondered whether >> setting member as the implied role for _member_ could pick up all >> those users who didn't manage to rotate their application credentials >> before switching to Zed Antelope. >> >> Is this a valid approach or could it cause problems? >> >> Kind regards, >> >> Christian >> >>