June 2024 - openstack-discuss - lists.openstack.org

[ironic] Failed when writing image to the device file /dev/sda
by James Leong 06 Jun '24

06 Jun '24

Hi all, I have been exploring ironic, with a simple setup. However, I get the following error messages whenever I attempt to deploy an ISO image to my baremetal node. 1. An agent token generation request is being refused as one is already present for the node 2. Writing image to device /dev/sda failed with exit code 32. stdout: . stderr: lsblk: /dev/sda2: not a block device 3. Invalid or missing agent_token received for node I am not quite sure what is required to be set on my node's property other than vendor and cpu_arch. Any help would be appreciated. Best, James

1 0

[kolla-ansible][Antelope][Keystone] Redelegation Not Working with Heat
by Devin Patterson 06 Jun '24

06 Jun '24

Hello! We are attempting to use role redelegation with Heat and Keystone to deploy a stack that deploys Zaqar subscriptions among other resources. When attempting to scale up the stack, we receive this error: TypeError: '<' not supported between instances of 'NoneType' and 'int') This appears to be sourced from this check in the code in the Keystone core.py file. # expiry times trust_expiry = trust.get('expires_at') redelegated_expiry = redelegated_trust['expires_at'] if trust_expiry: # redelegated trust is from backend and has no tzinfo if redelegated_expiry < trust_expiry.replace(tzinfo=None): raise exception.Forbidden( _('Requested expiration time is more ' 'than redelegated trust can provide')) The issue appears to be related to 'None' values for what is passed into 'expires_at' for 'redelegated_expiry'. We have this config in keystone.conf: [trust] allow_redelegation = True And this in heat.conf: reauthentication_auth_method = trusts allow_trusts_redelegation = True This is how we are scaling the stack: openstack stack resource signal <stack_name> scale_up_policy Any suggestions on how to fix this issue? Thanks, Devin

1 0

[masakari] DB migration failure during upgrade to 2024.1
by Dmitriy Rabotyagov 06 Jun '24

06 Jun '24

Hey folks, Just wanted to let you know, that upgrade of Masakari to 2024.1 is most likely borked due to issue in Alembic migration, where masakari considers taskflow DB as its own, as masakari-manage does not pass correct namespace for alembic. Upgrade path is not tested in CI, which made it possible to slip the issue from the radar. There's a patch existing [1] that fixes the root cause of the issue, though needs to be reviewed and backported. Thus, I wanted to call out Masakari Maintainers for some attention to this issue as well as give operators some awareness about the issue. [1] https://review.opendev.org/c/openstack/masakari/+/920034

1 0

[stackalytics] no "neutron" module for Commits
by Ihar Hrachyshka 06 Jun '24

06 Jun '24

Hi, I noticed that "neutron" is not an option in the Module list when Commits are selected, and no neutron commits are registered since ~Oct 2023. On the other hand, other Neutron program repos (e.g. neutron-lib and neutron-tempest-plugin) are available in the list and do register commits. Also, when Reviews are selected, neutron module is back as an option, and review stats are correctly updated. I checked governance yaml and it doesn't strike me as incorrect: https://opendev.org/openstack/governance/src/commit/5f93b99a3c90b452f6c7368… Any ideas what may have happened? Thanks, Ihar

4 7

[Horizon][Keystone] Multiple OIDC Providers and token deletion at logout
by MIchele Delli Veneri 06 Jun '24

06 Jun '24

Hi, We have configured our keystone and Horizon to work with multiple OIDC Providers and through API I can successfully authenticate with the two providers. When I authenticate through the Horizon dashboard, I can successfully authenticate with both providers if I use two separate browser sessions, If I use the same browser session, I log in with one provider, log out, and then log in with the other, given that the user token is stored, I get an authentication error. The reason for the error is that the token was issued by the first provider and thus it is not accepted as a valid by the second. Is it possible to configure Horizon to revoke/delete a user token at logout or provide session isolation without changes on the provider side? Thank you and best regards, Michele Delli Veneri

1 0

[Horizon][Keystone] Multiple OIDC Providers and token deletion at logout
by Michele Delli Veneri 06 Jun '24

06 Jun '24

Hi, We have configured our keystone and Horizon to work with multiple OIDC Providers and through API I can successfully authenticate with the two providers. When I authenticate through the Horizon dashboard, I can successfully authenticate with both providers if I use two separate browser sessions, If I use the same browser session, I log in with one provider, log out, and then log in with the other, given that the user token is stored, I get an authentication error. The reason for the error is that the token was issued by the first provider and thus it is not accepted as a valid by the second. Is it possible to configure Horizon to revoke/delete a user token at logout or provide session isolation without changes on the provider side? Thank you and best regards, Michele Delli Veneri

1 0

Re: [ops] help with manila
by Alvaro Soto 05 Jun '24

05 Jun '24

[ please hit reply all ] Hi! 1. Your cluster needs to be healthy for Manila to connect. Could you purge your Ceph cluster to free up space? 2. They are very open, but they are OK if they are just for testing. 3. Yes, many resources are out there, but I'll depend on what deployment tool you're using. Cheers!

1 0

Antelope Barbican/Keystone: failure to delete secrets (Keystone TrustNotFound)
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I've been validating Yoga -> Antelope on our preprod cloud. It was working successfully but "suddenly" (i.e. without a kwown/identified change!) when deleting Magnum clusters, the deletion of Barbican secrets (after stack deletion) fail. In the Magnum log, I find the following error: ---- Failed to delete trust: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a4400250-e8d5-46ec-a28a-dcb97a682512) (traceback skipped for readability) 2024-05-29 10:43:09.750 2078 INFO magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17… from Barbican. 2024-05-29 10:43:09.750 2078 INFO barbicanclient.base [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Calculated Containers uuid ref: containers/edd8487a-6cf1-4f4c-9f53-173dfae7a8b2 2024-05-29 10:43:09.760 2078 ERROR barbicanclient.client [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] 4xx Client error: Not Found: Secrets container not found. 2024-05-29 10:43:09.761 2078 ERROR magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Error recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17…: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. ---- In Keystone log, I find with the same timestamp the following error that seems related to the one above: ---- 2024-05-29 10:43:09.690 1713 WARNING keystone.api._shared.authentication [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Could not find trust: cf4b1b6082594025a1ff9e48df383566.: keystone.exception.TrustNotFound: Could not find trust: cf4b1b6082594025a1ff9e48df383566. 2024-05-29 10:43:09.693 1713 WARNING keystone.server.flask.application [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Authorization failed. The request you have made requires authentication. from 134.158.77.23: keystone.exception.Unauthorized: The request you have made requires authentication. 2024-05-29 10:43:09.746 1723 ERROR keystone.server.flask.application [None req-8c76f304-c2a3-418c-8372-02c706ed356e 44d8cb3192b3485790ababe29991491a - 5b67a8641dd041089e018ceea9884eac - 5b67a8641dd041089e018ceea9884eac -] Could not find user: 527ec32509d849e8a49b41c3dc4c4c32.: keystone.exception.UserNotFound: Could not find user: 527ec32509d849e8a49b41c3dc4c4c32. ---- It is not clear for me if the ERROR is releated to the 2 warnings as the request is not the same but it seems to happen every time this way. In Barbican logs, there is not much things related except a 401 status for the request corresponding ot the TrustNotFound. I'm looking for advices on how to troubleshoot this. In particular, when the trust should be created and whether I should find somewhere a matching error at the time of the trust creation. It seems the error is only affecting deletion but I may be wrong. for the record, when starting Magnum there is an error complaining about /etc/magnum/keystone_auth_default_policy.json being not found but ignored it so far. And in Keystone an error seems to occur at Magnum cluster creation time: --- 2024-05-29 10:17:17.530 1711 WARNING py.warnings [None req-f484cc35-3e54-4d7f-8688-e6442c90344f a22b1199c74f4a819a49f3ac037a5e44 1dd67113c8bb4ef0a31dfca53335ffbd - - defaul t default] /usr/lib/python3.9/site-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_services": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is re quired ---- Thanks in advance for any hint. Best regards, Michel

2 3

Antelope Magnum: creating a new cluster makes the/some others unhealthy
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I have been playing with Magnum (Antelope), creating a lot of clusters with the same template and varying the number of masters and/or nodes, having ~10 clusters started in the same project. I observed recently that cluster configuration that used to work well were not working any more (CREATE_FAILED during the master deployment in general). I have not found any evidence of the cause digging in various logs but I observed that with the current list of active clusters, if add a new one (even with a configuration as minimal as 1 master and 1 node), not only its creation fails but also the other (or at least) running clusters become unhealthy. If I delete the failed cluster, the other clusters return to the healthy state. It seems very reproducible (I did it more than 10 times) but I still don't see any message in the logs that could help to identify the cause. I've the feeling (but may be wrong) that it is related to either an insufficient project quota or an insufficient limit on one of the OpenStack service. Any idea on a possible cause or any advice on where to look for some information? Thanks in advance. Michel

2 6

Antelope: weird problem with member role after upgrade (Nova and Glance)
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I just upgraded Nova/Cinder/Glance of our production cloud from Yoga to Antelope (after upgrading Keystone yesterday) and since the upgrade, users who are not admin cannot do anything basically, despite we changed nothing to service configuration or user's roles since Yoga. We enabled scoped tokens a while ago (several months). For (bad) historical reasons, the role "member" was called "users" but it had no impact (I was surprised), despite we are using standard policies. We thought it may be a consequence of this and we renamed the role back to "member". It was not enough to fix the problem, even after restart memcached on all servers just in case. We thought that there was may be some caching done somewhere with the old role name and modified slightly the policy rules defining what is a member or read with: "project_member_api": "(role:member or role:users) and project_id:%(project_id)s" "project_reader_api": "(role:reader or role:users) and project_id:%(project_id)s" It first works but the change was reverted by mistake and now it doesn't work anymore. I am really completely stuck, without any clue about what happen and on how to troubleshoot it. I googled a bit but was not able to find something looking similar... Any help would be greatly appreciated. Best regards, Michel -- Michel

3 2