August 2025 - openstack-discuss - lists.openstack.org

[keystone][ops] Upgrade to 2025.1 and "corner cases"
by Dmitriy Rabotyagov 25 Aug '25

25 Aug '25

Hey folks, I wanted to raise awareness of operators regarding Keystone upgrade to Epoxy (2025.1), as found that existing release notes are slightly more vague about corner case definition. During analysis, it appeared that a lot of users (and under "a lot" I mean 4 figures number) in my really old deployments (which were deployed during Newton), still in use passwords hashed with sha512_crypt which has been dropped in 2025.1 [1]. As this was referenced as a "corner case" in upgrade docs, it may not bring enough attention to this matter reading notes, while sha512_crypt removal is mentioned in "Other" section which I personally missed when analyzing upgrade requirements. So I would highly recommend to ensure that all your users are using supported hashing algorithms before the upgrade to Epoxy, and inform them to reset their passwords if they don't. You can list users who are using unsupported hashing algorithm using following MySQL query: SELECT distinct l_u.user_id FROM keystone.local_user AS l_u JOIN keystone.password AS hash ON l_u.id = hash.local_user_id WHERE (hash.password_hash LIKE '$6$%' OR hash.password_hash LIKE '$5$%') AND hash.expires_at i s null; Hope this helps somebody to save some time in the future. [1] https://opendev.org/openstack/keystone/commit/4a4137d5f77ced3f4f011846fb9c2…

2 2

[ops][largescale-sig][scientific-sig] Ops Radio Hour on August 22, 2025
by Ildiko Vancsa 23 Aug '25

23 Aug '25

Hi OpenStack community, I'm reaching out to you with a friendly reminder that the next OpenStack Ops Radio Hour[1] is scheduled to happen on __August 22nd, 2025 at 1300UTC.__ The call series is a casual setting for operators, engineers, and contributors to come together, share knowledge, and collaborate on the complexities of managing and deploying OpenStack environments. Topics the next gathering will dive into: - (SWIFT) Compliance processes - GPUs - VMware migration - vTPM (live migrations or not?): reboot, start - Keystone security param bug - https://bugs.launchpad.net/keystone/+bug/2117217 - Consolidating Ops IRC and Matrix channels What other topics should we add to the agenda? A few reminders: - The group on the call agreed to try out Matrix as a chat platform for more synchronous communication outside calls + You can access the Matrix room here: https://matrix.to/#/#openstack-ops:opendev.org<https://matrix.to/#/%23openstack-ops:opendev.org> + Further details are here: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… - If you would like to read and catch up on notes from previous calls, you can find them in this etherpad: https://etherpad.opendev.org/p/openstack_ops_radiohour See you on the call on August 22nd! Bets Regards, Ildikó [1] https://wiki.openstack.org/wiki/Operations/Meetups

1 1

[cinder-backup]cinder backup show progress percentage
by Nguyễn Hữu Khôi 23 Aug '25

23 Aug '25

Hello. I would like to know if we have some methods to show backup progress percentage. I see this but it is old and work only with Swift. https://specs.openstack.org/openstack/cinder-specs/specs/kilo/backup-notifi… Nguyen Huu Khoi

1 0

[freezer]backup with cindernative mode
by Nguyễn Hữu Khôi 23 Aug '25

23 Aug '25

Hello. I am using openstack 2025.1 and freezer 2025.1 to test. I can backup and restore using cinder. but I won't do it with cindernative, could you tell me that we need swiff to use this mode? This is my command: freezer-agent --mode cindernative --cindernative-vol-id fcc22b9f-ef01-4d16-bfde-83af6312bce9 --log-file k.log --debug Critical Error: Endpoint for object-store not found - have you specified a region? Thank much. Nguyen Huu Khoi

2 11

Keystone "TokenNotFound: Failed to validate token"
by Marc Vorwerk 23 Aug '25

23 Aug '25

Hi all, I am running into an issue with our OpenStack Kolla-Ansible deployment (Release 2024.1, Keystone 25.0.1) where Keystone logs are being flooded (18k per day) with TokenNotFound: Failed to validate token stack traces[1]. From the traceback, it seems these requests are attempting to validate tokens that have already been revoked. The majority of these come from Nova and Cinder, but the request_id from Keystone does not appear in their logs. Instead of finding the matching requests in Nova or Cinder logs, I only see InvalidToken: Token authorization failed messages[2]. Most of the time, the platform does not show any user-visible issues in these areas, although occasional failures can be observed. Could this indicate an issue with token caching or revocation list synchronization between services? Memcached looks fine. Any ideas what could be triggering this behavior are welcome. If you need more Information please let me know. Thanks in advance. Best regards, Marc [1] ERROR keystone.server.flask.application [None req-326cbeaa-a363-4a82-b45b-498d6f2ad338 b30e2d031b6a4851b69f9b1791716919 d977f9d63d3043288bed6e549e507c9a - - default default] Failed to validate token: keystone.exception.TokenNotFound: Failed to validate token ERROR keystone.server.flask.application Traceback (most recent call last): ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 870, in full_dispatch_request ERROR keystone.server.flask.application rv = self.dispatch_request() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 855, in dispatch_request ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper ERROR keystone.server.flask.application resp = resource(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return] ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request ERROR keystone.server.flask.application resp = meth(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/auth.py", line 285, in get ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:validate_token') ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 422, in enforce_call ERROR keystone.server.flask.application subj_token_target_data = cls._extract_subject_token_target_data() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/rbac_enforcer/enforcer.py", line 261, in _extract_subject_token_target_data ERROR keystone.server.flask.application token = PROVIDER_APIS.token_provider_api.validate_token( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 147, in validate_token ERROR keystone.server.flask.application self._is_valid_token(token, window_seconds=window_seconds) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 201, in _is_valid_token ERROR keystone.server.flask.application self.check_revocation(token) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 138, in check_revocation ERROR keystone.server.flask.application return self.check_revocation_v3(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1632, in get_or_create_for_user_func ERROR keystone.server.flask.application return self.get_or_create( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1092, in get_or_create ERROR keystone.server.flask.application with Lock( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__ ERROR keystone.server.flask.application return self._enter() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create ERROR keystone.server.flask.application return self.creator() ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1046, in gen_value ERROR keystone.server.flask.application created_value = creator( ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/token/provider.py", line 134, in check_revocation_v3 ERROR keystone.server.flask.application PROVIDERS.revoke_api.check_token(token_values) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 110, in wrapped ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) ERROR keystone.server.flask.application File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/revoke/core.py", line 137, in check_token ERROR keystone.server.flask.application raise exception.TokenNotFound(_('Failed to validate token')) ERROR keystone.server.flask.application keystone.exception.TokenNotFound: Failed to validate token [2] $ grep f47b3827 */*log nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/72970654-cbaf-4e0e-94e0-70e7ba3e0db4" status: 200 len: 3747 microversion: 2.1 time: 0.104714 nova/nova-api.log: WARNING keystonemiddleware.auth_token [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] Authorization failed for token: keystonemiddleware.auth_token._exceptions.InvalidToken: Token authorization failed nova/nova-api.log: INFO nova.api.openstack.requestlog [None req-f47b3827-9070-4761-a9de-0f8c854ccc93 1f24a1c85bb24bc3aa83a22bc6b9a47e 2f764cefd79544f1959b3743281b072d - - 16cb1fce3b1b4842b3be2fb4e88c5c69 16cb1fce3b1b4842b3be2fb4e88c5c69] 10.70.142.1 "GET /v2.1/servers/detail?name=shoot--system--mon" status: 401 len: 114 microversion: - time: 0.078451

2 3

[ironic][magnum][tacker][telemetry][venus][vitrage] broken gate on master branch
by Elõd Illés 22 Aug '25

22 Aug '25

Hi teams in $SUBJECT, The release team usually checks the gate health of deliverables that did not have merged patches recently and it seems you have a project with broken gate: https://review.opendev.org/q/topic:release-health-check-flamingo+is:open Could you please check and fix the gate so that we don't release 2025.2 Flamingo with a broken state. Thanks in advance, Előd Illés irc: elodilles @ #openstack-release

3 2

[release] Release countdown for week R-5
by Thierry Carrez 22 Aug '25

22 Aug '25

Development Focus ----------------- We are getting close to the end of the 2025.2 "Flamingo" cycle! Next week on August 28th, 2025 is the Flamingo-3 milestone, also known as feature freeze. It's time to wrap up feature work in the services and their client libraries, and defer features that won't make it to the 2026.1 "Gazpacho" cycle. General Information ------------------- This coming week is the deadline for client libraries: their last feature release needs to happen before "Client library freeze" on August 28th, 2025. Only bugfix releases will be allowed beyond this point. When requesting those library releases, you can also include the stable/2025.2 branching request with the review. As an example, see the "branches" section here: https://opendev.org/openstack/releases/src/branch/master/deliverables/pike/… August 28th, 2025 is also the deadline for feature work in all OpenStack deliverables following the cycle-with-rc model. To help those projects produce a first release candidate in time, only bugfixes should be allowed in the master branch beyond this point. Any feature work past that deadline has to be raised as a Feature Freeze Exception (FFE) and approved by the team PTL. Finally, feature freeze is also a good time for submitting a first version of your cycle-highlights. Cycle highlights are the raw data that helps shape what is communicated in press releases and other release activity at the end of the cycle, avoiding direct contacts from marketing folks. See https://docs.openstack.org/project-team-guide/release-management.html#cycle… for more details. Upcoming Deadlines & Dates -------------------------- Flamingo-3 milestone (feature freeze): August 28th, 2025 (R-5 week) RC1 deadline: September 12th, 2025 (R-3 week) Final RC deadline: September 25th, 2025 (R-1 week) 2025.2 "Flamingo" final release: October 1st, 2025 OpenInfra Summit Europe: October 17th-19th, 2025 -- Thierry Carrez (ttx)

1 0

[blazar] Blazar-dashboard contains non-free files (ie: minified Javascripts)
by Thomas Goirand 22 Aug '25

22 Aug '25

Hi, I've been trying to package Blazar, all is already in Debian, but I can't get blazar-dashboard in, because upstream source contains non-free files. Namely: https://github.com/openstack/blazar-dashboard/blob/master/blazar_dashboard/… is a minifed version of a much more complex package: https://github.com/apexcharts/apexcharts.js/tree/v3.31.0 Reminder: minified Javascripts are considered by most distributions (this includes Red Hat, Ubuntu, and in my case Debian) as non-free files that cannot be shipped in the distribution without the underlying source code, and packages must build from the not minified version. Could the Blazar team work on un-vendoring apexcharts.min.js, possibly removing it completely, or replacing it with a source-full version? Cheers, Thomas Goirand (zigo)

4 6

[ops] [all] One Week until 2025 OpenStack User Survey Deadline
by Allison Price 21 Aug '25

21 Aug '25

Hi everyone, I wanted to remind everyone that next Friday, August 29 is the deadline to complete this year’s OpenStack User Survey. If you haven’t taken it before, this is a great opportunity to help channel feedback to the upstream community while helping us understand the global OpenStack footprint better. It should only take ~20 minutes and is incredibly valuable. If you have taken it before, it’s still important to review and update your data. This should only take a few minutes. https://openstack.org/usersurvey Thank you all! Cheers, Allison

1 0

[neutron] Drivers meeting canceled Friday, August 22nd
by Brian Haley 21 Aug '25

21 Aug '25

Hello Neutrinos: I am going to cancel tomorrow's Drivers meetings as there will not be quorum. Just a reminder there are still open specs that need updates and/or review, list is at [0]. Have a nice weekend! -Brian [0] https://review.opendev.org/q/status:open+project:openstack/neutron-specs

1 0