December 2025 - openstack-discuss

[Masakari][HA][Antelope] Host failure not detected in 3-node controller+compute cluster
by Thamanna Farhath 24 Dec '25

24 Dec '25

Hello OpenStack community, I am testing Masakari host failure recovery on OpenStack 2023.1 (Antelope) deployed using Kolla-Ansible. Environment: 3 nodes Each node runs controller + compute Pacemaker + Corosync deployed on all 3 nodes Masakari enabled (hostmonitor, instancemonitor, engine, api) pacemaker-remote not used (all nodes are full Pacemaker members) Masakari segment: service_type = COMPUTE recovery_method = auto Tests performed: Process-level failures (qemu, libvirt) → detected correctly nova-compute stop → detected (no evacuation, expected) Host failure test: Powered off one compute/controller node Also tested systemctl stop corosync Issue: Pacemaker shows the node as OFFLINE However: No Masakari HOST_DOWN notification is generated No evacuation is triggered masakari-hostmonitor logs do not show host failure handling Question: In a 3-node controller+compute Pacemaker cluster, is Masakari expected to detect host failure directly from Pacemaker membership events without pacemaker-remote? Are there any additional requirements needed for Masakari host failure detection in this topology? Any guidance or references would be greatly appreciated. Disclaimer : The content of this email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to which they are addressed. If you have received this email in error, please notify the sender and remove the messages from your system. If you are not the named addressee, it is strictly forbidden for you to share, circulate, distribute or copy any part of this e-mail to any third party without the written consent of the sender. E-mail transmission cannot be guaranteed to be secured or error free as information could be intercepted, corrupted, lost, destroyed, arrive late, incomplete, or may contain viruses. Therefore, we do not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. The recipient should check this e-mail and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email."

3 2

[ceilometer] ceilometer metrics for list fields in event payloads
by NITIN GUPTA 23 Dec '25

23 Dec '25

Hello Ceilometer Team! Good Day! Could you please suggest, how to create ceilometer metrics for list fields in event payloads when creating Gnocchi resources? Looks like Ceilometer doesn't iterate over list fields in event payloads when creating Gnocchi resources. Is this a known issue? From the below Sample payload, I'm looking to create a separate metric for "vcpus" field (for example) from each item in "reservations", which is a list of dictionary type of data in the payload. 'payload': { 'lease_id': 'f94182be-70e2-4f3f-bc8d-a1d1c2eea0aa', 'user_id': '2cf63f4ecf7740f5ac9ef4adfa4cdf5e', 'project_id': '07f1a09ab11543bb9080d260a98d812c', 'trust_id': '0037eb9121f04cdc94c39628a8354dde', 'start_date': '2025-12-15T16:18:00.000000', 'end_date': '2025-12-15T16:22:00.000000', 'status': 'PENDING', '*reservations*': [ { 'id': '1fffeb17-599d-441a-80e0-ca6d9044fd36', 'lease_id': 'f94182be-70e2-4f3f-bc8d-a1d1c2eea0aa', 'resource_id': 'c9daade3-4a85-4122-bec2-f34b7a849a01', 'resource_type': 'virtual:instance', 'status': 'pending', 'missing_resources': False, 'resources_changed': False, 'created_at': '2025-12-15 16:18:07', 'updated_at': '2025-12-15 16:18:10', '*vcpus*': 1, 'memory_mb': 1024, 'disk_gb': 20, 'amount': 1, 'affinity': None, 'resource_properties': '', 'flavor_id': '1fffeb17-599d-441a-80e0-ca6d9044fd36', 'aggregate_id': 292, 'server_group_id': None }, { 'id': '97df8162-d756-44c0-a27f-c832101577e8', 'lease_id': 'f94182be-70e2-4f3f-bc8d-a1d1c2eea0aa', 'resource_id': '4a3e20b1-5f63-46b1-a61b-13a267fd829a', 'resource_type': 'virtual:instance', 'status': 'pending', 'missing_resources': False, 'resources_changed': False, 'created_at': '2025-12-15 16:18:05', 'updated_at': '2025-12-15 16:18:07', '*vcpus*': 1, 'memory_mb': 1024, 'disk_gb': 20, 'amount': 1, 'affinity': None, 'resource_properties': '', 'flavor_id': '97df8162-d756-44c0-a27f-c832101577e8', 'aggregate_id': 291, 'server_group_id': None } ], 'events': [{'id': '4cea5a4f-53f4-4a69-969b-74a5869c7ab1', 'lease_id': 'f94182be-70e2-4f3f-bc8d-a1d1c2eea0aa', 'event_type': 'end_lease', 'time': '2025-12-15T16:22:00.000000', 'status': 'UNDONE', 'created_at': '2025-12-15 16:18:10', 'updated_at': None}, {'id': '63b51fc9-5eca-4729-8f5e-a3659e7db24b', 'lease_id': 'f94182be-70e2-4f3f-bc8d-a1d1c2eea0aa', 'event_type': 'start_lease', 'time': '2025-12-15T16:18:00.000000', 'status': 'UNDONE', 'created_at': '2025-12-15 16:18:10', 'updated_at': None}, {'id': 'bba382f6-47d8-4bb0-b3aa-0a65c6959ed9', 'lease_id': 'f94182be-70e2-4f3f-bc8d-a1d1c2eea0aa', 'event_type': 'before_end_lease', 'time': '2025-12-15T16:18:00.000000', 'status': 'UNDONE', 'created_at': '2025-12-15 16:18:10', 'updated_at': None}], 'name': 'lease-short-ins', 'created_at': '2025-12-15 16:18:05', 'updated_at': '2025-12-15 16:18:10', 'degraded': False } Best regards. -- NITIN GUPTA

1 0

[nova][cinder][neutron] Heartbeat improvement: checking if main thread responds to rabbit ping
by Thomas Goirand 23 Dec '25

23 Dec '25

tl;dr: I want to implement main thread ping in the heartbeat tread via rabbitmq to make sure the main thread is alive when reporting a service status, I'd like to have opinions. Hi, I already wrote about this on #openstack-cinder, but nobody replied to me. Anyways, I think it's a topic that is cross-project, so I want to open it in this list. After a rabbitmq outage due to a disk full in our infra, we restarted our rabbitmq cluster. Everything seemed to come back normally, but we later discovered that the main threads of cinder-volume and cinder-backup were dead, while the heartbeat thread was not, and was reporting the services as up. It took us some time to realize it, as this cluster isn't public cloud, and we could live with the errors. Though we don't want to leave things the way they are, as we want reliable monitoring to avoid painful support tickets from customers. Within Cinder, there's already all the infrastructure in place to enhance the hearbeat thread, and make it do more checks. However, currently, the Manager class isn't even instantiated and used when creating the service. Here's my proposal, currently only for Cinder, but hopefully, I'll also implement it in Nova and Neutron. When creating the cinder-volume using cinder/service.py, it's going to also create a manager, and attach it to the service. The ManagerVolume class will implement the is_working() method of the class Manager. In it, it's going to ping the main thread with rabbitmq, sending a ping message, to which the main thread must respond with a pong. If the pong isn't received within a timeout, is_working() will reply False. I started implementing it here: https://review.opendev.org/c/openstack/cinder/+/970796 Note that it's currently still a WIP, and there's more to work on, like some unit tests that are currently failing. I'm not posting the link to the patch to get a merge yet, and it's unfinished, but the current patch is probably enough to get the idea of what I want to implement. So, for each service, the heartbeat will do a ping, which will of course make more messages in Rabbit. Though I believe this is still a reasonable thing to do. With 200 Cinder service, and a heartbeat every 30 seconds per service, that's going to be an additional less than 15 messages per seconds. Considering a reasonable RabbitMQ service can eat 100k message per seconds, it should be ok. Now, this is for Cinder. Once it's done there, I intend to investigate Neutron next, but I'm not sure if it has all the facilities that Cinder offers. Maybe it'd be worth unifying this in oslo.messaging? I'm open to suggestions, but I'm unsure (yet) of what it would look like within one of the oslo libs (I'd have to look on how / where to do it: suggestions are welcome, as it would speed-up my work on it). Please let know your opinion on this, if this is a good idea or not, and if the Cinder team (and potentially Neutron and Nova) will be willing to accept such a patch. Cheers, Thomas Goirand (zigo)

2 3

[Nova] unified limit and pci alias combination support
by Masahito Muroi 23 Dec '25

23 Dec '25

Hi Nova folks, We're trying to use unified limit with alias PCI device information with following configuration. With this configuration, unified limit calculates the resources:CUSTOM_NVIDIA_T4 correctly, but nova-scheduler makes allocation_candidate request with duplicated CUSTOM_NVIDIA_T4 resources. From my quick check of the code, the unified limit and pci device alias usecase is not supported by the master branch now. I'm thinking to fix the issue and comming up with two idea to solve the issue. I want to hear the nova team's opinion which is better to fit the nova's code. Or please give me any comment if there is another better way. - idea1: nova-scheduler drops duplicated resources request if needed. Adding a configuration to drop duplicated situation and etc. - idea2: introduces a new flavor extra spec prefix, e.g. "placement:<RC>", to indicate the resource class comes from placement. # nova.conf [pci] device_spec = {"vendor_id":"8086", "product_id":"8d2d", "resource_class":"CUSTOM_NVIDIA_T4"} alias = {"name":"t4_pcie_16gb", "vendor_id":"8086", "product_id":"8d2d", "resource_class":"CUSTOM_NVIDIA_T4", "device_type":"type-PCI"} report_in_placement = true pci_in_placement = true [quota] driver = nova.quota.UnifiedLimitsDriver unified_limits_resource_strategy = require unified_limits_resource_list = class:CUSTOM_NVIDIA_T4, class:VCPU, class:MEMORY_MB, class:DISK_GB # openstack flavor show GPU_FLAVOR | Fields. | Value | +-------------+-----------------------------------------| | properties | pci_passthrough:alias='t4_pcie_16gb:1', | | | resources:CUSTOM_NVIDIA_T4='1' | # allocation candidate request query /allocation_candidates limit=1000 resources=CUSTOM_NVIDIA_T4:1 DISK_GB:10 MEMORY_MB:1024 VCPU:1 resourcesbb775ad3-f50c-4d3b-bb28-8993e4a664ca-0=CUSTOM_NVIDIA_T4:1 best regards, Masahito

1 0

[Masakari] Proposal to restart bi-weekly IRC meetings (poll for the 1st session)
by Sei Sano 23 Dec '25

23 Dec '25

Dear Kajinami, Sam Su (苏正伟) I’d like to restart the regular Masakari IRC meeting, which has been paused for a while, and resume it on a bi-weekly basis. Also, through the Japan OpenStack user community network, I have already reached out to a few people who are interested in Masakari (they are welcome to join as observers at first). As a first step, I propose holding a kick-off meeting to align on how we run the meeting going forward (bi-weekly schedule, agenda/minutes process, chair rotation, etc.). Could you please let me know your availability for the first meeting from the options below? * Option 1: Friday, 2025-12-19 06:00-07:00 (UTC) / 15:00-16:00 (JST) * Option 2: Monday, 2025-12-22 06:00-07:00 (UTC) / 15:00-16:00 (JST) * Option 3: Tuesday, 2025-12-23 06:00-07:00 (UTC) / 15:00-16:00 (JST) Please reply by 2025-12-17 23:59 UTC. If I don’t hear back by then, I will go ahead and schedule the first meeting on (tentatively) 2025-12-22 06:00-07:00 (UTC) / 15:00-16:00 (JST), so we can start moving again (and we can adjust for the next meeting if needed). Proposed location: OpenStack IRC (#openstack-masakari) Draft agenda: * Agreement to restart the regular meeting (bi-weekly cadence, day/time) * Near-term priorities (review backlog, bugs, releases/events, blueprints, etc.) * Lightweight meeting operations (chair/agenda/minutes) If you have any constraints (e.g., specific time windows that never work), please share them as well. Thanks, -- Sei Sano NTT DOCOMO BUSINESS, Inc. E-mail: sei.sano(a)ntt.com

2 5

[nova] Ephemeral storage potential features
by Karol Klimaszewski 22 Dec '25

22 Dec '25

Hello, In my company (CloudFerro) we have developed two features related to nova ephemeral storage (with libvirt virt driver) - "SPDK-based ephemeral storage backend" and "Multiple ephemeral storage backend handling" (both described below). Would these features be appropriate to be added to upstream? If yes, then should I start with creating a blueprint and proposing a spec for each of them so it is more clear what we want to introduce? Of course not both at once, since there is some code that makes one work with the other. For us it would probably be best if we upstreamed "SPDK-based ephemeral storage backend" first and then upstreamed "Multiple ephemeral storage backend handling" Description of SPDK-based ephemeral storage backend: We add a new possible value for [libvirt]/images_type in nova.conf: spdk_lvol. If this value is set then local disks of instances are handled as logical volumes (lvols). of locally run SPDK instance (on the same compute as VM). See https://spdk.io/doc/logical_volumes.html#lvol for docs on this subject. These are essentially a part of local NVMe disk managed by SPDK. We create and manage those lvols by making calls from nova-compute to SPDK instance with RPC (https://spdk.io/doc/jsonrpc.html) using provided python library (https://github.com/spdk/spdk/blob/master/python/README.md) We attach those lvols to instances by exposing them as vhost-blk devices (see https://spdk.io/doc/vhost.html) and specifying them as disks with source_type='vhostuser'. This method of exposing NVMe local storage allows for much better I/O performance than exposing them from local filesystem. We currently have working: creating, deleting, cold-migrating, shelving, snapshots, unshelving VMs with this storage backend. We don't have live migration working yet but have it in plans. This feature includes changes only to nova. Description of Multiple ephemeral storage backend handling: We add a new configuration option [libvirt]/supported_image_types to nova.conf of nova-compute and change the meaning of [libvirt]/images_type to mean default image type. If libvirt:image_type extra_spec is specified in flavor then VM is scheduled on compute with appropriate image type in its [libvirt]/supported_image_types. We use potentially multiple DISK_GB resource providers and construct appropriate request groups to handle this scheduling. This spec is also used to fill driver_info.libvirt.image_type field of BDMs with destination_type=local of this VM. (driver_info is new JSON serialized field of BDM) Then in compute if BDM specifies this field we use its value instead of [libvirt]/images_type to decide on imagebackend to use for it. This method of handling multiple backends only works after being enabled by administrators by setting libvirt:image_type on all their flavors and running nova-manage commands that update existing VMs. Without enabling everything works as it was before. This feature includes changes to nova and some new traits in os-traits (One per possible value of [libvirt]/images_type + one extra) Best regards, Karol Klimaszewski

2 5

[deployment][kolla][rabbitmq] RabbitMQ errors
by Taavi Ansper 22 Dec '25

22 Dec '25

Hi I am seeing a lot of errors regarding rabbitmq. Our Openstack deployments work, but it is littered with these kinds of messages. 2025-12-22 15:41:47.026 1078 ERROR oslo.messaging._drivers.impl_rabbit [None req-0f284b85-9d30-4696-8246-c79da103b149 - - - - - -] [89e1e73c-56eb-4618-b775-9055444c9015] AMQP server on 192.168.X.X:5672 is unreachable: connection already closed. Trying again in 1 seconds.: amqp.exceptions.RecoverableConnectionError: connection already closed 2025-12-22 15:41:46.792 1117 ERROR oslo.messaging._drivers.impl_rabbit [-] Unable to connect to AMQP server on 192.168.X.X:5672 after inf tries: Basic.consume: (406) PRECONDITION_FAILED - stream queue 'q-reports-plugin_fanout' in vhost '/' does not have a running replica on the local node: amqp.exceptions.PreconditionFailed: Basic.consume: (406) PRECONDITION_FAILED - stream queue 'q-reports-plugin_fanout' in vhost '/' does not have a running replica on the local node 2025-12-22 15:41:46.257146+02:00 [error] <0.39571548.0> operation basic.consume caused a channel exception precondition_failed: stream queue 'central_fanout' in vhost '/' does not have a running replica on the local node Why is this happening and what should I do or keep in mind? -- ---- Taavi Ansper taavi.ansper(a)cyber.ee

1 0

[watcher] Dec 25th and Jan 1st meetings cancelled
by Douglas Viroel 19 Dec '25

19 Dec '25

Hi all, The watcher team agreed[1] on cancelling the following two weekly meetings (Dec 25th and Jan 1st) due to holidays (in most of the countries). We will meet again on Jan 8th. Thank you all and happy holidays! BR, [1] https://meetings.opendev.org/meetings/watcher_meeting_11_dec_2025/2025/watc… -- Douglas Viroel - dviroel

1 0

[blazar] Meeting cancelled for 25 December 2025
by Pierre Riteau 19 Dec '25

19 Dec '25

Hi all, Due to lack of availability of the core team, the fortnightly Blazar meeting will be cancelled on December 25. Have a nice break! Pierre

1 0

[manila] Next two weekly meetings cancelled
by Carlos Silva 19 Dec '25

19 Dec '25

Hello everyone, The manila team is cancelling the two upcoming weekly meetings (Dec 25 and 31) as these are holidays in a bunch of countries and people are taking time off. We'll resume the manila weekly meetings on Jan 8th. Hope you have a great break! Regards, carloss

1 0