July 2025 - openstack-discuss - lists.openstack.org

[neutron][octavia] vip port missing virtual-parent in ovn
by Maximilian Stinsky-Damke 03 Jul '25

03 Jul '25

Greetings, I'm currently debugging a strange issue related to Octavia loadbalancer failovers and was wondering if anyone has encountered something similar or has any insights. We're running Neutron with ML2/OVN and the ovn-bgp-agent. When we create Octavia load balancers using the Amphora driver, they come up fine. After attaching a floating IP, the ovn-bgp-agent correctly announces the FIP. However, during load balancer failovers, we sometimes observe that the FIP is no longer being announced. After some debugging, we found that the ovn-bgp-agent stops announcing the FIP because the VIP port in OVN is down and no longer has a neutron:host_id set in external_ids. Digging deeper, we noticed that the VIP port's options:virtual-parents only includes a single port ID and it's not the one corresponding to the current VRRP master. As a result, OVN doesn't associate the VIP with the active amphora, causing the port to be up=false and leaving the external_ids:neutron:host_id field empty. Because of this, no bgp-agent sees itself as responsible for announcing the FIP. Looking at the Octavia amphora ports in Neutron, all of them have the correct allowed address pairs set. This should normally result in the correct virtual-parents being reflected in OVN. Interestingly, I was able to reproduce this behavior even with the SINGLE topology in Octavia, a failover sometimes removes the entire options fields related to virtual-parents and virtual-ip from the VIP port. Manually unsetting and re-setting the allowed address pair on the affected port always restores the correct virtual-parents, and the announcement resumes. Now the question is how could this happen? It seems completely random, but it's very reproducible in my environment. We're running Neutron 2025.1 (26.0.0) with OVN 24.03.1-5.el9. I haven't found any errors in the Neutron server logs indicating issues writing to OVN, or any relevant tracebacks. Thanks in advance for any input or ideas! Best regards, Max

2 3

[cinder] How to retype while maintaining volume usage
by Yuta Kambe (Fujitsu) 02 Jul '25

02 Jul '25

Hi everyone, We are using a Ceph cluster as the backend for Cinder. The Ceph cluster has two pools: volumes_data_hdd for HDDs and volumes_data_ssd for SSDs. These are assigned to the Cinder volume types ceph-hdd and ceph-ssd, respectively. To change the type of a ceph-hdd volume to ceph-ssd, I executed the following command: openstack volume set --type ceph-ssd --retype-policy on-demand <ID of ceph-hdd volume> After execution, checking the capacity with the 'ceph df' command shows an increase in usage after the type change. The following is an example of migrating a newly created, empty 10GB ceph-hdd type volume to ceph-ssd: $ sudo ceph df | grep volumes_data volumes_data_hdd 19 1 98 GiB 25.51k 195 GiB 2.24 4.2 TiB volumes_data_ssd 22 1 91 GiB 23.89k 181 GiB 3.55 2.4 TiB $ openstack volume list --long | grep 7fd16e56-71ec-430b-8499-8d216a57f1c6 | 7fd16e56-71ec-430b-8499-8d216a57f1c6 | test | available | 10 | ceph-hdd | false | | | $ openstack volume set --type ceph-ssd --retype-policy on-demand 7fd16e56-71ec-430b-8499-8d216a57f1c6 $ openstack volume list --long | grep 7fd16e56-71ec-430b-8499-8d216a57f1c6 | 7fd16e56-71ec-430b-8499-8d216a57f1c6 | test | available | 10 | ceph-ssd | false | | | $ sudo ceph df | grep volumes_data volumes_data_hdd 19 1 98 GiB 25.51k 195 GiB 2.24 4.2 TiB volumes_data_ssd 22 1 101 GiB 26.45k 201 GiB 3.95 2.4 TiB Despite the volume being empty, the usage of volumes_data_ssd increases by 10GB. I suspect this is due to zero-filling of unused space during the copy process. Is it possible to change the volume type using Cinder functionality while maintaining the same usage? If not, I think it would be better to use the 'rbd cp' command internally for type changes within Ceph, as it should be able to copy the image while preserving the image size. What are your thoughts on this approach? OpenStack and Ceph versions are: $ ceph -v ceph version 17.2.8 (f817ceb7f187defb1d021d6328fa833eb8e943b3) quincy (stable) $ dnf list --installed | grep openstack-cinder openstack-cinder.noarch 1:21.3.2-1.el9s @centos-openstack-zed Best regards,

2 1

[publiccloud-sig] Next meeting of Public Cloud SIG - July 2nd (today) at 0700 UTC
by Felix Kronlage-Dammers 01 Jul '25

01 Jul '25

Hi everyone, This upcoming Wednesday (which is today, July 2nd) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0700 UTC here: https://meetpad.opendev.org/publiccloud-sig See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG Sorry for the short notice! Have fun! cheers felix -- Felix Kronlage-Dammers Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[Nova] - Upgrading to Epoxy causes Ubuntu cloud images to fail DHCP
by Karl Kloppenborg 01 Jul '25

01 Jul '25

Hi Team, We’ve got a weird one where other images (cirros, Debian, fedora etc) work fine. However upgrading to Nova Epoxy ubuntu fails cloud-init because network timed out. Note network loads fine and DHCP is received on cirros etc and setting config-drive for the installation Ubuntu works, and continues to work. It smells of some driver issue/change but I haven’t put my finger on it yet. Boot log: https://gist.github.com/inventionlabsSydney/3d491bb08ea96d1f32ed16fd6b58179a Image was uploaded as such: openstack image create --file ~/noble-server-cloudimg-amd64.img --disk-format qcow2 \ --min-disk 8 \ --min-ram 1024 \ --public \ --property hw_firmware_type=uefi \ --property hw_disk_bus=scsi \ --property hw_scsi_model=virtio-scsi \ --property hw_machine_type=pc-q35-focal-hpb ’Noble Test' Has anyone else seen this yet? Karl Kloppenborg Chief Technology Officer m: +61 437 239 565 resetdata.com<https://resetdata.com/> [cid:reset_69557fc2-1d63-4932-b5fd-93bd4f39ca7b.png] ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit; https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-sta…<https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-sta…> This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact(a)resetdata.com.au<mailto:contact@resetdata.com.au>

1 0

Error Occurred After Multipath Configuration
by jiwooo.oh＠samsung.com 01 Jul '25

01 Jul '25

After setting up multipath on netapp, errors occur frequently when detaching a volume. Unexpected error while running command. Command: multipath -f 3600a098038314e7a4b24577351787462 Exit code: 1 Stderr: map in use' This occurs when executing the multipath -f command, and a 'map in use' error occurs. [Full error log] 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] Traceback (most recent call last): 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/block_device.py", line 413, in driver_detach 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] virt_driver.detach_volume(context, connection_info, instance, mp, 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 2860, in detach_volume 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] self._disconnect_volume(context, connection_info, instance, 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 2099, in _disconnect_volume 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] vol_driver.disconnect_volume( 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/volume/iscsi.py", line 74, in disconnect_volume 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] self.connector.disconnect_volume( 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 174, in trace_logging_wrapper 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] return f(*args, **kwargs) 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/base.py", line 78, in inner 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] return f(*args, **kwargs) 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/utils.py", line 415, in change_encrypted 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] res = func(**call_args) 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 894, in disconnect_volume 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] return self._cleanup_connection(connection_properties, force=force, 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/connectors/iscsi.py", line 948, in _cleanup_connection 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] multipath_name = self._linuxscsi.remove_connection( 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/linuxscsi.py", line 382, in remove_connection 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] self.flush_multipath_device(multipath_name) 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/initiator/linuxscsi.py", line 450, in flush_multipath_device 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] self._execute('multipath', '-f', device_map_name, run_as_root=True, 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/executor.py", line 55, in _execute 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] result = self.__execute(*args, **kwargs) 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/os_brick/privileged/rootwrap.py", line 172, in execute 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] return execute_root(*cmd, **kwargs) 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] return self.channel.remote_call(name, args, kwargs, 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 215, in remote_call 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] raise exc_type(*result[2]) 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266]oslo_concurrency.processutils.ProcessExecutiON-ERROR: Unexpected error while running command. Jun 4, 2025 @ 05:08:04.928 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] Jun 4, 2025 @ 05:08:04.928 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] Stderr: '9203485.810915 | 3600a098038315143433f58706d524b7a: map in use\n' Jun 4, 2025 @ 05:08:04.928 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] Stdout: '' Jun 4, 2025 @ 05:08:04.928 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] Exit code: 1 Jun 4, 2025 @ 05:08:04.928 1487897 ERROR nova.virt.block_device [instance: 402cec31-5d7e-4167-b057-09675dcf4266] Command: multipath -f 3600a098038315143433f58706d524b7a [Related config] use_multipath_for_image_xfer: true netapp_lun_space_reservation: disabled image_volume_cache_enabled: true image_volume_cache_max_size_gb: 0 image_volume_cache_max_count: 50 [Error Occurrence Case] - Create volume with image - Volume detach - Stop then start operation - vm resize Is there a solution to this situation?

1 0

[heat][ops] Removal of standalone api services (heat-api and heat-cfn-api)
by Takashi Kajinami 01 Jul '25

01 Jul '25

Hello, Historically heat has provided two methods to deploy api services (native heat api and heat CFN api). 1. standalone script based on eventlet 2. wsgi application run by server framework like apache or uwsgi Now the heat team is trying to get rid of eventlet following the global effort to replace it, and the current blocker is 1 because we have no good alternative to re-implement the standaloe service. The global migration strategy is to remove standalone api services using eventlet and provide only wsgi application. So I'm now sending an email to ask for any input, in case if any operator has concern about our current plan below. - Backport deprecation of standalone api scripts to 2025.1 - Remove these scripts during 2025.2 This means that any operators should use 3rd party server framework like uwsgi and apache+mod_wsgi to run heat api from 2025.2 . I understand backporting deprecation is something we won't accept usually, but I've seen this pattern was done in the past for a few exceptions like Hyper-V support removal. If we hear no objections/concerns for a week then, we aim to merge the removal during this cycle. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

1 1