August 2024 - openstack-discuss - lists.openstack.org

[glance][cinder] No format check on glance image uploading vs. thorough image format checks on cinder
by Florent Le Lain 21 Aug '24

21 Aug '24

Hello folks, Since resolution of vmdk related CVE<https://bugs.launchpad.net/nova/+bug/1996188>, Cinder performs extra checks on format of images used for specific actions, like creating a volume out of an image for instance. See related cinder commit: https://review.opendev.org/c/openstack/cinder/+/871625 Those new checks first led to failures on our tempest tests, on tests that were passing before. E.g.: tempest.api.compute.servers.test_create_server.ServersTestBootFromVolume It turned out that tests were failing because of a discrepancy between actual format of the image (qcow2 here) and declared format in glance db (raw). To us, this raises the question of format checks responsibility throughout images lifecycle, between Glance and Cinder. Today, afaik, Glance will: - tag as 'raw' any image uploaded without any format specified - tag the image with the format provided by the customer, whatever the actual format of the image is whereas Cinder is now more picky, refusing to deal with an image that claims to be of X format but is actually Y. If you do confirm these observations, in your opinion, what would be the best option among: 1. leave it as is, after all openstack customer is responsible for the accuracy of the information provided on image upload and should not be surprised by downstream mismatches when using the image 2. strengthen customer's responsibility of bullet 1. by making image format declaration mandatory on image upload 3. enforce format checks on glance on image upload 4. other way you may think of? something upstream I may have missed? Thank you for your opinion! Regards, Florent Le Lain SRE Team Lead – Public Cloud OVHcloud Rennes [1672217009737]

2 1

Openstack Backup and Restore
by meteccinar＠gmail.com 21 Aug '24

21 Aug '24

Hi everyone, we make cloud region for our company. Using Openstack with Ceph (as Block Storage). We configured backup driver field as BackupDriverCeph in the cinder.conf file. Also created volume and backups pools in Ceph Rados. We have a few problems/questions. We have developed an API that regularly backs up VMs (instances) that work with volumes. The process works as follows: First, we use the create_server_image() method from the OpenStack API to create an image of the instance. This results in an image of the instance being automatically created in OpenStack with a Snapshot type (including metadata such as direct_url, stores, schema, etc.). Simultaneously, a snapshot is also automatically created using with the create_server() command. We then take backups of all volumes attached to the instance. In our restore API, we use the image created from the instance backup mentioned above. We change block_device_mapping field and add the IDs of the volume backups to the block_device_mapping field of the image and use the create_server(image_name, volume_from_backed_volume) method in the OpenStack API to create a new instance. Then new volumes are created from volume backups and attach new volumes in new restored instance. The problem we're encountering is that when using the create_server command, the image created from the instance backup is tied to the direct_url, which points to the RBD URL of the snapshot. As a result, the instance does not running with new volume. Note: If we create a new VM with new volume from volume backup in Horizon. Instance running successfully. What steps should we take to resolve this? Is there a flaw in our approach when designing the API? What is the best practice we should follow when creating an API for backup and restore (especially for instances that boot from volumes)? Is there a configuration we need to adjust in Ceph? We would appreciate your advice on the best practices for backup and restore, particularly for instances that boot from volumes. Kind regards,

3 6

[neutron] Moving CI Meeting to Monday 1400 UTC
by Yatin Karel 21 Aug '24

21 Aug '24

Hello Neutron Team !! As discussed/agreed over the last couple of CI meetings, pushed the changes[1][2] for the new timing. [1] https://review.opendev.org/c/opendev/irc-meetings/+/926763 [2] https://etherpad.opendev.org/p/neutron-ci-meetings -- Thanks and Regards Yatin Karel

1 0

Magnum Caracal: cluster creation goes through state CREATION_FAILED before CREATION_COMPLETE
by Michel Jouvin 21 Aug '24

21 Aug '24

Hi, I continue my tests with Magnum Caracal and found another strange behaviour during cluster creation. It goes through state CREATION_FAILED before reaching the state CREATION_COMPLETE (HEALTHY). It is very confusing. Looking at Heat engine logs, it happens during load balancer creation, at the time the LB amphora has been created and Octavia is waiting to be able to connect it (typically one minue). During this period, Heat logs the following error: ----------- 2024-08-21 09:54:52.194 3200318 INFO heat.engine.resource [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] CREATE: LoadBalancer "loadbalancer" [670985ef-5808-4226-ad1b-b1244871de16] Stack "test-mj-1-30-m1n1-bis-ipufjomqmnnf-etcd_lb-d5fuqygvpp2j" [5c5905bf-f538-4a42-93ce-38d96b7c6603] 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource Traceback (most recent call last): 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 922, in _action_recorder 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource yield 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 1034, in _do_action 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource yield from self.action_handler_task(action, args=handler_args) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 984, in action_handler_task 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource done = check(handler_data) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/octavia/loadbalancer.py", line 169, in check_create_complete 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource return self._check_status() 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/octavia/octavia_base.py", line 29, in _check_status 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource raise exception.ResourceInError(resource_status=status) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource heat.common.exception.ResourceInError: Went to status ERROR due to "Unknown" 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource 2024-08-21 09:54:58.804 3200319 INFO heat.engine.scheduler [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] Task pause timed out 2024-08-21 09:54:59.340 3200318 INFO heat.engine.scheduler [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] Task pause timed out ----- that results in Magnum setting the state to CREATE_FAILED despite it is a transient state from Heat point of view and despite Heat stack creation continues successfully once the transient state is cleared and Magnum later updates the status to CREATE_COMPLETED. Unfortunately magnum-conductor log has nothing related to these events (it is filled with barbican_client related messages mainly). Any idea what could cause this behaviour? Best regards, Michel

2 3

[nova][cinder] Can not find device after change HBA
by Qi Xu (Fujitsu) 21 Aug '24

21 Aug '24

Hi everyone, I am currently using RHOSP 16.2.4 and launched a virtual machine (VM) using a volume. From the attach information, I confirmed that this volume was mounted on ports 1 and 2 through HBA1 and HBA2 on the compute node. When I checked the devices on the compute node, I found two devices, which is as expected. [root@overcloud-novacompute-0 heat-admin]# multipath -ll LUN-UUID dm-5 FUJITSU,ETERNUS_DXL size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=10 status=active |- 3:0:2:1 sdl 8:176 active ready running `- 5:0:5:1 sdv 65:80 active ready running [root@overcloud-novacompute-0 heat-admin]# ll /dev/disk/by-path/ lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA1-port1-lun-0 -> ../../sdv lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0a.0-fc-port1-lun-0 -> ../../sdv lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA2-port2-lun-0 -> ../../sdl lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdl I then shut down the VM and the compute node, and replaced HBA1 with HBA3 on the compute node. Under these circumstances, the volume should be mounted on ports 1 and 2 through HBA3 and HBA2 respectively. In other words, I think I should be able to find the following devices. device1: fc-HBA3-port1-lun-0 -> ../../sdl pci-0000:00:0c.0-fc-port1-lun-0 -> ../../sdl device2: fc-HBA2-port2-lun-0 -> ../../sdv pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdv However, after restarting the compute node and the VM, although the connection and read/write operations are functioning normally, the compute node fails to find the volume on HBA3. [root@overcloud-novacompute-0 heat-admin]# multipath -ll LUN-UUID dm-5 FUJITSU,ETERNUS_DXL size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:5:1 sdl 8:176 active ready running [root@overcloud-novacompute-0 heat-admin]# ll /dev/disk/by-path/ lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA2-port2-lun-0 -> ../../sdl lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdl Could you please advise on how I can make the compute node find the volume on HBA3 in this scenario? And I would like to know how to change the Initiator from HBA1 to HBA3 in the initiator_target_map of DB without detach volume? Regards, Inori.

1 0

[Ironic] Unable to deploy bare metal due to partx partition failure
by Junya Noguchi (Fujitsu) 21 Aug '24

21 Aug '24

To whom it may concern A bare metal deployment with Ironic failed. Deploy Command $ openstack server create --flavor "flavor_name" --image "image_name" --nic net-id="net_id" --security-group "group_name" --key-name "key_name" "server_name" This command powers on a bare metal node that was previously in a powered-off state and boots the deployment image. An error occurs during the deployment process on the deployment image and the server status is error. $ openstack server list +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ | "id" | "server_name " | ERROR | | "image_name" | "flavor_name" | +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ When I checked the journal log, I saw the following errors. Can you tell me how to resolve these errors to ensure a successful deployment? I look forward to your reply. 1. Partitions created by partx fail Aug 02 03:05:28 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:28.468 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:28 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:28.468 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:483 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.190 1047 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): partx -av /dev/sda execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:390 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.194 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.195 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:483 Aug 02 03:05:30 localhost.localdomain systemd[1]: systemd-hostnamed.service: Deactivated successfully. Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.586 1047 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): partx -av /dev/sda execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:390 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Not Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:479 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG ironic_lib.utils [-] Command stdout is: "partition: none, disk: /dev/sda, lower: 0, upper: 0 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: /dev/sda: partition table type 'gpt' detected Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: range recount: max partno=2, lower=0, upper=0 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: " _log /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_lib/utils.py:99 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG ironic_lib.utils [-] Command stderr is: "partx: /dev/sda: adding partition #1 failed: Device or resource busy Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: partx: /dev/sda: adding partition #2 failed: Device or resource busy Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: partx: /dev/sda: error adding partitions 1-2 2. The boot efi partition is empty Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 DEBUG ironic_python_agent.efi_utils [-] Looking for all efi files on /tmp/tmp1gqks5kg/boot/efi _get_efi_bootloaders /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_python_agent/efi_utils.py:241 Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 DEBUG ironic_python_agent.efi_utils [-] efi files found in /tmp/tmp1gqks5kg/boot/efi : [] _get_efi_bootloaders /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_python_agent/efi_utils.py:245 Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 WARNING ironic_python_agent.efi_utils [-] Empty EFI partition detected. OS in which the above events occurred ・CentOS Stream 9 ・Ubuntu 24 ・OpenSUSE 15 What I've tried (none of which worked) ・Reduce flavor disk size ・Vary local_gb in bare metal node properties Properties driver: irmc boot_interface: irmc-pxe console_interface: ipmitool-socat deplo y_interface: direct inspect_interface: no-inspect management_interface: irmc network_interface: flat power_interface: irmc raid_interfa ce: no-raid rescue_interface: no-rescue storage_interface: noop v endor_interface: no-vendor memory_mb: 64000 loca l_gb: 800 image_type: partition image_disk_format: raw image_chec ksum: None Create bare metal image command (ubuntu example) $ disk-image-create ubuntu baremetal dhcp-all-interfaces -a amd64 grub2 -o ubuntu_x86 Best regards, Junya Noguchi.

1 0

Heat Caracal: log file full of oslo_messaging._drivers.amqpdriver Expecting reply to msg/Received RPC response
by Michel Jouvin 21 Aug '24

21 Aug '24

Hi, On our cloud where Caracal Magnum/Heat is deployed, heat_engine log file is filled with many occurences of the following pair of messages (IDs change for every pair) when there is a cluster creation (and probably deletion) : 2024-08-20 09:21:54.458 78230 INFO oslo_messaging._drivers.amqpdriver [None req-bd38bf98-d978-482a-8758-62b1af72fa43 jouvin - - - - -] Expecting reply to msg 699a920fddb940 8a81fd799125835996 in queue reply_2d4fccf5059340818bbfa72887989a6e 2024-08-20 09:21:54.468 78230 INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 699a920fddb9408a81fd799125835996 Why these INFO messages from oslo that are polluting the logs? we have debug=false in Heat configuration. It doesn't seem to happen in Antelope that we run on another cloud. Any way to disable these messages? Best regards, Michel

2 4

[devstack][glance] glance-manage Error
by Josephine Seifert 21 Aug '24

21 Aug '24

Hi everyone, within my devstack I can execute cinder-manage db sync or version and everything works fine: ``` stack@devstack:~/devstack$ cinder-manage db version INFO dbcounter [-] Registered counter for database cinder DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=304079) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} DEBUG dbcounter [-] [304079] Writer thread running {{(pid=304079) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} INFO alembic.runtime.migration [-] Context impl MySQLImpl. INFO alembic.runtime.migration [-] Will assume non-transactional DDL. a8189e5afd9a ``` But when I try to do the same with glance-manage I get an Error: ``` stack@devstack:~/devstack$ glance-manage db version CRITICAL glance [-] Unhandled error: sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.plugins:dbcounter ERROR glance Traceback (most recent call last): ERROR glance File "/usr/bin/glance-manage", line 10, in <module> .... ``` my current version of sqlalchemy is: stack@devstack:~/devstack$ pip show sqlalchemy Name: SQLAlchemy Version: 1.4.31 Does anyone have an idea, how I can fix this Error? greetings Luzi

2 2

Report a potential risk of secret leakage in project(cloudkitty)
by jiawei_zhou＠seu.edu.cn 20 Aug '24

20 Aug '24

Dear developers of the project(cloudkitty), We are software security researchers, currently conducting research on secret detection and leakage risk within the open-source ecosystem. In our analysis, we identified potential secret leakage risks in your project, cloudkitty. We provide the detail of our findings in the attachment, which allows you to locate the potential leaked secrets. Below is an interpretation of the attached data: { 'file': '', #The file containing the secret #The project name, version or commit_hash may be reflected in the file path 'line_start': 1, #location: Start line of the secret 'line_end': 28, #location: End line of the secret 'col_start': 1, #location: Start column of the secret 'col_end': 1, #location: End column of the secret 'index_start': 0, #location: Start index of the secret 'index_end': 1675, #location: End index of the secret } Declaration: we hereby declare that we have *NOT* conducted any verification test or exploit on the identified secrets. we plan to publish related research papers in the future, and the relevant content MIGHT BE ACCESS TO THE PUBLIC due to the 90-day disclosure policy. Some advise: 1. If the leaked secret is sensitive and still valid, invalid and rotate the secret immediately. 2. Some secrets seem to be used only in testing environment. Although probably harmless, it is considered bad practices to include secrets for test environment in release builds. Best regards,

2 1

[blazar] Update of core reviewer group
by Pierre Riteau 20 Aug '24

20 Aug '24

Hello, The blazar-core group includes the following contributors who have not been active on the project for several years: - Jason Anderson - Masahito Muroi - Bertrand Souville - Tetsuro Nakamura I propose to remove them so the blazar-core group contains only active reviewers. In the absence of objections, I will perform the removal after August 19. I would like to thank them all for their valuable contributions to the Blazar project. They are welcome back anytime. Thanks, Pierre Riteau (priteau)

1 1