August 2024 - openstack-discuss - lists.openstack.org

Magnum Caracal: cluster creation goes through state CREATION_FAILED before CREATION_COMPLETE
by Michel Jouvin 21 Aug '24

21 Aug '24

Hi, I continue my tests with Magnum Caracal and found another strange behaviour during cluster creation. It goes through state CREATION_FAILED before reaching the state CREATION_COMPLETE (HEALTHY). It is very confusing. Looking at Heat engine logs, it happens during load balancer creation, at the time the LB amphora has been created and Octavia is waiting to be able to connect it (typically one minue). During this period, Heat logs the following error: ----------- 2024-08-21 09:54:52.194 3200318 INFO heat.engine.resource [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] CREATE: LoadBalancer "loadbalancer" [670985ef-5808-4226-ad1b-b1244871de16] Stack "test-mj-1-30-m1n1-bis-ipufjomqmnnf-etcd_lb-d5fuqygvpp2j" [5c5905bf-f538-4a42-93ce-38d96b7c6603] 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource Traceback (most recent call last): 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 922, in _action_recorder 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource yield 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 1034, in _do_action 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource yield from self.action_handler_task(action, args=handler_args) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 984, in action_handler_task 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource done = check(handler_data) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/octavia/loadbalancer.py", line 169, in check_create_complete 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource return self._check_status() 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/octavia/octavia_base.py", line 29, in _check_status 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource raise exception.ResourceInError(resource_status=status) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource heat.common.exception.ResourceInError: Went to status ERROR due to "Unknown" 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource 2024-08-21 09:54:58.804 3200319 INFO heat.engine.scheduler [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] Task pause timed out 2024-08-21 09:54:59.340 3200318 INFO heat.engine.scheduler [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] Task pause timed out ----- that results in Magnum setting the state to CREATE_FAILED despite it is a transient state from Heat point of view and despite Heat stack creation continues successfully once the transient state is cleared and Magnum later updates the status to CREATE_COMPLETED. Unfortunately magnum-conductor log has nothing related to these events (it is filled with barbican_client related messages mainly). Any idea what could cause this behaviour? Best regards, Michel

2 3

[nova][cinder] Can not find device after change HBA
by Qi Xu (Fujitsu) 21 Aug '24

21 Aug '24

Hi everyone, I am currently using RHOSP 16.2.4 and launched a virtual machine (VM) using a volume. From the attach information, I confirmed that this volume was mounted on ports 1 and 2 through HBA1 and HBA2 on the compute node. When I checked the devices on the compute node, I found two devices, which is as expected. [root@overcloud-novacompute-0 heat-admin]# multipath -ll LUN-UUID dm-5 FUJITSU,ETERNUS_DXL size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=10 status=active |- 3:0:2:1 sdl 8:176 active ready running `- 5:0:5:1 sdv 65:80 active ready running [root@overcloud-novacompute-0 heat-admin]# ll /dev/disk/by-path/ lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA1-port1-lun-0 -> ../../sdv lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0a.0-fc-port1-lun-0 -> ../../sdv lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA2-port2-lun-0 -> ../../sdl lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdl I then shut down the VM and the compute node, and replaced HBA1 with HBA3 on the compute node. Under these circumstances, the volume should be mounted on ports 1 and 2 through HBA3 and HBA2 respectively. In other words, I think I should be able to find the following devices. device1: fc-HBA3-port1-lun-0 -> ../../sdl pci-0000:00:0c.0-fc-port1-lun-0 -> ../../sdl device2: fc-HBA2-port2-lun-0 -> ../../sdv pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdv However, after restarting the compute node and the VM, although the connection and read/write operations are functioning normally, the compute node fails to find the volume on HBA3. [root@overcloud-novacompute-0 heat-admin]# multipath -ll LUN-UUID dm-5 FUJITSU,ETERNUS_DXL size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:5:1 sdl 8:176 active ready running [root@overcloud-novacompute-0 heat-admin]# ll /dev/disk/by-path/ lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA2-port2-lun-0 -> ../../sdl lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdl Could you please advise on how I can make the compute node find the volume on HBA3 in this scenario? And I would like to know how to change the Initiator from HBA1 to HBA3 in the initiator_target_map of DB without detach volume? Regards, Inori.

1 0

[Ironic] Unable to deploy bare metal due to partx partition failure
by Junya Noguchi (Fujitsu) 21 Aug '24

21 Aug '24

To whom it may concern A bare metal deployment with Ironic failed. Deploy Command $ openstack server create --flavor "flavor_name" --image "image_name" --nic net-id="net_id" --security-group "group_name" --key-name "key_name" "server_name" This command powers on a bare metal node that was previously in a powered-off state and boots the deployment image. An error occurs during the deployment process on the deployment image and the server status is error. $ openstack server list +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ | "id" | "server_name " | ERROR | | "image_name" | "flavor_name" | +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ When I checked the journal log, I saw the following errors. Can you tell me how to resolve these errors to ensure a successful deployment? I look forward to your reply. 1. Partitions created by partx fail Aug 02 03:05:28 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:28.468 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:28 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:28.468 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:483 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.190 1047 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): partx -av /dev/sda execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:390 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.194 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.195 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:483 Aug 02 03:05:30 localhost.localdomain systemd[1]: systemd-hostnamed.service: Deactivated successfully. Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.586 1047 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): partx -av /dev/sda execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:390 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Not Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:479 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG ironic_lib.utils [-] Command stdout is: "partition: none, disk: /dev/sda, lower: 0, upper: 0 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: /dev/sda: partition table type 'gpt' detected Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: range recount: max partno=2, lower=0, upper=0 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: " _log /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_lib/utils.py:99 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG ironic_lib.utils [-] Command stderr is: "partx: /dev/sda: adding partition #1 failed: Device or resource busy Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: partx: /dev/sda: adding partition #2 failed: Device or resource busy Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: partx: /dev/sda: error adding partitions 1-2 2. The boot efi partition is empty Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 DEBUG ironic_python_agent.efi_utils [-] Looking for all efi files on /tmp/tmp1gqks5kg/boot/efi _get_efi_bootloaders /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_python_agent/efi_utils.py:241 Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 DEBUG ironic_python_agent.efi_utils [-] efi files found in /tmp/tmp1gqks5kg/boot/efi : [] _get_efi_bootloaders /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_python_agent/efi_utils.py:245 Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 WARNING ironic_python_agent.efi_utils [-] Empty EFI partition detected. OS in which the above events occurred ・CentOS Stream 9 ・Ubuntu 24 ・OpenSUSE 15 What I've tried (none of which worked) ・Reduce flavor disk size ・Vary local_gb in bare metal node properties Properties driver: irmc boot_interface: irmc-pxe console_interface: ipmitool-socat deplo y_interface: direct inspect_interface: no-inspect management_interface: irmc network_interface: flat power_interface: irmc raid_interfa ce: no-raid rescue_interface: no-rescue storage_interface: noop v endor_interface: no-vendor memory_mb: 64000 loca l_gb: 800 image_type: partition image_disk_format: raw image_chec ksum: None Create bare metal image command (ubuntu example) $ disk-image-create ubuntu baremetal dhcp-all-interfaces -a amd64 grub2 -o ubuntu_x86 Best regards, Junya Noguchi.

1 0

Heat Caracal: log file full of oslo_messaging._drivers.amqpdriver Expecting reply to msg/Received RPC response
by Michel Jouvin 20 Aug '24

20 Aug '24

Hi, On our cloud where Caracal Magnum/Heat is deployed, heat_engine log file is filled with many occurences of the following pair of messages (IDs change for every pair) when there is a cluster creation (and probably deletion) : 2024-08-20 09:21:54.458 78230 INFO oslo_messaging._drivers.amqpdriver [None req-bd38bf98-d978-482a-8758-62b1af72fa43 jouvin - - - - -] Expecting reply to msg 699a920fddb940 8a81fd799125835996 in queue reply_2d4fccf5059340818bbfa72887989a6e 2024-08-20 09:21:54.468 78230 INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 699a920fddb9408a81fd799125835996 Why these INFO messages from oslo that are polluting the logs? we have debug=false in Heat configuration. It doesn't seem to happen in Antelope that we run on another cloud. Any way to disable these messages? Best regards, Michel

2 4

[devstack][glance] glance-manage Error
by Josephine Seifert 20 Aug '24

20 Aug '24

Hi everyone, within my devstack I can execute cinder-manage db sync or version and everything works fine: ``` stack@devstack:~/devstack$ cinder-manage db version INFO dbcounter [-] Registered counter for database cinder DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=304079) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} DEBUG dbcounter [-] [304079] Writer thread running {{(pid=304079) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} INFO alembic.runtime.migration [-] Context impl MySQLImpl. INFO alembic.runtime.migration [-] Will assume non-transactional DDL. a8189e5afd9a ``` But when I try to do the same with glance-manage I get an Error: ``` stack@devstack:~/devstack$ glance-manage db version CRITICAL glance [-] Unhandled error: sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.plugins:dbcounter ERROR glance Traceback (most recent call last): ERROR glance File "/usr/bin/glance-manage", line 10, in <module> .... ``` my current version of sqlalchemy is: stack@devstack:~/devstack$ pip show sqlalchemy Name: SQLAlchemy Version: 1.4.31 Does anyone have an idea, how I can fix this Error? greetings Luzi

2 2

Report a potential risk of secret leakage in project(cloudkitty)
by jiawei_zhou＠seu.edu.cn 20 Aug '24

20 Aug '24

Dear developers of the project(cloudkitty), We are software security researchers, currently conducting research on secret detection and leakage risk within the open-source ecosystem. In our analysis, we identified potential secret leakage risks in your project, cloudkitty. We provide the detail of our findings in the attachment, which allows you to locate the potential leaked secrets. Below is an interpretation of the attached data: { 'file': '', #The file containing the secret #The project name, version or commit_hash may be reflected in the file path 'line_start': 1, #location: Start line of the secret 'line_end': 28, #location: End line of the secret 'col_start': 1, #location: Start column of the secret 'col_end': 1, #location: End column of the secret 'index_start': 0, #location: Start index of the secret 'index_end': 1675, #location: End index of the secret } Declaration: we hereby declare that we have *NOT* conducted any verification test or exploit on the identified secrets. we plan to publish related research papers in the future, and the relevant content MIGHT BE ACCESS TO THE PUBLIC due to the 90-day disclosure policy. Some advise: 1. If the leaked secret is sensitive and still valid, invalid and rotate the secret immediately. 2. Some secrets seem to be used only in testing environment. Although probably harmless, it is considered bad practices to include secrets for test environment in release builds. Best regards,

2 1

[blazar] Update of core reviewer group
by Pierre Riteau 20 Aug '24

20 Aug '24

Hello, The blazar-core group includes the following contributors who have not been active on the project for several years: - Jason Anderson - Masahito Muroi - Bertrand Souville - Tetsuro Nakamura I propose to remove them so the blazar-core group contains only active reviewers. In the absence of objections, I will perform the removal after August 19. I would like to thank them all for their valuable contributions to the Blazar project. They are welcome back anytime. Thanks, Pierre Riteau (priteau)

1 1

[neutron] Bug deputy report for week of August 12th
by Brian Haley 19 Aug '24

19 Aug '24

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bug - yes, only one bug :) -Brian High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2076916 - VMs cannot access metadata when connected to a network with only IPv6 subnets with the ML2/OVS and ML2/LB backends - Could be issue with gate setup - https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926503

1 0

Nova buildrequest
by engineer2024 19 Aug '24

19 Aug '24

Hi, when I do a pdb inside a filter in scheduler, it is dropping inside the pdb shell. So far so good. But when I execute these statements inside it as shown below, the buildrequest function get_by_instance_uuid is failing to return any db object: ---------- class ComputeFilter(filters.BaseHostFilter): """Filter on active Compute nodes.""" RUN_ON_REBUILD = False def __init__(self): self.servicegroup_api = servicegroup.API() # Host state does not change within a request run_filter_once_per_request = True def host_passes(self, host_state, spec_obj): """Returns True for only active compute nodes.""" breakpoint(). <-------------- service = host_state.service if service['disabled']: LOG.debug("%(host_state)s is disabled, reason: %(reason)s", {'host_state': host_state, 'reason': service.get('disabled_reason')}) return False else: if not self.servicegroup_api.service_is_up(service): LOG.warning("%(host_state)s has not been heard from in a " "while", {'host_state': host_state}) return False return True ----- at the above breakpoint, it is dropping into the pdb shell. (pdb) from nova import objects as object (pdb) object.BuildRequest.get_by_instance_uuid(spec_obj._context, instance_uuid=spec_obj.instance_uuid)) Build request not found for instance .... ---- But if I place the above buildrequest inside the above same filter and run it normally without breakpoint(), it is working fine. So, why am I not able to do it manually while in pdb, and succeed when done normally ?

1 0

[ops][Image Encryption] Proposed Upgrade Workflow for the Image Encryption
by Josephine Seifert 19 Aug '24

19 Aug '24

Hi operators, we want to integrate image encryption into openstack. When the patches will be merged, there will be a change in a metadata name. Patches are here: https://review.opendev.org/q/topic:%22LUKS-image-encryption%22 We want to propose the following upgrade workflow: 1. install 2024.2 or later, new resources get os_encrypt_* metadata, legacy cinder_encrypt_* metadata is still honored for a deprication period 2. run glance-manage to update all the legacy metadata names to the new os_encrypt* 3. at this point, no openstack service will create cinder_encrypt* metadata, so anything from here must be created by a user 4. Operators need to reach out to the user and notify them, that shortly the cinder_encrypt* will no longer work We don't think the last step will occur often, because it is not documented anywhere, that it is possible to add these metadata to an image right now. If not done correctly (only possible if you know how Cinder does its volume encryption) it would render the image unusable. If you object to this workflow please let me know, we still have time to discuss this. greetings Josephine (Luzi)

1 0