August 2024 - openstack-discuss - lists.openstack.org

Heat Caracal: log file full of oslo_messaging._drivers.amqpdriver Expecting reply to msg/Received RPC response
by Michel Jouvin 21 Aug '24

21 Aug '24

Hi, On our cloud where Caracal Magnum/Heat is deployed, heat_engine log file is filled with many occurences of the following pair of messages (IDs change for every pair) when there is a cluster creation (and probably deletion) : 2024-08-20 09:21:54.458 78230 INFO oslo_messaging._drivers.amqpdriver [None req-bd38bf98-d978-482a-8758-62b1af72fa43 jouvin - - - - -] Expecting reply to msg 699a920fddb940 8a81fd799125835996 in queue reply_2d4fccf5059340818bbfa72887989a6e 2024-08-20 09:21:54.468 78230 INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 699a920fddb9408a81fd799125835996 Why these INFO messages from oslo that are polluting the logs? we have debug=false in Heat configuration. It doesn't seem to happen in Antelope that we run on another cloud. Any way to disable these messages? Best regards, Michel

2 4

[devstack][glance] glance-manage Error
by Josephine Seifert 21 Aug '24

21 Aug '24

Hi everyone, within my devstack I can execute cinder-manage db sync or version and everything works fine: ``` stack@devstack:~/devstack$ cinder-manage db version INFO dbcounter [-] Registered counter for database cinder DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=304079) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} DEBUG dbcounter [-] [304079] Writer thread running {{(pid=304079) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} INFO alembic.runtime.migration [-] Context impl MySQLImpl. INFO alembic.runtime.migration [-] Will assume non-transactional DDL. a8189e5afd9a ``` But when I try to do the same with glance-manage I get an Error: ``` stack@devstack:~/devstack$ glance-manage db version CRITICAL glance [-] Unhandled error: sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.plugins:dbcounter ERROR glance Traceback (most recent call last): ERROR glance File "/usr/bin/glance-manage", line 10, in <module> .... ``` my current version of sqlalchemy is: stack@devstack:~/devstack$ pip show sqlalchemy Name: SQLAlchemy Version: 1.4.31 Does anyone have an idea, how I can fix this Error? greetings Luzi

2 2

Report a potential risk of secret leakage in project(cloudkitty)
by jiawei_zhou＠seu.edu.cn 20 Aug '24

20 Aug '24

Dear developers of the project(cloudkitty), We are software security researchers, currently conducting research on secret detection and leakage risk within the open-source ecosystem. In our analysis, we identified potential secret leakage risks in your project, cloudkitty. We provide the detail of our findings in the attachment, which allows you to locate the potential leaked secrets. Below is an interpretation of the attached data: { 'file': '', #The file containing the secret #The project name, version or commit_hash may be reflected in the file path 'line_start': 1, #location: Start line of the secret 'line_end': 28, #location: End line of the secret 'col_start': 1, #location: Start column of the secret 'col_end': 1, #location: End column of the secret 'index_start': 0, #location: Start index of the secret 'index_end': 1675, #location: End index of the secret } Declaration: we hereby declare that we have *NOT* conducted any verification test or exploit on the identified secrets. we plan to publish related research papers in the future, and the relevant content MIGHT BE ACCESS TO THE PUBLIC due to the 90-day disclosure policy. Some advise: 1. If the leaked secret is sensitive and still valid, invalid and rotate the secret immediately. 2. Some secrets seem to be used only in testing environment. Although probably harmless, it is considered bad practices to include secrets for test environment in release builds. Best regards,

2 1

[blazar] Update of core reviewer group
by Pierre Riteau 20 Aug '24

20 Aug '24

Hello, The blazar-core group includes the following contributors who have not been active on the project for several years: - Jason Anderson - Masahito Muroi - Bertrand Souville - Tetsuro Nakamura I propose to remove them so the blazar-core group contains only active reviewers. In the absence of objections, I will perform the removal after August 19. I would like to thank them all for their valuable contributions to the Blazar project. They are welcome back anytime. Thanks, Pierre Riteau (priteau)

1 1

[neutron] Bug deputy report for week of August 12th
by Brian Haley 20 Aug '24

20 Aug '24

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bug - yes, only one bug :) -Brian High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2076916 - VMs cannot access metadata when connected to a network with only IPv6 subnets with the ML2/OVS and ML2/LB backends - Could be issue with gate setup - https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926503

1 0

Nova buildrequest
by engineer2024 19 Aug '24

19 Aug '24

Hi, when I do a pdb inside a filter in scheduler, it is dropping inside the pdb shell. So far so good. But when I execute these statements inside it as shown below, the buildrequest function get_by_instance_uuid is failing to return any db object: ---------- class ComputeFilter(filters.BaseHostFilter): """Filter on active Compute nodes.""" RUN_ON_REBUILD = False def __init__(self): self.servicegroup_api = servicegroup.API() # Host state does not change within a request run_filter_once_per_request = True def host_passes(self, host_state, spec_obj): """Returns True for only active compute nodes.""" breakpoint(). <-------------- service = host_state.service if service['disabled']: LOG.debug("%(host_state)s is disabled, reason: %(reason)s", {'host_state': host_state, 'reason': service.get('disabled_reason')}) return False else: if not self.servicegroup_api.service_is_up(service): LOG.warning("%(host_state)s has not been heard from in a " "while", {'host_state': host_state}) return False return True ----- at the above breakpoint, it is dropping into the pdb shell. (pdb) from nova import objects as object (pdb) object.BuildRequest.get_by_instance_uuid(spec_obj._context, instance_uuid=spec_obj.instance_uuid)) Build request not found for instance .... ---- But if I place the above buildrequest inside the above same filter and run it normally without breakpoint(), it is working fine. So, why am I not able to do it manually while in pdb, and succeed when done normally ?

1 0

[ops][Image Encryption] Proposed Upgrade Workflow for the Image Encryption
by Josephine Seifert 19 Aug '24

19 Aug '24

Hi operators, we want to integrate image encryption into openstack. When the patches will be merged, there will be a change in a metadata name. Patches are here: https://review.opendev.org/q/topic:%22LUKS-image-encryption%22 We want to propose the following upgrade workflow: 1. install 2024.2 or later, new resources get os_encrypt_* metadata, legacy cinder_encrypt_* metadata is still honored for a deprication period 2. run glance-manage to update all the legacy metadata names to the new os_encrypt* 3. at this point, no openstack service will create cinder_encrypt* metadata, so anything from here must be created by a user 4. Operators need to reach out to the user and notify them, that shortly the cinder_encrypt* will no longer work We don't think the last step will occur often, because it is not documented anywhere, that it is possible to add these metadata to an image right now. If not done correctly (only possible if you know how Cinder does its volume encryption) it would render the image unusable. If you object to this workflow please let me know, we still have time to discuss this. greetings Josephine (Luzi)

1 0

[publiccloud-sig] Reminder - next meeting August 21st - 0700 UTC
by Felix Kronlage-Dammers 19 Aug '24

19 Aug '24

Hi everyone, This Wednesday (August 21st) the next meeting of the public cloud sig is going to happen. We’ll be meeting at 0700 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you on Wednesday! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[cinder] Instance Volume Issue After Restarting tgtd Container During Upgrade
by Vahideh Alinouri 19 Aug '24

19 Aug '24

Hi Everyone, I'm currently upgrading an OpenStack deployment that was set up using Kolla-Ansible. I'm currently on Kolla version 14.8 (Yoga) and I am trying to upgrade to (Latest Yoga) 14.11, to eventually upgrade to the Zed release. During the upgrade process, I encountered an issue with the compute node that uses the Cinder LVM backend, which the instances rely on. Restarting the tgtd container caused the connection between the iscsid container and tgtd to be lost. In the Nova compute logs, I found the following error: Stderr: 'blockdev: cannot open /dev/sdb: No such device or address\n' Because of this, the instance transitions to an error state. I also observed this error in the tgtd container: login failed ISCSI_CONN_STATE_IN_LOGIN/R_STAGE_SESSION_REOPEN 500 10:18 Kernel reported iSCSI connection 6:0 error (1019 - ISCSI_ERR_XMIT_FAILED: Transmission of iSCSI packet failed) state (2). When I check the iSCSI session with the following command, it shows that the session is established: iscsiadm --mode session However, when I run the following command on the compute host (which handles the Cinder LVM backend via iSCSI), no output is returned: tgtadm --lld iscsi --op show --mode target I tried logging out and logging back into the session, but the logout operation failed! Here is my cinder.conf file on the compute host: [os-compXXXX-nvme] volume_group = vgnvme volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver volume_backend_name = os-compXXXX-nvme target_helper = tgtadm target_protocol = iscsi lvm_mirrors = 0 lvm_max_over_subscription_ratio = 1.0 The operating system on the compute node is Ubuntu 20.04. After further investigation, I found a temporary workaround to bring the instance back to an active running state: 1. Power off the compute node. 2. Evacuate the node (it gets stuck in rebuild, but that's okay). 3. Power on the compute node. 4. Reset the instance state to "active." 5. Check the volume status and update it if it's not in the "in-use" state. 6. Perform a hard reboot of the instance. 7. Restart the tgtd container. However, this is only a temporary solution. When I attempt to upgrade to the next version, I encounter the same issue again. Is there anything specific I should consider? Could this be a bug? I would greatly appreciate your help.

1 1

[MAGNUM] Fedora Core OS include NVIDIA GPU drivers
by Oliver Weinmann 19 Aug '24

19 Aug '24

Hi all, I'm running OpenStack 2023.2 (Bobcat) deployed with Kolla-Ansible. Currently I'm struggling to get the NVIDIA drivers to work in a FCOS 35 image. I know FCOS 35 is EOL and outdated, but for a K8s 1.21.11 cluster it is the only version that still works with Magnum. So far I managed to get the GPU Passthrough working for two NVIDIA Tesla T4 GPUs. I have attached the GPUs to an Ubuntu instance and ran geekbench without any issues. Then I tried to install the latest NVIDIA drivers in FCOS 35 by running: sudo rpm-ostree install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$ (rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$ (rpm -E %fedora).noarch.rpm systemctl reboot rpm-ostree install --apply-live akmod-nvidia akmods pciutils rpmdevtools xorg-x11-drv-nvidia-cuda It fails on: rpm-ostree install --apply-live akmod-nvidia akmods pciutils rpmdevtools xorg-x11-drv-nvidia-cuda error: Cannot download rpm-build-libs-4.17.0-4.fc35.x86_64.rpm: All mirrors were tried; Last error: Status code: 403 for https://fedoraproject-updates-archive.fedoraproject.org/fedora/35/x86_64/rp… I guess the binaries are no longer hosted? Running the same on FCOS 36 and latest FCOS 40 works fine. Next I have taken new images from the FCOS instances with the NVIDIA driver. openstack server image create --name fedora_coreos_36_nvidia_cuda fedora_coreos_35_latest_no_update --wait This takes ages but anyway I can successfully deploy new instances from the images with my gpu flavor and the GPU is working fine. Again tested with geekbench. But now when adding new K8s worker nodes in a fresh nodegroup on an existing K8s 1.21.11 cluster the nodes are not joining the cluster: openstack coe nodegroup create \ --node-count 1 \ --role worker-gpu \ --flavor gpuflavor \ --image fedora_coreos_36_nvidia_cuda \ $CLUSTER_ID worker-gpu I have tested the latest available images of FCOS36 and they work fine, so something must be wrong with how I created the new image. Is there some magic that needs to be done? I'm also not sure if this is the best way of doing this. Maybe there is already a ready FCOS image with nvidia drivers? I can find lots of outdated information on the internet. Generally I tend to use Vexxhost CAPI from now on, which works great. This is really just a one off attempt to have a working NVIDIA T4 GPU in a K8s 1.21.11 cluster. Best Regards, Oliver

5 16