- openstack-discuss - lists.openstack.org

[neutron] bug deputy report for week of 2024-08-19
by Bence Romsics 26 Aug '24

26 Aug '24

Hi All, We had the following new bugs last week: OVN-Octavia (not yet triaged): * https://bugs.launchpad.net/neutron/+bug/2077399 OVN-Octavia maintainers please have a look! High: * https://bugs.launchpad.net/neutron/+bug/2077790 [eventlet] RPC handler thread model is incompatible with eventlet fix proposed: https://review.opendev.org/c/openstack/neutron/+/926922 Medium: * https://bugs.launchpad.net/neutron/+bug/2077532 Two floatingip_agent_gateways are created which are bound to the same agent. assigned to Anton Low: * https://bugs.launchpad.net/neutron/+bug/2077351 "Error formatting log line" sometimes seen in l3-agent log fix proposed: https://review.opendev.org/c/openstack/neutron/+/926566 Incomplete: * https://bugs.launchpad.net/neutron/+bug/2077533 An error in processing one DVR router can lead to connectivity issues for other routers discussing with Anton whether a standalone bug or a repercussion of #2077532, will follow up RFE: * https://bugs.launchpad.net/neutron/+bug/2077596 [rfe][fwaas] Add normalized_cidr column to firewall rules Cheers, Bence

1 0

[release] [ironic] [swift] Missing releases
by Thierry Carrez 26 Aug '24

26 Aug '24

Quick reminder that we'll need a release very soon for a number of deliverables following a cycle-with-intermediary release model but which have not done *any* release yet in the Dalmatian cycle: bifrost ironic-prometheus-exporter ironic-python-agent-builder ironic-ui networking-baremetal networking-generic-switch swift Those should be released ASAP, and in all cases before September 13th, 2024, so that we have a release to include in the final 2024.2 release. -- Thierry Carrez (ttx)

2 1

[release] Release countdown for week R-5
by Thierry Carrez 23 Aug '24

23 Aug '24

Development Focus ----------------- We are getting close to the end of the 2024.2 "Dalmatian" cycle! Next week on August 29th, 2024 is the dalmatian-3 milestone, also known as feature freeze. It's time to wrap up feature work in the services and their client libraries, and defer features that won't make it to the 2025.1 "Epoxy" cycle. General Information ------------------- This coming week is the deadline for client libraries: their last feature release needs to happen before "Client library freeze" on August 29th, 2024. Only bugfix releases will be allowed beyond this point. When requesting those library releases, you can also include the stable/2024.2 branching request with the review. As an example, see the "branches" section here: https://opendev.org/openstack/releases/src/branch/master/deliverables/pike/… Dalmatian-3 is also the deadline for feature work in all OpenStack deliverables following the cycle-with-rc model. To help those projects produce a first release candidate in time, only bugfixes should be allowed in the master branch beyond this point. Any feature work past that deadline has to be raised as a Feature Freeze Exception (FFE) and approved by the team PTL. Finally, feature freeze is also a good time for submitting a first version of your cycle-highlights. Cycle highlights are the raw data that helps shape what is communicated in press releases and other release activity at the end of the cycle, avoiding direct contacts from marketing folks. See https://docs.openstack.org/project-team-guide/release-management.html#cycle… for more details. Upcoming Deadlines & Dates -------------------------- Dalmatian-3 milestone (feature freeze): August 29th, 2024 (R-5 week) Cycle Highlights Due: September 5th, 2024 (R-4 week) RC1 deadline: September 13th, 2024 (R-3 week) Final RC deadline: September 26th, 2024 (R-1 week) Final 2024.2 Dalmatian release: October 2nd, 2024 OpenInfra Summit Asia: September 3-4, 2024 -- Thierry Carrez (ttx)

1 0

Openstack Instance Snapshot and Rescue Problem
by meteccinar＠gmail.com 23 Aug '24

23 Aug '24

Hi everyone, We use Ceph in Cinder as Block Storage. Firsly, we create a VM with volume (volume create automatically) and we write a data in the VM and take an snapshot after wrote data. Then we write data in main VM again and take second snapshot. But, when we try to rescue instance related main VM with first snapshot, there was an error occured. Nova Log: 2024-08-22 08:23:25.594 19 WARNING nova.compute.api [None req-eb3b4856-c8be-4926-be51-99aaec4de0e0 41040b4a21e34904b6906c12b228d8e9 67ddd9f771be452a808ba3458af544f8 - - default default] Unable to rescue an instance using a volume snapshot image with img_block_device_mapping image properties set Horizon Output: Requested rescue image 'ddc2df28-c4f5-434b-9f9a-030e56425fc5' is not supported (HTTP 400) (Request-ID: req-bd8037a1-3cd4-43fe-a945-e809b1f8ce16) Ceph Block: https://imgtr.ee/images/2024/08/22/2108d8aacee12eaf800377593a229659.png Instance Snapshot Image: https://imgtr.ee/images/2024/08/22/ca1fa81683acc0518ac161e7f97fda0e.jpeg Could it be any configuration wrong ? We will wait for your answers. Thanks.

5 8

[nova][OpenStackSDK][vitrage][zaqar] broken gates in libraries
by Elõd Illés 23 Aug '24

23 Aug '24

Hi teams in $subject, Since we are close to Feature Freeze period and Dalmatian-3 milestone, the relmgt team checked the status of the gates of libraries [1] and the following repositories seem to have broken gates on their master branches: python-novaclient osc-lib python-vitrageclient python-zaqarclient Could the teams check the broken jobs please and fix them as soon as possible? [1] https://review.opendev.org/q/topic:release-health-check-cwi-dalmatian+is:op… Thanks in advance, Előd Illés irc: elodilles @ #openstack-release

1 0

[neutron] dsvm-fullstack test suite: issues with ovs datapaths when running locally?
by Ihar Hrachyshka 22 Aug '24

22 Aug '24

Hi all, This must be simple to do, but when I try to follow the guide in a vagrant box to run the fullstack suite locally, I am hitting a wall. I suspect I miss something obvious but I can't figure it out. I'm following this doc: https://docs.openstack.org/neutron/latest/contributor/testing/fullstack.htm… I start with an empty ubuntu 22.04 vagrant box with enough memory, disk, cpu. Then I check out the neutron repo and devstack into my home dir. I also `mkdir -p /opt/stack/ && chmod a+rwx /opt/stack` and clone `requirements` repo in /opt/stack/requirements. (Otherwise configure_for_func_testing script fails.) Then I execute: export VENV=dsvm-fullstack tools/configure_for_func_testing.sh ~/devstack -i (I am not sure setting VENV envvar is needed, but that's what is in the doc) It successfully completes (I see "Phew, we're done!" message from the script.) I then: $ tox -e dsvm-fullstack neutron.tests.fullstack.test_logging (or any other test case that triggers Open vSwitch) and the test just fails with timeout. And when I look into logs, I see interface additions failing and in vswitchd log: Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00032|bridge|ERR|failed to create bridge br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00031|ofproto|ERR|failed to open datapath br-eth68909740e: No such file or directory Aug 21 20:24:26 ubuntu ovs-vswitchd[42747]: ovs|00030|ofproto_dpif|ERR|failed to open datapath of type system: No such file or directory (there are more errors of the same...) I disabled apparmor as the doc suggests. I also confirmed that the `openvswitch` kernel module is loaded (it is not loaded after ovs and ovn are built from source as part of configure_for_func_testing.sh execution). I also tried to run the suite as root. (I don't think this is needed, but I figured I'll check.) Anyone else running into this? How do you guys execute the suite locally? Ihar

2 2

[all][elections][ptl][tc] Combined PTL/TC 2025.1 cycle Election Nominations Kickoff
by Slawomir Kaplonski 22 Aug '24

22 Aug '24

Nominations for OpenStack PTLs (Project Team Leads) and TC (Technical Committee) positions (4 positions) are now open and will remain open until Aug 28, 2024 23:45 UTC. All nominations must be submitted as a text file to the openstack/election repository as explained at https://governance.openstack.org/election/#how-to-submit-a-candidacy Please make sure to follow the candidacy file naming convention: candidates/2025.1/<Project_Name or "TC">/<email_address> (for example, "candidates/2025.1/TC/stacker(a)example.org"). The name of the file should match an email address for your current OpenStack Foundation Individual Membership. Take this opportunity to ensure that your OSF member profile contains current information: https://www.openstack.org/profile/ Any OpenStack Foundation Individual Member can propose their candidacy for an available, directly-elected seat on the Technical Committee. In order to be an eligible candidate for PTL you must be an OpenStack Foundation Individual Member. PTL candidates must also have contributed to the corresponding team during the 2024.1 to 2024.2 timeframe, Sep 15, 2023 00:00 UTC - Aug 28, 2024 00:00 UTC. Your Gerrit account must also have a verified email address matching the one used in your candidacy filename. Both PTL and TC elections will be held from Sep 04, 2024 23:45 UTC through to Sep 18, 2024 23:45 UTC. The electorate for the TC election are the OpenStack Foundation Individual Members who have a code contribution to one of the official teams over the 2024.1 to 2024.2 timeframe, Sep 15, 2023 00:00 UTC - Aug 28, 2024 00:00 UTC, as well as any Extra ACs who are acknowledged by the TC. The electorate for a PTL election are the OpenStack Foundation Individual Members who have a code contribution over the 2024.1 to 2024.2 timeframe, Sep 15, 2023 00:00 UTC - Aug 28, 2024 00:00 UTC, in a deliverable repository maintained by the team which the PTL would lead, as well as the Extra ACs who are acknowledged by the TC for that specific team. The list of project teams can be found at https://governance.openstack.org/tc/reference/projects/ and their individual team pages include lists of corresponding Extra ACs. Please find below the timeline: PTL + TC nomination starts @ Aug 14, 2024 23:45 UTC PTL + TC nomination ends @ Aug 28, 2024 23:45 UTC TC campaigning starts @ Aug 28, 2024 23:45 UTC TC campaigning ends @ Sep 04, 2024 23:45 UTC PTL + TC elections start @ Sep 04, 2024 23:45 UTC PTL + TC elections end @ Sep 18, 2024 23:45 UTC Shortly after election officials approve candidates, they will be listed on the https://governance.openstack.org/election/ page. The electorate is requested to confirm their email addresses in Gerrit prior to 2024-08-28 00:00:00+00:00, so that the emailed ballots are sent to the correct email address. This email address should match one which was provided in your foundation member profile as well. Gerrit account information and OSF member profiles can be updated at https://review.openstack.org/#/settings/contact and https://www.openstack.org/profile/ accordingly. If you have any questions please be sure to either ask them on the mailing list or to the elections officials: https://governance.openstack.org/election/#election-officials -- Best regards Slawek Kaplonski skaplons(a)redhat.com

5 5

[all][elections][ptl][tc] Combined PTL/TC 2024.2 cycle Election Nominations progress
by Sławek Kapłoński 22 Aug '24

22 Aug '24

Hi, Here is report about PTL/TC 2024.2 cycle nomination progress so far. If you consider to nominate yourself for the PTL or TC role, please do so in next 6 days. Nominations started @ 2024-08-14 23:45:00 UTC Nominations end @ 2024-08-28 23:45:00 UTC Nominations duration : 14 days, 0:00:00 Nominations remaining : 6 days, 11:07:57 Nominations progress : 53.83% --------------------------------------------------- Projects[1] : 41 Projects with candidates : 8 ( 19.51%) Projects with election : 0 ( 0.00%) --------------------------------------------------- Need election : 0 () Need appointment : 33 (Adjutant Barbican Cinder Cloudkitty Cyborg Designate Glance Heat Keystone Kury r Magnum Manila Masakari Mistral Monasca Neutron Nova Octavia OpenStackAnsible OpenStackSDK Puppet_OpenStack Q uality_Assurance Rally Skyline Storlets Swift Tacker Telemetry Trove Venus Vitrage Watcher Zun) --------------------------------------------------- TC Candidates : 3 (4 Required) =================================================== Stats gathered @ 2024-08-22 12:37:03 UTC -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

Troubleshooting ERROR status after starting instance from Ubuntu Image
by klaus.becker＠webmen.de 22 Aug '24

22 Aug '24

Hi OpenStackers! After a vacation I continue to get my first OpenStack work (one node guided install). What works: I managed to install OpenStack on Ubuntu 22.04.4 LTS on an older DELL server with plenty of RAM and Disks I used Snap Install I can use horizon to create my VMs Problem: After I made a few instances from the Ubuntu templates with different flavors, I am Getting the status „ERROR“ for each of these instances. Trying to shutdown an instance I am getting an Error: Questions: What is the best practice for troubleshooting regarding my installation type (single node, snap)? Where do I find log-files? Google brings different solutions, but the paths and files do not exist for my installation? I tried a few suggestions: Openstack instance generating status error – Fix it now <https://bobcares.com/blog/openstack-instance-generating-status-error/> ( /var/lib/nova/ and /var/lib/nova and /var/log/libvirt/qemu do not exist ) I found: ~snap/juju ~snap/openstack ~/snap/openstack$ ls -ahl total 20K drwxr-xr-x 5 webmen webmen 4.0K Aug 13 10:54 . drwx------ 4 webmen webmen 4.0K Jul 24 13:59 .. drwxr-xr-x 4 webmen webmen 4.0K Jul 19 12:53 576 drwxr-xr-x 4 webmen webmen 4.0K Jul 19 12:53 577 drwxr-xr-x 4 webmen webmen 4.0K Jul 17 12:16 common lrwxrwxrwx 1 webmen webmen 3 Jul 24 10:04 current -> 560 The link "current —> 560 has no target here! In „common", there are sunbeam-logs Maybe there is a misconfiguration with the soft-network. I had a hard time understanding the different networks I need to access VMs from the LAN. Finaly I found then logs using "find -name nova“ /var/snap/openstack-hypervisor/common/log webmen@ada:/var/snap/openstack-hypervisor/common/log$ ls -ahl total 92K drwxr-xr-x 7 root root 4.0K Jul 17 12:39 . drwxr-xr-x 8 root root 4.0K Jul 24 12:54 .. drwxr-xr-x 3 root root 4.0K Jul 17 12:39 libvirt drwxr-xr-x 2 root root 4.0K Jul 17 12:39 neutron -rw-r--r-- 1 root root 58K Aug 16 13:06 neutron.log drwxr-xr-x 2 root root 4.0K Jul 17 12:39 nova drwxr-xr-x 2 root root 4.0K Jul 17 12:39 openvswitch drwxr-xr-x 2 root root 4.0K Jul 17 12:39 ovn webmen@ada:/var/snap/openstack-hypervisor/common/log$ But most dirs are empty webmen@ada:/var/snap/openstack-hypervisor/common/log$ tree . ├── libvirt │ └── qemu ├── neutron ├── neutron.log ├── nova ├── openvswitch │ ├── ovsdb-server.log │ └── ovs-vswitchd.log └── ovn └── ovn-controller.log But I did not find the logs of glance webmen@ada:/$ sudo find -name glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/995/fs/var/lib/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/995/fs/etc/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1192/fs/var/lib/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1192/fs/etc/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/915/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/894/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/891/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/205/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/var/log/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/var/lib/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/usr/share/doc/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/usr/bin/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/usr/lib/python3/dist-packages/glance ./var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/234/fs/etc/glance ./snap/openstack-hypervisor/153/usr/bin/glance ./snap/openstack/577/bin/glance ./snap/openstack/577/lib/python3.10/site-packages/muranoclient/glance ./snap/openstack/576/bin/glance ./snap/openstack/576/lib/python3.10/site-packages/muranoclient/glance find: ‘./proc/3340174’: No such file or directory find: ‘./proc/3361710’: No such file or directory find: ‘./proc/3361712’: No such file or directory find: ‘./proc/3361763’: No such file or directory find: ‘./proc/3364581’: No such file or directory find: ‘./proc/3364766’: No such file or directory find: ‘./proc/3364791’: No such file or directory htop: server lose very high The server still has a load average of 4.68 running for 4 days by now. The hungry processes are: 5 processes of /charm/bin/pebble run —create-dies —hold —http :38813 —verbose /snap/micro8s/7038/kubelite/… Here a copy of top (htop did not let me copy the list) PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1272 root 20 0 11.4g 238528 59048 S 83.3 0.1 3596:35 containerd 21051 root 20 0 1638280 260508 7088 S 72.5 0.1 266:36.35 pebble 21945 root 20 0 1305356 52048 7088 S 71.8 0.0 91:01.52 pebble 20543 root 20 0 1704412 220848 7088 S 69.5 0.1 265:03.19 pebble 2994 root 20 0 1854984 606616 104600 S 37.0 0.3 2957:41 kubelite 1275 root 20 0 3586472 340048 19788 S 23.6 0.2 1822:17 k8s-dqlite 3340174 root 20 0 2135364 31952 19516 S 6.2 0.0 0:00.19 alertmanager 1749555 snap_da+ 20 0 3242712 924312 42656 S 5.9 0.5 67:37.06 mysqld 78926 snap_da+ 20 0 3490924 1.1g 42428 S 5.6 0.6 91:53.33 mysqld 1267 root 20 0 4276408 2.3g 55224 S 5.2 1.2 529:17.09 mongod So… how do I get rid of that status ERROR in Horizon? Best regards Klaus Becker Webmen Internet GmbH Faulenstraße 12 28195 Bremen +49 421 24 34 94-0 info(a)webmen.de www.webmen.de Geschäftsführerin: Christiane Niebuhr-Redder Prokuristin: Kathleen Marx-Colonius Amtsgericht Bremen HRB 16685 USt-ID DE 179413676 Facebook Instagram LinkedIn

2 2

[glance][cinder] No format check on glance image uploading vs. thorough image format checks on cinder
by Florent Le Lain 22 Aug '24

22 Aug '24

Hello folks, Since resolution of vmdk related CVE<https://bugs.launchpad.net/nova/+bug/1996188>, Cinder performs extra checks on format of images used for specific actions, like creating a volume out of an image for instance. See related cinder commit: https://review.opendev.org/c/openstack/cinder/+/871625 Those new checks first led to failures on our tempest tests, on tests that were passing before. E.g.: tempest.api.compute.servers.test_create_server.ServersTestBootFromVolume It turned out that tests were failing because of a discrepancy between actual format of the image (qcow2 here) and declared format in glance db (raw). To us, this raises the question of format checks responsibility throughout images lifecycle, between Glance and Cinder. Today, afaik, Glance will: - tag as 'raw' any image uploaded without any format specified - tag the image with the format provided by the customer, whatever the actual format of the image is whereas Cinder is now more picky, refusing to deal with an image that claims to be of X format but is actually Y. If you do confirm these observations, in your opinion, what would be the best option among: 1. leave it as is, after all openstack customer is responsible for the accuracy of the information provided on image upload and should not be surprised by downstream mismatches when using the image 2. strengthen customer's responsibility of bullet 1. by making image format declaration mandatory on image upload 3. enforce format checks on glance on image upload 4. other way you may think of? something upstream I may have missed? Thank you for your opinion! Regards, Florent Le Lain SRE Team Lead – Public Cloud OVHcloud Rennes [1672217009737]

2 1

Openstack Backup and Restore
by meteccinar＠gmail.com 22 Aug '24

22 Aug '24

Hi everyone, we make cloud region for our company. Using Openstack with Ceph (as Block Storage). We configured backup driver field as BackupDriverCeph in the cinder.conf file. Also created volume and backups pools in Ceph Rados. We have a few problems/questions. We have developed an API that regularly backs up VMs (instances) that work with volumes. The process works as follows: First, we use the create_server_image() method from the OpenStack API to create an image of the instance. This results in an image of the instance being automatically created in OpenStack with a Snapshot type (including metadata such as direct_url, stores, schema, etc.). Simultaneously, a snapshot is also automatically created using with the create_server() command. We then take backups of all volumes attached to the instance. In our restore API, we use the image created from the instance backup mentioned above. We change block_device_mapping field and add the IDs of the volume backups to the block_device_mapping field of the image and use the create_server(image_name, volume_from_backed_volume) method in the OpenStack API to create a new instance. Then new volumes are created from volume backups and attach new volumes in new restored instance. The problem we're encountering is that when using the create_server command, the image created from the instance backup is tied to the direct_url, which points to the RBD URL of the snapshot. As a result, the instance does not running with new volume. Note: If we create a new VM with new volume from volume backup in Horizon. Instance running successfully. What steps should we take to resolve this? Is there a flaw in our approach when designing the API? What is the best practice we should follow when creating an API for backup and restore (especially for instances that boot from volumes)? Is there a configuration we need to adjust in Ceph? We would appreciate your advice on the best practices for backup and restore, particularly for instances that boot from volumes. Kind regards,

3 6

[neutron] Moving CI Meeting to Monday 1400 UTC
by Yatin Karel 21 Aug '24

21 Aug '24

Hello Neutron Team !! As discussed/agreed over the last couple of CI meetings, pushed the changes[1][2] for the new timing. [1] https://review.opendev.org/c/opendev/irc-meetings/+/926763 [2] https://etherpad.opendev.org/p/neutron-ci-meetings -- Thanks and Regards Yatin Karel

1 0

Magnum Caracal: cluster creation goes through state CREATION_FAILED before CREATION_COMPLETE
by Michel Jouvin 21 Aug '24

21 Aug '24

Hi, I continue my tests with Magnum Caracal and found another strange behaviour during cluster creation. It goes through state CREATION_FAILED before reaching the state CREATION_COMPLETE (HEALTHY). It is very confusing. Looking at Heat engine logs, it happens during load balancer creation, at the time the LB amphora has been created and Octavia is waiting to be able to connect it (typically one minue). During this period, Heat logs the following error: ----------- 2024-08-21 09:54:52.194 3200318 INFO heat.engine.resource [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] CREATE: LoadBalancer "loadbalancer" [670985ef-5808-4226-ad1b-b1244871de16] Stack "test-mj-1-30-m1n1-bis-ipufjomqmnnf-etcd_lb-d5fuqygvpp2j" [5c5905bf-f538-4a42-93ce-38d96b7c6603] 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource Traceback (most recent call last): 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 922, in _action_recorder 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource yield 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 1034, in _do_action 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource yield from self.action_handler_task(action, args=handler_args) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resource.py", line 984, in action_handler_task 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource done = check(handler_data) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/octavia/loadbalancer.py", line 169, in check_create_complete 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource return self._check_status() 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource File "/usr/lib/python3.9/site-packages/heat/engine/resources/openstack/octavia/octavia_base.py", line 29, in _check_status 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource raise exception.ResourceInError(resource_status=status) 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource heat.common.exception.ResourceInError: Went to status ERROR due to "Unknown" 2024-08-21 09:54:52.194 3200318 ERROR heat.engine.resource 2024-08-21 09:54:58.804 3200319 INFO heat.engine.scheduler [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] Task pause timed out 2024-08-21 09:54:59.340 3200318 INFO heat.engine.scheduler [None req-610b24bd-7511-4d31-9f8c-b922162a7648 jouvin - - - - -] Task pause timed out ----- that results in Magnum setting the state to CREATE_FAILED despite it is a transient state from Heat point of view and despite Heat stack creation continues successfully once the transient state is cleared and Magnum later updates the status to CREATE_COMPLETED. Unfortunately magnum-conductor log has nothing related to these events (it is filled with barbican_client related messages mainly). Any idea what could cause this behaviour? Best regards, Michel

2 3

[nova][cinder] Can not find device after change HBA
by Qi Xu (Fujitsu) 21 Aug '24

21 Aug '24

Hi everyone, I am currently using RHOSP 16.2.4 and launched a virtual machine (VM) using a volume. From the attach information, I confirmed that this volume was mounted on ports 1 and 2 through HBA1 and HBA2 on the compute node. When I checked the devices on the compute node, I found two devices, which is as expected. [root@overcloud-novacompute-0 heat-admin]# multipath -ll LUN-UUID dm-5 FUJITSU,ETERNUS_DXL size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=10 status=active |- 3:0:2:1 sdl 8:176 active ready running `- 5:0:5:1 sdv 65:80 active ready running [root@overcloud-novacompute-0 heat-admin]# ll /dev/disk/by-path/ lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA1-port1-lun-0 -> ../../sdv lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0a.0-fc-port1-lun-0 -> ../../sdv lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA2-port2-lun-0 -> ../../sdl lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdl I then shut down the VM and the compute node, and replaced HBA1 with HBA3 on the compute node. Under these circumstances, the volume should be mounted on ports 1 and 2 through HBA3 and HBA2 respectively. In other words, I think I should be able to find the following devices. device1: fc-HBA3-port1-lun-0 -> ../../sdl pci-0000:00:0c.0-fc-port1-lun-0 -> ../../sdl device2: fc-HBA2-port2-lun-0 -> ../../sdv pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdv However, after restarting the compute node and the VM, although the connection and read/write operations are functioning normally, the compute node fails to find the volume on HBA3. [root@overcloud-novacompute-0 heat-admin]# multipath -ll LUN-UUID dm-5 FUJITSU,ETERNUS_DXL size=10G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='service-time 0' prio=10 status=enabled `- 5:0:5:1 sdl 8:176 active ready running [root@overcloud-novacompute-0 heat-admin]# ll /dev/disk/by-path/ lrwxrwxrwx. 1 root root 9 Aug 20 08:51 fc-HBA2-port2-lun-0 -> ../../sdl lrwxrwxrwx. 1 root root 9 Aug 20 08:51 pci-0000:00:0b.0-fc-port2-lun-0 -> ../../sdl Could you please advise on how I can make the compute node find the volume on HBA3 in this scenario? And I would like to know how to change the Initiator from HBA1 to HBA3 in the initiator_target_map of DB without detach volume? Regards, Inori.

1 0

[Ironic] Unable to deploy bare metal due to partx partition failure
by Junya Noguchi (Fujitsu) 21 Aug '24

21 Aug '24

To whom it may concern A bare metal deployment with Ironic failed. Deploy Command $ openstack server create --flavor "flavor_name" --image "image_name" --nic net-id="net_id" --security-group "group_name" --key-name "key_name" "server_name" This command powers on a bare metal node that was previously in a powered-off state and boots the deployment image. An error occurs during the deployment process on the deployment image and the server status is error. $ openstack server list +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ | "id" | "server_name " | ERROR | | "image_name" | "flavor_name" | +--------------------------------------+-----------------+--------+----------+-------------------+---------------------+ When I checked the journal log, I saw the following errors. Can you tell me how to resolve these errors to ensure a successful deployment? I look forward to your reply. 1. Partitions created by partx fail Aug 02 03:05:28 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:28.468 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:28 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:28.468 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:483 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.190 1047 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): partx -av /dev/sda execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:390 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.194 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:30 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:30.195 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:483 Aug 02 03:05:30 localhost.localdomain systemd[1]: systemd-hostnamed.service: Deactivated successfully. Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.586 1047 DEBUG oslo_concurrency.processutils [-] Running cmd (subprocess): partx -av /dev/sda execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:390 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG oslo_concurrency.processutils [-] CMD "partx -av /dev/sda" returned: 1 in 0.004s execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:428 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG oslo_concurrency.processutils [-] 'partx -av /dev/sda' failed. Not Retrying. execute /opt/ironic-python-agent/lib64/python3.9/site-packages/oslo_concurrency/processutils.py:479 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG ironic_lib.utils [-] Command stdout is: "partition: none, disk: /dev/sda, lower: 0, upper: 0 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: /dev/sda: partition table type 'gpt' detected Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: range recount: max partno=2, lower=0, upper=0 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: " _log /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_lib/utils.py:99 Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:31.591 1047 DEBUG ironic_lib.utils [-] Command stderr is: "partx: /dev/sda: adding partition #1 failed: Device or resource busy Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: partx: /dev/sda: adding partition #2 failed: Device or resource busy Aug 02 03:05:31 localhost.localdomain ironic-python-agent[1047]: partx: /dev/sda: error adding partitions 1-2 2. The boot efi partition is empty Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 DEBUG ironic_python_agent.efi_utils [-] Looking for all efi files on /tmp/tmp1gqks5kg/boot/efi _get_efi_bootloaders /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_python_agent/efi_utils.py:241 Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 DEBUG ironic_python_agent.efi_utils [-] efi files found in /tmp/tmp1gqks5kg/boot/efi : [] _get_efi_bootloaders /opt/ironic-python-agent/lib64/python3.9/site-packages/ironic_python_agent/efi_utils.py:245 Aug 02 03:05:32 localhost.localdomain ironic-python-agent[1047]: 2024-08-02 03:05:32.525 1047 WARNING ironic_python_agent.efi_utils [-] Empty EFI partition detected. OS in which the above events occurred ・CentOS Stream 9 ・Ubuntu 24 ・OpenSUSE 15 What I've tried (none of which worked) ・Reduce flavor disk size ・Vary local_gb in bare metal node properties Properties driver: irmc boot_interface: irmc-pxe console_interface: ipmitool-socat deplo y_interface: direct inspect_interface: no-inspect management_interface: irmc network_interface: flat power_interface: irmc raid_interfa ce: no-raid rescue_interface: no-rescue storage_interface: noop v endor_interface: no-vendor memory_mb: 64000 loca l_gb: 800 image_type: partition image_disk_format: raw image_chec ksum: None Create bare metal image command (ubuntu example) $ disk-image-create ubuntu baremetal dhcp-all-interfaces -a amd64 grub2 -o ubuntu_x86 Best regards, Junya Noguchi.

1 0

Heat Caracal: log file full of oslo_messaging._drivers.amqpdriver Expecting reply to msg/Received RPC response
by Michel Jouvin 21 Aug '24

21 Aug '24

Hi, On our cloud where Caracal Magnum/Heat is deployed, heat_engine log file is filled with many occurences of the following pair of messages (IDs change for every pair) when there is a cluster creation (and probably deletion) : 2024-08-20 09:21:54.458 78230 INFO oslo_messaging._drivers.amqpdriver [None req-bd38bf98-d978-482a-8758-62b1af72fa43 jouvin - - - - -] Expecting reply to msg 699a920fddb940 8a81fd799125835996 in queue reply_2d4fccf5059340818bbfa72887989a6e 2024-08-20 09:21:54.468 78230 INFO oslo_messaging._drivers.amqpdriver [-] Received RPC response for msg 699a920fddb9408a81fd799125835996 Why these INFO messages from oslo that are polluting the logs? we have debug=false in Heat configuration. It doesn't seem to happen in Antelope that we run on another cloud. Any way to disable these messages? Best regards, Michel

2 4

[devstack][glance] glance-manage Error
by Josephine Seifert 21 Aug '24

21 Aug '24

Hi everyone, within my devstack I can execute cinder-manage db sync or version and everything works fine: ``` stack@devstack:~/devstack$ cinder-manage db version INFO dbcounter [-] Registered counter for database cinder DEBUG oslo_db.sqlalchemy.engines [-] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=304079) _check_effective_sql_mode /opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/engines.py:342}} DEBUG dbcounter [-] [304079] Writer thread running {{(pid=304079) stat_writer /opt/stack/data/venv/lib/python3.10/site-packages/dbcounter.py:102}} INFO alembic.runtime.migration [-] Context impl MySQLImpl. INFO alembic.runtime.migration [-] Will assume non-transactional DDL. a8189e5afd9a ``` But when I try to do the same with glance-manage I get an Error: ``` stack@devstack:~/devstack$ glance-manage db version CRITICAL glance [-] Unhandled error: sqlalchemy.exc.NoSuchModuleError: Can't load plugin: sqlalchemy.plugins:dbcounter ERROR glance Traceback (most recent call last): ERROR glance File "/usr/bin/glance-manage", line 10, in <module> .... ``` my current version of sqlalchemy is: stack@devstack:~/devstack$ pip show sqlalchemy Name: SQLAlchemy Version: 1.4.31 Does anyone have an idea, how I can fix this Error? greetings Luzi

2 2

Report a potential risk of secret leakage in project(cloudkitty)
by jiawei_zhou＠seu.edu.cn 21 Aug '24

21 Aug '24

Dear developers of the project(cloudkitty), We are software security researchers, currently conducting research on secret detection and leakage risk within the open-source ecosystem. In our analysis, we identified potential secret leakage risks in your project, cloudkitty. We provide the detail of our findings in the attachment, which allows you to locate the potential leaked secrets. Below is an interpretation of the attached data: { 'file': '', #The file containing the secret #The project name, version or commit_hash may be reflected in the file path 'line_start': 1, #location: Start line of the secret 'line_end': 28, #location: End line of the secret 'col_start': 1, #location: Start column of the secret 'col_end': 1, #location: End column of the secret 'index_start': 0, #location: Start index of the secret 'index_end': 1675, #location: End index of the secret } Declaration: we hereby declare that we have *NOT* conducted any verification test or exploit on the identified secrets. we plan to publish related research papers in the future, and the relevant content MIGHT BE ACCESS TO THE PUBLIC due to the 90-day disclosure policy. Some advise: 1. If the leaked secret is sensitive and still valid, invalid and rotate the secret immediately. 2. Some secrets seem to be used only in testing environment. Although probably harmless, it is considered bad practices to include secrets for test environment in release builds. Best regards,

2 1

[blazar] Update of core reviewer group
by Pierre Riteau 21 Aug '24

21 Aug '24

Hello, The blazar-core group includes the following contributors who have not been active on the project for several years: - Jason Anderson - Masahito Muroi - Bertrand Souville - Tetsuro Nakamura I propose to remove them so the blazar-core group contains only active reviewers. In the absence of objections, I will perform the removal after August 19. I would like to thank them all for their valuable contributions to the Blazar project. They are welcome back anytime. Thanks, Pierre Riteau (priteau)

1 1

[neutron] Bug deputy report for week of August 12th
by Brian Haley 20 Aug '24

20 Aug '24

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bug - yes, only one bug :) -Brian High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2076916 - VMs cannot access metadata when connected to a network with only IPv6 subnets with the ML2/OVS and ML2/LB backends - Could be issue with gate setup - https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/926503

1 0

Nova buildrequest
by engineer2024 19 Aug '24

19 Aug '24

Hi, when I do a pdb inside a filter in scheduler, it is dropping inside the pdb shell. So far so good. But when I execute these statements inside it as shown below, the buildrequest function get_by_instance_uuid is failing to return any db object: ---------- class ComputeFilter(filters.BaseHostFilter): """Filter on active Compute nodes.""" RUN_ON_REBUILD = False def __init__(self): self.servicegroup_api = servicegroup.API() # Host state does not change within a request run_filter_once_per_request = True def host_passes(self, host_state, spec_obj): """Returns True for only active compute nodes.""" breakpoint(). <-------------- service = host_state.service if service['disabled']: LOG.debug("%(host_state)s is disabled, reason: %(reason)s", {'host_state': host_state, 'reason': service.get('disabled_reason')}) return False else: if not self.servicegroup_api.service_is_up(service): LOG.warning("%(host_state)s has not been heard from in a " "while", {'host_state': host_state}) return False return True ----- at the above breakpoint, it is dropping into the pdb shell. (pdb) from nova import objects as object (pdb) object.BuildRequest.get_by_instance_uuid(spec_obj._context, instance_uuid=spec_obj.instance_uuid)) Build request not found for instance .... ---- But if I place the above buildrequest inside the above same filter and run it normally without breakpoint(), it is working fine. So, why am I not able to do it manually while in pdb, and succeed when done normally ?

1 0

[ops][Image Encryption] Proposed Upgrade Workflow for the Image Encryption
by Josephine Seifert 19 Aug '24

19 Aug '24

Hi operators, we want to integrate image encryption into openstack. When the patches will be merged, there will be a change in a metadata name. Patches are here: https://review.opendev.org/q/topic:%22LUKS-image-encryption%22 We want to propose the following upgrade workflow: 1. install 2024.2 or later, new resources get os_encrypt_* metadata, legacy cinder_encrypt_* metadata is still honored for a deprication period 2. run glance-manage to update all the legacy metadata names to the new os_encrypt* 3. at this point, no openstack service will create cinder_encrypt* metadata, so anything from here must be created by a user 4. Operators need to reach out to the user and notify them, that shortly the cinder_encrypt* will no longer work We don't think the last step will occur often, because it is not documented anywhere, that it is possible to add these metadata to an image right now. If not done correctly (only possible if you know how Cinder does its volume encryption) it would render the image unusable. If you object to this workflow please let me know, we still have time to discuss this. greetings Josephine (Luzi)

1 0

[publiccloud-sig] Reminder - next meeting August 21st - 0700 UTC
by Felix Kronlage-Dammers 19 Aug '24

19 Aug '24

Hi everyone, This Wednesday (August 21st) the next meeting of the public cloud sig is going to happen. We’ll be meeting at 0700 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you on Wednesday! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[cinder] Instance Volume Issue After Restarting tgtd Container During Upgrade
by Vahideh Alinouri 19 Aug '24

19 Aug '24

Hi Everyone, I'm currently upgrading an OpenStack deployment that was set up using Kolla-Ansible. I'm currently on Kolla version 14.8 (Yoga) and I am trying to upgrade to (Latest Yoga) 14.11, to eventually upgrade to the Zed release. During the upgrade process, I encountered an issue with the compute node that uses the Cinder LVM backend, which the instances rely on. Restarting the tgtd container caused the connection between the iscsid container and tgtd to be lost. In the Nova compute logs, I found the following error: Stderr: 'blockdev: cannot open /dev/sdb: No such device or address\n' Because of this, the instance transitions to an error state. I also observed this error in the tgtd container: login failed ISCSI_CONN_STATE_IN_LOGIN/R_STAGE_SESSION_REOPEN 500 10:18 Kernel reported iSCSI connection 6:0 error (1019 - ISCSI_ERR_XMIT_FAILED: Transmission of iSCSI packet failed) state (2). When I check the iSCSI session with the following command, it shows that the session is established: iscsiadm --mode session However, when I run the following command on the compute host (which handles the Cinder LVM backend via iSCSI), no output is returned: tgtadm --lld iscsi --op show --mode target I tried logging out and logging back into the session, but the logout operation failed! Here is my cinder.conf file on the compute host: [os-compXXXX-nvme] volume_group = vgnvme volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver volume_backend_name = os-compXXXX-nvme target_helper = tgtadm target_protocol = iscsi lvm_mirrors = 0 lvm_max_over_subscription_ratio = 1.0 The operating system on the compute node is Ubuntu 20.04. After further investigation, I found a temporary workaround to bring the instance back to an active running state: 1. Power off the compute node. 2. Evacuate the node (it gets stuck in rebuild, but that's okay). 3. Power on the compute node. 4. Reset the instance state to "active." 5. Check the volume status and update it if it's not in the "in-use" state. 6. Perform a hard reboot of the instance. 7. Restart the tgtd container. However, this is only a temporary solution. When I attempt to upgrade to the next version, I encounter the same issue again. Is there anything specific I should consider? Could this be a bug? I would greatly appreciate your help.

1 1

[MAGNUM] Fedora Core OS include NVIDIA GPU drivers
by Oliver Weinmann 19 Aug '24

19 Aug '24

Hi all, I'm running OpenStack 2023.2 (Bobcat) deployed with Kolla-Ansible. Currently I'm struggling to get the NVIDIA drivers to work in a FCOS 35 image. I know FCOS 35 is EOL and outdated, but for a K8s 1.21.11 cluster it is the only version that still works with Magnum. So far I managed to get the GPU Passthrough working for two NVIDIA Tesla T4 GPUs. I have attached the GPUs to an Ubuntu instance and ran geekbench without any issues. Then I tried to install the latest NVIDIA drivers in FCOS 35 by running: sudo rpm-ostree install https://mirrors.rpmfusion.org/free/fedora/rpmfusion-free-release-$ (rpm -E %fedora).noarch.rpm https://mirrors.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-$ (rpm -E %fedora).noarch.rpm systemctl reboot rpm-ostree install --apply-live akmod-nvidia akmods pciutils rpmdevtools xorg-x11-drv-nvidia-cuda It fails on: rpm-ostree install --apply-live akmod-nvidia akmods pciutils rpmdevtools xorg-x11-drv-nvidia-cuda error: Cannot download rpm-build-libs-4.17.0-4.fc35.x86_64.rpm: All mirrors were tried; Last error: Status code: 403 for https://fedoraproject-updates-archive.fedoraproject.org/fedora/35/x86_64/rp… I guess the binaries are no longer hosted? Running the same on FCOS 36 and latest FCOS 40 works fine. Next I have taken new images from the FCOS instances with the NVIDIA driver. openstack server image create --name fedora_coreos_36_nvidia_cuda fedora_coreos_35_latest_no_update --wait This takes ages but anyway I can successfully deploy new instances from the images with my gpu flavor and the GPU is working fine. Again tested with geekbench. But now when adding new K8s worker nodes in a fresh nodegroup on an existing K8s 1.21.11 cluster the nodes are not joining the cluster: openstack coe nodegroup create \ --node-count 1 \ --role worker-gpu \ --flavor gpuflavor \ --image fedora_coreos_36_nvidia_cuda \ $CLUSTER_ID worker-gpu I have tested the latest available images of FCOS36 and they work fine, so something must be wrong with how I created the new image. Is there some magic that needs to be done? I'm also not sure if this is the best way of doing this. Maybe there is already a ready FCOS image with nvidia drivers? I can find lots of outdated information on the internet. Generally I tend to use Vexxhost CAPI from now on, which works great. This is really just a one off attempt to have a working NVIDIA T4 GPU in a K8s 1.21.11 cluster. Best Regards, Oliver

5 16