December 2024 - openstack-discuss

[openstack helm] [Glance] Glance showing OSError: unable to receive chunked part when uploading image
by daniel890723＠gmail.com 10 Jun '25

10 Jun '25

Hello i'm testing using Openstack-helm to install all component but after the installation complete i can't upload the OS image through both openstack CLI or Horizon it stuck at Queueing and when i checking Glance Api log it shows 2024-11-07 07:31:19.362 7 INFO glance.api.v2.image_data [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. 2024-11-07 07:31:19.400 7 ERROR glance.api.v2.image_data [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Failed to upload image data due to internal error: OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Caught error: unable to receive chunked part: OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi Traceback (most recent call last): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1302, in __call__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1345, in dispatch 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return method(*args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 415, in wrapped 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/api/v2/image_data.py", line 299, in upload 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi with excutils.save_and_reraise_exception(): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.force_reraise() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise self.value 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/api/v2/image_data.py", line 162, in upload 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/notifier.py", line 492, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi with excutils.save_and_reraise_exception(): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.force_reraise() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise self.value 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/notifier.py", line 443, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/quota/__init__.py", line 322, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/location.py", line 596, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/location.py", line 487, in _upload_to_store 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/driver.py", line 294, in add_adapter 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/capabilities.py", line 176, in op_checker 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/_drivers/filesystem.py", line 764, in add 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise errors.get(e.errno, e) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/_drivers/filesystem.py", line 746, in add 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi for buf in utils.chunkreadable(image_file, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/common/utils.py", line 73, in chunkiter 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi chunk = fp.read(chunk_size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 294, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi result = self.data.read(i) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 121, in readfn 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi result = fd.read(*args) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/format_inspector.py", line 956, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi chunk = self._source.read(size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1045, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi data = uwsgi.chunked_read() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi Ceph Cluster Status: NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL FSID ceph /var/lib/rook 3 82m Ready Cluster created successfully HEALTH_OK 5cf8b9bb-13e1-4c31-b47d-9e7d92c3c05d Ceph osd status: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 0.35156 - 360 GiB 3.3 GiB 493 MiB 0 B 2.8 GiB 357 GiB 0.91 1.00 - root default -5 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 964 MiB 119 GiB 0.92 1.01 - host instance-lu-1 2 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 964 MiB 119 GiB 0.92 1.01 195 up osd.2 -3 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 - host instance-lu-2 0 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 195 up osd.0 -7 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 - host instance-lu-3 1 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 197 up osd.1 TOTAL 360 GiB 3.3 GiB 493 MiB 0 B 2.8 GiB 357 GiB 0.91 MIN/MAX VAR: 1.00/1.01 STDDEV: 0.01 now i'm able to create Empty Volume using horizon(can't create volume using default cirros image ) and i can start an Instance without volume and mount the empty volume in(Ceph also can detect when i use that volume) is there something i need to set when starting glance? because i'm following the Openstack-helm installation document to install openstack with helm

4 7

[all][dev][ops][tc] Bridging the gap between community and contributing orgs
by Jeremy Stanley 20 May '25

20 May '25

The tl;dr on this is that I'm opening a broad community discussion on how we can, collectively through improved communication, better bring together our established contributors with casual and prospective contributors who struggle to find successful patterns of contribution, so that we might all support each other in order to benefit OpenStack as a whole. Read on for details... OpenStack's TC has requested that foundation staff, when in conversation with representatives of member organizations, encourage participation in the project and collect feedback on any related challenges those organizations encounter in their attempts to do so. In order to better understand the feedback and brainstorm achievable recommendations for the broader community[*], Community Managers on the foundation staff started to work with a small focus group of established contributors with a solid understanding of what successful patterns of contribution look like. It also came to light that established contributors experience many of the same sorts of challenges, even more so if they don't have the luxury of focusing full-time upstream, and so success often comes down to knowing how to effectively navigate those challenges. The community's goal with this exercise is to improve everyone's experience, and your help is needed to do that! The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. Ideas so far to address these gaps include: * Review bandwidth - Ensure contributors and companies understand the importance and benefits of code review - Incentivize and credit meaningful "+1" reviews more - Understand that the volume of changes and limited reviewer bandwidth often means proposals aren't reviewed quickly (or at all), and their proponents may need some additional tenacity and engagement with the team to get their work noticed - Utilize "review dashboards" as a way to find stale changes which have fallen through the cracks * Review strategy and etiquette - Be clear about the meaning of votes on changes, especially negative ones, and set explicit timelines for things like "procedural -2" or WIP blocks when their approval needs to be delayed - Focus on comments and requests that affect whether or not the change can be merged after the fixes are submitted, rather than requesting trivial adjustments to an unsuitable change which has more fundamental problems - Reciprocate with reviews of changes from new reviewers you see leaving insubstantial reviews, demonstrating to them what deeper and more meaningful reviewing looks like * Mentoring newcomers - Established contributors with sufficient bandwidth can help mentoring newcomers, potential core reviewers and new leaders - Mentoring can happen as part of internship programs as well as by helping newcomers determined to become active contributors; both ways should be embraced and utilized What challenges are you facing? And, how would you improve them? Two weeks from now, we'll have a forum session at the OpenInfra Summit Asia in Suwon[**] where our community can refine these possible approaches and integrate others, as well as bring attention to them and promote their application. I've also proposed a similar forum session for OpenInfra Days North America in Indianapolis a month after that, where we can hopefully continue this conversation. The goal for this multi-stage effort is to improve the contribution experience for established participants, casual contributors and newcomers alike. While it's easy to jump to blaming in these situations, it will be far more productive if we can focus on the challenges and experiences that we would each like to remove or improve for ourselves and others in our community. As we find ways together to improve our efficiency, we can reduce the load on all contributors and lower the barriers for people to join and participate. [*] https://etherpad.opendev.org/p/r.2205024e55689bccb82c20c960853cb5 [**] https://2024.openinfraasia.org/a/schedule#view=calendar&title=Bridging%20th… -- Jeremy Stanley

2 9

[manila] Removal of the Standalone CephFS NFS protocol helper in Epoxy
by Ashley Rodriguez 02 May '25

02 May '25

Hi zorillas! As per subject line, we will be deprecating the standalone CephFS NFS protocol helper in this cycle. This is following our deprecation warning we put out in 2024.1 / Caracal [1] [2]. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the ceph-orch-deployed clustered NFS service; this code was backported to stable/2023.1 (Antelope). Although the upgrade path is disruptive, there is a UX improvement in Manila to ease this migration through surfacing “preferred” export paths. Moving forward, deployments must have “cephfs_nfs_cluster_id” in your configuration, otherwise the driver will fail to start. Manila will not make any export changes to NFS-Ganesha, so upgrading manila without an NFS cluster will mean that only the control plane is not going to work; but, data plane will remain unaffected. To assist migration, admins can also configure "cephfs_ganesha_export_ips" (or alternatively, "cephfs_ganesha_server_ip") alongside "cephfs_nfs_cluster_id". Setting these options will allow the CephFS driver to report additional export paths. These additional export paths will have the "preferred" metadata key set to False. Best, Manila team [1] https://review.opendev.org/c/openstack/manila/+/912648 [2] https://docs.openstack.org/manila/2024.2/configuration/shared-file-systems/…

5 7

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

[oslo] python-consul2 is unmaintained
by Takashi Kajinami 21 Apr '25

21 Apr '25

Hello, I recently noticed that python-consul2 library[1], which is used by the consul coordination driver in tooz, hasn't been updated for 3+ years. Looking at issues in the project, there is an indication that the library is not compatible with ACL API in consul >= 1.11[2]. There is another fork created[3] and this includes fixes to support new API endpoints. Although the driver in tooz does not directly use the feature, I think it's better to discuss our plan to mitigate risks caused by the unmaintained library. As far as I can think of, there are three options now 1. Deprecate consul driver and remove it 2. Keep using python-consul2 now. At least basic functionality is proven to work by our functional tests (Note that the tests do not cover all configuration patterns) 3. Replace python-consul2 by py-consul Before we start the overall discussions I'd like to ask a few questions to the wider audience, as our inputs. - Is anyone using the consul coordination driver now ? - Does anyone have experience with python-consul2 and/or py-consul ? Note that this discussion affects the http messaging driver[4] currently proposed, because it also requires client library to communicate with Consul. Actually I noticed the situation when I noticed that the requirement check job is failing because the library is not part of global requirements list... Thank you, Takashi Kajinami [1] https://github.com/poppyred/python-consul2 [2] https://github.com/poppyred/python-consul2/issues/42 [3] https://github.com/criteo-forks/py-consul [4] https://review.opendev.org/c/openstack/oslo.messaging/+/912499 -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 3

[octavia] Purging ERROR'ed amphora and load balancers stuck in transitional states
by Paul Browne 11 Apr '25

11 Apr '25

Hello Octavia users, For a while I've been trying to find a systematic way to remove Octavia LBs+Amphora that have become stuck in ERROR or transitional PENDING_* states ; $ openstack loadbalancer amphora list -f value | grep ERROR a2323f14-3aaa-418a-8249-111aaa9c21fe 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR MASTER 10.8.0.242 192.168.3.247 e5b236ba-e7ee-4ed7-9f58-57ce7a408489 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR BACKUP 10.8.0.190 192.168.3.247 6b556f28-93c9-49dd-b6ee-4379288e7957 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR MASTER 10.8.0.39 192.168.3.126 c669db5d-8686-4d5c-9e95-e02030b34301 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR BACKUP 10.8.0.174 192.168.3.126 $ openstack loadbalancer list -f value | grep -vi active 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe k8s-clusterapi-cluster-default-ci-6386871107-kube-upgrade-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.247 PENDING_UPDATE amphora d5e402fe-2c4b-49af-a700-532cb408cee5 k8s-clusterapi-cluster-default-ci-6386871107-latest-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.126 PENDING_DELETE amphora These resources are marked immutable and so cannot be failed over or deleted; $ openstack loadbalancer amphora failover a2323f14-3aaa-418a-8249-111aaa9c21fe Load Balancer 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe is immutable and cannot be updated. (HTTP 409) (Request-ID: req-6e66c4e8-c3d3-4549-a03c-367017c8c8b3) $ openstack loadbalancer failover 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe Invalid state PENDING_UPDATE of loadbalancer resource 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe (HTTP 409) (Request-ID: req-84b44212-e7a8-4101-a16f-18c774c0577e) The backing Nova instances for these Amphora do seem to exist and be in good working order. Is there any API way to purge these out of Octavia's service state, or would (very careful) DB hackery be required here? Many thanks, Paul Browne ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

4 6

GPU PCI passthrough woes.
by Andy Speagle 10 Apr '25

10 Apr '25

Hey Folks, I could use a little assistance getting GPU passthrough working. I had this working already for one flavor of nvidia gpu... and I've added some hosts with a much newer gpu... I've updated the pci_alias and pci_passthrough variables and those seem to be getting set properly in nova.conf passthrough_whitelist = [{"vendor_id":"10de", "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": "10de"} I believe I have all of the iommu stuff configured and have the pci- stub module entries... dmesg output shows that the GPUs are being claimed by the stub module. $ openstack flavor show t1.small_gpu_l40s +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 0 | | id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | | name | t1.small_gpu_l40s | | os-flavor-access:is_public | True | | properties | pci_passthrough:alias='gpu-l40s:1' | | ram | 2048 | | rxtx_factor | 1.0 | | swap | 0 | | vcpus | 1 | +----------------------------+--------------------------------------+ Yet... I can't seem to get an instance to run using that new flavor... keeps complaining about there not being enough hosts available. Fault: code: 500 created: 2024-10-30T02:04:33Z message: "No valid host was found. There are not enough hosts available." details: | Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1580, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 940, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist- packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist- packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Any clues on how/where to dig into this more to see what might be missing? Thanks. -- Andy Speagle

3 2

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 17 Mar '25

17 Mar '25

Hello Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [Call Trace] ./stack.sh:1126:start_keystone /opt/stack/devstack/lib/keystone:549:die [ERROR] /opt/stack/devstack/lib/keystone:549 keystone did not start Error on exit World dumping... see /opt/stack/logs/worlddump-2024-11-18-150808.txt for details ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) Thanks in advance, Ishan

3 5

[all][tc][ptl][qa] Migrating upstream CI/CD jobs to Ubuntu Noble (24.04)
by Ghanshyam Mann 15 Mar '25

15 Mar '25

Hello Everyone, As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. This is the etherpad which has the migration plan and will be used for the tracking: - https://etherpad.opendev.org/p/migrate-to-noble I have prepared the change(WIP) for base jobs migration on Noble (openstack tox jobs, devstack, and tempest) which will migrate most of the projects jobs. Tempest change[3] include dependency on openstack tox, devstack base jobs changes so testing project gate using the the tempest change will test the complete set. Refer to the etherpad for the complete migration plan/steps. * Deadline: Nov 29th 2024 (R18) is the deadline to test/fix the projects jobs and to merge the base jobs running on Noble (openstck tox, devstack, and tempest). [1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubunt… [3] https://review.opendev.org/c/openstack/tempest/+/932156 -gmann

2 9

[kolla] [nova] API flags ignored in Wallaby
by Albert Braden 13 Mar '25

13 Mar '25

After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list: Train: openstack server list --all-projects --locked --os-compute-api-version 2.73 <empty list because nothing is locked> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> Wallaby (locked flag is ignored): openstack server list --all-projects --locked --os-compute-api-version 2.73 <list of every VM in the cluster> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> The second problem is the hypervisor list. We can’t get “long” output in Wallaby: Train: openstack hypervisor list --long | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | Wallaby (long flag is ignored): openstack hypervisor list --long | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |

6 14

[nova]Intel RDT CAT Support Missing from OpenStack
by Junya Noguchi (Fujitsu) 25 Feb '25

25 Feb '25

To whom it may concern I have a question regarding the lack of Intel RDT CAT implementation in OpenStack We are planning to commit Resource Monitoring functionality to Openstack for ARM, and are exploring the implementation of Intel RDT on Openstack. We found the following blueprint for the Intel RDT CAT implementation in nova. https://blueprints.launchpad.net/nova/+spec/cat-support However, the Intel RDT CAT feature is not currently implemented in OpenStack. Has the implementation of Intel RDT CAT functionality in Openstack been rejected? Or is it just a development delay? Best regards, Junya Noguchi.

5 9

[tc][all][monasca ] Retiring the Monasca project
by Ghanshyam Mann 16 Feb '25

16 Feb '25

Hello everyone, As you might know, the Monasca project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project. I'm sending this email to propose the Monasca retirement and based on the response, I will proceed next. [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 -gmann

6 17

[nova] How to gain approval for a new feature on unavailable hardware
by taketani.ryo＠fujitsu.com 14 Feb '25

14 Feb '25

To Whom It May Concern, We are considering to propose and develop a new feature on OpenStack Nova for building Arm Confidential Computing Architecture (Arm CCA)-encrypted instances. I have some questions for doing it. * While specification of Arm CCA is already open(https://developer.arm.com/documentation/den0125/0300/Overview), No Arm CCA-enabled hardware is currently available to our knowledge. Therefore, we plan to conduct testing in an internal environment. * Would a proposal for a feature utilizing publicly unavailable hardware be acceptable to OpenStack? Alternatively, in this case, what steps or preparations are required for approval? * Should the blueprint and specification be submitted simultaneously? * The specification is currently incomplete. Building Arm CCA instances requires libvirt APIs, similar to AMD SEV instances(https://blueprints.launchpad.net/nova/+spec/amd-sev-libvirt-suppo…, but specification of these are not yet determined publicly. We are considering submitting the blueprint first to inform the community of our ongoing development efforts. Regards

3 6

[ironic]About the OS supported by Diskimage-builder
by Junya Noguchi (Fujitsu) 03 Feb '25

03 Feb '25

To whom it may concern. My name is Junya Noguchi. I have some questions about the supported OS for Diskimage-builder operation. Please respond. References https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html Does it mean that the OS that says "work but receive only limited testing and support" works but is not supported? References https://docs.openstack.org/diskimage-builder/latest/ This document describes RHEL and other operating systems that are not supported in the previous question. Are these operating systems supported? Also, are there any plans to support more operating systems in the future? Best regards, Junya Noguchi

5 7

[Nova] Nova-compute service flapping on Antelope
by Jakub Darmach 29 Jan '25

29 Jan '25

Hello, Recently I encountered an interesting issue - nova-compute service started to temporarily lose Rabbit connectivity to regain it after a few seconds on Antelope. The issue was replicated on the test environment, without specific steps to replicate though - it just starts to happen after some time. I pasted a log example in the bug I opened [1]. First we can see rabbit logging closing AMQP connection, followed by nova-compute reporting rabbit server being unreachable, with next message being successful reconnection. Restarting nova-compute services helps temporarily, the issue starts to manifest after a few days. Initially disconnects once every few hours, each time in shorter intervals, to the point it happens every minute or so. Did anyone encounter something similar? [1] https://bugs.launchpad.net/nova/+bug/2092297 Pozdrawiam / Best regards, *Jakub Darmach*

3 2

[glance][ops] proposal to remove the glance-scrubber
by Brian Rosmaita 20 Jan '25

20 Jan '25

Hello Glance users, The Glance project team wants to determine whether anyone is using the glance-scrubber utility [0]. If not, we are thinking of deprecating it this cycle to be removed in the appropriate future cycle. If you have not previously heard of the glance-scrubber, that's most likely because it's only useful for particular deployment configurations of which yours is not one. See the doc [0] for details if you are curious. Our suspicion is that the glance-scrubber may have outlived its usefulness for modern OpenStack configurations. (The last time I remember it being discussed at a PTG was during the snowstorm in Dublin, which I believe was the Rocky development cycle). On the other hand, the team also has some ideas for improving the scrubber, but there's no point doing that if it's not actually being used in the first place. So if removal of the glance-scrubber would negatively affect you, please respond to this email so we can get some data points. Thanks! [0] https://docs.openstack.org/glance/latest/cli/glancescrubber.html

2 2

[i18n][infra][horizon] Zanata translation job failures after Python 3.8 deprecation
by Ian Y. Choi 19 Jan '25

19 Jan '25

Hi all, During last week's PTG, the I18n SIG identified Zanata translation job failures for Horizon (and likely related plugins). The root cause appears to be Python 3.8 deprecation on Horizon [1]. Failed jobs have been observed on Zuul [2], and translation proposals were made on Gerrit [3]. Since translation sync on Horizon stable branches is functioning correctly, the error message seems to be related to Python 3.8 deprecation: ERROR: Cannot install -r doc/requirements.txt (line 3) due to conflicting package version dependencies. To address this, I'd like to raise two points: 1/ To resolve the issue, would you support updating translation sync jobs with Bionic-based Zuul jobs by either upgrading to Jammy or modifying the Bionic image to support Python >3.8? 2/ At least me from I18n SIG acknowledges that recognizing these translation job errors was delayed. Moving forward, would it be beneficial to implement periodic (e.g., monthly) reporting on translation job successes and failures? Thank you, /Ian [1] https://docs.openstack.org/releasenotes/horizon/2024.2.html#upgrade-notes [2] https://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-up… [3] https://review.opendev.org/q/owner:proposal-bot+project:openstack/horizon+b…

2 2

[openstack][octavia] How clients can debug their LB
by Nguyễn Hữu Khôi 15 Jan '25

15 Jan '25

Hello. I would like to know how clients can debug their LB such as access log or monitor LB performance? Thank you. Regards Nguyen Huu Khoi

5 7

[cinder][kubernetes] pvc stuck in reserved state
by Eugen Block 10 Jan '25

10 Jan '25

Hi, we have a couple of k8s clusters in our openstack environment (Victoria) and are facing some problems when pods get moved to different worker nodes. The PVCs get stuck in "reserved" state, and they only get attached after resetting the volumes to "available" state. First I thought it was due to the missing multiattach volume type which seems to work fine in our customer's cluster. I also found the "duplicate connector" error messages in the nova-compute.log: > Unable to update attachment.(Invalid volume: duplicate connectors > detected on volume But after adding the multiattach volume type and retyping the affected volumes, the PVCs still get stuck in "reserved" state while I don't see any error messages on the hypervisors anymore. So it seems that something has changed/improved, but now I don't have a trace where to look next. Could this be a cinder-csi driver issue or something? Our cinder backend is ceph (Pacific). Not sure which information could help so please ask for anything I can provide. Any pointers are highly appreciated! Thanks! Eugen

1 2

Nova create vm with special ipv4 and ipv6
by Alex Song (宋文平) 08 Jan '25

08 Jan '25

Hi Nova team, does nova support create vm with special ipv4 and ipv6? As port create with ipv4 and ipv6 works by: openstack port create 111 --network 10851e96-8995-4988-b819-0514b38d7353 --fixed-ip subnet=a564ac90-3ff3-47d7-8ae5-20e2666eaa3c,ip-address=172.24.4.54 --fixed-ip subnet=afa3c858-c3d3-4784-8964-2c46840b714d,ip-address=2001:db8::183 but vm create with ipv4 and ipv6 with the cmd openstack server create --flavor 1 --image 62254c92-a579-4c8a-a2f4-f2624cef8ec1 --nic net-id=10851e96-8995-4988-b819-0514b38d7353,v4-fixed-ip=172.24.4.53,v6-fixed-ip=2001:db8::188 test raise exception: openstack server create: error: argument --nic: Invalid argument net-id=10851e96-8995-4988-b819-0514b38d7353,v4-fixed-ip=172.24.4.53,v6-fixed-ip=2001:db8::188; either 'v4-fixed-ip' or 'v6-fixed-ip' should be specified but not both. So why nova doesnot support?

2 3

Failing at TASK [glance : Copy over multiple ceph configs for Glance]
by Karera Tony 05 Jan '25

05 Jan '25

Hello Team, I am deploying Openstack release 2024.02 with External Ceph. However I am encountering the error below. But it is failing here TASK [glance : Copy over multiple ceph configs for Glance] Please assist Regards Tony Karera

2 1

hexchat irc channel
by Alex Song (宋文平) 31 Dec '24

31 Dec '24

Hello, I update hexchat and which network is available now? Thanks [cid:image001.png@01DB5782.29D8F150] [cid:image002.png@01DB5782.29D8F150] [cid:image003.png@01DB5782.29D8F150]

3 5

How to find the official documentation for installing OpenStack services and how to locate the historical documentation
by ゞPositive î 31 Dec '24

31 Dec '24

 https://docs.openstack.org/install-guide/openstack-services.html# I am installing the OpenStack zed version. How can I find the zed version document in this official document, as there are only the last four versions available ゞPositive ? 451526204(a)qq.com  

2 1

OpenStack zed version seeking help
by 451526204＠qq.com 30 Dec '24

30 Dec '24

Identity service Image service Placement service Compute service Networking service Dashboard Block Storage service In the official documentation, when installing OpenStack services, there are only the last four versions in the interface of the above services. So where can I find the zed version when installing it?

2 1

How to find the official documentation for installing OpenStack services and how to locate the historical documentation
by ゞPositive î 29 Dec '24

29 Dec '24

 https://docs.openstack.org/install-guide/openstack-services.html# I am installing the OpenStack zed version. How can I find the zed version document in this official document, as there are only the last four versions available ゞPositive ? 451526204(a)qq.com  

1 0

[ceilometer]Intel RDT CMT support on OpenStack
by Junya Noguchi (Fujitsu) 26 Dec '24

26 Dec '24

To whom it may concern I have a question about the support status of Intel RDT CMT in OpenStack. We are planning to commit Resource Monitoring functionality to Openstack for ARM, and are exploring the implementation of Intel RDT on Openstack. We found a document published by Intel on the Intel RDT CMT on OpenStack.[1] However, when I checked the source code for nova and ceilometer, it was modified to ignore the settings in the documentation[1].[2] In addition, ceilometer was written to perform resource monitoring by itself. Is resource monitoring with ceilometer supported? Or is it just old code that is not used? [1] https://www.intel.com/content/www/us/en/developer/articles/training/how-to-… [2] https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L… Best regards, Junya Noguchi.

2 2

[nova] force pinning instances to AZ
by Arnaud Morin 24 Dec '24

24 Dec '24

Hey all, I have a region with multiple AZ. If the server was not created in a specific zone then it is free to be moved to other zones. Is there any way in nova config to prevent this? Regards, Arnaud.

7 10

[tacker] Cancelling the next two weekly meetings
by Yasufumi Ogawa 24 Dec '24

24 Dec '24

Hi all, Due to the holidays, I'd skip the next two meetings. So, the next our meeting is Jan 14th. Thanks, Yasufumi

1 1

ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version
by Swogat Pradhan 24 Dec '24

24 Dec '24

Hi, I am currently trying to deploy openstack wallaby using tripleo arch. I created the network jinja templates, ran the following commands also: #openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml # openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml # openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml and used the environment files in the openstack overcloud deploy command: (undercloud) [stack@hkg2director ~]$ cat deploy.sh #!/bin/bash THT=/usr/share/openstack-tripleo-heat-templates/ CNF=/home/stack/ openstack overcloud deploy --templates $THT \ -r $CNF/templates/roles_data.yaml \ -n $CNF/workplace/custom_network_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e $CNF/templates/node-info.yaml \ -e $CNF/templates/scheduler-hints.yaml \ -e $CNF/workplace/networks-deployed-environment.yaml \ -e $CNF/workplace/vip-deployed-environment.yaml \ -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ -e $CNF/workplace/custom-net-bond-with-vlans.yaml Now when i run the ./deploy.sh script i encounter an error stating: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command. 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1 Can someone tell me where the mistake is? With regards, Swogat Pradhan

4 4

[all] Installation guide for openSUSE and SUSE Linux Enterprise
by Takashi Kajinami 24 Dec '24

24 Dec '24

Hello, As you may know, we have been maintaining installation guide for three linux distributions below. - oepnSUSE and SUSE Linux Enterprise - Red Hat Enterprise Linux and CentOS - Ubuntu However, as far as I know, the packages for openSUSE and SLES haven't been built/updated for some time. Also openSUSE was removed from tested runtimes in wallaby and hasn't been tested. So I doulbt that users can use these operating systems to install OpenStack nowadays. Can/Should we remove the guide for openSUSE/SLES Now ? I'm not very sure if we can maintain the guide for these with no details available nowadays. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 1

[neutron] Bug deputy Dec 16 - Dec 22
by Rodolfo Alonso Hernandez 23 Dec '24

23 Dec '24

Hello Neutrinos: This is the bug list from last week: HIGH * api slow due to missing LRP index ( https://bugs.launchpad.net/neutron/+bug/2092133) Assigned and patch proposed: https://review.opendev.org/c/openstack/ovsdbapp/+/938004 MEDIUM * [RFE] OVN: allow to configure activation-strategy (change it to a different strategy from rarp; or disable completely) ( https://bugs.launchpad.net/neutron/+bug/2092250) Assigned and patch proposed: https://review.opendev.org/c/openstack/neutron/+/938106 LOW * [OVN] Migrate the OVN L3 scheduler to use HA_Chassis_Group ( https://bugs.launchpad.net/neutron/+bug/2092271) Assigned. * Deprecate and remove vlan_transparent config option ( https://bugs.launchpad.net/neutron/+bug/2092174) *NOT ASSIGNED*. Regards.

1 0

[freezer][freezer-dr] Proposal to retire Freezer DR project
by Dmitriy Rabotyagov 23 Dec '24

23 Dec '24

Hi folks, As an ongoing effort to return Freezer project back to the list of Active projects and recovering gate health for repositories owned by Freezer, I've come across freezer-dr[1] Freezer DR is a separate from freezer/freezer-api instance-ha project which is highly pluggable by design with the main goal to evacuate instances from the compute node once service detects that the serving compute node is down. And that is what is currently achieved by the Masakari project. While pluggable design for the service, can be treated as a good benefit in favor of it. Another relatively valuable benefit of freezer-dr, is that it uses nova service state to detect if the compute is down, rather then require 3rd party cluster tooling to independently evaluate hypervisor health. Regardless, I don't see enough value to keep maintenance of that project and instead it's better to keep focus on Masakari maintenance/development instead. With that being said, I suggest the retirement of the freezer-dr project. If someone feels in favor of keeping maintenance of a separate project for instance evacuation - please feel free to reply in thread until December 10, 2024. After that date I will initiate the project retirement process. [1] https://opendev.org/openstack/freezer-dr

2 2

[magnum][cluster api] Heads-up - Octavia OVN driver and latest CAPO
by Stig Telfer 21 Dec '24

21 Dec '24

Hi Magnum folks - Heads up - we wanted to make people aware of a recent upstream CAPO change that could break your Magnum setup if using Cluster API and the Octavia OVN driver. What we've seen is that a recent contribution to CAPO tightens the security group rules for NodePorts to the local tenant network. This works fine for Octavia's Amphora driver, in which HAproxy becomes the source for all back-end API requests. The Octavia OVN driver doesn't have that behaviour, which results in inbound requests getting filtered. The CAPO release is v0.11.0 (https://github.com/kubernetes-sigs/cluster-api-provider-openstack/releases/…) The issue is described here: https://github.com/kubernetes-sigs/cluster-api-provider-openstack/issues/23… The Magnum drivers will pin CAPO driver versions so it's more of a developer / power-user concern than for conventional operators. Cheers, Stig

1 0

[masakari] is this project still active and can be used in production environment
by luckydog xf 20 Dec '24

20 Dec '24

Hi, list, I saw this project hasn't been active for months. So I'm wondering if it is still actively developed and can it be used in production? Are there any use cases ? Thanks.

4 8

Openstack ansible Upgrades
by engineer2024 20 Dec '24

20 Dec '24

Is it possible to upgrade openstack ansible from one major release to another, along with the os distribution upgrade all in one flow ? I mean, as of now we are doing the upgrade as two step process. first we are upgrading the os of the hosts and then the os of the lxc containers and then re running the existing openstack version playbooks. Next , we are upgrading to major release of OSA. This two step process is doubling our effort and time to upgrade to a major version. Is it possible to download and run the major release playbooks soon after the os upgrades ? instead of rerunning the existing playbook again and then download and running next version playbooks ?

2 1

[openstack][magnum] user who created cluster was deleted from keystone
by Nguyễn Hữu Khôi 20 Dec '24

20 Dec '24

Hello. I have a case: What will we do if the user who created the cluster was deleted from the keystone? In this case, we cannot add or remove node groups or use autoscale and autoheal. I tried to create a new application credential and patch my cluster. Then I can add or remove node groups but new node groups cannot join the cluster(nodes not ready) Thank you. Regards Nguyen Huu Khoi

2 2

[cinder] Next 2 Meetings cancelled
by Jon Bernard 20 Dec '24

20 Dec '24

Hello all, the next two weekly meetings (Dec. 25 and Jan 1) are cancelled for the holidays. I hope everyone has a nice break! Catch you at the Jan 8. meeting. Take care, -- Jon

1 0

[manila] Cancelling Dec 26th and Jan 2nd weekly meeting
by Carlos Silva 20 Dec '24

20 Dec '24

Hello zorillas, As announced in the previous weekly meetings, we will not have the upstream weekly meeting in the next two weeks, as most of the Zorillas will be taking a break over the year-end holidays. We will be back to our regular schedule on Jan 9th. Thank you very much for your collaboration throughout the year. This has been an incredible year and we got a lot done together. You definitely make this community more special. Wishing you a very well-deserved break! Cheers, carloss

1 0

[manila] Removal of the Standalone CephFS NFS Protocol Helper in Epoxy
by Ashley Rodriguez 20 Dec '24

20 Dec '24

Hi zorillas! As per subject line, we will be deprecating the standalone CephFS NFS protocol helper in this cycle. This is following our deprecation warning we put out in 2024.1 / Caracal [1] [2]. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the ceph-orch-deployed clustered NFS service; this code was backported to stable/2023.1 (Antelope). Although the upgrade path is disruptive, there is a UX improvement in Manila to ease this migration through surfacing “preferred” export paths. Moving forward, deployments must have “cephfs_nfs_cluster_id” in your configuration, otherwise the driver will fail to start. Manila will not make any export changes to NFS-Ganesha, so upgrading manila without an NFS cluster will mean that only the control plane is not going to work; but, data plane will remain unaffected. To assist migration, admins can also configure "cephfs_ganesha_export_ips" (or alternatively, "cephfs_ganesha_server_ip") alongside "cephfs_nfs_cluster_id". Setting these options will allow the CephFS driver to report additional export paths. These additional export paths will have the "preferred" metadata key set to False. Best, Manila team [1] https://review.opendev.org/c/openstack/manila/+/912648 [2] https://docs.openstack.org/manila/2024.2/configuration/shared-file-systems/…

1 0

[Glance] Cancelling weekly meeting for Next 2 weeks
by Mridula Joshi 19 Dec '24

19 Dec '24

Hello, Due to year end holidays, most of the team members are not around for the next two weeks, so cancelling weekly meetings . Next Glance meeting will be on January 9th !! Thanks, Mridula Joshi

1 0

[kolla] No weekly meetings until 8th January 2025
by Michał Nasiadka 19 Dec '24

19 Dec '24

Hello Koalas, Weekly meetings until 8th January (i.e. 25th Dec and 1st Jan) are cancelled. See you on 8th of January! Cheers, Michal

1 0

OpenStack APIs
by rushit.ajudiya＠crestdata.ai 19 Dec '24

19 Dec '24

Hello, We want to retrieve all hypervisors available in OpenStack using the Hypervisor API documented at https://docs.openstack.org/api-ref/compute/#list-hypervisors-details. The documentation states that the "limit" and "marker" parameters can be used to manage pagination, but they don’t seem to work as expected. Is this a problem with the API, or is there another parameter not mentioned in the documentation that we should use?

2 1

[placement] isolating aggregates on kolla-ansible
by przemyslaw.kuczynski＠ochk.pl 19 Dec '24

19 Dec '24

Did someone manage to configure Windows host aggregate ? I tried same is in https://docs.openstack.org/nova/2024.2/reference/isolate-aggregates.html 1. Created 2 aggregates in 2 AZ , added 1 host to each 2. Set the scheduler.enable_isolated_aggregate_filtering config option to true in nova.conf and restart the nova-scheduler service cat /etc/kolla-pilot/config/nova.conf [libvirt] hw_machine_type=x86_64=q35 [scheduler] enable_isolated_aggregate_filtering = True 3. Add trait CUSTOM_LICENSED_WINDOWS to the resource providers for kvm01 and kvm03 in the Placement service.openstack resource provider list +--------------------------------------+-------+------------+--------------------------------------+----------------------+ | uuid | name | generation | root_provider_uuid | parent_provider_uuid | +--------------------------------------+-------+------------+--------------------------------------+----------------------+ | 36c0ae68-cc5f-4219-94bb-e46bcd3311d9 | kvm01 | 257 | 36c0ae68-cc5f-4219-94bb-e46bcd3311d9 | None | | 8abb4b5e-0407-405d-97d1-7a03a9a06cbe | kvm03 | 129 | 8abb4b5e-0407-405d-97d1-7a03a9a06cbe | None | | 14ed0962-d8ce-4bed-b90a-243b3a069f5d | kvm02 | 1096 | 14ed0962-d8ce-4bed-b90a-243b3a069f5d | None | | 8714da8e-7e9b-4823-93bb-df819c143e99 | kvm04 | 87 | 8714da8e-7e9b-4823-93bb-df819c143e99 | None | +--------------------------------------+-------+------------+--------------------------------------+----------------------+ openstack --os-placement-api-version 1.6 trait create CUSTOM_LICENSED_WINDOWStraits=$(openstack --os-placement-api-version 1.6 resource provider trait list -f value 36c0ae68-cc5f-4219-94bb-e46bcd3311d9 | sed 's/^/--trait /')openstack --os-placement-api-version 1.6 resource provider trait set $traits --trait CUSTOM_LICENSED_WINDOWS 36c0ae68-cc5f-4219-94bb-e46bcd3311d9traits=$(openstack --os-placement-api-version 1.6 resource provider trait list -f value 8abb4b5e-0407-405d-97d1-7a03a9a06cbe | sed 's/^/--trait /') openstack --os-placement-api-version 1.6 resource provider trait set $traits --trait CUSTOM_LICENSED_WINDOWS 8abb4b5e-0407-405d-97d1-7a03a9a06cbe openstack --os-compute-api-version 2.53 aggregate set --property trait:CUSTOM_LICENSED_WINDOWS=required WindowsW1 openstack --os-compute-api-version 2.53 aggregate set --property trait:CUSTOM_LICENSED_WINDOWS=required WindowsW2 Flavor metadata trait:CUSTOM_LICENSED_WINDOWS with value required nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Got no allocation candidates from the Placement API.

1 0

#eventlet-removal Thank you for 2024
by Herve Beraud 18 Dec '24

18 Dec '24

You are no doubt aware that we are currently facing a real challenge. We are living in a complex situation where we have to engage resources to exit from a problematic technology, eventlet, without especially gaining new features. This migration, although without immediate direct functional value, represente for us a considerable investment in time and resources. But such context is not far different from the Python 2/Python 3 migration. Remember that era. The core Python maintainers found themselves in a difficult situation: continue to maintain their code on Python 2, which had become insecure and unsupported, or go through the effort of completely migrating their codebase and toolchain to Python 3. All the community had to make a choice. Third-party libraries, frameworks were also on the edge case, to be upgraded or replaced with Python 3 compatible equivalents. At the time of the decision, these efforts appeared to represent a "non-value-added burden". These efforts did not bring any new functionality, did not respond to any new customer need, and even slowed down other more innovative developments. The Python maintainers could have chosen the easy way. They could have chosen to continue to support Python 2 again for years, they could have chosen to live in statu-quo, but that's not what happened [1]. They chose to fight the problem and they chose to engage strong efforts in this migration. They ensured the security and the perenniality of the Python programming language. Once the migration was completed, the benefits quickly became apparent. The innovation cadence increased. The community united behind one uniq major version. Better performances. More innovations. And last but not the least, a simpler life for the distros maintainers and for end-users, and so a reinforced confidence in Python by our industry. Python is now the number 1 technology, leaving Java and C in the dust [2]. This is the lesson we must learn. Necessary actions without immediate direct functional value can ultimately hide treasures of innovation. Engagement and dedication is the key to a sustainable future. It guarantees the long-term viability of a project, reduces risks, and opens the way to faster and more peaceful innovation. Today, I can tell you that I'm proud that our community engaged the same way. I'm proud to see all the milestones we reached during this exciting year. I'm proud to observe so much engagement coming from every side, from everybody, and from all our companies. That's a strong signal of trust from our industry. Our companies support this initiative, our companies support open source, our companies invest in the future of OpenStack. On behalf of this community initiative, let me thank you very much for all your efforts and engagements in 2024. Let me wish you happy holidays and enjoy your loved ones. [1] https://devguide.python.org/versions/ [2] https://www.tiobe.com/tiobe-index/ -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[neutron] Hold the rechecks
by Yatin Karel 18 Dec '24

18 Dec '24

With the latest Ubuntu jammy kernel released, jobs running with openvswitch/linux bridge are impacted https://bugs.launchpad.net/neutron/+bug/2091990. Almost all branches are impacted, so hold any recheck until it's fixed. Other projects should also be seeing the issue when running such jobs. -- Thanks and Regards Yatin Karel

1 1

[qa] Team Organization Updates
by Martin Kopec 18 Dec '24

18 Dec '24

Hello everyone, I'm changing my work focus in the new year, therefore, I will have reduced bandwidth for QA and PTL duties for the rest of the Epoxy cycle. As a result, I'd like to announce a few changes: 1. Ghanshyam will take over moderating office hours starting in the new year. To better accommodate his timezone, the office hours will be held *bi-weekly on Wednesdays 18:00 UTC (even weeks).* The first office hour in the new year will be on *Wednesday January 8th*. 2. Ghanshyam will also act as PTL proxy for any emergencies, e.g. patches that require an approval asap. However, feel free to add me as a reviewer to patches that need a final review. 3. Luigi Toscano volunteered to help us with reviews, mainly with Tempest-related ones as he is a long-term contributor to cinder-tempest-plugin and several other projects - https://review.opendev.org/q/owner:ltoscano@redhat.com Thanks, -- Martin Kopec Red Hat EMEA IM: kopecmartin

1 0

[tc][all] No weekly meetings over the holidays (2024-12-24 and 2024-12-31)
by Goutham Pacha Ravi 18 Dec '24

18 Dec '24

Hello Stackers, As we approach the end-of-the-year holidays, please note that the OpenStack Technical Committee weekly meetings [1] previously scheduled for 2024-12-24 and 2024-12-31 are cancelled. We will resume our weekly meetings on 2025-01-07, and I will share the agenda closer to that date. Wishing everyone a happy and restful holiday season! Thank you, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[openstack-ansible] Weekly meetings on Dec 24 and Dec 31 cancelled
by Dmitriy Rabotyagov 18 Dec '24

18 Dec '24

Hi, Last week we decided to cancel the weekly meetings on December 24 and December 31 2024 due to Christmas holidays as many people will not be around for the meeting. This means that the next meeting will happen only on January 7 2025. With that being said, I want to congratulate the community with the upcoming Christmas and New Year and wish to have plenty of fun during holidays! And thank for all the hard work that was put in during 2024.

1 0

[neutron] End of year meeting schedule
by Brian Haley 17 Dec '24

17 Dec '24

Hello Neutrinos: Just wanted to make sure everyone was aware of our meeting schedule for the rest of the year. As discussed in our meeting Tuesday, next week will be the last week of meetings for the year, specifically: December 13th - Drivers meeting cancelled December 16th - Neutron CI meeting December 17th - Neutron meeting December 20th - Drivers meeting cancelled December 23rd - Neutron CI meeting (?) will leave up to Yatin Meetings the week of December 23rd and December 30th are cancelled, starting up again the week of January 6th. As usual, reach out on IRC/gerrit/email with questions for me or the core team, but assume a potentially slow turn-around. I will be offline from December 19th to January 8th, checking-in occasionally to make sure there are no fires, Miguel has offered to lead the team meeting on January 7th in my absence (thanks Miguel!). -Brian, Neutron PTL

2 2

Vault cannot authorize approle
by Andy Speagle 17 Dec '24

17 Dec '24

Hey Team, I'm working with charmed openstack and it looks like I let my certs expire stupidly... and now I'm seeing this from vault: Vault cannot authorize approle Further info from the logs indicate this: unit-vault-0: 08:42:04 WARNING unit.vault/0.juju-log InternalServerError: Unable to authorize approle. This may indicate failure to communicate with the database unit-vault-0: 08:42:04 ERROR unit.vault/0.juju-log Traceback (most recent call last): File "/var/lib/juju/agents/unit-vault- 0/charm/reactive/vault_handlers.py", line 896, in client_approle_authorized vault.get_local_client() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 339, in wrapped_f return self(f, *args, **kw) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 430, in __call__ do = self.iter(retry_state=retry_state) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 378, in iter raise retry_exc.reraise() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 206, in reraise raise self.last_attempt.result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result return self.__get_result() File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/tenacity/__init__.py", line 433, in __call__ result = fn(*args, **kwargs) File "/var/lib/juju/agents/unit-vault-0/charm/lib/charm/vault.py", line 258, in get_local_client client.auth_approle(app_role_id) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/v1/__init__.py", line 2072, in auth_approle return self.auth('/v1/auth/{0}/login'.format(mount_point), json=params, use_token=use_token) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/v1/__init__.py", line 1726, in auth return self._adapter.auth( File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 159, in auth response = self.post(url, **kwargs).json() File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 103, in post return self.request('post', url, **kwargs) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/adapters.py", line 233, in request utils.raise_for_error(response.status_code, text, errors=errors) File "/var/lib/juju/agents/unit-vault-0/.venv/lib/python3.10/site- packages/hvac/utils.py", line 39, in raise_for_error raise exceptions.InternalServerError(message, errors=errors) hvac.exceptions.InternalServerError: internal error The database seems to be up and accessible... so, I gotta believe it's a cert issue... so, I feel like this might be a chicken-egg issue where it can't connect to the DB due to certs... but then can't issue certs because it can't connect to the DB. Any thoughts? ----- Andy Speagle

2 4

[neutron] bug deputy report for week of 2024-12-09+
by Ihar Hrachyshka 17 Dec '24

17 Dec '24

Hi all, Bugs ------- Gate issues ========= 1. fullstack failures in l3ha agent, placeholder: https://bugs.launchpad.net/neutron/+bug/2091351 May be related to https://bugs.launchpad.net/neutron/+bug/2083609 There's a fix for the latter: https://review.opendev.org/c/openstack/neutron/+/937758 2. functional suite OOMing when CPU number is low: https://bugs.launchpad.net/neutron/+bug/2091855 There's a fix to split the suite in serial runs: https://review.opendev.org/c/openstack/neutron/+/937759 The bug alludes that memory consumption could be improved; I'd recommend to limit the scope of this bug to fixing the CI failure and let the memory concerns be addressed elsewhere / by another owner if needed. 3. tempest failure in stable branch in psql job: https://bugs.launchpad.net/neutron/+bug/2091711 No fix; no owner. API / policy ========= All assigned to Slawek; not sure if he really wants to own all of them. 1. no quota on the number of tags breaks api: https://bugs.launchpad.net/neutron/+bug/2091410 DoS when too many tags are created; CVE adjacent. Need quotas for tags? 2. policy mechanism allows to work around some limitations on external connectivity when fwaas is used: https://bugs.launchpad.net/neutron/+bug/2091536 Not sure if it has a security angle or affects anything but fwaas. 3. custom policy rule failing due to missing fields: https://bugs.launchpad.net/neutron/+bug/2091493 Needs owner ========== 1. Unicast leakage on flat network: https://bugs.launchpad.net/neutron/+bug/2091211 The next step would be to set up a 2-compute devenv setup with flat and see if ml2/ovs openvswitch driver can reproduce. Additional tests to functional / fullstack / tempest validating leakage is not happening could be helpful too. Already fixed ========== 1. 500 when backend doesn't support vlan-transparent: https://bugs.launchpad.net/neutron/+bug/2091613 2. pylint change broke ovn agent delete: https://bugs.launchpad.net/neutron/+bug/2091071 New RFEs -------------- 1. use unicast mode for keepalived to avoid vxlan flooding when mcast is off: https://bugs.launchpad.net/neutron/+bug/2091599 Ihar

1 0

[Neutron] DHCP port status inactive
by Vinicius Afonso 17 Dec '24

17 Dec '24

Hello, I am new to Openstack environment and I need help with a Neutron problem. When I create a new network, the DHCP port launches with status inactive. Even though I tried to create a VM, the DHCP generate a port for the machine, However it takes a few minutes to give an error: "Build of instance ed6c8835-f2d4-4730-9638-74ded388361b aborted: Failed to allocate the network(s), not rescheduling." The only alternative solution to overcome this issue is to change the DHCP agent of the network. If I change it, the new DHCP port will have an active status and then I can launch a VM. Nothing conclusive was found in the logs, just the error to bind the port. It doesn't say anything to why the port is inactive, just that the port was created and with status active. Things that I already tried: -> See nova and neutron logs -> Restart neutron-server, neutron-dhcp-agent, neutron-openvswitch-agent -> Change manually the DHCP agent Anyone had the same problem as this one? Any tips to troubleshoot? Any help is welcome! Thanks.

2 1

Dalmatian: Dimensions of instance console
by berndbausch＠gmail.com 16 Dec '24

16 Dec '24

This is a Dalmatian-based Devstack. The instance console's height is too large for my screen. Documentation claims that console dimensions can be configured in a file named _detail_vnc.html [1], but there is no file with this name anywhere in the filesystem. I just submitted a bug [2] but wonder if somebody can help me out right now. How can the console dimensions be configured? Thanks! Bernd Bausch Instructor and tinkerer Here is my bug description: Under Frequently Asked Questions (https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#fre quently-asked-questions), the answer to Q: How do I adjust the dimensions of the VNC window image in the OpenStack dashboard? is incorrect. My Devstack installation contains no file named _detail_vnc.html. I found a file with the interesting name _detail_console.html under /opt/stack/horizon/openstack_dashboard/dashboards/project/instances/template s/instances, but it doesn't contain the answer's code for width and height. [1] https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#freq uently-asked-questions [2] https://docs.openstack.org/nova/2024.2/admin/remote-console-access.html#freq uently-asked-questions

1 0

[publiccloud-sig] Reminder: next meeting December 18th - 0800 UTC
by Felix Kronlage-Dammers 16 Dec '24

16 Dec '24

Hi everyone, This upcoming Wednesday (December 18th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See ya in our pre-christmas special edition of the public cloud SIG :) felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-15)
by Goutham Pacha Ravi 16 Dec '24

16 Dec '24

Hello Stackers, We're 15 weeks away from the 2025.1 ("Epoxy") release date (2024-04-02) [1]. During the last week, the Technical Committee continued discussing the state of "unmaintained" code branches across OpenStack repositories and election officials proposed dates for the combined TC+PTL elections for the next release cycle [2]. We have decided to cancel the weekly meetings previously scheduled for 24th December and 31st December in lieu of the holidays. === Weekly Meeting === The OpenStack Technical Committee met over IRC on 2024-12-10 [3]. We revisited challenges with using CentOS 10 in the community CI system due to lack of AVX/AVX2 CPU flags on current cloud providers. Discussions revolved around finding providers with suitable hardware or reducing reliance on CentOS testing. We then revisited the state of "unmaintained" branches across OpenStack repositories. The oldest "unmaintained" branch tracks the "victoria" release (GA: 2020-10-14). The consensus amongst TC members was to enforce the policy to default old branches to EOL unless explicitly opted-in with required maintenance assurances. To do this, we need tooling to automate the transition, and a way to track and validate volunteer opt-ins for unmaintained branches. Volunteers stepped up to transition branches up until the 2023.1 ("Antelope") release to "end-of-life", beginning with the Victoria release branch [4]. If you're interested in keeping a particular branch open, for any repository, please express your interest directly on the gerrit change. All EOL transition patches will be merged after 1 month of community review. The next meeting of the OpenStack Technical Committee is on 2024-12-17 at 1800 UTC. This meeting will take place in the #openstack-tc channel on OFTC's IRC server. Please find the meeting agenda on the wiki [5]. Please remember that you can propose topics for upcoming TC meetings by directly editing the wiki. If you're unable to do so, please holler at me (IRC nick: gouthamr) or in the TC's IRC channel (#openstack-tc). === Governance Proposals === ==== Merged ==== - Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 ==== Open for Review ==== - Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 - Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 - Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 - Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 - Allow more than 2 weeks for elections | https://review.opendev.org/c/openstack/governance/+/937741 === How to Contact the TC === You can reach the TC in several ways: - Email: send an email with the tag [tc] on this mailing list. - IRC: Ping us using the tc-members keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meeting: The Technical Committee meets every week on Tuesdays at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] 2025.2/"F" elections proposal: https://review.opendev.org/c/openstack/election/+/937408 [3] TC Meeting IRC Log 2024-12-10: https://meetings.opendev.org/meetings/tc/2024/tc.2024-12-10-18.01.log.html [4] Transition unmaintained/victoria to EOL: https://review.opendev.org/c/openstack/releases/+/937515 [5] TC Meeting Agenda, 2024-12-17: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[nova] Next 3 meetings are CANCELLED
by Sylvain Bauza 15 Dec '24

15 Dec '24

Hi folks, Apologies for this email, but I was needing to travel this week so I can't be around for this week's meeting and then I will be on PTO for Dec 24 and 31. Thanks, Sylvain

1 0

Diagnosing Sluggish Instance Boot Performance in OpenStack Cluster
by Abhijit Singh Anand 14 Dec '24

14 Dec '24

Dear All, Problem Statement: Instances remain stuck in the “Block Device Mapping” state for an extended period (~9 minutes) before eventually booting. This issue is observed consistently in one of my OpenStack clusters, while another cluster built on similar hardware and configuration spins up instances much faster. Environment Details: 1. Cluster Configuration: * Deployed using Kolla-Ansible on similar hardware specifications. * Both clusters connect to a common Ceph SSD-backed storage cluster. * Connection: 20 Gbps bonded interface over Cisco Nexus switches. * Same volumes used for images, VMs, backups. * Ceph keyrings and configurations are identical. 2. Performance Comparison: * “Good” cluster: Instances (even with large images >20 GB) boot to a login prompt in ~1-2 minutes. * “Bad” cluster: Instances with the same images and size take ~8-9 minutes. 3. Storage Backend Architecture: * “Good” cluster: Cinder-volume service is co-resident on compute nodes. * “Bad” cluster: Cinder-volume service runs only on controller nodes. 4. Instance Images: * Performance issue occurs with both RAW and QCOW2 images. * Example: A Windows 10 Pro RAW image boots in 1 min 40 sec in the "Good" cluster and takes 8 min 45 sec in the "Bad" cluster. What I’ve Tried So Far: * Configuration Matching: Reviewed and aligned as many configuration parameters as possible between the two clusters. * Resource Monitoring: Verified CPU and memory usage on nodes during instance launches—no significant resource contention observed. * Ceph Performance: Checked Ceph cluster performance; it seems healthy with no latency or throughput issues. * Network Testing: Verified 20 Gbps bond interfaces on both clusters; no observed bottlenecks. Hypothesis: I suspect the cinder-volume architecture could be contributing to the issue. In the "Good" cluster, cinder-volume is co-located with compute nodes, possibly optimizing data transfer paths. In contrast, the "Bad" cluster relies solely on controller-hosted cinder-volume services, which might introduce delays in data availability to compute nodes. Request for Community Input: 1. Does the difference in the placement of the cinder-volume service (co-resident vs. controller-only) align with the observed performance discrepancy? 2. Could there be other subtle configuration or architectural nuances that might explain the delays in the "Bad" cluster? 3. What specific logs or metrics would be most insightful in diagnosing this issue further? * I’ve reviewed Nova and Cinder logs but could dive deeper if pointed in the right direction. 4. Are there recommended best practices for tuning Cinder, Nova, or Ceph for environments with high storage performance requirements? Thank You! I appreciate any insights, suggestions, or pointers you can provide to help narrow down the root cause of this frustrating performance discrepancy. Regards, Abhijit S Anand

2 2

[manila] Epoxy Bugsquash: Dec 17th to 19th
by Carlos Silva 14 Dec '24

14 Dec '24

Hello zorillas and interested stackers, The first bugsquash of this cycle is right around the corner! Our bug backlog has been increasing for a while and we are seeing a trend where bugs are staying in progress for some time. For that reason, we will focus on reviews and following up on changes for the next week's bugsquash. Vida has kindly helped us put a list of bugs together [1]. We will be meeting at 15 UTC on Tuesday using meetpad [2], as agreed and we will use the next week's weekly meeting as a checkpoint for the bugsquash, where we'll discuss the progress of the bugs. [1] https://ethercalc.net/a4lehhy9ilp5 [2] https://meetpad.opendev.org/manila-epoxy-bugsquash I'm looking forward to it! carloss

1 0

[release] Release countdown for week R-15
by Thierry Carrez 13 Dec '24

13 Dec '24

Development Focus ----------------- The Epoxy-2 milestone will happen next month, on January 9th, 2025. 2025.1 "Epoxy"-related specs should now be finalized so that teams can move to implementation ASAP. Some teams observe specific deadlines on the second milestone (mostly spec freezes): please refer to https://releases.openstack.org/epoxy/schedule.html for details. General Information ------------------- Please remember that libraries need to be released at least once per milestone period. At milestone 2, the release team will propose releases for any library that has not been otherwise released since milestone 1. Other non-library deliverables that follow the cycle-with-intermediary release model should have an intermediary release before milestone-2. Those who haven't will be proposed to switch to the cycle-with-rc model, which is more suited to deliverables that are released only once per cycle. At milestone-2 we also freeze the contents of the final release. If you have a new deliverable that should be included in the final release, you should make sure it has a deliverable file in: https://opendev.org/openstack/releases/src/branch/master/deliverables/epoxy You should request a beta release (or intermediary release) for those new deliverables by milestone-2. We understand some may not be quite ready for a full release yet, but if you have something minimally viable to get released it would be good to do a 0.x release to exercise the release tooling for your deliverables. See the MembershipFreeze description for more details: https://releases.openstack.org/epoxy/schedule.html#e-mf Finally, now may be a good time for teams to check on any stable releases that need to be done for your deliverables. If you have bugfixes that have been backported, but no stable release getting those. If you are unsure what is out there committed but not released in the openstack/releases repo, running the command "tools/list_stable_unreleased_changes.sh <cycle_name>" gives a nice report. Upcoming Deadlines & Dates -------------------------- Epoxy-2 Milestone: January 9th, 2025 2025.1 Epoxy final release: April 2nd, 2025 -- Thierry Carrez (ttx)

1 0

[Kayobe] kolla_upper_constraints_file empty and getting openstacksdk version incompatability error
by Ali Hussain 13 Dec '24

13 Dec '24

Hi Maksim and Stackers, Trying out Kayobe, seed and control host are ready but upon executing deploy command " kayobe seed service deploy". The following task failed. Looks like "kolla_upper_constraints_file" is empty. What would be the value for it? TASK [ironic-inspector-rules : Ensure introspection rules exist] ************************************************************************************************************ failed: [seed] (item={'description': 'Set IPMI driver_info if no credentials', 'conditions': [{'field': 'node://driver_info.ipmi_username', 'op': 'is-empty'}, {'field': 'node://driver_info.ipmi_password', 'op': 'is-empty'}], 'actions': [{'action': 'set-attribute', 'path': 'driver_info/ipmi_username', 'value': ''}, {'action': 'set-attribute', 'path': 'driver_info/ipmi_password', 'value': ''}]}) => {"ansible_loop_var": "item", "changed": false, "item": {"actions": [{"action": "set-attribute", "path": "driver_info/ipmi_username", "value": ""}, {"action": "set-attribute", "path": "driver_info/ipmi_password", "value": ""}], "conditions": [{"field": "node://driver_info.ipmi_username", "op": "is-empty"}, {"field": "node://driver_info.ipmi_password", "op": "is-empty"}], "description": "Set IPMI driver_info if no credentials"}, "msg": "Incompatible openstacksdk library found: Version MUST be >=0.36 and <=0.98.999, but 1.0.2 is larger than maximum version 0.98.999."} Any suggestions? Thank you Best,

2 3

[cinder][ceph rbd driver] create volume in EC pool by download, but not clone
by Tony Liu 13 Dec '24

13 Dec '24

Hi, With Antelope, when create a volume in EC pool from image, Cinder downloads the image and upload it to volume, instead of cloning it as create volume in replicated pool. May I know the intention behind it? Why not clone it? Also, in case the intention is to remove dependency, clone and flatten works better. Thanks! Tony

2 7

Fwd: Regarding Trove-Openstack error
by skumari 13 Dec '24

13 Dec '24

---------- Original Message ---------- From: skumari <skumari(a)cdac.in> To: jimmy(a)openstack.org, communitymngr(a)openstack.org Date: December 9, 2024 at 10:03 AM Subject: Regarding Trove-Openstack error Hi Sir, In trove-openstack while iam trying to create a database instance iam getting this error: Polling request timed out. | | fault_date | 2024-12-06T10:45:59 | | fault_details | Traceback (most recent call last): | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 208, in wait_for_task | | | return polling_task.wait() | | | ^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/eventlet/event.py", line 124, in wait | | | result = hub.switch() | | | ^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/eventlet/hubs/hub.py", line 310, in switch | | | return self.greenlet.switch() | | | ^^^^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_service/loopingcall.py", line 154, in _run_loop | | | idle = idle_for_func(result, self._elapsed(watch)) | | | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/oslo_service/loopingcall.py", line 349, in _idle_for | | | raise LoopingCallTimeOut( | | | oslo_service.loopingcall.LoopingCallTimeOut: | | | Looping call timed out after 1808.67 seconds | | | | | | During handling of the above exception, another exception occurred: | | | | | | Traceback (most recent call last): | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/taskmanager/models.py", line 447, in wait_for_instance | | | utils.poll_until(self._service_is_active, | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 224, in poll_until | | | return wait_for_task(task) | | | ^^^^^^^^^^^^^^^^^^^ | | | File "/var/lib/kolla/venv/lib/python3.11/site-packages/trove/common/utils.py", line 210, in wait_for_task | | | raise exception.PollTimeOut | | | trove.common.exception.PollTimeOut: Polling request timed out. I even increased poll request timeout also . can you please help me to solve the error Thankyou Kumari. ------------------------------------------------------------------------------------------------------------ [ C-DAC is on Social-Media too. Kindly follow us at: Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ] This e-mail is for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies and the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email is strictly prohibited and appropriate legal action will be taken. ------------------------------------------------------------------------------------------------------------

4 4

[nova][tc] Removing md5 in password injection
by Thomas Goirand 13 Dec '24

13 Dec '24

Hi, I didn't feel it would be controversial, though it seems removing md5 password injection is still up to debate: https://review.opendev.org/c/openstack/nova/+/935512 Of course, I'd like the TC to agree with me that injecting md5-hashed passwords is, in 2024, to be considered a security problem that should be fixed (and backported) ASAP. BTW, IMO this patch could be using the new feature from oslo_utils.secretutils that Takashi managed to get in: https://review.opendev.org/c/openstack/oslo.utils/+/931899 https://review.opendev.org/c/openstack/oslo.utils/+/935525 These, IMO, should also be backported to earlier oslo.utils releases, so we can fix earlier OpenStack releases in a nicer way. Cheers, Thomas Goirand (zigo)

6 14

[blazar] Next IRC meeting cancelled
by Pierre Riteau 13 Dec '24

13 Dec '24

Hello, Due to the holiday season, the next IRC meeting scheduled for December 26 is cancelled. We will meet again on January 9. I wish you all happy holidays. Cheers, Pierre Riteau (priteau)

1 0

[watcher] Canceling Dec 26th and Jan 2nd weekly meetings
by Douglas Viroel 12 Dec '24

12 Dec '24

Hi all o/ Since most of us will be out during the holidays, we agreed[1] on canceling watcher's weekly meetings[2] on Dec 26th and Jan 2nd. The last meeting of 2024 will be next week, on Dec 19th, and the first one of 2025 will be on Jan 9th. BR, [1] https://meetings.opendev.org/meetings/watcher/2024/watcher.2024-12-12-12.00… -- Douglas Viroel - dviroel

1 0

[cinder][ceph rbd] change volume type
by Tony Liu 12 Dec '24

12 Dec '24

Hi, With Antelope, old volume type is on replicated pool and new volume type is on EC pool. Change volume type doesn't work, volume type remains and no error returns. Is such change supported? I also tried change volume type from one replicated pool to another replicated pool. Doesn't work either. Am I missing anything? Thanks! Tony

2 1

[Openstack-ansible][Rabbitmq] Rabbitmq role is installing latest version of rabbitmq even after setting package version in variable file
by Danish Khan 12 Dec '24

12 Dec '24

Dear All, I have upgraded few of my openstack clusters from Ussuri to Wallaby successfully. But it seems rabbitmq repo is changed and now rabbitmq is getting its packages from: https://dl.cloudsmith.io/public/rabbitmq/rabbitmq-server/deb/ubuntu Everything was working fine until I face below issue while running rabbitmq playbook: TASK [rabbitmq_server : Apply rabbitmq policies] ******************************************************************************************************* failed: [rabbit_mq_container] (item={'name': 'HA', 'pattern': '^(?!(amq\\.)|(.*_fanout_)|(reply_)).*', 'priority': 0, 'tags': 'ha-mode=all'}) => {"ansible_loop_var": "item", "changed": false, "cmd": "/usr/sbin/rabbitmqctl -q -n '' set_policy -p / HA '^(?!(amq\\.)|(.*_fanout_)|(reply_)).*' '{\"ha-mode\": \"all\"}' --priority 0", "item": {"name": "HA", "pattern": "^(?!(amq\\.)|(.*_fanout_)|(reply_)).*", "priority": 0, "tags": "ha-mode=all"}, "msg": "Error:\nValidation failed\n\n[{<<\"ha-mode\">>,<<\"all\">>}] are not recognised policy settings", "rc": 69, "stderr": "Error:\nValidation failed\n\n[{<<\"ha-mode\">>,<<\"all\">>}] are not recognised policy settings\n\n", "stderr_lines": ["Error:", "Validation failed", "", "[{<<\"ha-mode\">>,<<\"all\">>}] are not recognised policy settings", ""], "stdout": "", "stdout_lines": []} It seems ha-mode=all is deprecated in latest version of Rabbitmq (4.0.4). I tried to install 3.8 or some other compatible version of rabbitmq but anyway the latest version of rabbitmq is getting installed. I have updated below entries of file /etc/ansible/roles/rabbitmq_server/vars/debian.yml : _rabbitmq_package_version: "3.8.14-1" and _rabbitmq_erlang_version_spec: "{{ (rabbitmq_install_method == 'external_repo') | ternary('1:23.3.4.15-1', '1:22.*') }}" Can someone please help me with this? I guess either I need to find a away to install specific version of rabbitmq to fox rabbitmq policy issue or I need to update parameters of command while applying policy. Regards, Danish Khan

2 2

[KOLLA-Ansible] default values for cpu_allocation_ratio
by Oliver Weinmann 12 Dec '24

12 Dec '24

Hi all, what are the default values for: cpu_allocation_ratio ram_allocation_ratio disk_allocation_ratio when using kolla-ansible? I can't seem to find any clear statement about it. Currently I assume they are 1.0 for each. Cheers, Oliver

2 2

[magnum] Cluster type (vm, ubuntu, kubernetes) not supported
by Satish Patel 12 Dec '24

12 Dec '24

Folks, I am trying to create a template for magnum but getting the following error. I am running the 2024.1 release. Anything changed or missing here? openstack image create ${IMAGE_NAME} --disk-format=qcow2 --container-format=bare --property os_distro=${OS_DISTRO} --file=${IMAGE_NAME}.qcow2; \ openstack coe cluster template create \ --image $(openstack image show ${IMAGE_NAME} -c id -f value) \ --external-network public \ --dns-nameserver 8.8.8.8 \ --master-lb-enabled \ --master-flavor gen.c4-m8-d30 \ --flavor gen.c4-m8-d30 \ --network-driver calico \ --docker-storage-driver overlay2 \ --coe kubernetes \ --label kube_tag=${version} \ k8s-${version}; Cluster type (vm, ubuntu, kubernetes) not supported (HTTP 400) (Request-ID: req-7c7f27b2-d67c-4f5b-aacf-01754b35bba8)

4 7

Re: [nova][security-sig] Communicating SSH host keys on
by Jeremy Stanley 12 Dec '24

12 Dec '24

first boot (was: Removing md5 in password injection) Reply-To: In-Reply-To: <8a163fa2-f071-4744-aae3-2182c048351c(a)redhat.com> On 2024-12-11 16:30:40 +0000 (+0000), Sean Mooney wrote: [...] > > Tangential, sorry, but could this sort of mechanism also be used to > > provide an out-of-band verification of SSH hostkeys generated on > > first boot for *nix type guests? We used to splat them to the > > primary terminal and then try to scrape them from the console log, > > but having a structured API response would be a lot cleaner (or did > > something similar already get added and I missed it?). [...] > i see the merrit in supprot some kind of reporting for limited > trusted boot type use cases such as providing the ssh host key > fingerprint so you can validate that the vm is the one you expect > it to be. > > im not sure how i would feel about allowing arbiarty metadata to > be store this way. > > is there something beyond the host key that would be useful for > you to record on boot? Nothing else comes to mind. Basically it's the desire to avoid ToFU style finger-crossing that the SSH host key you get on initially connecting to a newly booted server instance is really coming from that instance and not a MitM or honeypot. It's possible in most providers to create an authenticated backchannel by abusing the console log, as I described, but that has its own challenges. I suppose there might be alternative management protocols similar to SSH which rely on asymmetric key algorithms to authenticate hosts, but I'm not personally familiar with any (none with the near ubiquity of SSH at any rate). > perhaps we can try and capture this is a spec or separate thread > and work out what an mvp would look like. there would basically be > two part, one extednign the metadata api to support recording the > host key fingerprint somewhere, and then making that discoverable > via the main api perhaps in server show. That would make sense, and it would probably also be a good idea to try to involve e.g. a cloud-init maintainer in any such design discussion, as this would need client-side integration eventually anyway in whatever's going to communicate the key back to nova. > is we allowed read write access to all instance metadata we would > would not actually need to modify the main api as you can already > list metadata on an instance but im worried that could be abused > so i don't know if that is a good idea. Yes, that sounds like a security nightmare. > the nova metadata api is not intended a a high performance key > value store so i dont necessarily think it would be a good idea to > extned this to a generic crud api but we certainly could provide > limited support for targeted uscease liek the one you raise. Agreed, I suppose the field could be given a more generic name in case certain kinds of guests wanted to store functionally similar things in it, but as I said I don't know what else that would end up being anyway. -- Jeremy Stanley

1 1

Some questions about ironic
by Johnny Hao 12 Dec '24

12 Dec '24

Hi Buddies, I am new to ironic, after some study, I have several questions about it, could anyone help to answer? Thanks a lot. 1. According to the workflow chart below, what's the communication method between nova-compute and Ironic API? HTTP API call or RPC call through mq like rabbitmq? So are the communications between Ironic Conductor and other components of Neutron, Glance and Cinder/Swift etc? 2. Where do Ironic services run (Ironic API/Conductor/Drivers and DB)? In the controller node? Or a separate node is also supported? ** -- Best regards QingFeng Hao

2 1

[nova] riscv64 emulation support
by James Page 12 Dec '24

12 Dec '24

Hi Nova Team We've had some initial support for enabling RISCV64 support in Nova in a review for sometime which I refreshed this week: https://review.opendev.org/c/openstack/nova/+/889137 Sean suggested that this really needs a blueprint and discussion at the next Nova meeting (which I can't make) so proposing discussion via ML instead: https://blueprints.launchpad.net/nova/+spec/riscv64-emulation-support I'm also going to look at the nova-emulation job to see if that can be extended to cover RISCV64 as well - there are a few nuances with dependent packages (opensbi) in Ubuntu 24.04 but I think we can come up with something workable for testing. Review and comments are appreciated. Thanks James

2 1

OpenStack metrics and logs
by Timothé Bauge 11 Dec '24

11 Dec '24

Hello Stackers, I would like to know what are you all using to monitor and supervise your Openstack clusters ? We are in the process of setting up our own private cloud based on RedHat OpenStack Platform 17.1, we choose not to go with the RedHat Service Telemetry Platform, and were strongly advise against Ceilometer and aodh. At the moment, we built our own stack based on Prometheus (with the plenty of exporters) for the metrics, Graylog + OpenSearch for the logs, and Grafana for the visualisation. For now, we are only looking to retrieve basic information in order to know if a dysfunction occurs, but the end goal might be to go as far as to be able to count the cpu/mem/disk usage per hour per vm per project, etc. So, what are you all using and how did you implement it ? Best regards, [cid:659cb34d-63ad-4c5a-94e4-734ef252c20b] Timothé BAUGÉ timothe.bauge(a)covage.com<mailto:pr%C3%A9nom.nom@covage.com> COVAGE I Wholesale B2B du Groupe Altitude

3 3

Re: [openstack-helm] [Cinder] Cinder-Scheduler oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
by daniel890723＠gmail.com 11 Dec '24

11 Dec '24

Hello Daniel here I have been test and trying to solve the oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start Error on openstack-helm deployed Cinder After the test with kolla-ansible i can confirm the problem is from kubernetes So i perform the test with all Cinder Service with Priviliged true setting in Value.yaml: pod: security_context: volume_usage_audit: pod: runAsUser: 42424 container: cinder_volume_usage_audit: readOnlyRootFilesystem: true privileged: true allowPrivilegeEscalation: true cinder_api: pod: runAsUser: 42424 container: ceph_coordination_volume_perms: runAsUser: 0 readOnlyRootFilesystem: true cinder_api: readOnlyRootFilesystem: true allowPrivilegeEscalation: true privileged: true cinder_scheduler: pod: runAsUser: 42424 container: ceph_coordination_volume_perms: runAsUser: 0 readOnlyRootFilesystem: true cinder_scheduler: capabilities: add: - SYS_ADMIN privileged: true readOnlyRootFilesystem: true allowPrivilegeEscalation: true Also Glance-api container too: pod: security_context: glance: pod: runAsUser: 42424 container: glance_perms: readOnlyRootFilesystem: true runAsUser: 0 ceph_keyring_placement: readOnlyRootFilesystem: true runAsUser: 0 glance_api: privileged: true readOnlyRootFilesystem: true allowPrivilegeEscalation: true nginx: readOnlyRootFilesystem: false runAsUser: 0 With Cinder and Glance container set as priviliged container the problem is gone and i can create volume with image normally So it is confirmed that the ERROR is because the pod permission Also my System Setup: | OS | Ubuntu:22.04 | | Kubernetes | 1.30.4 | | Openstack-helm | 2024.1 | P.S. i haven't test with a complete clean enviroment with only Ceph and Openstack component installed Daniel Lu

3 3

[nova] Specs review day is TODAY and Spec soft freeze is TOMORROW
by Sylvain Bauza 11 Dec '24

11 Dec '24

Hi folks, As we discussed on yesterday's nova meeting, today is our second specs review day and tomorrow (Dec 12th) will be the last day for providing a new spec to be reviewed. Please see our schedule here: https://releases.openstack.org/epoxy/schedule.html Thanks, -Sylvain

1 0

Open Stack APIs
by foram.thakkar＠crestdata.ai 10 Dec '24

10 Dec '24

Hello, We want to fetch all the servers present in all the projects a user has access to. Currently we have to provide any one project name in the body of the authentication call. Is there a way we can fetch all the projects without providing any project name?

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-16)
by Goutham Pacha Ravi 10 Dec '24

10 Dec '24

Hello Stackers, We're 16 weeks away from the 2025.1 ("Epoxy") release date (2024-04-02) [1]. OpenStack project teams are planning bug squashes and review jams in the next couple of weeks before contribution activity slows due to end-of-the-year holidays around the world. The OpenStack Technical Committee did not merge any new governance changes this week; however, a few proposals are iterating with community and TC review. === Weekly Meeting === The OpenStack Technical Committee held its monthly video conference last week, which was simultaneously recorded via text on IRC [2][3]. The discussion tracked several TC initiatives for this release cycle [4]. Most notably, the recent migration of base integration jobs to Ubuntu 24.04 LTS and Python 3.12 has caused CI jobs to break for several code repos [5]. I’d like to reiterate that project teams must resolve these issues or temporarily pin the nodeset of their test jobs to "ubuntu-jammy" to buy some time for fixes. We then checked on the status of "unmaintained" branches. The release team wanted the TC to solidify processes for two specific issues: nominating "unmaintained" liaisons and establishing guidelines for retiring these branches. The lack of such processes has resulted in the proliferation of "unmaintained" code branches across the community and failure to implement the TC resolution that created these branches [6]. The oldest unmaintained branch is "unmaintained/victoria", and it’s unclear if anyone benefits from keeping it around. Discussion on cleanup has been delayed due to uncertainty about what to do [7]. Meanwhile, branches like Yoga may still have high usage and active contributors, emphasizing the need for a clear process to decide their fate. The technical committee may consider advising the release team to eliminate branches with low usage, but this doesn’t solve the problem of finding volunteers to maintain high-usage branches. The idea is that projects can opt to maintain more branches if there is sufficient user interest and support, but there is currently no process for this, nor is it clear how to recruit volunteers. We agreed to continue this discussion at the next meeting. However, in the meantime, there were no objections to EOL-ing the Victoria, Wallaby, and Xena branches. Clark Boylan (clarkb) noted an issue with building CentOS Stream 10 images using Disk Image Builder, as they cannot boot on several clouds due to the x86_64 v3 CPU instruction set requirement. Only a few provider clouds, including OpenMetal and Rackspace Flex, meet the CPU requirements to boot CentOS 10. This limitation may affect options for regular test nodes, and project teams are advised to take notice. Until all Zuul/Nodepool cloud providers meet the CPU requirements, teams may need to adjust their testing strategies. Jeremy Stanley (fungi) reminded us that nominations for individual member representatives for the OpenInfra Foundation Board of Directors are still open [8]. Ten nominations from members are required to make it onto the ballot. The OpenStack TC's next weekly meeting is today, 2024-12-10. This will be an IRC meeting hosted on #openstack-tc at 1800 UTC. Please find the agenda on the meeting wiki [9]. I look forward to seeing you there. === Governance Proposals === ==== Open for Review ==== Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC Meeting Video Recording, 2024-12-03: https://youtu.be/KlYbFxjhACs [3] TC Meeting IRC Log 2024-12-03: https://meetings.opendev.org/meetings/tc/2024/tc.2024-12-03-18.00.log.html [4] TC Tracker for 2025.1: https://etherpad.opendev.org/p/tc-2025.1-tracker [5] Runtime update to Ubuntu 24.04/Python3.12: https://etherpad.opendev.org/p/migrate-to-noble [6] TC Resolution to create "unmaintained" code branches: https://governance.openstack.org/tc/resolutions/20230724-unmaintained-branc… [7] Transition "victoria" code branch to EOL: https://review.opendev.org/c/openstack/releases/+/935373 [8] Nominations for OpenInfra Individual Board Member Elections: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/th… [9] TC Meeting Agenda, 2024-12-10: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

Kolla-Ansible: Why is handling kolla_*_fqdn in /etc/hosts out of scope?
by Matthias Tinnemeier 10 Dec '24

10 Dec '24

Dear Community, According to the documentation (https://docs.openstack.org/kolla-ansible/2024.2/admin/advanced-configuratio…) the provisioning of kolla_*_fqdn Name Resolution via /etc/hosts must be managed outside of the scope of Kolla. The question, therefore, is why :-) Given the existence of "customize_etc_hosts", there should be at least some logic to implement this behaviour in an automatic way. Is there a particular reason why this is not being implemented via Kolla? Alternatively, would it be worth proposing a patch for this? With kind regards Matthias

1 0

a suggestion: use LDAP to manage all openstack components configuration
by Shi Wei 10 Dec '24

10 Dec '24

Hi, everyone, I have a suggestion. we can use LDAP service to mange all openstak components and nodes(including physical nodes and vm nodes).  Because ldap is easy to read and ldap can support DNS and DHCP service. if a node get ip address by DHCP server with ldap. You and get the Mac address information from the LDAP server. and if all components are communicated by domain name. all records are on the LDAP server. if the one node will change IP address, you can just modify on the ldap server.  The ldap server can easy to backup and extend. use ldap command to export as a ldf file.   you can easy to backup before update configuration. if a new component will join, you can extend the schema by .sch file.  

2 1

[ops][largescale-sig] Join the Large Scale OpenStack virtual meetup with Rackspace !
by Thierry Carrez 09 Dec '24

09 Dec '24

Hi everyone, This cycle the Large Scale SIG is organizing virtual meetups, where operators of large-scale OpenStack deployments come to discuss their experiences, challenges and solutions with fellow operators. On Thursday, December 5, at 16 UTC (10am CT, 5pm Paris), we'll have our first one. The special guest for this meetup will be Kevin Carter, operator of the Rackspace Cloud. It will last one hour and will happen through a video meeting. To join us, please fill this form and we'll send you the link for the event the day before! https://forms.gle/92o1DTavHwivN8tn6 Hoping to see you there. -- Thierry Carrez (ttx)

4 6

OpenStack API Ports
by darsh.mehta＠crestdata.ai 09 Dec '24

09 Dec '24

I’m working with OpenStack APIs and noticed that the following endpoints use specific ports: Port 5000: - /auth/tokens?nocatalog - /auth/projects Port 8774: - /servers/detail - /flavors/detail - /images/detail - /os-hypervisors/detail I would like to know: Are these default port assignments (5000 for authentication and 8774 for compute services) configurable? Under what circumstances might these ports change? For example, would they differ in custom deployments or under certain OpenStack configurations?

3 2

[upgrades][openstack-ansible][upgrade from Wallaby to Antelope or Caracal]
by mmunther＠creativeadvtech.com 09 Dec '24

09 Dec '24

Hi Everyone, Kindly we would like to upgrade from Wallaby to something more recent such as Antelope or Caracal. We also have Ubuntu 20.04 and would like to upgrade the OS to Ubuntu 22.04. What would be the correct upgrade path for such a scenario using openstack ansible. Thanks Best Regards Monis

3 2

[neutron] Bug deputy report for week of December 2nd
by Miguel Lavalle 09 Dec '24

09 Dec '24

I was the Neutron bug deputy last week. Following is a brief summary of the reported bugs: Critical bugs ----------------- https://bugs.launchpad.net/neutron/+bug/2091021 Fullstack ha failover tests from neutron.tests.fullstack.test_l3_agent.TestHAL3Agent failing due to "destination ip 42.0.0.15 is replying to ping from namespace test-3dc5c664-ddba-4c2e-a9a9-48bbda478506, but it shouldn't". Assignee Slawek Kaplonski. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/937097 High bugs -------------- https://bugs.launchpad.net/neutron/+bug/2091071 ovn-agent: exception when deleing agent. Assignee Kien Nguyen. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/936988 Medium bugs ------------------ https://bugs.launchpad.net/neutron/+bug/2090825 AAP arp entry has incorrect mac address. Assignee Vasyl Saienko. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/936807 https://bugs.launchpad.net/neutron/+bug/2090913 [ovn]Allow multiple IPv6 ports on router from same network on ovn_l3. Assigned to Liu Xie. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/936931 Low bugs ------------- https://bugs.launchpad.net/neutron/+bug/2091106 OVN information in networking-ovn, Assignee: Brian Haley. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/937296 Under investigation -------------------------- https://bugs.launchpad.net/neutron/+bug/2090921 The OVN database is missing 6 ACL for security group default after each redeployment. Assignee Brian Haley https://bugs.launchpad.net/neutron/+bug/2091211 network leakage for flat network. Marked as incomplete for now pending more information from submitter

1 0

about accelerating database initialization
by Jiatong Shen 08 Dec '24

08 Dec '24

Hello all, I am trying to find out if there are ways to accelerate the database initialization process. We are using openstack-helm for deployment victoria OpenStack cluster and internally openstack-helm uses nova-manage db sync command to bootstrap the database. It works but a little bit slower than expected. Right now, I am measuring this bootstrap command could took around 2 minutes to complete. I'd like to get some advices on accelerating the process as well as what is the reasonable time to have. Thank you very much for your insights. -- Best Regards, Jiatong Shen

2 1

[HORIZON] configure for multiple labs at once
by Ishan Shanware (ishanwar) 08 Dec '24

08 Dec '24

Hey Stackers, I wanted to know if it’s possible to view the horizon dashboard for multiple openstack labs at once from the same UI ? thanks Ishan

3 2

[Kolla] Transition Yoga and Zed to EOL
by Jakub Darmach 05 Dec '24

05 Dec '24

Hello, As agreed during the latest IRC meeting, Kolla Yoga and Zed deliverables (Kolla, Kolla-Ansible, Kayobe) are going EOL. [1] Antelope remains as unmaintained, while Bobcat, Caracal and Dalmatian are our current stable releases. Yoga and Zed branches were unmaintained for a while, Kolla community does not have resources to support those branches actively. All changes to those branches have been abandoned. [1] https://review.opendev.org/c/openstack/releases/+/937101 Best regards, *Jakub Darmach*

1 0

Open Stack APIs
by darsh.mehta＠crestdata.ai 05 Dec '24

05 Dec '24

Hello, We want to create an integration with OpenStack for which we require the REST APIs to connect to OpenStack. However, OpenStack provides 2 offerings i.e. Cloud and On-Prem. We are referring to the OpenStack API documentation available over here https://docs.openstack.org/api-quick-start/api-quick-start.html#. But it does not specify anywhere whether the APIs are for cloud or on-prem. Are the APIs same for both or do we have separate documentation for each of these? Thanks in advance!

4 3

[Neutron] WSGI migration issues and eventlet requirements bump
by Rodolfo Alonso Hernandez 05 Dec '24

05 Dec '24

Hello Neutrinos: There is a patch [1] that is trying to bump the eventlet upper version. However, the tempest job is failing because of Neutron. There is a series of patches [2][3][4][5] that are trying to address the problems we are having with the migration to WSGI. I would ask you to review them soon in order to unblock the requirements patch. On top of this series there is a testing patch with several tempest jobs running. We still have some problems but they are limited. Thanks! [1]https://review.opendev.org/c/openstack/requirements/+/933257 [2]https://review.opendev.org/c/openstack/neutron/+/936428 [3]https://review.opendev.org/c/openstack/neutron/+/936813 [4]https://review.opendev.org/c/openstack/neutron/+/936838 [5]https://review.opendev.org/c/openstack/neutron/+/934409

1 0

OpenStack is Now on Bluesky
by Helena Spease 05 Dec '24

05 Dec '24

Hello Everyone, We’re thrilled to announce that OpenStack and the OpenInfra Foundation now have a presence on Bluesky! This is part of our continued effort to connect with our community and share updates in new and engaging ways. Here’s where you can now find us: - OpenStack on Bluesky: https://bsky.app/profile/openstack.org - OpenInfra Foundation on Bluesky: https://bsky.app/profile/openinfra.org Don’t forget, that OpenStack is already active on these platforms: - Mastodon: https://social.openinfra.dev/@OpenStack - Twitter: https://x.com/OpenStack - LinkedIn: https://www.linkedin.com/company/openstack/ And, the OpenInfra Foundation is already active on these platforms: - Mastodon: https://social.openinfra.dev/@OpenInfra - Twitter: https://x.com/openinfradev - Facebook: https://www.facebook.com/openinfradev - LinkedIn: https://www.linkedin.com/company/openinfra-foundation Join the conversation and engage with the OpenInfra community wherever you are most active! Cheers, Helena helena(a)openinfra.dev Community Programs Coordinator The OpenInfra Foundation <https://openinfra.dev/>

1 0

OSA adding compute without deploy host
by Derek O keeffe 05 Dec '24

05 Dec '24

Hi all, I have a controller and 2 compute setup that was deployed some time ago with OSA. It's been running fine since it was deployed but we now have the need for more compute and I was looking at the deploy host to add a compute host to the cluster. Unfortunately it seems the deploy host was repurposed and I have lost all the config. So two questions, 1) Is it possible to manually add a compute to the cluster (I fairly confident this is a yes) and 2) if so is there any documentation or how to that someone could point me to/share? any help would be greatly appreciated. Regards,Derek

2 2

[kolla-ansible][neutron][openvswitch]
by Adam Tomas 04 Dec '24

04 Dec '24

Hi everyone. I have strange problem with neutron ml2 plugin. I have a little strange setup: controllers in networkA/subnetA computesA in networkA/subnetA computesB in networkB/subnetB (separate L2, controllers can be connected via routing) Because computesB are on different LAN I had to put on two computes in that lan neutron-dhcp-agents and add them to networks served for instances in locationB. In addition aggregates are created so, that if an instance has network from locationB then it has to start on computes in locationB. computes and controllers have one flat network (vlan 21) ans three vlan networks (221, 51 and 251),, vlans 21 and 221 are in locationA, 51 and 251 in locationB, the ml2_plugin is configured like below: [ml2_type_vlan] network_vlan_ranges = physnet1:221:221,physnet2:51:51,physnet3:251:251 bond0 is the neutron_external_interface and has vlan 21 as native and 221,51 and 251 as tagged vlans. Openstack networks are mapped to corresponding physnets. dhcp_agents in location A are working without any problems, vm's are started without any problem in vlans 21 and 221 with addresses from all subnets from networks in these vlans. In location B when I add dhcp_agent the port is doen (namespaces qdhcpxxxxxxxxx are created on computesB and are replying to their own address, but not the gateway or any other address), port status of dhcp_agent is DOWN, and in logs I have: Device Port(admin_state_up=True,allowed_address_pairs=[],binding_levels=[],bindings=[PortBinding],created_at=2024-12-01T17:52:25Z,data_plane_status=None,description='',device_id='dhcpfd2f8da2-397b-5456-a75c-f9c5bbccb8b1-81b9ac42-6267-48e2-8352-0ae5c09d85fa',device_owner='network:dhcp',device_profile=<?>,dhcp_options=[],distributed_bindings=[],dns=None,fixed_ips=[IPAllocation,IPAllocation],hints=None,id=08c60b9d-ea8f-4a2d-a3b7-5f787d15f45d,mac_address=fa:16:3e:37:27:0c,name='',network_id=81b9ac42-6267-48e2-8352-0ae5c09d85fa,numa_affinity_policy=<?>,project_id='c0e2f3d69c064ae5b5596fd150d550d7',qos_network_policy_id=None,qos_policy_id=None,revision_number=2,security=PortSecurity(08c60b9d-ea8f-4a2d-a3b7-5f787d15f45d),security_group_ids=set([]),status='DOWN',updated_at=2024-12-01T17:52:25Z) is not bound. Device 08c60b9d\-ea8f\-4a2d\-a3b7\-5f787d15f45d not defined on plugin or binding failed Port 08c60b9d-ea8f-4a2d-a3b7-5f787d15f45d cannot update to ACTIVE because it failed. _port_provisioned /var/lib/kolla/venv/lib/python3.10/site-packages/neutron/plugins/ml2/plugin.py:357 Why the same configuration works in LocationA? In location B the same situation is with all ports - even if I create a port "by hand". The error message (in debug mode!) says not much - Device 08c60b9d\-ea8f\-4a2d\-a3b7\-5f787d15f45d not defined on plugin or binding failed, how can I find what is wrong? I would appreciate any help. Best regards, Adam Tomas

1 1

[kolla-ansible][nova] Refresh cell cache in nova scheduler
by Satish Patel 04 Dec '24

04 Dec '24

Folks, What is wrong here, I do have the following hostname resolvable from all the nodes even if they are in the /etc/hosts file, Why is the following task saying UNREACHABLE and from where its trying to reach ? PLAY [Refresh nova scheduler cell cache] ***************************************************************************************************************************************************** TASK [nova : Refresh cell cache in nova scheduler] ******************************************************************************************************************************************* fatal: [os2-bos-ctrl01]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl01\". Make sure this host can be reached over ssh: ", "unreachable": true} fatal: [os2-bos-ctrl02]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl02\". Make sure this host can be reached over ssh: ", "unreachable": true} fatal: [os2-bos-ctrl03]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl03\". Make sure this host can be reached over ssh: ", "unreachable": true}

4 5

[octavia] [lbaas] [ovs] Is octavia sr-iov supporting tunnels?
by yardalgedal＠gmail.com 04 Dec '24

04 Dec '24

Hi, Openstack Team, Is new octavia sr-iov feature supporting vxlan tunnels like in ovs or it can work only with provider networks (vlan)?

3 2

[kayobe] Running seed on ansible control host
by Olaf Trygve Berglihn 04 Dec '24

04 Dec '24

Dear list, Can someone explain how to configure kayobe to set up the seed hypervisor and seed vm on the ansible control host (where you run 'kayobe seed hypervisor host configure' and 'kayobe seed vm provision')? Thanks! -Olaf

2 5

[masakari] failures with new version of Eventlet
by Herve Beraud 04 Dec '24

04 Dec '24

Hey Masakari maintainers, Masakari is failing with new the version of Eventlet https://review.opendev.org/c/openstack/requirements/+/933257 This job failure indicates that Masakari will surely stop working correctly once this version bump will be merged, and we need that requirements bump to support newer versions of Python in OpenStack, so... Masakari is a non-voting job, so even if this job is failing, the requirement bump will proceed, but that would be better if every project is compatible and ready to go. If you need help do not hesitate to: - join us on #openstack-eventlet-removal; - join us at our next meeting (Dec 17) https://wiki.openstack.org/wiki/Eventlet-removal#Recurring_Events. Thanks for your hard work. -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[horizon] Upstream weekly meeting cancelled Dec 4
by Tatiana Ovchinnikova 04 Dec '24

04 Dec '24

Hi team, Since half of our team is on PTO tomorrow, I am cancelling the meeting. See you all on December 11! With kind regards, Tatiana

1 0

[OSSA-2024-005] Neutron: Authorization bypassed when setting tags on Neutron networks (CVE-2024-53916)
by Jay Faulkner 04 Dec '24

04 Dec '24

=========================================================================== OSSA-2024-005: Authorization bypassed when setting tags on Neutron networks =========================================================================== :Date: December 03, 2024 :CVE: CVE-2024-53916 Affects ~~~~~~~ - Neutron: >=23.0.0 <23.2.1, >=24.0.0 <24.0.2, >=25.0.0 <25.0.1 Description ~~~~~~~~~~~ Tore Anderson of Redpill Linpro AS discovered that Neutron does not apply the proper policy check for changing network tags. An unprivileged tenant is able to change (add and clear) tags on network objects which do not belong to the tenant, and this action is not being subjected to the proper policy authorization check. Patches ~~~~~~~ -https://review.opendev.org/c/openstack/neutron/+/936849 (2023.2/bobcat) -https://review.opendev.org/c/openstack/neutron/+/936846 (2024.1/caracal) -https://review.opendev.org/c/openstack/neutron/+/936843 (2024.2/dalmatian) -https://review.opendev.org/c/openstack/neutron/+/935883 (2025.1/epoxy) Credits ~~~~~~~ - Tore Anderson from Redpill Linpro AS (C, V, E, -, 2, 0, 2, 4, -, 5, 3, 9, 1, 6) References ~~~~~~~~~~ -https://launchpad.net/bugs/2088986 -http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-53916 -- Jay Faulkner OpenStack VMT

1 0

[publiccloud-sig] Reminder: next meeting December 4th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 04 Dec '24

04 Dec '24

Hi everyone, This Wednesday, tomorrow, (December 4th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you tomorrow! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[nova] Rescuing a volume-backed instance
by Francesco Di Nucci 03 Dec '24

03 Dec '24

Hello, I am using Openstack 2024.1 and I have issues with an instance with a corrupted FS - I wanted to start it in rescue mode, but I'm getting stuck After reading nova#1926601 <https://bugs.launchpad.net/nova/+bug/1926601> , downloaded System Rescue CD and created an image with properties [otherwise it always failed with "Cannot rescue a volume-backed instance (HTTP 400)"] : openstack image create --file systemrescue-11.02-amd64.iso --disk-format iso --container-format bare --property hw_rescue_bus=virtio --property hw_rescue_device=disk systemrescuecd Then tried to rescue the instance: openstack --os-compute-api-version 2.96 server rescue --image systemrescuecd (my-instance-id) Now it fails with "Instance (my-instance-id) cannot be rescued: Driver Error: Image (my-image-id) is unacceptable: Image not in a supported format" How should I proceed? Thanks in advance -- Francesco Di Nucci System Administrator Compute & Networking Service, INFN Naples Email: francesco.dinucci(a)na.infn.it

3 3