November 2024 - openstack-discuss

[all][dev][ops][tc] Bridging the gap between community and contributing orgs
by Jeremy Stanley 08 Dec '25

08 Dec '25

The tl;dr on this is that I'm opening a broad community discussion on how we can, collectively through improved communication, better bring together our established contributors with casual and prospective contributors who struggle to find successful patterns of contribution, so that we might all support each other in order to benefit OpenStack as a whole. Read on for details... OpenStack's TC has requested that foundation staff, when in conversation with representatives of member organizations, encourage participation in the project and collect feedback on any related challenges those organizations encounter in their attempts to do so. In order to better understand the feedback and brainstorm achievable recommendations for the broader community[*], Community Managers on the foundation staff started to work with a small focus group of established contributors with a solid understanding of what successful patterns of contribution look like. It also came to light that established contributors experience many of the same sorts of challenges, even more so if they don't have the luxury of focusing full-time upstream, and so success often comes down to knowing how to effectively navigate those challenges. The community's goal with this exercise is to improve everyone's experience, and your help is needed to do that! The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. Ideas so far to address these gaps include: * Review bandwidth - Ensure contributors and companies understand the importance and benefits of code review - Incentivize and credit meaningful "+1" reviews more - Understand that the volume of changes and limited reviewer bandwidth often means proposals aren't reviewed quickly (or at all), and their proponents may need some additional tenacity and engagement with the team to get their work noticed - Utilize "review dashboards" as a way to find stale changes which have fallen through the cracks * Review strategy and etiquette - Be clear about the meaning of votes on changes, especially negative ones, and set explicit timelines for things like "procedural -2" or WIP blocks when their approval needs to be delayed - Focus on comments and requests that affect whether or not the change can be merged after the fixes are submitted, rather than requesting trivial adjustments to an unsuitable change which has more fundamental problems - Reciprocate with reviews of changes from new reviewers you see leaving insubstantial reviews, demonstrating to them what deeper and more meaningful reviewing looks like * Mentoring newcomers - Established contributors with sufficient bandwidth can help mentoring newcomers, potential core reviewers and new leaders - Mentoring can happen as part of internship programs as well as by helping newcomers determined to become active contributors; both ways should be embraced and utilized What challenges are you facing? And, how would you improve them? Two weeks from now, we'll have a forum session at the OpenInfra Summit Asia in Suwon[**] where our community can refine these possible approaches and integrate others, as well as bring attention to them and promote their application. I've also proposed a similar forum session for OpenInfra Days North America in Indianapolis a month after that, where we can hopefully continue this conversation. The goal for this multi-stage effort is to improve the contribution experience for established participants, casual contributors and newcomers alike. While it's easy to jump to blaming in these situations, it will be far more productive if we can focus on the challenges and experiences that we would each like to remove or improve for ourselves and others in our community. As we find ways together to improve our efficiency, we can reduce the load on all contributors and lower the barriers for people to join and participate. [*] https://etherpad.opendev.org/p/r.2205024e55689bccb82c20c960853cb5 [**] https://2024.openinfraasia.org/a/schedule#view=calendar&title=Bridging%20th… -- Jeremy Stanley

2 16

[openstack helm] [Glance] Glance showing OSError: unable to receive chunked part when uploading image
by daniel890723＠gmail.com 08 Nov '25

08 Nov '25

Hello i'm testing using Openstack-helm to install all component but after the installation complete i can't upload the OS image through both openstack CLI or Horizon it stuck at Queueing and when i checking Glance Api log it shows 2024-11-07 07:31:19.362 7 INFO glance.api.v2.image_data [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. 2024-11-07 07:31:19.400 7 ERROR glance.api.v2.image_data [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Failed to upload image data due to internal error: OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Caught error: unable to receive chunked part: OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi Traceback (most recent call last): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1302, in __call__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1345, in dispatch 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return method(*args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 415, in wrapped 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/api/v2/image_data.py", line 299, in upload 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi with excutils.save_and_reraise_exception(): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.force_reraise() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise self.value 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/api/v2/image_data.py", line 162, in upload 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/notifier.py", line 492, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi with excutils.save_and_reraise_exception(): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.force_reraise() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise self.value 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/notifier.py", line 443, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/quota/__init__.py", line 322, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/location.py", line 596, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/location.py", line 487, in _upload_to_store 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/driver.py", line 294, in add_adapter 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/capabilities.py", line 176, in op_checker 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/_drivers/filesystem.py", line 764, in add 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise errors.get(e.errno, e) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/_drivers/filesystem.py", line 746, in add 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi for buf in utils.chunkreadable(image_file, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/common/utils.py", line 73, in chunkiter 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi chunk = fp.read(chunk_size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 294, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi result = self.data.read(i) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 121, in readfn 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi result = fd.read(*args) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/format_inspector.py", line 956, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi chunk = self._source.read(size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1045, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi data = uwsgi.chunked_read() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi Ceph Cluster Status: NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL FSID ceph /var/lib/rook 3 82m Ready Cluster created successfully HEALTH_OK 5cf8b9bb-13e1-4c31-b47d-9e7d92c3c05d Ceph osd status: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 0.35156 - 360 GiB 3.3 GiB 493 MiB 0 B 2.8 GiB 357 GiB 0.91 1.00 - root default -5 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 964 MiB 119 GiB 0.92 1.01 - host instance-lu-1 2 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 964 MiB 119 GiB 0.92 1.01 195 up osd.2 -3 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 - host instance-lu-2 0 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 195 up osd.0 -7 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 - host instance-lu-3 1 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 197 up osd.1 TOTAL 360 GiB 3.3 GiB 493 MiB 0 B 2.8 GiB 357 GiB 0.91 MIN/MAX VAR: 1.00/1.01 STDDEV: 0.01 now i'm able to create Empty Volume using horizon(can't create volume using default cirros image ) and i can start an Instance without volume and mount the empty volume in(Ceph also can detect when i use that volume) is there something i need to set when starting glance? because i'm following the Openstack-helm installation document to install openstack with helm

5 8

[openstack][kolla-ansible]about group node config overwrite
by Nguyễn Hữu Khôi 30 Oct '25

30 Oct '25

Hello Koller. I want to ask if we can overwrite config for a group of nodes. For example, compute01 >> compute10 will have the same nova configuration compute11 >> compute20 will have the same nova configuration. It will be very nice if we support this way. Thank you. Regards Nguyen Huu Khoi

4 7

[i18n][infra][horizon] Zanata translation job failures after Python 3.8 deprecation
by Ian Y. Choi 09 Jul '25

09 Jul '25

Hi all, During last week's PTG, the I18n SIG identified Zanata translation job failures for Horizon (and likely related plugins). The root cause appears to be Python 3.8 deprecation on Horizon [1]. Failed jobs have been observed on Zuul [2], and translation proposals were made on Gerrit [3]. Since translation sync on Horizon stable branches is functioning correctly, the error message seems to be related to Python 3.8 deprecation: ERROR: Cannot install -r doc/requirements.txt (line 3) due to conflicting package version dependencies. To address this, I'd like to raise two points: 1/ To resolve the issue, would you support updating translation sync jobs with Bionic-based Zuul jobs by either upgrading to Jammy or modifying the Bionic image to support Python >3.8? 2/ At least me from I18n SIG acknowledges that recognizing these translation job errors was delayed. Moving forward, would it be beneficial to implement periodic (e.g., monthly) reporting on translation job successes and failures? Thank you, /Ian [1] https://docs.openstack.org/releasenotes/horizon/2024.2.html#upgrade-notes [2] https://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-up… [3] https://review.opendev.org/q/owner:proposal-bot+project:openstack/horizon+b…

2 3

Problems with the Placement service
by Ron Gage 24 Jun '25

24 Jun '25

5 7

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

[oslo] python-consul2 is unmaintained
by Takashi Kajinami 21 Apr '25

21 Apr '25

Hello, I recently noticed that python-consul2 library[1], which is used by the consul coordination driver in tooz, hasn't been updated for 3+ years. Looking at issues in the project, there is an indication that the library is not compatible with ACL API in consul >= 1.11[2]. There is another fork created[3] and this includes fixes to support new API endpoints. Although the driver in tooz does not directly use the feature, I think it's better to discuss our plan to mitigate risks caused by the unmaintained library. As far as I can think of, there are three options now 1. Deprecate consul driver and remove it 2. Keep using python-consul2 now. At least basic functionality is proven to work by our functional tests (Note that the tests do not cover all configuration patterns) 3. Replace python-consul2 by py-consul Before we start the overall discussions I'd like to ask a few questions to the wider audience, as our inputs. - Is anyone using the consul coordination driver now ? - Does anyone have experience with python-consul2 and/or py-consul ? Note that this discussion affects the http messaging driver[4] currently proposed, because it also requires client library to communicate with Consul. Actually I noticed the situation when I noticed that the requirement check job is failing because the library is not part of global requirements list... Thank you, Takashi Kajinami [1] https://github.com/poppyred/python-consul2 [2] https://github.com/poppyred/python-consul2/issues/42 [3] https://github.com/criteo-forks/py-consul [4] https://review.opendev.org/c/openstack/oslo.messaging/+/912499 -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 3

[octavia] Purging ERROR'ed amphora and load balancers stuck in transitional states
by Paul Browne 11 Apr '25

11 Apr '25

Hello Octavia users, For a while I've been trying to find a systematic way to remove Octavia LBs+Amphora that have become stuck in ERROR or transitional PENDING_* states ; $ openstack loadbalancer amphora list -f value | grep ERROR a2323f14-3aaa-418a-8249-111aaa9c21fe 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR MASTER 10.8.0.242 192.168.3.247 e5b236ba-e7ee-4ed7-9f58-57ce7a408489 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR BACKUP 10.8.0.190 192.168.3.247 6b556f28-93c9-49dd-b6ee-4379288e7957 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR MASTER 10.8.0.39 192.168.3.126 c669db5d-8686-4d5c-9e95-e02030b34301 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR BACKUP 10.8.0.174 192.168.3.126 $ openstack loadbalancer list -f value | grep -vi active 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe k8s-clusterapi-cluster-default-ci-6386871107-kube-upgrade-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.247 PENDING_UPDATE amphora d5e402fe-2c4b-49af-a700-532cb408cee5 k8s-clusterapi-cluster-default-ci-6386871107-latest-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.126 PENDING_DELETE amphora These resources are marked immutable and so cannot be failed over or deleted; $ openstack loadbalancer amphora failover a2323f14-3aaa-418a-8249-111aaa9c21fe Load Balancer 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe is immutable and cannot be updated. (HTTP 409) (Request-ID: req-6e66c4e8-c3d3-4549-a03c-367017c8c8b3) $ openstack loadbalancer failover 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe Invalid state PENDING_UPDATE of loadbalancer resource 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe (HTTP 409) (Request-ID: req-84b44212-e7a8-4101-a16f-18c774c0577e) The backing Nova instances for these Amphora do seem to exist and be in good working order. Is there any API way to purge these out of Octavia's service state, or would (very careful) DB hackery be required here? Many thanks, Paul Browne ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

4 6

GPU PCI passthrough woes.
by Andy Speagle 09 Apr '25

09 Apr '25

Hey Folks, I could use a little assistance getting GPU passthrough working. I had this working already for one flavor of nvidia gpu... and I've added some hosts with a much newer gpu... I've updated the pci_alias and pci_passthrough variables and those seem to be getting set properly in nova.conf passthrough_whitelist = [{"vendor_id":"10de", "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": "10de"} I believe I have all of the iommu stuff configured and have the pci- stub module entries... dmesg output shows that the GPUs are being claimed by the stub module. $ openstack flavor show t1.small_gpu_l40s +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 0 | | id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | | name | t1.small_gpu_l40s | | os-flavor-access:is_public | True | | properties | pci_passthrough:alias='gpu-l40s:1' | | ram | 2048 | | rxtx_factor | 1.0 | | swap | 0 | | vcpus | 1 | +----------------------------+--------------------------------------+ Yet... I can't seem to get an instance to run using that new flavor... keeps complaining about there not being enough hosts available. Fault: code: 500 created: 2024-10-30T02:04:33Z message: "No valid host was found. There are not enough hosts available." details: | Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1580, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 940, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist- packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist- packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Any clues on how/where to dig into this more to see what might be missing? Thanks. -- Andy Speagle

3 2

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 17 Mar '25

17 Mar '25

Hello Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [Call Trace] ./stack.sh:1126:start_keystone /opt/stack/devstack/lib/keystone:549:die [ERROR] /opt/stack/devstack/lib/keystone:549 keystone did not start Error on exit World dumping... see /opt/stack/logs/worlddump-2024-11-18-150808.txt for details ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) Thanks in advance, Ishan

3 5

[all][tc][ptl][qa] Migrating upstream CI/CD jobs to Ubuntu Noble (24.04)
by Ghanshyam Mann 14 Mar '25

14 Mar '25

Hello Everyone, As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. This is the etherpad which has the migration plan and will be used for the tracking: - https://etherpad.opendev.org/p/migrate-to-noble I have prepared the change(WIP) for base jobs migration on Noble (openstack tox jobs, devstack, and tempest) which will migrate most of the projects jobs. Tempest change[3] include dependency on openstack tox, devstack base jobs changes so testing project gate using the the tempest change will test the complete set. Refer to the etherpad for the complete migration plan/steps. * Deadline: Nov 29th 2024 (R18) is the deadline to test/fix the projects jobs and to merge the base jobs running on Noble (openstck tox, devstack, and tempest). [1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubunt… [3] https://review.opendev.org/c/openstack/tempest/+/932156 -gmann

2 9

[kolla] [nova] API flags ignored in Wallaby
by Albert Braden 12 Mar '25

12 Mar '25

After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list: Train: openstack server list --all-projects --locked --os-compute-api-version 2.73 <empty list because nothing is locked> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> Wallaby (locked flag is ignored): openstack server list --all-projects --locked --os-compute-api-version 2.73 <list of every VM in the cluster> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> The second problem is the hypervisor list. We can’t get “long” output in Wallaby: Train: openstack hypervisor list --long | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | Wallaby (long flag is ignored): openstack hypervisor list --long | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |

6 14

[tc][all][monasca ] Retiring the Monasca project
by Ghanshyam Mann 16 Feb '25

16 Feb '25

Hello everyone, As you might know, the Monasca project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project. I'm sending this email to propose the Monasca retirement and based on the response, I will proceed next. [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 -gmann

6 17

[nova] How to gain approval for a new feature on unavailable hardware
by taketani.ryo＠fujitsu.com 14 Feb '25

14 Feb '25

To Whom It May Concern, We are considering to propose and develop a new feature on OpenStack Nova for building Arm Confidential Computing Architecture (Arm CCA)-encrypted instances. I have some questions for doing it. * While specification of Arm CCA is already open(https://developer.arm.com/documentation/den0125/0300/Overview), No Arm CCA-enabled hardware is currently available to our knowledge. Therefore, we plan to conduct testing in an internal environment. * Would a proposal for a feature utilizing publicly unavailable hardware be acceptable to OpenStack? Alternatively, in this case, what steps or preparations are required for approval? * Should the blueprint and specification be submitted simultaneously? * The specification is currently incomplete. Building Arm CCA instances requires libvirt APIs, similar to AMD SEV instances(https://blueprints.launchpad.net/nova/+spec/amd-sev-libvirt-suppo…, but specification of these are not yet determined publicly. We are considering submitting the blueprint first to inform the community of our ongoing development efforts. Regards

3 6

[ironic]About the OS supported by Diskimage-builder
by Junya Noguchi (Fujitsu) 03 Feb '25

03 Feb '25

To whom it may concern. My name is Junya Noguchi. I have some questions about the supported OS for Diskimage-builder operation. Please respond. References https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html Does it mean that the OS that says "work but receive only limited testing and support" works but is not supported? References https://docs.openstack.org/diskimage-builder/latest/ This document describes RHEL and other operating systems that are not supported in the previous question. Are these operating systems supported? Also, are there any plans to support more operating systems in the future? Best regards, Junya Noguchi

5 7

[glance][ops] proposal to remove the glance-scrubber
by Brian Rosmaita 20 Jan '25

20 Jan '25

Hello Glance users, The Glance project team wants to determine whether anyone is using the glance-scrubber utility [0]. If not, we are thinking of deprecating it this cycle to be removed in the appropriate future cycle. If you have not previously heard of the glance-scrubber, that's most likely because it's only useful for particular deployment configurations of which yours is not one. See the doc [0] for details if you are curious. Our suspicion is that the glance-scrubber may have outlived its usefulness for modern OpenStack configurations. (The last time I remember it being discussed at a PTG was during the snowstorm in Dublin, which I believe was the Rocky development cycle). On the other hand, the team also has some ideas for improving the scrubber, but there's no point doing that if it's not actually being used in the first place. So if removal of the glance-scrubber would negatively affect you, please respond to this email so we can get some data points. Thanks! [0] https://docs.openstack.org/glance/latest/cli/glancescrubber.html

2 2

[cinder][kubernetes] pvc stuck in reserved state
by Eugen Block 10 Jan '25

10 Jan '25

Hi, we have a couple of k8s clusters in our openstack environment (Victoria) and are facing some problems when pods get moved to different worker nodes. The PVCs get stuck in "reserved" state, and they only get attached after resetting the volumes to "available" state. First I thought it was due to the missing multiattach volume type which seems to work fine in our customer's cluster. I also found the "duplicate connector" error messages in the nova-compute.log: > Unable to update attachment.(Invalid volume: duplicate connectors > detected on volume But after adding the multiattach volume type and retyping the affected volumes, the PVCs still get stuck in "reserved" state while I don't see any error messages on the hypervisors anymore. So it seems that something has changed/improved, but now I don't have a trace where to look next. Could this be a cinder-csi driver issue or something? Our cinder backend is ceph (Pacific). Not sure which information could help so please ask for anything I can provide. Any pointers are highly appreciated! Thanks! Eugen

1 2

Failing at TASK [glance : Copy over multiple ceph configs for Glance]
by Karera Tony 04 Jan '25

04 Jan '25

Hello Team, I am deploying Openstack release 2024.02 with External Ceph. However I am encountering the error below. But it is failing here TASK [glance : Copy over multiple ceph configs for Glance] Please assist Regards Tony Karera

2 1

ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version
by Swogat Pradhan 24 Dec '24

24 Dec '24

Hi, I am currently trying to deploy openstack wallaby using tripleo arch. I created the network jinja templates, ran the following commands also: #openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml # openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml # openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml and used the environment files in the openstack overcloud deploy command: (undercloud) [stack@hkg2director ~]$ cat deploy.sh #!/bin/bash THT=/usr/share/openstack-tripleo-heat-templates/ CNF=/home/stack/ openstack overcloud deploy --templates $THT \ -r $CNF/templates/roles_data.yaml \ -n $CNF/workplace/custom_network_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e $CNF/templates/node-info.yaml \ -e $CNF/templates/scheduler-hints.yaml \ -e $CNF/workplace/networks-deployed-environment.yaml \ -e $CNF/workplace/vip-deployed-environment.yaml \ -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ -e $CNF/workplace/custom-net-bond-with-vlans.yaml Now when i run the ./deploy.sh script i encounter an error stating: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command. 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1 Can someone tell me where the mistake is? With regards, Swogat Pradhan

4 4

[all] Installation guide for openSUSE and SUSE Linux Enterprise
by Takashi Kajinami 23 Dec '24

23 Dec '24

Hello, As you may know, we have been maintaining installation guide for three linux distributions below. - oepnSUSE and SUSE Linux Enterprise - Red Hat Enterprise Linux and CentOS - Ubuntu However, as far as I know, the packages for openSUSE and SLES haven't been built/updated for some time. Also openSUSE was removed from tested runtimes in wallaby and hasn't been tested. So I doulbt that users can use these operating systems to install OpenStack nowadays. Can/Should we remove the guide for openSUSE/SLES Now ? I'm not very sure if we can maintain the guide for these with no details available nowadays. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 1

[ops][largescale-sig] Join the Large Scale OpenStack virtual meetup with Rackspace !
by Thierry Carrez 09 Dec '24

09 Dec '24

Hi everyone, This cycle the Large Scale SIG is organizing virtual meetups, where operators of large-scale OpenStack deployments come to discuss their experiences, challenges and solutions with fellow operators. On Thursday, December 5, at 16 UTC (10am CT, 5pm Paris), we'll have our first one. The special guest for this meetup will be Kevin Carter, operator of the Rackspace Cloud. It will last one hour and will happen through a video meeting. To join us, please fill this form and we'll send you the link for the event the day before! https://forms.gle/92o1DTavHwivN8tn6 Hoping to see you there. -- Thierry Carrez (ttx)

4 6

[kolla-ansible][nova] Refresh cell cache in nova scheduler
by Satish Patel 04 Dec '24

04 Dec '24

Folks, What is wrong here, I do have the following hostname resolvable from all the nodes even if they are in the /etc/hosts file, Why is the following task saying UNREACHABLE and from where its trying to reach ? PLAY [Refresh nova scheduler cell cache] ***************************************************************************************************************************************************** TASK [nova : Refresh cell cache in nova scheduler] ******************************************************************************************************************************************* fatal: [os2-bos-ctrl01]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl01\". Make sure this host can be reached over ssh: ", "unreachable": true} fatal: [os2-bos-ctrl02]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl02\". Make sure this host can be reached over ssh: ", "unreachable": true} fatal: [os2-bos-ctrl03]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl03\". Make sure this host can be reached over ssh: ", "unreachable": true}

4 5

[octavia] [lbaas] [ovs] Is octavia sr-iov supporting tunnels?
by yardalgedal＠gmail.com 04 Dec '24

04 Dec '24

Hi, Openstack Team, Is new octavia sr-iov feature supporting vxlan tunnels like in ovs or it can work only with provider networks (vlan)?

3 2

[kayobe] Running seed on ansible control host
by Olaf Trygve Berglihn 04 Dec '24

04 Dec '24

Dear list, Can someone explain how to configure kayobe to set up the seed hypervisor and seed vm on the ansible control host (where you run 'kayobe seed hypervisor host configure' and 'kayobe seed vm provision')? Thanks! -Olaf

2 5

[kayobe] [bifrost] Enabling and Deploying compute node
by alidot-subs Mail 30 Nov '24

30 Nov '24

Hi Everyone, Its be good experience to deploy Kayobe in all in one mode and see cirros image getting deployed and acesssible. Moving forward for multi-node deployment of baremetal machines. i tried to enable bifrost bit i doesn't seems to et enable by setting configurations in bifrost.yaml. What it takes to enable and use Bifrost using kayobe ? It may sound basic question. Thank you Ali

2 3

[cinder] What is the "official" name for Cinder in the service catalog?
by Stephen Finucane 29 Nov '24

29 Nov '24

o/ I've proposed a series of patches to "clean up" the service catalog that DevStack creates in a deployment. Currently, DevStack creates two endpoints for Cinder: one called 'cinderv3' of type 'volumev3' and another called 'cinder' of type 'block-storage'. For example: { "services": [ { "name": "cinderv3", "description": "Cinder Volume Service V3", "id": "65460706a9864e71926f13042f526aeb", "type": "volumev3", "enabled": true, "links": { "self": "http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb" } }, { "name": "cinder", "description": "Cinder Volume Service", "id": "cd406a0e296d47dca1e481e992851bb2", "type": "block-storage", "enabled": true, "links": { "self": "http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2" } }, // ... ], "links": { "next": null, "self": "http://10.0.108.50/identity/v3/services", "previous": null } } They both ultimately point to the same API, the Cinder v3 API, meaning one of these endpoints is redundant. My assumption was 'block-storage' was the correct one, since that's the official service type [1], and that 'cinder' was the appropriate name since it mirrors what we use for other services (nova, neutron, keystone, glance, ...). However, tkajinam raised the point [2] that the Install Guide currently recommends use of the 'volumev3' type and that projects like Nova and Glance have an '[cinder] catalog_info' that defaults to 'volumev3'. If so, is 'block-storage' really the official service type, or is it 'volumev3'. If it is, should we be changing the Install Guide and the various config option defaults? Cheers, Stephen [1] https://specs.openstack.org/openstack/service-types-authority/

6 18

[docs][arch] Architecture Guide, ch. Control Plane Design
by just.me.on.reddit＠protonmail.com 29 Nov '24

29 Nov '24

I am sorry for proprietary tag - I hope it helps to narrow down the scope of this mail. Tags in order generic to more specific. Hello, If I only understand the term "cloud controller" properly these are those OpenStack services which collectively control whole cloud, inclusively those services exposed to cloud user. Cloud control can run on single hosting node, or can be distributed to a group of nodes. Please correct me if I am wrong. My role is on cloud operative side (a learner to become OS user). Architecture Guide, chapter as in this mail subject states following "It’s also possible to use virtual machines for all or some of the services that the cloud controller manages, such as the message queuing. In this guide, we assume that all services are running directly on the cloud controller." It is hard to continue the lecture of this Guide due to unclear description at following points. 1. reg. "all services are running directly on the cloud controller" Can't actually understand this statement. The "on" suggests cloud controller be hosting services in mind. Furthermore, cloud controller includes only those services acting under the goal of cloud control. Service whose major purpose is to provide functionality to cloud user won't, I suppose, be part of cloud controller. Hence, this statement might be more generic as it should be. 2. reg. "use virtual machines for all or some of the services that the cloud controller manages, such as the message queuing" Will cloud controller really mange message queuing instead of include comprise it? Which use of virtual machine exactly is meant here * placing services to virtual machine during O.S. deployment to hosting IT * the IT hosting OpenStack-based cloud? Should the former bullet apply, wasn't is said in some other Guide, abstraction level rather high, that use of virtual machines as cloud host is recommended only for test and evaluation purposes?

1 0

[openstack-ansible] : GPU computes on openstack-ansible
by Rambo Rambo 29 Nov '24

29 Nov '24

Hi Team Openstack-Ansible Openstack Victoria Release Ubuntu 20.04 I need to add computes having GPU on them. So I need recommendations on the procedure to add them and also what all configurations through openstack-ansible are required for nova_api, nova_scheduler on control node nova_api containers and compute node nova_compute configurations? Also does anyone has experience using and installing openstack service : cyborg for GPU management. Regards Rambo

3 5

[heat][horizon][octavia][oslo][qa][skyline][takcer][telemetry][zaqar] Need attention for tomorrow (Nov 29th) CI migration to Ubutnu Noble
by Ghanshyam Mann 29 Nov '24

29 Nov '24

Hello Team, As you know, we have been trying very hard to test and fix the project CI for the Ubuntu Noble migration[1] and there are 9 projects that are still failing on Noble. Check the etherpad having testing change for me information on failures: - https://etherpad.opendev.org/p/migrate-to-noble The migration is planned for tomorrow (Nov 29th); request you to check/fix the failure before that; otherwise, it will break your project gate. [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… -gmann

1 0

[docs][kayobe] Architecture, terms used
by just.me.on.reddit＠protonmail.com 28 Nov '24

28 Nov '24

Dear Kayoube documentation, If possible please be bit more specific at few points - document Architecture, section Networks refers to bare metal overcloud hosts, however the chapter "Hosts in the System" presents only bare metal compute hosts, nor any other text passage clarifies the term "overcloud". This way it is hard to get full understanding of term "Overcloud out-of-band network". Furthermore, if you talk about an out-of-band stuff it gets hard to see what is out-of-band without term in-band be precisely clear.

1 3

[watcher][tc] watcher-drivers launchpad membership
by Sean Mooney 28 Nov '24

28 Nov '24

Hi folks o/ Following up form the PTG discussion https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L65 and the recently merged governance patch https://github.com/openstack/governance/commit/1e547ae39f9913dc9cfd9c755b24… I wanted to raise the topic of the watcher-drivers team https://launchpad.net/~watcher-drivers membership and ownership. during the TC sessions we discussed that that the <project>-coresec teams should be owned by the https://launchpad.net/~openstack-admins team https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b#L123 watcher does not currently have a sperate watcher-coresec team in launchpadd only a a watcher-drivers team i also think that the watcher driver team should be owned by the https://launchpad.net/~openstack-adminsteam. https://launchpad.net/~watcher-drivers is currently owned by https://launchpad.net/~david-tardivel and the only other member with admin rights is https://launchpad.net/~li-canwei2 Can i ask that one of the existing admin either grant my membership request and delegate admin rights to me or add the https://launchpad.net/~openstack-admins team as admins and make them the owner of the team? licanwei, david-tardivel, would either of ye be willing and able to make those changes in launchpad? i also think it would be prudent to add other active member (the current liaisons) as admins/moderators of the watcher-driver team but i can do that once i have access or https://launchpad.net/~openstack-adminscan. having all launchpad teams owned by https://launchpad.net/~openstack-adminsteam mitigates the risk of loosing access by the current admins becoming inactive, but it would be nice to not need to escalation to them for simple tasks like moderation of the group. that is why i want to add the other liaisons to the admins of the team. I am asking for this access for 2 reasons: Fist to add https://launchpad.net/~openstack-adminsas admins of the team and the launchpad project, Second so that i can create new launchpad series for current releases to do blueprint and bug tracking. at present because we do not have launchpad series fro 2025.1 we cannot properly track bugs/blueprints for the current cycle. In the future if there are no objections form the existing contributors i would also like to create a separate open bug team with less project wide permission so that anyone can contribute to bug triage without needing access to the drivers team. this is in lien with how other project like nova work for example https://launchpad.net/~nova-bugs i also think we should be creating a separate watcher-coresec team to allow dan as the security liaison and the vulnerability management team to properly manage security reports. once we have access to the watcher-drivers team we should be able to update the launchpad project to make that possible. Regards sean.

3 4

[watcher][tc][tempest] watcher gerrit group membership
by Sean Mooney 28 Nov '24

28 Nov '24

Hi everyone o/ As many will be aware, over the last month or two, I and some others have been looking to help maintain and extend the Watcher project. At the ptg, I proposed several topics to cover the administrivia of reviving the project. To bring it to a healthy state where features, bugs and general maintenance could be performed on an ongoing basis and avoid its retirement at the end of the 2025.1 cycle. One of the follow-up items to enable that was the extension of the watcher Gerrit groups to allow others to supplement the largely inactive core groups. https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L47 <https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L47> To that end, I would like to ask the existing watcher core team and TC (or other community members :) ) to provide input on the following proposal. 0.) I want to state upfront that I do not want to take away review rights or diminish the experience and efforts of any of the past or present members of the watcher team. I am sending this proposal to enable the watcher project to function effectively and I am welcome to feedback or objections. 1.) To facilitate the changes in the rest of the proposal I would like to formally ask to be added to all of the existing watcher Gerrit groups * watcher-core https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47… <https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47…> * watcher-release https://review.opendev.org/admin/groups/b265e6b6457de71c7ecfc2a3c70f0c52cde… <https://review.opendev.org/admin/groups/b265e6b6457de71c7ecfc2a3c70f0c52cde…> * watcher-dashboard-core https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47… <https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47…> * watcher-dashboard-release https://review.opendev.org/admin/groups/904be58ec9fa0e811166ce6144d6314cc7b… <https://review.opendev.org/admin/groups/904be58ec9fa0e811166ce6144d6314cc7b…> * watcher-specs-core https://review.opendev.org/admin/groups/747af616eb8748aee5b17d9493f9ebbc49e… <https://review.opendev.org/admin/groups/747af616eb8748aee5b17d9493f9ebbc49e…> * watcher-tempest-core https://review.opendev.org/admin/groups/09a91d8e24af9ce44b80062c4851a1d2fa3… <https://review.opendev.org/admin/groups/09a91d8e24af9ce44b80062c4851a1d2fa3…> 2.) watcher-release and watcher-dashboard-release Historically, there has been a benefit to having separate core teams for the master and the stable branches. As the velocity of projects has decreased and projects have become more mature the delineation between the maintainers of the master branch and the stable branches has lessened. I believe there is still merit in having the flexibility to add people directly to the release team independently from the core team so I am not proposing replacing the *-release teams with *-core. To that end, I would like to make the following changes to the membership of watcher-release and watcher-dashboard-release - add stable-maint to both release groups - add watcher-core to watcher-release, as we have done in other projects the ask to the watcher-core team is, if you do not have time to review stable branches, or do not feel comfortable with applying stable policy, then there is no expectation to review on the stable branches. - add watcher-dashboard-core to watcher-dashboard-release - me (sean-k-mooney as release-liaison) to facilitate backports and release activities and Gerrit membership admin. 3.) watcher-core and watcher-dashboard-core As it currently stands the core groups are relatively inactive https://tinyurl.com/bdz9k47n full url: https://openstack.biterg.io/app/dashboards?security_tenant=openstack#/view/…:'Gerrit%20Approvals%20panel%20by%20Bitergia',filters:!(('$state':(store:appState),meta:(alias:Bots,disabled:!f,index:gerrit,key:author_bot,negate:!t,params:(query:!t),type:phrase),query:(match:(author_bot:(query:!t,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:gerrit,key:project,negate:!f,params:(query:watcher),type:phrase),query:(match_phrase:(project:watcher)))),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),query:(language:lucene,query:''),timeRestore:!f,title:'Gerrit%20Approvals',viewMode:view <https://openstack.biterg.io/app/dashboards?security_tenant=openstack#/view/…:'Gerrit%20Approvals%20panel%20by%20Bitergia',filters:!(('$state':(store:appState),meta:(alias:Bots,disabled:!f,index:gerrit,key:author_bot,negate:!t,params:(query:!t),type:phrase),query:(match:(author_bot:(query:!t,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:gerrit,key:project,negate:!f,params:(query:watcher),type:phrase),query:(match_phrase:(project:watcher)))),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),query:(language:lucene,query:''),timeRestore:!f,title:'Gerrit%20Approvals',viewMode:view>) Dan Smith and Slawek Kaplonski were added to help with the retirement/gate issues on behalf of the TC While Dan has offered to help bootstrap the core team we need to be cognizant that both Dan and Slawek have other commitments. Chenker has previously expressed that they intend to review when time allows but they have limited time to do so. To that end, I would like to propose that I be added to both core teams, To have a health core team we will eventually need to have at least 3 active reviewers. Watcher will still be 1-2 reviews short of that, but I hope that can be addressed over the coming months. As others become active over the 2025.1 cycles I would like to revisit the membership and extend the core team after milestone 2 or more realistically after milestone 3 if others demonstrate interest and capacity to review. 4.) watcher-tempest-core This is a newly created group added by https://review.opendev.org/c/openstack/project-config/+/934357 <https://review.opendev.org/c/openstack/project-config/+/934357> This group intends to allow the review team for the tempest plugin to diverge from the main watcher core group. To enable effective review and maintenance of the tempest plugin I would like to propose the following membership changes. - tempest-core should be added as a fallback, to guide best practices, and review if they have time. - gmann has volunteered to help with this too so I propose they be added as a direct member - I would like to be added to have a second active reviewer available. Note, while I can provide some review input I have less knowledge of tempest in general as such I will rely on Gmann's guidance in this area. 5.) watcher-specs Several years ago Nova expanded the nova-specs core group to include nova-core there were a number of motivating factors but one of them was the large overlap between the two groups. another factor was the role of a spec core (focusing on the design constraints, upgrade impact, API impact ectra) is different from the implementation considerations(testing, code quality, performance) often the line between the two is subtle and blurred. To that end, I would like to propose the following changes. - first, I would like to be directly added to the specs core group - second, I would like to add the watcher-core group to watcher-specs. If this proposal is accepted I would like to try and move forward with this in the next week or so to unblock the pending review backlog. Regards sean PS many of the open reviews are cleanup i have submitted so this does not directly address review bandwidth for my personal patches but i think it still has value in general.

3 3

Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster
by Ishan Shanware (ishanwar) 28 Nov '24

28 Nov '24

For everyone’s reference From: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com> Date: Wednesday, 27 November 2024 at 10:01 PM To: Sven Kieske <kieske(a)osism.tech> Subject: Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster Hi Sven, Thanks for getting back to me. We have configured kolla to use 3 controllers in each cluster. Each controller has one rabbitmq container created by kolla. The process we follow move a controller out of rotation is as follows: 1. We first failover all the l3 and dhcp agents running on the controller to the 2 other controllers. 2. We then turn of the server and then delete any neutron agents on it. Currently the rabbitmq cluster only consists of only 2 containers. The problem specifically occurs during the reinstall of the rabbitmq container in the kolla deploy stage. Specifically, after it executes the checking rabbitmq containers role. It triggers a restart for the contains on all the blades. After this stage completed we observe that the masakari-engine disables all the nova-computes. Let me know if you need any other information for discussion Thanks Ishan From: Sven Kieske <kieske(a)osism.tech> Date: Tuesday, 26 November 2024 at 11:36 PM To: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster Hi Ishan, my first question would be, how many controller nodes and specifically, rabbitmq nodes you are running inside your Openstack cluster? You should be following the general guidelines for running any raft consensus based distributed system and only run an odd number of systems, e.g. 3 or 5 control nodes. Can you confirm that this is the case in your setup? If you e.g. run a two node setup such errors are indeed expected. See also our production architecture guide: https://docs.openstack.org/kolla-ansible/latest/admin/production-architectu… > Control - Cloud controller nodes which host control services like APIs and databases. This group should have odd number of nodes for quorum. If you are running an odd number of control nodes and you're still facing this issue, I would be curious to know the rabbitmq cluster state before you add or remove a node, because this should theoretically just work. But maybe there is another issue with your cluster. HTH -- Sven Kieske Senior Cloud Engineer Mail: kieske(a)osism.tech Web: https://osism.tech OSISM GmbH / Talweg 8 / 75417 Mühlacker / Deutschland Geschäftsführer: Christian Berendt Unternehmenssitz: Mühlacker Amtsgericht: Stuttgart, HRB 756139

1 1

[watcher] Canceling Dec 5th weekly meeting
by Douglas Viroel 28 Nov '24

28 Nov '24

Hi all o/ Since most of our audience will be out next week, we agreed[1] on canceling watcher's weekly meeting[2], on Dec 5th. We will meet again in two weeks, on Dec 12th. [1] https://meetings.opendev.org/meetings/watcher/2024/watcher.2024-11-28-12.00… [2] https://meetings.opendev.org/#Watcher_Team_Meeting BR, -- Douglas Viroel - dviroel

1 0

[kolla-ansible] nova-cell can't not find nova.keyring file when deploy
by daniel890723＠gmail.com 28 Nov '24

28 Nov '24

hello Daniel here Previously i was testing openstack-helm but it doesn't work so i switch to kolla-ansible with all-in-one mode i was following the official guide and put the keyring file i generate to the config folder: (venv) root:~# nano /etc/kolla/config/nova/ ceph.client.nova.keyring ceph.conf localhost/ nova-compute.conf But when i try to deploy it show the error below TASK [nova-cell : Check nova keyring file] ***************************************************************************************************************** fatal: [localhost]: FAILED! => {"msg": "No file was found when using first_found."} Just in case i also put the keyring file in /config/localhost/nova Still shows the same error But when i put the cinder keyring file in nova folder like this: (venv) root@:~# nano /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf localhost/ nova-compute.conf It can find the cinder keyring file but this time the nova-compute container show error below: unable to find a keyring on /etc/ceph/ceph.client.nova.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Did i do anything wrong or something makes it not able to detect the nova.keyring file P.S. Cinder and Glance works fine and Ceph can be used normally

2 5

[manila] Cancelling Nov 28th upstream weekly meeting
by Carlos Silva 27 Nov '24

27 Nov '24

Hello Zorillas, At least half of our audience tomorrow will not be online tomorrow, as it is thanksgiving in the US, so let's cancel the meeting. We'll resume the weekly meetings on Dec 5th. Regards, carloss

1 0

[ops][kolla-ansible][openstack-ansible] Deploying OpenStack using Ansible to LXC while project is used to grab for Kolla-Ansible
by just.me.on.reddit＠protonmail.com 27 Nov '24

27 Nov '24

Hi, I am learning OS because my imminent mission uses a cloud-based solution (cloud-app) which runs in OpenStack cloud. How do you estimate the probability for a cloud-app to feature built-in Docker dependency (runtime) in situation where the project of mentioned app appears to be used to apply Kolla Ansible in order to deploy OpenStack cloud? It means the cloud-app to depend at runtime on Docker-specific artefacts. 3rd-party cloud-app going to be in use will play a major role in my project. It is in responsibility of cloud-app operator to do provisioning of OpenStack cloud before mentioned app can be deployed to cloud. Recommendations found in cloud-app's documentation read to use Kolla Ansible in order to conduct mentioned provision. Myself learned that Kolla uses Docker. Docker use is that component which triggers me to ask this question. My own preference would be rather to deploy OpenStack services to LXC-containers. One can find in OpenStack web the project Ansible-based deployment to LXC containers. I wonder following if cloud-app developers could had placed in it any built-in Docker (eco-system) runtime dependency and they achieved it by the use of Ansible Kolla for OpenStack deployment. Considering my very flat knowledge of Docker (only familiarized with its purpose and very few exercises made in far past) there exists just hypothetical risk for cloud project in my focus to feature built-in docker eco-system dependency. Should such dependency be a fact (not just a risk) my aim to use OpenStack services in LXC-containers has good chance to fail. It is hard to contact 3rd-party cloud project developer regarding my questions. Question 2 Any common conclusions had been drawn so far if one of the two approaches be of remarkably higher suitableness to host OpenStack-based cloud services? * placing OpenStack-based cloud services to Docker-based containers * placing services of OpenStack-based cloud to Linux containers (LXC/LXD/...)

1 0

[os-vif][nova][neutron] Proposing Rodolfo as an os-vif core
by Sean Mooney 27 Nov '24

27 Nov '24

hi folks o/ I intended to send this last week but didn't find time till now. I'm sure many are familiar with Rodolfo form the work that he has done in neutron over the years. I have worked with them for a long time and over the years they have also contributed too and reviewed changes too os-vif. I think having a neutron core on the os-vif team would certainly help bubble up pain points form them and address such issue faster in os-vif going forward. Rodolfo is familiar with the code and with the upstream review process so i think they would be a good fit. If there are no objection to this ill bring this up again in next weeks nova team meeting and add rodoflo to the core list. reagrds sean.

2 2

[Octavia] Cancelling today's meeting
by Gregory Thiemonge 27 Nov '24

27 Nov '24

Hi Folks, Due to personal reasons, I have to cancel this week's meeting, sorry for the short notice, See you next week, Greg

1 0

Openstack ansible variables
by engineer2024 27 Nov '24

27 Nov '24

I want to print the values of all the ansible variables at the playbook and task level. If we execute os_glance.yml , I want to see the variable values of all the tasks/playbooks involved in this glance playbook. Is it also possible doing it from any one file ? Thanks elinux

1 0

[largescale-sig] Next meeting: November 27, 9utc
by Thierry Carrez 27 Nov '24

27 Nov '24

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20241127T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[devstack][neutron] DNS Resolution Issue After Server Reboot with DevStack
by xuq.fnstxz＠fujitsu.com 27 Nov '24

27 Nov '24

Hi everyone, I am reaching out to seek your assistance regarding an issue I am experiencing with my DevStack installation. After successfully installing DevStack stable/2024.2 on ubuntu 22.04, I configured the network and set up DNS for private subnet. stack@controller:~$ openstack subnet list +--------------------------------------+---------------------+--------------------------------------+---------------------+ | ID | Name | Network | Subnet | +--------------------------------------+---------------------+--------------------------------------+---------------------+ | 2b6411d5-9c19-4a45-bd47-58dc748f5e00 | public-subnet | 45502b6a-773b-40cf-866a-e8b18397ffcb | 192.168.1.0/24 | | 69629170-2e0e-49c0-b741-61560558001c | ipv6-public-subnet | 45502b6a-773b-40cf-866a-e8b18397ffcb | 2001:db8::/64 | | c74bf062-b67a-45cb-8a50-73583498dfcd | ipv6-private-subnet | 8fb37e61-bd59-4624-a4be-1fc1841987d7 | fd3a:2f86:ca98::/64 | | f2e10d3d-f785-4cf9-8804-1969269623c2 | private-subnet | 8fb37e61-bd59-4624-a4be-1fc1841987d7 | 10.0.0.0/26 | +--------------------------------------+---------------------+--------------------------------------+---------------------+ stack@controller:~$ openstack subnet set --dns-nameserver 10.12.255.254 --dns-nameserver 192.168.1.95 f2e10d3d-f785-4cf9-8804-1969269623c2 10.12.255.254 is the real DNS server address and 192.168.1.95 is my server IP address. Initially, when I launched a VM instance, it was able to resolve the proxy and connect to the external network without any issues. jenkins@np0000000531:~$ curl opendev.org <a href="https://opendev.org/">Temporary Redirect</a>. However, after rebooting the server and launching the instance again, the instance is unable to resolve the proxy. jenkins@np0000000531:~$ curl opendev.org curl: (5) Could not resolve proxy: my.proxy.com I have verified that the systemd-resolved service is in an active state, but the logs show a large number of entries with the following message: Nov 13 11:53:51 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 10.12.255.254. Nov 13 11:54:01 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.95. Nov 13 11:54:01 np0000000531 systemd-resolved[568]: Using degraded feature set TCP instead of UDP for DNS server 10.12.255.254. Nov 13 11:54:11 np0000000531 systemd-resolved[568]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.95. Nov 13 11:54:21 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 10.12.255.254. I have checked the network and routing table of the VM instance, as well as the bridge settings on the server. Everything appears to be the same as it was before the reboot. Could you please help me identify the cause of this issue and suggest any potential solutions? If you need any additional information, please let me know, and I will be happy to provide it. Thank you for your time and assistance. Best regards, Inori.

2 3

[all] 2025 Open Infrastructure Foundation Individual Director nominations are open
by Jeremy Stanley 27 Nov '24

27 Nov '24

To those who aren't following the foundation mailing list, please be aware that the nomination period for the 2025 Open Infrastructure Foundation Board of Director Individual Member elections is open, and will close at 23:59 UTC on December 13. If you've considered taking on some additional leadership responsibility, this could be your opportunity! See Wes's post for more information: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/th… -- Jeremy Stanley

1 0

[kolla] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster
by Ishan Shanware (ishanwar) 26 Nov '24

26 Nov '24

Hi Kolla Community, I would like to understand the current guidelines for adding or removing a controller node from an OpenStack cluster. With the introduction of quorum queues, we’ve observed that adding a new controller node causes a loss of quorum within the cluster. Consequently, during the reinstallation of a controller node, Masakari disables all the nova-compute services. Is this behaviour expected with quorum queues, or could this be addressed differently? Additionally, I’ve noticed that as soon as the Kolla containers are restarted in the cluster, clients are unable to connect, causing the cluster to go down. Could you please share insights or recommendations on how to handle this scenario effectively? Looking forward to hearing your thoughts. Thanks, Ishan Cisco Systems

2 2

[keystone] Weekly meeting canceled 27-Nov-2024
by Dave Wilde 26 Nov '24

26 Nov '24

Hello, Due to the Thanksgiving holiday in the US I'm canceling the weekly meeting and reviewathon this week. Please reach out here or on IRC if you need anything. Thanks! /Dave

1 0

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

[openstack helm] [Ansible] Openstack-helm Ansible inventory.yaml missing some value
by daniel890723＠gmail.com 26 Nov '24

26 Nov '24

Hello today i'm testing installation with completely new enviroment i was following the guide from Openstack-helm document and install k8s cluster using ansible and deploy-env playbook but i encounter this error when setup ssh cluster TASK [deploy-env : Generate ssh key pair] ****************************************************************************************************************** skipping: [node-1] fatal: [primary]: FAILED! => {"msg": "Failed to change ownership of the temporary files Ansible (via chmod nor setfacl) needs to create despite connecting as a privileged user. Unprivileged become user would be unable to read the file."} and the log with -vvv output: debug2: mux_client_hello_exchange: master version 4 debug3: mux_client_forwards: request forwardings: 0 local, 0 remote debug3: mux_client_request_session: entering debug3: mux_client_request_alive: entering debug3: mux_client_request_alive: done pid = 114038 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 chown: invalid user: ‘zuul’ debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 And by checking the playbook we can see the playbook is using "{{ client_ssh_user }}" on primary node and "{{ cluster_ssh_user }}" with cluster node which is not defined in inventory.yaml Causing it using default user zuul by adding client_ssh_user & cluster_ssh_user in ansible inventory it works normally Is this meant to be like this? or the document missing some variables in inventory.yaml

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-18)
by Goutham Pacha Ravi 26 Nov '24

26 Nov '24

Hello Stackers, We're 18 weeks away from the coordinated release for 2025.1 ("Epoxy") [1]. We're beginning a period of low activity due to year-end holidays across the world. Last week, the TC didn't merge any new governance proposals; however, several are currently in progress. === Weekly Meeting === The TC met in OFTC's #openstack-tc IRC channel on 2024-11-19 [2]. We discussed a series of global requirements upper constraint bumps that were committed and the ensuing CI flakiness. We appreciated the effort to proactively update dependencies to their latest versions earlier in the release cycle so we can have more time to test and stabilize OpenStack software. Sometimes, these failures may interrupt other project priorities. The best way we can help is if project teams identify a relatively quick-running and stable integration job to run against changes to the global requirements repository. The TC and OpenDev administrators worked with Launchpad administrators to regain control of trackers belonging to the Watcher project team. It was noted that Launchpad trackers for any project should not be owned/administered by individuals. Instead, there must be a Launchpad team driving all trackers associated with an OpenStack initiative, and "openstack-admins" must own this team. This approach ensures the sustainability of project teams and protects projects from the impact of changing individual priorities. We also briefly discussed the "eventlet-removal" goal proposal. The goal document is being rewritten [3] to simplify what is being done and the timelines we're aiming for. I'd love for the community to take a look at this change and review it. The next meeting of the OpenStack Technical Committee is today, 2024-11-25, at 1800 UTC in OFTC's #openstack-tc channel. The agenda for this meeting is in our meeting wiki [4]. I hope you're able to join us. === Governance Proposals === ==== Open for Review ==== Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 ==== Merged ==== Fix doc job for pillow 11.0.0 | https://review.opendev.org/c/openstack/governance/+/935967 Add Ubuntu Noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meetings: Join us at our weekly meeting. The Technical Committee meets every week on Tuesdays at 1800 UTC [4]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC Meeting IRC Log 2024-11-19: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.log.html [3] Eventlet removal Community goal: https://review.opendev.org/c/openstack/governance/+/931254 [4] TC Meeting Agenda, 2024-11-26: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[docs][dev] replace_global_vars not working in project-deploy-guide
by Jakub Darmach 25 Nov '24

25 Nov '24

Hello everyone, I've been updating documentation in kolla-ansible and noticed that while replacing variable content works in https://docs.openstack.org/kolla-ansible it fails to replace the variable with proper content in https://docs.openstack.org/project-deploy-guide/kolla-ansible. If I understand correctly, it's being substituted with var having proper content in here: https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… Later on used in https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… And now for the example - |KOLLA_BRANCH_NAME| variable: Working substitution: https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#t… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… Failed to render content: https://docs.openstack.org/project-deploy-guide/kolla-ansible/latest/quicks… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… The var seems to be used the same in both examples, I suspect slightly different configuration on project end of the things. Can anyone advise where should I start looking? Best regards, Jakub

2 1

[neutron] Bug deputy report for week of November 18th
by Brian Haley 25 Nov '24

25 Nov '24

Hi, I was the Neutron bug deputy last week. Below is a short summary about the reported bugs. -Brian Critical bugs ------------- * https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - https://review.opendev.org/c/openstack/neutron/+/935804 High bugs --------- * https://bugs.launchpad.net/neutron/+bug/2088453 - [UT] Neutron tests failing with eventlet 0.37.0 - https://review.opendev.org/c/openstack/neutron/+/935524 * https://bugs.launchpad.net/neutron/+bug/2089157 - [neutron-specs] CI job "openstack-tox-docs" broken - https://review.opendev.org/c/openstack/neutron-specs/+/935836 * https://bugs.launchpad.net/neutron/+bug/2089250 - neutron-vpnaas: Automatic VPN Agent failover not working with OVN driver - needs owner * https://bugs.launchpad.net/neutron/+bug/2089252 - neutron-vpnaas: LibreSwan 4 connections not listed as ACTIVE - Submitter said he would look this week Medium ------ * https://bugs.launchpad.net/neutron/+bug/2089169 - [OVN] Test ``TestCreateNeutronPgDrop.test_non_existing`` is randomly failing - https://review.opendev.org/c/openstack/neutron/+/935802

1 0

[cinder] Meeting canceled this week, 27 Nov
by Jon Bernard 25 Nov '24

25 Nov '24

Hello all, schools are closed this week so I won't be able to host the weekly meeting due to the holidays. I'll see you all the following week. Take care, -- Jon

1 0

[horizon][ceph] object storage container storage policy missing
by Satish Patel 25 Nov '24

25 Nov '24

Folks, Openstack Release: Zed Recently I have integrated ceph RGW service with openstack/keystone and testing with command line tool "openstack container create foo1" and it works but when I go to Horizon UI I am seeing it doesn't have storage policy dropdown list and because of that I am not able to create container or upload file from Horizon GUI https://ibb.co/KhYW6Rd It works with commands like but not horizon. $ openstack container create foo1 +---------------------------------------+-----------+------------------------------------------------+ | account | container | x-trans-id | +---------------------------------------+-----------+------------------------------------------------+ | AUTH_d4d82849885e4126aef402c408ad2c5c | foo1 | tx00000c057454841b13497-0065b558b4-42ff2eb-ahm | +---------------------------------------+-----------+------------------------------------------------+ $ openstack container list +------+ | Name | +------+ | foo1 | +------+

2 2

[Neutron] - Unable to ping my instance from my Openstack controller node
by ABATCHA OLLOH 23 Nov '24

23 Nov '24

Hello Everyone, Could you give me any advice and help me to better understand neutron. On my VirtualBox VM I properly installed Openstack using packstack (all-in-one installation). I have access to the horizon dashboard. I'm able to launch an instance and associate floating IP but from the controller node I cannot reach my instance. - Here are my interfaces config: - enp0s3: for mgmt - enp0s8: for openstack services - enp0s9: for neutron_external_interface (used as a public interface) - enp0s10: interface on my controller to be connected to the neutron_external_interface - VirtualBox Promiscuous mode : Allow All [root@packstack ~(keystone_admin)]# ip -br -c a lo UNKNOWN 127.0.0.1/8 ::1/128 enp0s3 UP 9.10.93.8/24 fe80::a00:27ff:fe2e:150a/64 enp0s8 UP 9.11.93.8/24 fe80::a00:27ff:fec7:56ab/64 enp0s9 UP fe80::a00:27ff:fef9:3cc7/64 enp0s10 UP 9.12.93.15/24 fe80::a00:27ff:feff:3641/64 ovs-system DOWN br-tun DOWN br-int DOWN br-ex UNKNOWN 9.12.93.8/24 fe80::b021:85ff:fe8a:9d44/64 qbr9eefea66-89 UP qvo9eefea66-89@qvb9eefea66-89 UP fe80::1409:2aff:feb4:e37d/64 qvb9eefea66-89@qvo9eefea66-89 UP fe80::8c84:15ff:fe7d:8896/64 tap9eefea66-89 UNKNOWN fe80::fc16:3eff:fec6:5f5c/64 - Security Group Rules associated to my instance - Egress IPv4 Any Any 0.0.0.0/0 - Ingress IPv4 ICMP Any 0.0.0.0/0 - Ingress IPv4 TCP 22 (SSH) 0.0.0.0/0 - Ingress IPv4 TCP 80 (HTTP) 0.0.0.0/0 - Instances status [root@packstack ~(keystone_admin)]# openstack server list +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | testrocky | ACTIVE | private1=10.0.0.202, 9.12.93.203 | N/A (booted from volume) | m1.devops | +--------------------------------------+-----------+--------+----------------------------------+--------------------------+-----------+ - From the router inside my openstack I can ping the floating ip [root@packstack ~(keystone_admin)]# ip netns list qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. 64 bytes from 9.12.93.203: icmp_seq=1 ttl=64 time=6.67 ms 64 bytes from 9.12.93.203: icmp_seq=2 ttl=64 time=1.51 ms9.12.93.203 [root@packstack ~(keystone_admin)]# ip netns qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e (id: 2) qdhcp-2a02741e-35f0-4a61-81b0-abd4b5a09f36 (id: 0) qdhcp-bcc1c132-074f-45d5-a715-a2d371cdb5be (id: 1) [root@packstack ~(keystone_admin)]# ip netns exec qrouter-a4c63603-b8e8-460a-bbc7-47503fe6cc8e ip route default via 9.12.93.1 dev qg-68351942-28 proto static 9.12.93.0/24 dev qg-68351942-28 proto kernel scope link src 9.12.93.201 10.0.0.0/24 dev qr-5ba4fd5b-b5 proto kernel scope link src 10.0.0.1 - Trying to reach the floating IP from the controller node [root@packstack ~(keystone_admin)]# ping PING 9.12.93.203 (9.12.93.203) 56(84) bytes of data. From icmp_seq=1 Destination Host Unreachable ...9.12.93.2039.12.93.8 - Check floating ip status [root@packstack ~(keystone_admin)]# openstack port list --network private1 +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ | 5ba4fd5b-b5e1-4f56-b681-963cf2c9ac8a | | fa:16:3e:14:0b:4a | ip_address='10.0.0.1', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | 9eefea66-897b-4682-883c-b61e22c3c09a | | fa:16:3e:c6:5f:5c | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | | a73cd16c-8480-4461-99f2-7837e10f27b1 | | fa:16:3e:91:5f:98 | ip_address='10.0.0.200', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | ACTIVE | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 9eefea66-897b-4682-883c-b61e22c3c09a +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | packstack.example.local | | binding_profile | | | binding_vif_details | bound_drivers.0='openvswitch', bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' | | binding_vif_type | ovs | | binding_vnic_type | normal | | created_at | 2024-11-15T11:38:07Z | | data_plane_status | None | | description | | | device_id | 8d2e2f04-8080-44df-923f-9728ebabe9e5 | | device_owner | compute:nova | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='10.0.0.202', subnet_id='48bc8201-25ca-48f5-b8b2-3daf3fc9d1f2' | | hardware_offload_type | None | | hints | | | id | 9eefea66-897b-4682-883c-b61e22c3c09a | | ip_allocation | None | | mac_address | fa:16:3e:c6:5f:5c | | name | | | network_id | 2a02741e-35f0-4a61-81b0-abd4b5a09f36 | | numa_affinity_policy | None | | port_security_enabled | True | | project_id | bbba2507099c4209a529acd64352939e | | propagate_uplink_status | None | | resource_request | None | | revision_number | 10 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | 2a206a8d-fb54-4fa7-9db7-e5f197dcc107 | | status | ACTIVE | | tags | | | trunk_details | None | | updated_at | 2024-11-18T08:15:49Z | +-------------------------+--------------------------------------------------------------------------------------------------------------------------------------------+ [root@packstack ~(keystone_admin)]# openstack port list --network public1 +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ | 2b215f41-edf8-4c61-8969-383143340444 | | fa:16:3e:30:7e:08 | ip_address='9.12.93.200', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | fa:16:3e:28:39:a9 | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | N/A | | 68351942-28a1-4df3-8661-bf157fcd5982 | | fa:16:3e:bf:66:56 | ip_address='9.12.93.201', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | ACTIVE | +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+ [root@packstack ~(keystone_admin)]# openstack port show 31d7b194-50a0-4a25-b102-542210e5f3f3 +-------------------------+----------------------------------------------------------------------------+ | Field | Value | +-------------------------+----------------------------------------------------------------------------+ | admin_state_up | UP | | allowed_address_pairs | | | binding_host_id | | | binding_profile | | | binding_vif_details | | | binding_vif_type | unbound | | binding_vnic_type | normal | | created_at | 2024-11-15T15:57:42Z | | data_plane_status | None | | description | | | device_id | 3dc9d9c3-28eb-4dfb-a41b-9bbfac9f96da | | device_owner | network:floatingip | | device_profile | None | | dns_assignment | None | | dns_domain | None | | dns_name | None | | extra_dhcp_opts | | | fixed_ips | ip_address='9.12.93.203', subnet_id='01aff9ec-e22c-47d3-b92e-192b01c8281a' | | hardware_offload_type | None | | hints | | | id | 31d7b194-50a0-4a25-b102-542210e5f3f3 | | ip_allocation | None | | mac_address | fa:16:3e:28:39:a9 | | name | | | network_id | bcc1c132-074f-45d5-a715-a2d371cdb5be | | numa_affinity_policy | None | | port_security_enabled | False | | project_id | | | propagate_uplink_status | None | | resource_request | None | | revision_number | 2 | | qos_network_policy_id | None | | qos_policy_id | None | | security_group_ids | | | status | N/A | | tags | | | trunk_details | None | | updated_at | 2024-11-15T15:57:43Z | +-------------------------+----------------------------------------------------------------------------+ - I couldn't find something related to the binding port in these logs file. - tail -f /var/log/neutron/openvswitch-agent.log - tail -f /var/log/neutron/server.log - tail -f /var/log/neutron/l3-agent.log [image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…>[image: r/openstack - [Neutron] - Unable to ping my instance from my Openstack controller node] <https://preview.redd.it/neutron-unable-to-ping-my-instance-from-my-openstac…> Any advise will be welcome

2 2

Live Migration problems
by Gilles Mocellin 22 Nov '24

22 Nov '24

Hi Stackers, I have 2 recurring problems with live migration. If someone already face that and have recommendations... My env : - OpenStack 2023.1, deployed by OpenStack-Ansible 27.5.1 - Network : ML2/LinuxBridges - Some shared Networks on a VLAN provider - Most private networks on VXLAN - OS : Ubuntu 20.04 - libvirt-daemon-system: 8.0.0-1ubuntu7.10~cloud0 - qemu-system-x86: 1:4.2-3ubuntu6.30 Each time we have maintenance to do on compute nodes, we live-migrate the instances, with `nova host-evacuate-live` (still no openstack CLI command to do that). 1. Never ending / failed migrations In general, it works, but each time we have 1 or 2 (on 40 instances) that don't want to complete. It's some memory active instances (Kubernetes controller nodes with etcd...). Even if I had configured live_migration_permit_auto_converge to True. Now, if we try to force complete (openstack server migration force complete), migration fails, instance return to ACTIVE status on source. And I can see libvirt errors : [...] libvirt.libvirtError: operation failed: migration out job: unexpectedly failed Not really helpful. I tried removing live_migration_permit_auto_converge, reduce live_migration_completion_timeout to 60 and set live_migration_timeout_action to force_complete. Same problems. 2. Instance unreachable More problematic, as it's not easy to see : some instances are not reachable after migration. And after a while, DHCP leases expire. The only way to recover is to restart neutron-linuxbridge-agent on the instance compute node. In fact, on all compute nodes, because instances evacuation spread them on all the compute nodes. Then, if DHCP has not expired, the instance is joinable again. We made a script to find those instances, it lists all ports bound to a nova instance. Then connects to each control/network nodes, enter qdhcp network namespaces (found with the MAC adresse in DHCP leases files), and try to ping the instance private IP. The only log I see, but don't know if it has something to do with that is : Device tapxxx-yy has no active binding in host But I can see that message even since, without unjoinable instances... Here it is, if something ring a bell, for an existing bug or configuration error, please let us know ! Thanks. -- Gilles

3 4

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 21 Nov '24

21 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading, the hypervisor is not showing correct numbers of VMs on compute nodes from browser. But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

3 5

[keystone][magnum] Policy doesn't allow certificate:get to be performed
by Satish Patel 21 Nov '24

21 Nov '24

Folks, I have configured keystone with LDAP authentication and it works but when I deploy magnum + magnum-cluster-api. But when I am trying to create a cluster and obtain credentials I am getting policy errors. Few month back I had similar issue [1] and I have resolved withadding [assignment] driver = sql But somehow It's back and started getting errors again. This time I have tried everything but still getting the same error and no idea where to find and how to find a workaround here. My users in LDAP only authenticate and the rest of the things work from SQL. I have assigned role readers to users also. But that didn't resolve the issue. (venv-openstack) root@os-eng-ctrl-01:~# openstack coe cluster config ssr-dev3 identity version 3% is not in supported versions: 2, 2.0, 3 Policy doesn't allow certificate:get to be performed (HTTP 403) (Request-ID: req-d49586c0-fa88-4bbc-9a12-5d07c2b06c0e) My openstack release version is 2023.1 1. https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack…

1 1

[ironic][networking] Interest in starting a baremetal networking working group?
by Julia Kreger 20 Nov '24

20 Nov '24

Greetings folks, During the PTG this past week, the Ironic community including developers and operators reached a conclusion that we need to start back up a networking focused working group to try and move the current state of capabilities forward when bare metal machines are involved. Specifically, management of physical switch connections to bare metal *and* programming DPU/IPU devices for networking attachments. As we're all quite aware, it is a complex topic with many different threads. Given resurgent interest and a mutual desire to find some better solutions, it just makes sense to have regular discussions. That being said we need to identify an ideal day of week, frequency, and preferred time. I'd recommend sometime between 1500 UTC and 1800 UTC. Preferably a Tuesday or Wednesday. Would every two weeks work? Or would folks prefer to meet weekly to try and build the most possible velocity through regular interaction? Thoughts? -Julia

10 16

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 20 Nov '24

20 Nov '24

Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:image001.png@01DB39FB.69BBA000] Thanks in advance. Regards, Ishan

2 1

[kolla] DHCP not accessible on compute nodes and VM IP Assignment Issue in Compute Nodes - Kolla Ansible 2024.1
by Jaganathan, Keerthana (SCC) 20 Nov '24

20 Nov '24

Hello Kolla Ansible community, We are experiencing difficulties with VM IP address assignment in our OpenStack deployment using Kolla Ansible 2024.1 on Rocky Linux 9.2. Here are the details of the issue: 1. Problem description: IP addresses are successfully assigned to VMs created on control nodes, but VMs created on compute-only nodes fail to receive IP addresses. This suggests that DHCP requests originating from the compute nodes are not reaching the DHCP agents on the control nodes. 2. Error messages/logs: Console log of a Cirros VM on a compute node shows: checking http://169.254.169.254/2009-04-04/instance-id failed 1/20: up 0.88. request failed ... failed 20/20: up 54.89. request failed failed to read iid from metadata. tried 20 failed to get instance-id of datasource 3. Configuration: - 7 nodes in total: 5 control/compute/network nodes, 2 compute-only nodes - Single bonded interface (bond0) with 3 VLANs (bond0.10, bond0.20, bond0.30) - network_interface: bond0.10 - api_interface: bond0.20 - tunnel_interface: same as network_interface - neutron_external_interface: bond0 - Provider network: VLAN 41 - ml2_conf.ini [ml2_type_vlan] network_vlan_ranges = physnet1:41:41 - 5 neutron DHCP agents, one on each control node 4. Troubleshooting steps taken: - Verified br-tun OpenFlow rules are dropping packets - Confirmed 5 DHCP agents are running, one on each control node - Openvswitch containers: agent and Here are the openvswitch open flow rules (dump) from the compute node (node16) for br-ex, br-in and br-tun (openvswitch-vswitchd)# ovs-ofctl dump-flows br-ex cookie=0xa9774500bf4a6912, duration=175366.856s, table=0, n_packets=3093, n_bytes=945104, priority=4,in_port="phy-br-ex",dl_vlan=1 actions=mod_vlan_vid:41,NORMAL cookie=0xa9774500bf4a6912, duration=177050.619s, table=0, n_packets=2, n_bytes=176, priority=2,in_port="phy-br-ex" actions=drop cookie=0xa9774500bf4a6912, duration=177050.621s, table=0, n_packets=8178, n_bytes=2363856, priority=0 actions=NORMAL (openvswitch-vswitchd)# ovs-ofctl dump-flows br-int cookie=0x98223299c10a59d3, duration=177059.641s, table=0, n_packets=0, n_bytes=0, priority=65535,dl_vlan=4095 actions=drop cookie=0x98223299c10a59d3, duration=175373.945s, table=0, n_packets=123, n_bytes=9594, priority=10,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136 actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=175373.944s, table=0, n_packets=0, n_bytes=0, priority=10,arp,in_port="qvoab4fc3cb-a2" actions=resubmit(,24) cookie=0x98223299c10a59d3, duration=177059.625s, table=0, n_packets=20, n_bytes=1668, priority=2,in_port="int-br-ex" actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=0, n_packets=3028, n_bytes=939610, priority=9,in_port="qvoab4fc3cb-a2" actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=175375.861s, table=0, n_packets=2256, n_bytes=219762, priority=3,in_port="int-br-ex",dl_vlan=41 actions=mod_vlan_vid:1,resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.643s, table=0, n_packets=2, n_bytes=176, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.644s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.946s, table=24, n_packets=123, n_bytes=9594, priority=2,icmp6,in_port="qvoab4fc3cb-a2",icmp_type=136,nd_target=fe80::f816:3eff:fe4f:8020 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=175373.945s, table=24, n_packets=0, n_bytes=0, priority=2,arp,in_port="qvoab4fc3cb-a2",arp_spa=141.52.56.110 actions=resubmit(,25) cookie=0x98223299c10a59d3, duration=177059.641s, table=24, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x98223299c10a59d3, duration=175373.947s, table=25, n_packets=2970, n_bytes=935510, priority=2,in_port="qvoab4fc3cb-a2",dl_src=fa:16:3e:4f:80:20 actions=resubmit(,30) cookie=0x98223299c10a59d3, duration=177059.640s, table=30, n_packets=2970, n_bytes=935510, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.639s, table=31, n_packets=0, n_bytes=0, priority=0 actions=resubmit(,58) cookie=0x98223299c10a59d3, duration=177059.642s, table=58, n_packets=5351, n_bytes=1165042, priority=0 actions=resubmit(,60) cookie=0x98223299c10a59d3, duration=177059.642s, table=60, n_packets=5351, n_bytes=1165042, priority=1 actions=NORMAL cookie=0x98223299c10a59d3, duration=177059.640s, table=62, n_packets=0, n_bytes=0, priority=3 actions=NORMAL (openvswitch-vswitchd)[root@os16 /]# ovs-ofctl dump-flows br-tun cookie=0xad1cb3762184465d, duration=178072.704s, table=0, n_packets=4815, n_bytes=1114796, priority=1,in_port="patch-int" actions=resubmit(,2) cookie=0xad1cb3762184465d, duration=178072.703s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.703s, table=2, n_packets=1164, n_bytes=69852, priority=1,arp,dl_dst=ff:ff:ff:ff:ff:ff actions=resubmit(,21) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=7, n_bytes=528, priority=0,dl_dst=00:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,20) cookie=0xad1cb3762184465d, duration=178072.702s, table=2, n_packets=3644, n_bytes=1044416, priority=0,dl_dst=01:00:00:00:00:00/01:00:00:00:00:00 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.701s, table=3, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.701s, table=6, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0xad1cb3762184465d, duration=178072.700s, table=10, n_packets=0, n_bytes=0, priority=1 actions=learn(table=20,hard_timeout=300,priority=1,cookie=0xad1cb3762184465d,NXM_OF_VLAN_TCI[0..11],NXM_OF_ETH_DST[]=NXM_OF_ETH_SRC[],load:0->NXM_OF_VLAN_TCI[],load:NXM_NX_TUN_ID[]->NXM_NX_TUN_ID[],output:OXM_OF_IN_PORT[]),output:"patch-int" cookie=0xad1cb3762184465d, duration=178072.699s, table=20, n_packets=7, n_bytes=528, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=21, n_packets=1164, n_bytes=69852, priority=0 actions=resubmit(,22) cookie=0xad1cb3762184465d, duration=178072.699s, table=22, n_packets=4815, n_bytes=1114796, priority=0 actions=drop I would greatly appreciate any guidance on resolving this. Best regards, le3188

1 0

[kolla] IRC meeting cancelled - 20.11.2024
by Bartosz Bezak 20 Nov '24

20 Nov '24

Hello, Unfortunately, this week’s Kolla IRC meeting is cancelled. There are multiple conferences happening this week. Apologies for the late notice. Best regards, Bartosz Bezak

1 0

[openstack-ansible][horizon]The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope
by Danish Khan 20 Nov '24

20 Nov '24

Dear All, I have upgraded one of my test openstack cluster from victoria to Wallaby. After upgrading the hypervisor is not showing correct numbers of VMs on compute nodes: [image: image.png] But openstack CLI is giving correct number of VMs. I am getting below error in horizon container when I click on "hypervisor" from browser: Nov 20 11:17:29 horizon-container-2b244cbd apache2[43317]: [wsgi:error] [pid 43317:tid 139897181898496] [remote x.x.x.151:37180] /openstack/venvs/horizon-23.3.1.dev4/lib/python3.8/site-packages/oslo_policy/policy.py:1065: UserWarning: Policy os_compute_api:os-migrate-server:migrate failed scope check. The token used to make the request was domain scoped but the policy requires ['system', 'project'] scope. This behavior may change in the future where using the intended scope is required I did try few thing but that did not work. Regards, Danish Khan

1 0

[Cinder] Restore volume snaptshots while volume is in use
by Jan 19 Nov '24

19 Nov '24

Hi, for providing a more enterprise like (VMware, HyperV, Proxmox) snapshot feature, it would be great to be able to revert snapshots in use. With the "revert-to-snapshot" functions everything is there to support this feature. IMO if the storage backend (driver) does not support the operation, it should raise an error (ex. resetting a ceph rbd volume while a vm is running is not possible). But if it supports the operation, we should not stop the user doing this, even if the volume is "in-use". One might introduce a force switch, to prevent triggering it unintended. A frontend component like skyline can also show additional information like the status of an attached instance. Since the "Detach and attach boot volumes" spec [1] was never fully implemented, there is no way to revert an instance to a snapshot state without completely rebuilding it. Like described in the spec draft, this is not always suitable and at least always cumbersome. -- Jan [1] https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/detac… --- suggested patch ----- https://github.com/jmussmann/cinder/commit/4f971d77a94806e9ebdccab45392058c… From 4f971d77a94806e9ebdccab45392058c25005761 Mon Sep 17 00:00:00 2001 From: jmussmann <jmussmann+github(a)posteo.de> Date: Thu, 31 Oct 2024 09:13:45 +0100 Subject: [PATCH] add: allow reverting snapshots of volumes in use --- cinder/volume/api.py | 5 +++-- cinder/volume/manager.py | 4 ++-- cinder/volume/rpcapi.py | 4 ++-- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/cinder/volume/api.py b/cinder/volume/api.py index 0e20baff0..9aa160740 100644 --- a/cinder/volume/api.py +++ b/cinder/volume/api.py @@ -390,8 +390,9 @@ class API(base.Base): """revert a volume to a snapshot""" context.authorize(vol_action_policy.REVERT_POLICY, target_obj=volume) + volume_state = volume.status v_res = volume.update_single_status_where( - 'reverting', 'available') + 'reverting', ['available', 'in-use']) if not v_res: msg = (_("Can't revert volume %(vol_id)s to its latest snapshot " "%(snap_id)s. Volume's status must be 'available'.") @@ -408,7 +409,7 @@ class API(base.Base): "snap_id": snapshot.id}) raise exception.InvalidSnapshot(reason=msg) - self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot) + self.volume_rpcapi.revert_to_snapshot(context, volume, snapshot, volume_state) def delete(self, context: context.RequestContext, diff --git a/cinder/volume/manager.py b/cinder/volume/manager.py index 30de5d8c7..18914c073 100644 --- a/cinder/volume/manager.py +++ b/cinder/volume/manager.py @@ -1150,7 +1150,7 @@ class VolumeManager(manager.CleanableManager, self.create_snapshot(context, snapshot) return snapshot - def revert_to_snapshot(self, context, volume, snapshot) -> None: + def revert_to_snapshot(self, context, volume, snapshot, original_volume_state) -> None: """Revert a volume to a snapshot. The process of reverting to snapshot consists of several steps: @@ -1207,7 +1207,7 @@ class VolumeManager(manager.CleanableManager, LOG.exception(msg, msg_args) v_res = volume.update_single_status_where( - 'available', 'reverting') + original_volume_state, 'reverting') if not v_res: msg_args = {"id": volume.id, "status": 'available'} diff --git a/cinder/volume/rpcapi.py b/cinder/volume/rpcapi.py index 58de69c1c..0b36cebdd 100644 --- a/cinder/volume/rpcapi.py +++ b/cinder/volume/rpcapi.py @@ -180,11 +180,11 @@ class VolumeAPI(rpc.RPCAPI): volume=volume) @rpc.assert_min_rpc_version('3.15') - def revert_to_snapshot(self, ctxt, volume, snapshot): + def revert_to_snapshot(self, ctxt, volume, snapshot, volume_state): version = self._compat_ver('3.15') cctxt = self._get_cctxt(volume.service_topic_queue, version) cctxt.cast(ctxt, 'revert_to_snapshot', volume=volume, - snapshot=snapshot) + snapshot=snapshot, original_volume_state=volume_state) def delete_volume(self, ctxt: context.RequestContext, -- 2.47.0

3 2

#eventlet-removal - HTTP SGI working group call for discussion
by Herve Beraud 19 Nov '24

19 Nov '24

tl;dr: We should have a cross discussion concerning the replacement of Eventlet.wsgi and we want to invite you to join the discussion and hence help us to define a standardized approach in a collaborative way. The Eventlet WSGI module can be used for 2 things: 1. as a WSGI server to serve any kind of web framework (flask, routes, etc...) [1] 2. and in some cases as a framework for web applications [2] Some OpenStack deliverables are using it in one way or another. Some deliverables like Glance already have credible migrations opportunities. But some of you may still be wondering what is your option to replace the Eventlet WSGI features? That's a good question. Now that the specs defining the removal of Eventlet from oslo.service are merged we have to define the way we want to proceed for the WSGI features. Indeed, we decided to drop the WSGI support from oslo.service. That means that we have to define, at least, a "default" way to replace it. If we want to avoid having multiple ways to manage things, we have to provide some guidance and standardizations. For this reason, after the PTG, we encouraged the creation of the HTTP SGI Working Group [4]. This working group aim to answer which (W|A)SGI servers are needed or credible alternatives. Which framework we have/want/need to use in parallel with the SGI server. The outcomes created by this group will serve as "a default" to guide the community through the replacement of the WSGI features of Eventlet. We prepared comparisons of the available alternatives for each layer: - https://wiki.openstack.org/wiki/SGI-servers-comparison - https://wiki.openstack.org/wiki/web-frameworks-comparison - https://wiki.openstack.org/wiki/Http-libraries-comparison Feel free to comment and to put your pros and cons on the given related etherpads (see the links in the comparison). This topic will be discussed at each irc meeting on the #openstack-eventlet-removal channel [5], until we find a default official option. Next meeting date is: 2024 December 3rd [6]. It would be good to reach an official consensus before the Milestone 3 of Epoxy. The goal is simply to ensure that our community looks in the same direction and that everyone is on the same page. We want to ensure that nobody is left behind. Feel free to reply to this thread or to put your comments directly on the various etherpads [6][7][8][9]. Thanks for reading. [1] https://eventlet.readthedocs.io/en/latest/reference/api/eventlet.html#modul… [2] https://eventlet.readthedocs.io/en/latest/modules/wsgi.html [3] https://review.opendev.org/c/openstack/oslo-specs/+/927503 [4] https://wiki.openstack.org/wiki/Eventlet-removal#The_HTTP_SGI_Working_Group [5] https://meetings.opendev.org/meetings/openstack_eventlet_removal/ [6] https://etherpad.opendev.org/p/epoxy-eventlet-tracking [7] https://etherpad.opendev.org/p/eventlet-removal-SGI-servers-comparison [8] https://etherpad.opendev.org/p/eventlet-removal-web-frameworks-comparison [9] https://etherpad.opendev.org/p/eventlet-removal-http-libraries-comparison -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

Kolla TLS
by devispain＠mail.com 19 Nov '24

19 Nov '24

Reading the Kolla docs it says: The back-end TLS cert/key can be the same certificate that is used for the VIP, as long as those certificates are configured to allow requests from both the VIP and internal networks. But there is no advice on how to do this, can someone explain how to use letsencrypt vip certs for backend tls, i have found nowhere that explains it at all. Trying to use a self signed cert for backend tls seems to break everything when using proper certs for vips

1 0

python3-dns 2.6.1 breaks Nova
by Francesco Di Nucci 19 Nov '24

19 Nov '24

Hi all, maybe this is already known (https://bugs.launchpad.net/glance/+bug/2054732) but also python3-dns-2.6.1 seems to break Nova and other services, with the error reported in the issue, that the DNS "answered udp() got an unexpected keyword argument 'ignore_errors';" It seems that Almalinux has the new version in the repos, while Rocky Linux still has 2.3.0 Best regards Francesco Di Nucci

2 2

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 19 Nov '24

19 Nov '24

Hello Stackers, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base The image includes several packages (such as Apache) that don’t appear necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though all containers don’t use them. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

Oslo Spec Review, Eventlet Removal Proposal.
by Daniel Bengtsson 19 Nov '24

19 Nov '24

Hi everyone, I hope you’re all doing well. During yesterday’s Oslo IRC meeting, the main topic was the removal of eventlet in oslo.service[1]. It seems that everyone is in agreement with this proposal, and no objections were raised. I would appreciate it if you could take another look and review the code. If we don’t receive any further objections by Monday, I propose we proceed with merging the spec. Thanks in advance for your help and feedback! [1] https://review.opendev.org/c/openstack/oslo-specs/+/927503?usp=dashboard -- Daniel Bengtsson Software engineer

2 1

[publiccloud-sig] Reminder: next meeting November 20th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 19 Nov '24

19 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 20th) the next meeting of the publiccloud-sig is going to happen. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> As I’ll be traveling, I’ll not be able to participate myself. felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 19 Nov '24

19 Nov '24

Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:image001.png@01DB39FF.4A36CA30] Thanks in advance, Ishan

4 3

[octavia] [lbaas] CPU pinning & multiqueue
by yardalgedal＠gmail.com 19 Nov '24

19 Nov '24

Hi, openstack team, is someone tested CPU pinning together with enabled multiqueue for VM? I'm seeing huge performance degradation (750k RPS => 200k RPS) on flavor with 16 vCPUs and 16 gb ram when using image built with CPU pinning vs image without it, we're using zed (I've backported commits to check, but I think newer versions can also be affected), haproxy 2.4, ubuntu 22.04.

4 4

[Glance] Status of image clone feature
by Masahito Muroi 19 Nov '24

19 Nov '24

Hello Glance folks, I'm wondering the development status of the image clone feature[1]. We provide multi region OpenStack cluster and Glance API is deployed in each region individually. We as a cloud operator want to clone same glance image attributes and images across all the regions using the cloning feature. I have checked the image interoperable feature[2] is available, but the feature looks for cloud users, not for cloud operators. If the development is suspended by some reason, we're happy to start talking the cloning feature. 1. https://wiki.openstack.org/wiki/Glance-image-cloning 2. https://docs.openstack.org/glance/latest/admin/interoperable-image-import.h… thank you, Masahito

2 2

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-19)
by Goutham Pacha Ravi 19 Nov '24

19 Nov '24

Hello Stackers, We're now past milestone-1 of the 2025.1 ("Epoxy") release cycle [1]. As a contributor/maintainer, you have probably made plans to adjust your goals and project deliverables in light of the likely drop in capacity for the year-end holiday season that's ahead. In the past week, the OpenStack Technical Committee didn't commit any new governance changes; however, we're actively working on our action items from the PTG [2]. === Weekly Meeting === The TC's weekly meeting last week was well attended [3]. Clark Boylan (clarkb) noted that contributors were reaching out to OpenDev Infrastructure Maintainers via IRC and the Zuul email list seeking help to set up and configure "third-party" CI systems to meet the requirements set by several projects (Nova, Neutron, Cinder, Manila, ...). Contributors often lack clear documentation and support for setting up these systems, leading to misplaced queries on Zuul or OpenDev channels. We agreed that there is a lack of centralized, up-to-date documentation regarding CI systems put together to test things outside of the purview of the community's Zuul CI. We have lost mindshare regarding third-party CI setup and maintenance, and this hurts the participation of vendor driver maintainers. The TC would love to see interested parties come together to re-form the "Third-Party CI SIG." If we're unable to meaningfully help contributors with this, we must evaluate whether third-party CI should remain a mandatory requirement for service projects. It was highlighted that Nova maintains a dedicated weekly topic to discuss specific third-party CI issues, a model other projects might emulate. We also discussed kicking off the election planning for the 2025.2 cycle. There will be 5 seats available on the OpenStack TC, and every OpenStack project team will elect a PTL or liaison [4]. Election officials will convene over the next few weeks to propose election dates and other details. There's still time to volunteer to be an election official. Please contact me off-list if you're interested! The next meeting of the OpenStack Technical Committee is today (11/18/2024) at 1800 UTC. This meeting will be held via IRC on OFTC's #openstack-tc channel. Please find the meeting agenda on the meeting wiki [5]. I hope you're able to attend. Please remember, you can always propose new topics for this meeting, and it's a great way to synchronously engage with the TC every week. === Governance Proposals === ==== Open for Review ==== rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 ==== Merged ==== Add ubuntu noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] OpenStack TC's Epoxy PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [3] TC Meeting IRC Log 2024-11-12: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-12-18.00.log.html [4] OpenStack Election website: https://governance.openstack.org/election/ [5] TC Meeting Agenda, 2024-11-19: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

Re: [Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 19 Nov '24

19 Nov '24

Hey Martin, Artemio, Thanks for responding. The VM is sufficiently large for development with about 60 GB of root storage. I was able to fix the keystone failure, by changing the OVS_RUNDIR like this OVS_RUNDIR=$OVS_PREFIX/var/run/ovn. However the script hangs here, and doesn’t move ahead. +functions:wait_for_compute:505 echo 'Didn'\''t find service registered by hostname after 60 seconds' Didn't find service registered by hostname after 60 seconds +functions:wait_for_compute:506 openstack --os-cloud devstack-admin --os-region RegionOne compute service list I am trying to setup a development environment for the nova service. Thanks Ishan From: Martin Kopec <mkopec(a)redhat.com> Date: Tuesday, 19 November 2024 at 4:05 AM To: Artemio Mendoza <artemiom681(a)gmail.com> Cc: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [Devstack] Trying to setup development environment on VM Hi there, the VM should be big enough, the snip from the logs doesn't suggest any reason why keystone didn't start. There is one job that runs devstack on debian every day - https://zuul.openstack.org/builds?job_name=devstack-platform-debian-bookwor… You can try to compare your local.conf with the one used in the jobs. On Mon, 18 Nov 2024 at 21:08, Artemio Mendoza <artemiom681(a)gmail.com<mailto:artemiom681@gmail.com>> wrote: Hi Ishan, In my experience, I was able to set up devstack in the Ubuntu 22.04 (Jammy) VM. I never tried Debian. Hopefully, it can help you. Thank you, Artemio On Mon, Nov 18, 2024 at 12:46 PM Ishan Shanware (ishanwar) <ishanwar(a)cisco.com<mailto:ishanwar@cisco.com>> wrote: Hi Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [cid:ii_19340db3c084cff311] Thanks in advance, Ishan -- Martin Kopec IRC: kopecmartin

1 0

[tc][ptg] 2025.1 "Epoxy" Technical Committee PTG Summary
by Goutham Pacha Ravi 18 Nov '24

18 Nov '24

Hello Stackers, The following is a summary of the Technical Committee's virtual PTG discussions during the last week. For longer notes / recordings, please consult the Etherpad [1] where we kept minutes, or recordings within a YouTube playlist [2]. === Reviving the Inclusive language discussion === The TC brainstormed with the Diversity and Inclusion Working Group [3] about the renaming of the "master" development branch in repositories. We evaluated whether a TC resolution is necessary to provide a clear, standardized approach. Much of OpenDev tooling allows projects the ability to customize their default branch names, with an understanding that this could sometimes be confusing for tools and end-users. Some open-source communities have attempted similar renaming efforts but faced pushback or incomplete adoption, as highlighted by Kubernetes, Ceph, and GNOME. Additionally, there remain concerns over the feasibility, workload, and support available in the OpenStack community to undertake a coordinated renaming effort. The discussion did not yield a consensus and we will not be pursuing an OpenStack wide effort to renaming all the development branches in OpenStack. This doesn't prevent a future change of course, however. If we do undertake this in the future, we will begin by assessing the community's bandwidth for the effort and we'll identify contributors who will lead and assist before proceeding. The conversation also considered the broader scope of inclusive language. The D&I WG does not strongly recommend renaming the "master" branch of git repositories as it is not linked to problematic contexts like slavery in this case. More concerning terminology, like "slave," should be prioritized: https://wiki.openstack.org/wiki/Diversity/Inclusivity Action Items: - Evaluate and Draft a TC Resolution (gouthamr): TC resolution will propose formalizing OpenStack's stance on inclusive language, which would include specific recommendations about branch naming. - Encourage a timeline to deal with non-Inclusive language in code and documentation (tc) === Community Leaders meeting === Subtopic 1: Translating OpenStack documentation and i18n SIG's challenges The OpenStack i18n team discussed challenges and potential improvements for translating OpenStack documentation, particularly due to a slow shift from Zanata to Weblate. The translation team currently has limited activity and faces infrastructure issues, such as Weblate integration problems after a recent cloud upgrade. Key goals are to engage Asian communities (e.g., Korea, Vietnam, Indonesia) to support documentation translations, focusing initially on a subset of projects like Nova, Cinder, Neutron, and Glance. To streamline the process, the i18n team proposed starting with machine translations, which would be refined by regional language moderators. They also discussed AI-generated translations and compliance with OpenInfra's AI policy, requiring clear indication of AI involvement in commits. The i18n team needs infrastructure support, including Zuul automation for translation updates, and guidance for project core teams on reviewing translations. The discussion continued in the i18n SIG meetings during the rest of the week: https://etherpad.opendev.org/p/oct2024-ptg-i18n Action Items - I18n SIG will work on weblate migration with priority, and that will need resolving infrastructure issues as well as figuring out Zuul integration. - Sylvain Bauza (bauzas) takes on the role of the i18n TC liaison, and will highlight the SIGs issues to the TC - Begin translation efforts for a subset of projects (starting with Nova) by translating content in the “docs” folders. - Apply machine translation as a first step, with periodic updates pushed to each project until Zuul automation is set up. - Clearly mark AI-assisted translations in commit messages and add a notice on documentation pages to indicate machine translation usage, in line with OpenInfra’s AI policy. - Define and document processes for core teams to review translated content, considering creating specialized review teams for translation directories. Subtopic 2: Vulnerability Management and your project We discussed the processes within the OpenStack Vulnerability Management Team (VMT) and the need for consistency/coherence in the face of evolving regulation across the world, besides our interest in maintaining good security hygiene. A key concern expressed was the need to keep core security contacts up to date, and the potential for automatic inclusion of all OpenStack projects under the VMT's purview. Adding all project teams and deliverables under the VMT will provide consistency in our security bug management process. It will also allow the VMT to formally help smaller project teams to maintain a good security hygiene. It will allow for cross-project bugs to be resolved quickly while still maintaining the tenets of embargoed disclosures. We also discussed the possibility of a Common Vulnerabilities and Exposures (CVE) authority within OpenStack. Jeremy Stanley (fungi) emphasised that as a community we must be prioritizing alignment and avoiding premature optimizations while regulations are still being drafted. There were suggestions that the VMT wanted the TC to act as a path of escalation to projects where the core security contacts were unresponsive. If we include all TC governed projects under the VMT, there were also concerns about the VMT's workload increasing. fungi clarified factors that drain the VMT's time (mainly chasing project security contacts to triage issues in a timely manner), and didn't think that adding all OpenStack projects would be a burden on the VMT. Having consistent processes may in fact reduce the VMT's workload. There is a strong desire however to grow the VMT with dedicated volunteers. So if you're reading this and are interested, please hop into OFTC's #openstack-security channel and join us. Action Items: - Project PTLs (or DPLs) must update their core-sec teams on Launchpad/Storyboard, removing inactive members where necessary (e.g., Nova to review its `nova-coresec` members) - PTLs of each project must confirm or designate a security liaison for each project, or confirm that the PTL will take on this role. - gouthamr will propose a TC resolution to include all governed projects automatically included in VMT oversight - the security-sig will continue engaging with the Open Regulatory Compliance Working Group to monitor and influence relevant regulatory developments (without immediate process changes on our end at this time) - We will also create a path of escalation for projects not actively following security processes, which may include notifying TC without detailed specifics or, as a last resort, considering project removal from OpenStack. - The VMT and OpenStack Security Team will evaluate the need and feasibility of a common CVE authority for OpenStack and assess community feedback on its potential value. Subtopic 3: Remove Postgres CI jobs from the projects We discussed whether to discontinue Postgres CI jobs across projects, with Ironic and Neutron teams already planning to do so. This stems from a lack of testing coverage and recurring errors in some projects, particularly Neutron. The group raised concerns about the need for a consistent, community-wide approach to database support, especially given the challenges of switching database backends. Current user survey data indicate that about 5% of deployments use Postgres, but maintaining its support is not feasible without dedicated resources. Action Items - Update OpenStack documentation to clearly outline the level of support and testing for each database backend, helping users understand support status before deploying. - Projects (like Ironic, Neutron) must drop their postgres CI jobs, earlier in this cycle so we can prevent unexpected CI failures in interconnected projects - Reviving postgres support will need volunteers. The TC last published a resolution in 2017 [4] explaining the state of support for postgresql. There must be a follow up to state that non-MySQL backends are not tested within the community. Subtopic 4: Migrate CI testing from Ubuntu 22.04 to Ubuntu 24.04 (this implies python3.12 support/usage) Ghanshyam Mann (gmann) is championing the goal to ensure that the advertised runtimes for 2025.1 are enforced in the community CI jobs. Base CI job changes are already underway Action Items: - gmann will share the goal progress with the community as this progresses - project teams must triage integration job failures with Python3.12/Ubuntu Noble and prioritize fixes. They may temporarily pin selective test jobs to python3.9/Ubuntu Jammy however, we expect these issues to be resolved sooner within this release cycle. === Eventlet Removal === This was a "kick-off" cross project discussion on the proposed goal to drop the usage of the "eventlet" library across OpenStack repositories [5]. The migration plan includes identifying suitable alternatives (e.g., native threads, asyncio, awaitlet) to meet different services' needs while addressing challenges around async compatibility with existing infrastructure (e.g., WSGI). Hervé Beraud (hberaud) led this discussion, and we discussed the pop-up team that has been formed to guide this transition. I'd refer you to the summary that hberaud shared on the openstack-discuss ML earlier for a more in-depth summary of further discussions [6] Action Items: - The TC and the community must review the proposal [7] for outcomes, timeline and select the "Eventlet Removal" goal as a cross-community goal - hberaud will set up working groups focused on specific migration needs: background tasks, networking/HTTP, and database interactions. Details are outlined on the Eventlet Removal Wiki [4] - Mike Bayer (zzzeek) and hberaud will work with project teams to assess where asyncio may be beneficial or problematic, and provide guidelines on selecting alternative solutions - We need volunteers from project teams to join the pop-up team and evaluate eventlet usages in the projects they maintain. Please head to OFTC's #openstack-eventlet-removal channel to participate in the discussion === Bridging the gap between community and contributing organizations === Ildiko Vancsa (ildikov) and Jeremy Stanley (fungi) brought the fourth of a series of community discussions to the TC and the community. We brainstormed ways to increase contributions from OpenInfra member organizations, improve contributor experience, and address blockers to community growth and engagement. The goals outlined are to bridge gaps between casual contributors and experienced community members, optimize review processes, and enhance visibility of contribution pathways and expectations. The challenges discussed included visibility of documentation, contributor engagement with maintainers, review delays, and difficulty navigating IRC. The brainstorming that followed proposed some ideas to try, such as contributor spotlights for maintainers (e.g., Superuser articles or video interviews). Please review the "bridging the gap" PTG etherpad for further discussions around the topic [8] Action Items: - Project PTLs must ensure we have clear documentation on patch review and escalation paths. This includes clearly highlighting the core maintainers, liaisons and PTL and ways to contact them. Teams must also list any code review rules that projects have evolved in the documentation. - Project maintainers must consider “review days” to prioritize and expedite small patches. - Develop a Matrix/Element connection guide for OpenStack rooms as an alternative to IRC promoting the guide across "so you want to contribute" pages. - Formalize guidelines around code review pitfalls === OceanBase Database === Members of the OceanBase project community had a cross-project discussion with OpenStack maintainers through this session. OceanBase was stated to be a compatible alternative to MySQL within OpenStack, without necessitating code changes in OpenStack services. The OceanBase team aims to support broader community involvement by contributing integration code for devstack and deployment tools like Kolla, OpenStack-Helm, and OpenStack-Ansible (OSA). The team learned about resource constraints in the community CI jobs, and we brainstormed approaches to support integration testing. Action Items: - OceanBase contributors will implement OceanBase support in devstack, potentially as a plugin if setup is complex. - The team will explore an OpenStack’s CI job by working with the OpenStack QA team, targeting tempest testing - The team will initiate feature requests within deployment projects (e.g., Kolla, OpenStack-Helm, OSA), and potentially collaborate with OpenStack's Large Scale SIG === Updating the OpenStack tenant policy on CI === As a community, we have been discouraging indiscriminate "recheck"s on failed CI jobs; however, our recent experiences landing interdependent CVE fixes within several repositories necessitated fighting repeated unrelated failures. The discussion was around some desire to avoid this in the future and several approaches were brainstormed. Clark Boylan (clarkb) helped the TC understand OpenStack's policy regarding "clean check" practices in the Zuul CI [9]. This policy is intended to reduce repeated gate failures and encourage debugging. It would be an anti-pattern to explore a way to bypass this. The idea of dropping this policy was discussed and the attendees overwhelmingly agreed that there were significant downsides to doing this, including the infeasibility of re-orienting each project team towards how to responsibly craft their CI to avoid pitfalls. There was a suggestion to allocate a limited “infra budget” per project, allowing them to assess re-check frequency and test requirements based on specific project stability and job complexity. There was a tangential discussion on why Zuul's philosophy of testing does not allow re-running single failed jobs, and several contributors shared their experience with CI elsewhere where this convenience came at the expense of introducing serious bugs. Currently, the community can reach out in OFTC's #opendev channel to get help merging a change despite Zuul's disagreements, de-queing a change, or re-enqueuing it on Zuul. These bypass mechanisms were deemed sufficient for the problem at hand. Action Items: - The need to recheck stems from unstable test jobs. Teams must try to move flaky tests to experimental or periodic test queues and attempt to isolate and fix issues with them. - If you spot common failure patterns (e.g., timeouts, mirror issues, OOM errors), please raise awareness to the project teams or to the Infra administrators via the openstack-discuss mailing list or the #opendev channel on OFTC - Zuul CI could use documentation that clarifies why re-running individual jobs in a failed buildset is not a supported action to manage contributor expectations. === Testing and shipping non-OSI compatible software within OpenStack binary artifacts === This discussion was an opportunity to clarify the community's stance on adding, testing, maintaining, supporting, documenting and shipping OpenStack software that pertains to components that are not licensed with an OSI-approved license [10]. A specific recent example was Masakari's Consul integration. Consul ships with a non-OSI-compliant BSL 1.1 license, and is as such incompatible with OpenStack's license policy. However, the general rule has been that the community would strive to support and test only open source solutions. While there may be integrations to proprietary or non-OSI licensed components, they will not be tested with Community Infrastructure, and in each case, a free and open source alternative must be available. In addition, if an OpenStack service supports a feature, there must be a OSI compatible implementation that the community will support and test with CI, and there can be any number of implementations for the feature concerning non-OSI components, but these must be tested by vendors or users of such integrations. Some projects (like Neutron and Cinder) have documented guidelines for this, other projects (like Nova and Manila) enforce this as an unwritten rule. This creates ambiguity around licensing and integration expectations. At the tail end, there was a suggestion to explore the feasibility of hosting an OpenStack container registry (e.g., registry.openstack.org) This could help us better manage binary artifacts and maintain container consistency across services, though permanent registry storage remains a concern. The discussion around this item was tabled. Action Items: - The TC will formally define and document a policy requiring open-source alternatives for non-OSI software integrations. We must ensure that it is applied consistently across all OpenStack projects. - In continuation of this topic, we will review and clarify how to handle existing integrations with dependencies that have changed licenses (e.g., Redis). Determine whether to seek alternatives, discontinue first-party CI testing, or require third-party CI for these integrations. That's a wrap! It was great fun seeing you all at the PTG. I look forward to working on these AIs with you! Thanks, On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] PTG Etherpad for TC discussions: https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b [2] Recordings of the TC PTG: https://www.youtube.com/playlist?list=PLhwOhbQKWT7XGjIwT0mtPpixpuY-tKYoh [3] PTG Etherpad of the D&I working group: https://etherpad.opendev.org/p/r.4f257833bbd7e0284c26f34e2bbc87c6 [4] TC Resolution on the state of testing of database systems: https://governance.openstack.org/tc/resolutions/20170613-postgresql-status.… [5] Eventlet removal wiki: https://wiki.openstack.org/wiki/Eventlet-removal [6] Eventlet removal PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [7] Eventlet removal Goal Proposal: https://review.opendev.org/c/openstack/governance/+/931254 [8] "Bridging the Gap" PTG Etherpad: https://etherpad.opendev.org/p/r.522712847f6a5a1d7bd2031566cde4e9 [9] "Clean Check" CI policy: https://docs.openstack.org/contributors/common/zuul-status.html#why-do-chan… [10] OSI Approved Licenses: https://opensource.org/licenses

2 1

[watcher] re-starting a weekly irc meeting
by Marios Andreou 18 Nov '24

18 Nov '24

Hello OpenStack o/, As discussed at PTG [1] and since there are now a few of us looking to contribute to watcher, it seems like a good time to re-start a regular irc meeting in #openstack-watcher for plans, reviews or any other relevant topics. It seems that Tuesday or Thursday work well for most of the current/new contributors. I started a doodle poll with these two options there https://doodle.com/meeting/participate/id/aMoRKp5d to see if there is any clear preference between the two. I provisionally propose 1200 UTC to (try) and accommodate different time zones (obviously there is no perfect solution for this). If you are interested but the time and/or days do not work for you please reply here so we can try and work it out (If there is enough interest we may consider something like alternate times but first lets see if there is a preference or problem with the current proposal), thanks! marios [1] https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L1

2 1

[Glance] Survey: Property protection file
by Cyril Roelandt 18 Nov '24

18 Nov '24

Hello, Glance allows admins to define custom rules regarding CRUD (create/read/update/delete) operations on custom image properties[1]. This is done by setting the property_protection_file option in glance-api.conf to the path to an INI file defining such permissions. We believe this feature has not seen a lot of use, and are thinking about removing it. We would like to know what you think about this. We set up an online form[2] that you can fill to help us understand real-world use cases for this feature. 1) Are you using a property protection file? 2) Are you using roles, polices, or both for property protection? (Or "None" if you do not use this feature) 3) Can you give examples of config files you use? 4) Would it be an issue for you if Glance were to remove the feature? Only the role-base protection? Only the policy-based protection? On behalf of the Glance team, Cyril [1] https://docs.openstack.org/glance/latest/admin/property-protections.html [2] https://framaforms.org/survey-glance-property-protection-file-1731944193

1 0

[neutron][requirements] Lock failures within Neutron with Eventlet 0.37.0
by Herve Beraud 18 Nov '24

18 Nov '24

Hey Neutrinos, Mid-September we released Eventlet 0.37.0 [1]. Then Jens diligently proposed a patch [2] to bump our global requirements to this newest version of Eventlet. Unfortunately this requirement upgrade is stalled by the Neutron cross job [3]. There are plenty of tests failing with lock errors in threading context (`RuntimeError: cannot notify on un-acquired lock`, please see the logs [3]). A new patch [4] changed the way we manage RLocks and threading at runtime. This patch Upgrade RLocks as the last thing we do. Green existing locks after patching modules, since patching modules might involve imports that create new locks. Neutron tests might potentially need some adjustment to properly work with this new Eventlet version. The support of Python 3.13 is coming and it includes this behavioural change, so if we want the Python 3.13 support of Eventlet in OpenStack, we need first to adjust our OpenStack deliverables to prepare the ground. Can you please check this issue with your Neutron expert eyes. Feel free to also take a look at the changelog of the version 0.37.0 of Eventlet [5], but I think the other patches are irrelevant to this context and can be safely ignored as they are unrelated to locks or threading. To finish Masakari and Mistral also failed but the errors seem a bit different than the Neutron ones. In all the cases Masakri and Mistral cross jobs are non-voting jobs. Thanks for reading. [1] https://github.com/eventlet/eventlet/issues/979 [2] https://review.opendev.org/c/openstack/requirements/+/933257 [3] https://zuul.opendev.org/t/openstack/build/e399c76765f34bfda62e2125b80caf84… [4] https://github.com/eventlet/eventlet/commit/06ec82896ebb9a26edaf6e1ad4d6339… [5] https://github.com/eventlet/eventlet/issues/979 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

2 2

[neutron] Bug deputy - week of Nov 11th
by Sławek Kapłoński 18 Nov '24

18 Nov '24

Hi, I was Neutron bug deputy last week. Here's summary of what new bugs were opened during that period: ## Critical - https://bugs.launchpad.net/neutron/+bug/2088423 - [functional test] test_ovn_nb_sync* tests fails randomly with Internal Server Error - needs assignment, - https://bugs.launchpad.net/neutron/+bug/2088285 - [neutron-tempest-plugin] neutron-dynamic-routing using os-ken Quagga are failing - assignent to Lajos, fix already proposed, we have https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935366 and https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/935368 ## High - https://bugs.launchpad.net/neutron/+bug/2088166 - os-ken ``DockerImage`` class failing during the image creation - assigned to Rodolfo and patch is proposed already, - https://bugs.launchpad.net/neutron/+bug/2088286 - external port created by neutron-vpnaas with ovn driver is not fully configured - needs assignment, this seems to be bug related to the `ovn-bgp-agent` and `vpnaas` ## Medium - https://bugs.launchpad.net/neutron/+bug/2088154 - ``ProcessManager.disable`` failing if the process doesn't exist - assigned to Rodolfo and patch is proposed already, ## Low - https://bugs.launchpad.net/neutron/+bug/2087909 - When network log exists, neutron-ovn-db-sync-util has a wrong log message - needs assignment, ## Whishlist - https://bugs.launchpad.net/neutron/+bug/2087863 - fullstack job doesn't collect keepalived and other backend logs - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087939 - [eventlet-deprecation] Remove the usage of eventlet in the OVS agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087941 - [eventlet-deprecation] Remove the usage of eventlet in the OVN metadata agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087942 - [eventlet-deprecation] Remove the usage of eventlet in the OVN agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087943 - [eventlet-deprecation] Remove the usage of eventlet in the L3 agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087944 - [eventlet-deprecation] Remove the usage of eventlet in the DHCP agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087946 - [eventlet-deprecation] Remove the usage of eventlet in the SR-IOV agent - needs assignment - https://bugs.launchpad.net/neutron/+bug/2087947 - [eventlet-deprecation] Remove the usage of eventlet in the Linux Bridge agent - needs assignment, - https://bugs.launchpad.net/neutron/+bug/2087953 - [eventlet-deprecation] Remove the usage of eventlet in the Neutron API - needs assignment There is bunch of the "eventlet-deprecation" bugs reported this week. I marked all of them with the tag "eventlet-deprecation" so that all can be easily found: https://bugs.launchpad.net/neutron/+bugs?field.tag=eventlet-deprecation -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[neutron] external port created by neutron-vpnaas with ovn driver is not fully configured
by przemyslaw.kuczynski＠ochk.pl 18 Nov '24

18 Nov '24

Guys can You help with https://bugs.launchpad.net/neutron/+bug/2088286 Hello We are deploying OpenStack/Neturon with OVN, BGP (ovn-bgp-agent) and VPNaaS and we encountered a problem - because of incomplete configuration of VPN services external port bgp agent isn't announcing it's address. Summary: external port created by neutron-vpnaas with ovn plugin/driver is not fully/properly configured and because of this ovn-bgp-agent won't announce/publish/configure routing path for this address. $ /var/lib/kolla/venv/bin/neutron-ovn-vpn-agent --version neutron-ovn-vpn-agent 24.0.2.dev42 $ openstack network show public1 +---------------------------+--------------------------------------+ | Field | Value | +---------------------------+--------------------------------------+ | admin_state_up | UP | | availability_zone_hints | | | availability_zones | | | created_at | 2024-09-06T15:48:18Z | | description | | | dns_domain | None | | id | ad8c81c1-08fd-4503-833f-912675d1c6d8 | | ipv4_address_scope | None | | ipv6_address_scope | None | | is_default | False | | is_vlan_transparent | None | | mtu | 1500 | | name | public1 | | port_security_enabled | True | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | provider:network_type | vlan | | provider:physical_network | physnet1 | | provider:segmentation_id | 2 | | qos_policy_id | None | | revision_number | 2 | | router:external | External | | segments | None | | shared | True | | status | ACTIVE | | subnets | 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 | | tags | | | tenant_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | updated_at | 2024-09-06T15:48:19Z | +---------------------------+--------------------------------------+ $ openstack subnet show 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 +----------------------+--------------------------------------+ | Field | Value | +----------------------+--------------------------------------+ | allocation_pools | 203.0.113.226-203.0.113.254 | | cidr | 203.0.113.224/27 | | created_at | 2024-09-06T15:48:19Z | | description | | | dns_nameservers | | | dns_publish_fixed_ip | None | | enable_dhcp | False | | gateway_ip | 203.0.113.225 | | host_routes | | | id | 306ea02d-a5ec-4c1b-bd2f-bff1a88750d6 | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | public1 | | network_id | ad8c81c1-08fd-4503-833f-912675d1c6d8 | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | | tags | | | updated_at | 2024-09-06T15:48:19Z | +----------------------+--------------------------------------+ $ openstack port list --network ad8c81c1-08fd-4503-833f-912675d1c6d8 --long +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | 1dc2b248-1d21-44ff-923f-2405f9f28f4e | | fa:16:3e:3e:97:10 | ip_address='203.0.113.228', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | N/A | None | network:floatingip | | | 71a1d9eb-11bb-4278-979b-6a1d83e87ecc | | fa:16:3e:15:5e:b5 | ip_address='203.0.113.229', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | 7398b975-552a-408f-8289-25e52d5cb8fc | | fa:16:3e:e7:dd:5d | ip_address='203.0.113.252', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | ccbb42e0-3258-4ade-a5e6-c873ca0530b7 | | fa:16:3e:97:9d:d7 | | DOWN | None | network:distributed | | | e6a43606-2ea3-4967-9aa2-967c800cbdbe | | fa:16:3e:4e:e9:d8 | ip_address='203.0.113.243', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ $ openstack vpn service create --router router_vpn test_vpn_service +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | Description | | | Flavor | None | | ID | 0d8436f0-6cc5-4af5-84f7-10b54ac99784 | | Name | test_vpn_service | | Project | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | Router | 30e448d5-078e-40fe-9418-901f8195b6cb | | State | True | | Status | PENDING_CREATE | | Subnet | None | | external_v4_ip | 203.0.113.246 | | external_v6_ip | None | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | +----------------+--------------------------------------+ $ openstack vpn ipsec site connection create --peer-id 1.2.3.4 --peer-address 1.2.3.4 --psk 1234 --vpnservice test_vpn_service --ikepolicy ikepolicy --ipsecpolicy ipsecpolicy test_vpn_tunnel --peer-endpoint-group west-peer-epg --local-endpoint-group local_network +--------------------------+----------------------------------------------------+ | Field | Value | +--------------------------+----------------------------------------------------+ | Authentication Algorithm | psk | | Description | | | ID | ca7f63dc-685f-4f9d-bc23-47c7b5b1c577 | | IKE Policy | 1457dc12-c1ec-4574-8985-9e93dcf06f56 | | IPSec Policy | c68d172f-220f-455c-b46c-b9ff8b9e46e4 | | Initiator | bi-directional | | Local Endpoint Group ID | 0f7f645c-8ae9-4dc8-a286-40afabbc2dd7 | | Local ID | | | MTU | 1500 | | Name | test_vpn_tunnel | | Peer Address | 1.2.3.4 | | Peer CIDRs | | | Peer Endpoint Group ID | 842cc6e8-d124-4258-b9d8-8b901d06cd97 | | Peer ID | 1.2.3.4 | | Pre-shared Key | 1234 | | Project | 0cfa2dc8d9024b7fa0462a9be5d8b832 | | Route Mode | static | | State | True | | Status | PENDING_CREATE | | VPN Service | 0d8436f0-6cc5-4af5-84f7-10b54ac99784 | | dpd | {'action': 'hold', 'interval': 30, 'timeout': 120} | | project_id | 0cfa2dc8d9024b7fa0462a9be5d8b832 | +--------------------------+----------------------------------------------------+ $ openstack port list --network ad8c81c1-08fd-4503-833f-912675d1c6d8 --long +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | Security Groups | Device Owner | Tags | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ | 1dc2b248-1d21-44ff-923f-2405f9f28f4e | | fa:16:3e:3e:97:10 | ip_address='203.0.113.228', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | N/A | None | network:floatingip | | | 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 | vpn-gw-30e448d5-078e-40fe-9418-901f8195b6cb | fa:16:3e:e5:e0:69 | ip_address='203.0.113.246', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:vpn_router_gateway | | | 71a1d9eb-11bb-4278-979b-6a1d83e87ecc | | fa:16:3e:15:5e:b5 | ip_address='203.0.113.229', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | 7398b975-552a-408f-8289-25e52d5cb8fc | | fa:16:3e:e7:dd:5d | ip_address='203.0.113.252', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | | ccbb42e0-3258-4ade-a5e6-c873ca0530b7 | | fa:16:3e:97:9d:d7 | | DOWN | None | network:distributed | | | e6a43606-2ea3-4967-9aa2-967c800cbdbe | | fa:16:3e:4e:e9:d8 | ip_address='203.0.113.243', subnet_id='306ea02d-a5ec-4c1b-bd2f-bff1a88750d6' | ACTIVE | None | network:router_gateway | | +--------------------------------------+---------------------------------------------+-------------------+------------------------------------------------------------------------------+--------+-----------------+----------------------------+------+ routing table on gateway/network node: 203.0.113.228 dev vrf1d63891f-e7 scope link 203.0.113.243 dev vrf1d63891f-e7 scope link 203.0.113.252 dev vrf1d63891f-e7 scope link 203.0.113.229 dev vrf1d63891f-e7 scope link $ ovn-nbctl lsp-list `ovn-nbctl ls-list | grep ad8c81c1-08fd-4503-833f-912675d1c6d8 | cut -f1 -d\ ` | while read a b; do echo -n "$b "; ovn-nbctl lsp-get-addresses $a; done (3d2dbb2a-b004-4955-a4ca-4d4a88d2f702) unknown (71a1d9eb-11bb-4278-979b-6a1d83e87ecc) fa:16:3e:15:5e:b5 203.0.113.229/27 (7398b975-552a-408f-8289-25e52d5cb8fc) fa:16:3e:e7:dd:5d 203.0.113.252/27 (ccbb42e0-3258-4ade-a5e6-c873ca0530b7) fa:16:3e:97:9d:d7 (e6a43606-2ea3-4967-9aa2-967c800cbdbe) fa:16:3e:4e:e9:d8 203.0.113.243/27 (provnet-5aa931a9-ac56-4144-ab7d-c61819a46c2a) unknown 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 (vpn-gw-30e448d5-078e-40fe-9418-901f8195b6cb) stands out from other ports, has no addresses $ ovn-nbctl lsp-set-addresses 3d2dbb2a-b004-4955-a4ca-4d4a88d2f702 "fa:16:3e:e5:e0:69 203.0.113.246/27" $ ovn-nbctl lsp-list `ovn-nbctl ls-list | grep ad8c81c1-08fd-4503-833f-912675d1c6d8 | cut -f1 -d\ ` | while read a b; do echo -n "$b "; ovn-nbctl lsp-get-addresses $a; done (3d2dbb2a-b004-4955-a4ca-4d4a88d2f702) fa:16:3e:e5:e0:69 203.0.113.246/27 (71a1d9eb-11bb-4278-979b-6a1d83e87ecc) fa:16:3e:15:5e:b5 203.0.113.229/27 (7398b975-552a-408f-8289-25e52d5cb8fc) fa:16:3e:e7:dd:5d 203.0.113.252/27 (ccbb42e0-3258-4ade-a5e6-c873ca0530b7) fa:16:3e:97:9d:d7 (e6a43606-2ea3-4967-9aa2-967c800cbdbe) fa:16:3e:4e:e9:d8 203.0.113.243/27 (provnet-5aa931a9-ac56-4144-ab7d-c61819a46c2a) unknown routing table after: 203.0.113.228 dev vrf1d63891f-e7 scope link 203.0.113.243 dev vrf1d63891f-e7 scope link 203.0.113.246 dev vrf1d63891f-e7 scope link 203.0.113.252 dev vrf1d63891f-e7 scope link 203.0.113.229 dev vrf1d63891f-e7 scope link And now it's possible for tunnel to connect. I tried to identify code of ovn driver/plugin in neutron-vpnaas responsible for external port allocation, but I wasn't able to find it. --

1 0

[ironic] Having issue with deploying ISO image to node
by James Leong 18 Nov '24

18 Nov '24

Hi all, I am trying to play around with ironic, and I am having an issue with deploying an Ubuntu 20.04 ISO image onto my bare metal node. Since I am quite new to bare metal, I am not entirely sure what went wrong as I couldn't find any error messages shown on my log file (ironic-conductor, ironic-api, ironic_neutron_agent, and nova-compute-ironic). This might be my guess; the ironic inspection is hitting timeout, which is causing the deployment to timeout. However, I have no idea why the inspection is failing at timeout. Any help would be appreciated. Best, James

6 7

[openstack-helm] [Cinder] Cinder-Scheduler oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start
by daniel890723＠gmail.com 18 Nov '24

18 Nov '24

My Enviroment: | OS | Ubuntu 22.04 | | Kernel | 5.15.0-87-generic | | kubernetes | v1.30.3 | (tested with both v1.28.14 and v1.30.3) Hello I'm testing with Openstack-helm the installation works fine but when i trying to create volume from image and when creating Cinder-Scheduler shows error and the log output: Cinder-Scheduler log: 2024-11-18 01:13:21.651 1 ERROR cinder.scheduler.filter_scheduler [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Error scheduling e838be3b-80a2-4c6c-98cc-53cc84527236 from last vol-service: cinder-volume-worker@rbd1#rbd1 : ['Traceback (most recent call last):\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/taskflow/engines/action_engine/executor.py", line 52, in _execute_task\n result = task.execute(**arguments)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 1241, in execute\n model_update = self._create_from_image(context,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/utils.py", line 669, in _wrapper\n return r(f, *args, **kwargs)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 379, in __call__\n do = self.iter(retry_state=retry_state)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 314, in iter\n return fut.result()\n', ' File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result\n return self.__get_result()\n', ' File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result\n raise self._exception\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/tenacity/__init__.py", line 382, in __call__\n result = fn(*args, **kwargs)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 1115, in _create_from_image\n model_update, cloned = self._prepare_image_cache_entry(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/decorator.py", line 232, in fun\n return caller(func, *(extras + args), **kw)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/coordination.py", line 239, in _synchronized\n return f(*a, **k)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 894, in _prepare_image_cache_entry\n model_update = self._create_from_image_cache_or_download(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/volume/flows/manager/create_volume.py", line 978, in _create_from_image_cache_or_download\n with image_utils.TemporaryImages.fetch(\n', ' File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__\n return next(self.gen)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 1375, in fetch\n fetch_verify_image(context, image_service, image_id, tmp)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 861, in fetch_verify_image\n data = get_qemu_data(image_id, has_meta, format_raw,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 700, in get_qemu_data\n data = qemu_img_info(dest,\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/cinder/image/image_utils.py", line 169, in qemu_img_info\n format_name = cinder.privsep.format_inspector.get_format_if_safe(\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 269, in _wrap\n self.start()\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/priv_context.py", line 283, in start\n channel = daemon.RootwrapClientChannel(context=self)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 374, in __init__\n super(RootwrapClientChannel, self).__init__(sock, context)\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 187, in __init__\n self.exchange_ping()\n', ' File "/var/lib/openstack/lib/python3.10/site-packages/oslo_privsep/daemon.py", line 201, in exchange_ping\n raise FailedToDropPrivileges(msg)\n', 'oslo_privsep.daemon.FailedToDropPrivileges: Privsep daemon failed to start\n'] 2024-11-18 01:13:21.652 1 INFO cinder.message.api [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Creating message record for request_id = req-d4455f4e-a374-4251-ab04-7c3523809b6f 2024-11-18 01:13:21.667 1 ERROR cinder.scheduler.flows.create_volume [None req-d4455f4e-a374-4251-ab04-7c3523809b6f d7a3b766f6274ea6b83d6f4b28639b6e 5fc62074ca69400f8192bf00941616d5 - - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid backend was found. Exceeded max scheduling attempts 3 for resource e838be3b-80a2-4c6c-98cc-53cc84527236: cinder.exception.NoValidBackend: No valid backend was found. Exceeded max scheduling attempts 3 for resource e838be3b-80a2-4c6c-98cc-53cc84527236 I also tried several way to test by set SYS_ADMIN on Cinder_scheduler pod change the version to 2023.2 but non of the above method works Is this because the pod permission or there is something i need to set when deploy using openstack-helm ? P.S : i can create Empty Volume and mount to instance

1 0

[FOSDEM] Reminder about call for participation - Virtualization and Cloud infrastructure Room 2025
by Piotr Kliczewski 17 Nov '24

17 Nov '24

All, This yeah submission deadline is enforced for all devrooms so this year our due date is on 1st of December not as all previous years on 8th. This is our gentle reminder to submit [0] your proposals. See you soon at FOSDEM! [0] https://fosdem.org/submit

1 0

[ironic] PTG Summary and Workitems for 2025.1
by Jay Faulkner 15 Nov '24

15 Nov '24

Hi all, The Ironic team, as usual, has whittled our PTG notes down into a document describing the items we'll be focusing on this cycle. The usual disclaimer applies: this is open source, and we're talking about what we /want/ to do, not making any promises about what /will/ be done. The document can be found at https://specs.openstack.org/openstack/ironic-specs/priorities/2025-1-workit…. I hesitate to include this in the same email; because that ^ document is semi-official and this is not; but I also made a short video describing these priorities if you prefer that medium; https://www.youtube.com/watch?v=oy25mATgbrY Thanks to everyone in the Ironic community for an awesome PTG, and a special thanks to our PTL Riccardo for doing an excellent job organizing and moderating it. Thanks, Jay Faulkner

1 0

[ec2-api][kuryr][murano][sahara][senlin][solum][winstackers] Move 2023.1 Antelope and older branches to End of Life
by Elõd Illés 15 Nov '24

15 Nov '24

Hi, The projects in $SUBJECT are retired projects without any activity or maintenance for some time now. That is why instead of moving their stable/2023.1 (antelope) branch to Unmaintained we plan to directly move them to End of Life. To be able to do that we also have to move older branches to End of Life and delete them too. We don't expect any objection, but if someone wants to step up and bring these projects back to life, then this is the last chance before we move them to End of Life. Thanks, Előd Illés irc: elodilles @ #openstack-release

2 1

[release] Release countdown for week R-19, Nov 18 - 22
by Elõd Illés 15 Nov '24

15 Nov '24

Development Focus ----------------- We are now past the Epoxy-1 milestone. Teams should now be focused on feature development and completion of goals [0]. [0] https://governance.openstack.org/tc/goals/selected/index.html General Information ------------------- Our next milestone in this development cycle will be Epoxy-2, on January 9th, 2025. This milestone is when we freeze the list of deliverables that will be included in the 2025.1 Epoxy final release, so if you plan to introduce new deliverables in this release, please propose a change to add an empty deliverable file in the deliverables/epoxy directory of the openstack/releases repository. Now is also generally a good time to look at bugfixes that were introduced in the master branch that might make sense to be backported and released in a stable release. If you have any question around the OpenStack release process, feel free to ask on this mailing-list or on the #openstack-release channel on IRC. Upcoming Deadlines & Dates -------------------------- Epoxy-2 Milestone: January 9th, 2025 2025.1 Epoxy final release: April 2nd, 2025 Előd Illés irc: elodilles @ #openstack-release

1 0

CLI or Steps to find installed OpenStack version
by Manohar Kundapura (makundap) 15 Nov '24

15 Nov '24

Hi All, Looking for the steps to find out which release/version of OpenStack from the below link, is installed on a Server. https://releases.openstack.org/ Thanks, Manohar

3 2

[horizon] SCSS compilers
by Radomir Dopieralski 15 Nov '24

15 Nov '24

The openstack dashboard uses scss for styling, and so far we have used a python scss library for compiling those files to css for use in the browser. However, that library (pyscss) had in recent years some problems, and is not developed or supported by the authors anymore. So it's time to switch to something with a better support. The django-compressor plugin that we use for handling the compilation of files allows us to use any command-line tool to do the compilation, so we can switch to an scss compiler that is supported and used by the wider community. However, there are several options we can choose from: sassc python-libsass pysassc ruby-sass ruby-sassc node-node-sass Each of those would introduce a non-python dependency on the dashboard. The question for the packagers and distro maintainer is which of those tools would be preferable in terms of support, packaging, and maintenance for the long term? Do you have opinions? Maybe there is another tool that would be even better that I didn't find?

2 3

[neutron] Drivers meeting agenda for Friday, November 15th
by Brian Haley 15 Nov '24

15 Nov '24

Hello Neutrinos: Sorry for the late notice but this is the agenda for tomorrow's drivers meeting: * Configurable Agent Termination on OVS Restart * https://bugs.launchpad.net/neutron/+bug/2086776 * There has been good discussion in this bug from Liu * L3 Agent Readiness Status for HA Routers * https://bugs.launchpad.net/neutron/+bug/2086794 * OVS Agent Synchronization Status for Container Environments * https://bugs.launchpad.net/neutron/+bug/2086799 - related to 2086794 * Add OVN Ganesha Agent to setup path for VM connectivity to NFS-Ganesha * https://bugs.launchpad.net/neutron/+bug/2087541 Unfortunately I will not be able to attend due to a last-minute conflict, so if someone else could lead the meeting I would appreciate it, otherwise we will postpone until next Friday. -Brian

1 0

[watcher] unmaintained branch cleanup
by Sean Mooney 14 Nov '24

14 Nov '24

Hi everyone, As some may now we are currently trying to revive the maintenance of the watcher project, i have volunteered to be the release liaison for the project. At the ptg a few weeks ago we briefly discussed the state of the unmaintained and stable branches watcher currently had 5 unmaintained branches from the oldest unmaintained/victoria <https://opendev.org/openstack/watcher/src/branch/unmaintained/victoria> to the newly created unmaintained/2023.1 <https://opendev.org/openstack/watcher/src/branch/unmaintained/2023.1> https://opendev.org/openstack/watcher/branches since the unmaintained branches have been create, they have not received any meaningful patches, including in some cases the auto generated patches from our openstack patch bots. during the ptg i proposed moving all unmaintained branches older than 2023.1 to end of life. recently https://review.opendev.org/c/openstack/releases/+/935041 has been proposed to "Transition zed-eom branches to EOL" which aligned to my proposed plan for watcher. Per the ptg discussion i committed to reaching out via the mailing list to see if there was any objection to this plan https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L79 please respond here or on https://review.opendev.org/c/openstack/releases/+/935041 if you have an objection to the unmaintained branch transition. regards sean

1 0

[nova] mdev management for vgpu
by Mickael Razzouk 14 Nov '24

14 Nov '24

Hello, I am tying a few things with vGPU on nova and I struggle with the mdev management Nova does. My environment: - Nvidia A2 pGPU - openstack caracal What I did : - installed the nvidia grid host driver on the compute - enabled the virtual function of the pGPU - added in nova.conf the mdev type wanted and the pci address of the virtual functions - restarted nova - created a flavor with the property resources:VGPU=1 (basicaly followed this documentation : https://docs.openstack.org/nova/latest/admin/virtual-gpu.html) the only diference is that I did not put the pGPU pci address in the nova.conf but the address of its virtual functions, so in my case 0000:41:00.4 and not 0000:41:00.0 (the RP were not detected otherwise) From my understanding of this specs sheet (https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/a…) and some experimentation, When a vgpu is requested by the end user : - Nova looks at the mdevs already running and use them if available. - If there is no mdev available, Nova looks at the RP tree to find mdev capable pci devices that have the specific mdev type available. - If there is such pci device available, Nova create the mdev with a UUID and create the domain XML with this UUID inside. - If no such device is available, Nova returns no host available as an error. My questions are : - Mdev are not persistent across reboot, but when it happens, nova crash at boot because the mdevs are missing, just recreating mdev does not fix the issue as they need to have the same UUID. One could fix the issue by making mdev persistent using a tool like mdevctl, but by creating the mdev the first time I seem to me that Nova tries to manage mdev, is it normal for it to require manual intervention afterward ? Or am I missing something ? - In the nova configuration, is it normal to diverge from the documentation and enter the VF PCI addresses to detect the RF ? Thank you in advance for your help.

4 4

[nova] Spec review day this Thursday 14 Nov 2024
by Sylvain Bauza 14 Nov '24

14 Nov '24

Hey folks, As agreed at the PTG, we'll have a spec review day this week on Thursday. Please mark your calendars for that day as we'll do a first round of spec reviews. Thanks all, -Sylvain

1 1

[OpenInfra Days NA] Recording/Slides from Workshop From zero to code
by Kaster, Jörn 13 Nov '24

13 Nov '24

Hello together, I have seen that on the OpenInfra Days NA there was a Session named 'From zero to code: A hands-on guide to making your first OpenStack code contribution' [1]. Unfortunately I can't attend the OpenInfraDays NA. Now the open question is the following: Is there any Recording/Slides or other Material available? Thank you for any information. [1]https://oidiu2024.sched.com/event/1fH6M/from-zero-to-code-a-hands-on-guid… Mit freundlichen Grüßen / Best Regards i.A. Jörn Kaster Cloud Platform Engineer Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. EPG – Ehrhardt Partner Group EPX Ehrhardt + Partner Xtended GmbH Alte Römerstraße 3 56154 Boppard-Buchholz Germany Phone: (+49) 67 42 / 87 27 0 Fax: (+49) 67 42 / 87 27 50 E-Mail: info(a)epg.com Internet: www.epg.com __________________________________________ CEO: Marco Ehrhardt, Markus Derksen Commercial register Koblenz HRB 22546 Registered office: Boppard

2 1

[ironic] [unmaintained] Retire Bifrost UM? Retire Victoria?
by Jay Faulkner 13 Nov '24

13 Nov '24

Hi all, At the PTG, Ironic cores discussed the state of unmaintained branches in Ironic, this email serves as documentation of this conversation as well as a call to action for anyone who uses (some) unmaintained projects. During this discussion, it was very clear that we believe two things are ready for being fully end-of-life'd: - Bifrost; all unmaintained branches. Bifrost is more of a deployment utility than a service; it takes considerable effort to keep it working past the normal maintenance period. It's the consensus of Ironic cores that all current unmaintained branches for bifrost should be closed, and we should avoid opening them in the future. - Victoria: all projects. There are no longer Ironic core reviewers with vested interest in keeping victoria releases of Ironic (and related projects under BM governance) around anymore. This decision is also informed by the significant effort required to get patches tested against victoria during the recent Ironic OSSAs/CVEs. Since traditionally we have handled our own unmaintained branches, we're making a call for anyone who'd like to keep this alive -- including the current unmaintenance team. This is awkward, because if these were branches Ironic fully controlled we would simply end-of-life them. However, since the idea is for these branches to be able to be maintained without the core teams' interaction, we were going to make this call out. If you have interest in keeping maintenance for these branches, please let us know -- otherwise we will begin efforts to end of life them. Thanks, Jay Faulkner

3 2

[election][all][tc] A call for election officials for 2025.2
by Goutham Pacha Ravi 12 Nov '24

12 Nov '24

Hello Stackers, I'd like to express my gratitude to Ian Y Choi (ianychoi) and Slawek Kaplonski (slaweq) for their service as OpenStack's election officials for the 2025.1 term. A hallmark of our community has been its leadership and the guiding principles [1] we use when defining them. In Feb-Mar 2025, we're going to be electing the next set of OpenStack Project Team Leads as well as 5 members of the OpenStack Technical Committee. We've run a combined election for PTLs and TC seats in the past few years. We envision doing the same for this upcoming election. Most of the election process is automated, and election officials help execute and oversee this process [2]. If you're interested in being an election official, please reach out to me. If you've previously served as an election official, please do not hesitate to reach out. If we have enough volunteers, you may do us a great favor by mentoring those doing this for the first time. Thank you, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] https://governance.openstack.org/tc/reference/principles.html [2] https://governance.openstack.org/election/process.html

1 0

[OpenStack-Ansible] Major Upgrade from Yoga to Zed/Antelope
by Roby G 12 Nov '24

12 Nov '24

Hello everyone, Current environment: - Ubuntu 20.04 - OpenStack-Ansible Yoga - no LXC, services are installed bare metal - `install_method` is `distro` My goal is to upgrade this environment to Zed, or skip Zed and upgrade directly to Antelope, since Zed is also in Unmaintained state. What would you recommend? Which order should I follow - upgrade operating system or OpenStack first? The decision is a bit difficult as all 3 releases (Yoga, Zed and Antelope) support different Ubuntu versions in install_method distro and not the same one.

2 1

[kolla-ansible]: guidance needed for neutron setup
by turkey automation 12 Nov '24

12 Nov '24

Hey folks, I am new to Openstack and want to try it on KVM setup with 1 controller and 2 compute VMs. In my first attempt, I was able to install Openstack using kolla-ansible(2024.1 version) and also run-initonce completed without any issues. Now, I want to create instances that can access my back-bone i.e. KVM virtual network(192.168.121.x/24) but either floating IP concept or extending my KVM network to openstack(I don't know if it's feasible!) Some info about my infra: a. Controller 1: Interface eth0 has 192.168.121.5 IP address. eth1 has no IP. b. compute 1: Interface eth0: 192.168.121.10, eth1 has no IP. c. compute 2: interface eth0: 192.168.121.15, eth1 has no IP. What configuration do I need to enable in globals.yml and what all I need to do post deployment of openstack to ensure I have virtual instances on openstack that have IPs from 192.168.121.x network(dhcp can be controlled by KVM network?) and are accessible from other KVM based VMs that I created outside of openstack. I am a learner to please be elaborate if possible. Thanks in advance to the community. - Best regards!

3 6

Upstream Oslo IRC Meetings Are Back.
by Daniel Bengtsson 12 Nov '24

12 Nov '24

Hello everyone, I hope you are all doing well! I sent an email a while ago to reestablish the Upstream Oslo IRC meetings. With the removal of Eventlet, this makes even more sense. I have received two positive responses, and we also mentioned it during the PTG last week. I propose to hold these meetings every other Monday, starting today, at 4:00 PM UTC. Does this work for everyone? Thank you for your feedback! -- Daniel Bengtsson Software engineer

6 18

Oslo Meeting day and agenda.
by Daniel Bengtsson 12 Nov '24

12 Nov '24

Hi everyone, I hope you’re doing well! As a reminder, we’ve chosen the day and time for the Oslo meetings. They will take place every Wednesday with the following schedule: • Even weeks: Europe/US • Odd weeks: Europe/Asia I’ve proposed a patch[1] to update the meeting times. We’ll also have a meeting tomorrow at 13:00 UTC. The topic for tomorrow will be the proposed removal of Eventlet[2] in Oslo service. To help track our meetings, I’ve created an Etherpad document[3], which you can view here: Oslo Etherpad. I used the release structure as inspiration, keeping only the essentials for Oslo. Thanks, and see you tomorrow! [1] https://review.opendev.org/c/opendev/irc-meetings/+/934648 [2] https://review.opendev.org/c/openstack/oslo-specs/+/927503 [3] https://etherpad.opendev.org/p/epoxy-oslo-meeting-tracking -- Daniel Bengtsson Software engineer

1 0

#eventlet-removal - OFTC IRC meeting time slot poll - Express your choice!
by Herve Beraud 12 Nov '24

12 Nov '24

Hey Stackers, We propose to schedule a one hour OFTC IRC meeting (#opentack-eventlet-removal) each first and third week of each month. Let's find together the right UTC time slot for our one hour OFTC IRC meeting. Once the vote is done, meetings will be scheduled each first and third <selected time slot in the poll> of each month. Please, vote here: https://framadate.org/rUghO3j6ZkRhKD7m Votes will be considered as closed by Nov. 8th First meeting will be forced to 4pm UTC on Tuesday Nov 5th and then moved to the selected time slot for all the following meetings. See the proposed time slots in the poll link above. Join us on the #opentack-eventlet-removal OFTC IRC channel. Thanks in advance for your votes -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 2

OpenStack Cinder - Volume-ID in cinder DB
by Harsh Ailani 12 Nov '24

12 Nov '24

Hello contributors, I am working on clone operation for a volumegroup and in that the volume ID auto-generated by OpenStack is not the one with which the clone volumes are being created on my IBM storage backend. This is as per design that IBM Storage backend will take the source-volume ID in the source-volumegroup and add a suffix to it while cloning. Now since there is a mismatch in the IDs between cinder and storage backend, because of which I am unable to perform any volume operations on this cloned volume. So, I was wondering if I can change the volume ID in cinder DB? I know that it’s a unique key and there is a limit of 36 characters for this field in cinder DB, but since I cannot change the volume in storage backend, I wanted to check if there is a way I can alter the volume ID in cinder DB? Kindly share your thoughts/suggestions on this matter. Thanks and Regards, Harsh Ailani

2 4

[OpenInfra Days NA] Recording/Slides from Workshop From zero to code
by Kaster, Jörn 12 Nov '24

12 Nov '24

Hello together, I have seen that on the OpenInfra Days NA there was a Session named 'From zero to code: A hands-on guide to making your first OpenStack code contribution' [1]. Unfortunately I can't attend the OpenInfraDays NA. Now the open question is the following: Is there any Recording/Slides or other Material available? Thank you for any information. Mit freundlichen Grüßen / Best Regards i.A. Jörn Kaster Cloud Platform Engineer Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you. EPG – Ehrhardt Partner Group [EPG_Logo_withClaim_rgb_transparent-background.png]<https://www.epg.com/de/> __________________________________________ EPX Ehrhardt + Partner Xtended GmbH Alte Römerstraße 3 56154 Boppard-Buchholz Germany Phone: (+49) 67 42 / 87 27 0 Fax: (+49) 67 42 / 87 27 50 E-Mail: info(a)epg.com Internet: www.epg.com __________________________________________ CEO: Marco Ehrhardt, Markus Derksen Commercial register Koblenz HRB 22546 Registered office: Boppard

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-20)
by Goutham Pacha Ravi 12 Nov '24

12 Nov '24

Hello Stackers, This week is milestone-1 of the 2025.1 ("Epoxy") release cycle [1]. We've been hard at work implementing the action items from the recent Virtual Project Teams Gathering (PTG) [2]. This past week, we didn’t pursue any new governance changes, though several remain in draft as follow-ups from the PTG. We’re also excited about the extended deadline for the Call for Proposals for the OpenInfra Days event at SCaLE 2025. The new deadline is Nov 15, 2024 [3]. === Weekly Meeting === Last week, the Technical Committee met concurrently on video [4] and IRC [5]. Allison Price (aprice) and Jimmy McArthur (jimmymcarthur) from the OpenInfra VMware Migration Working Group [6] joined us to discuss the group's charter and activities so far. The working group’s goal is not only to address gaps and workarounds but also to influence the future development of OpenStack software. The group has published a whitepaper [7] for operators planning to migrate cloud workloads. The TC expressed interest in collaboration between the working group and the current maintainers of the VMware hypervisor driver in OpenStack Nova. We also highlighted low contributor numbers in several critical project teams mentioned in the whitepaper, including those maintaining OpenStack Compute High Availability (Masakari), OpenStack Storage Backup and Restore (Freezer), OpenStack Resource Optimization (Watcher), and OpenStack Database Service (Trove). The working group plans to hold regular meetings soon and will share details via this mailing list. The TC reviewed the maintenance status of the OpenStack documentation styling extension for Python Sphinx, openstackdocstheme. This library is outdated, as it depends on older versions, but there aren’t enough maintainers to review changes, which could impact documentation across all projects. Ideally, maintainers of this library should have expertise in Sphinx and JavaScript, making the TC a less-than-ideal group to support it. We encourage migrating documentation away from this custom tool, as Sphinx styling extensions have advanced and may now suffice without heavy customization. While there’s no immediate migration plan, we’re looking for volunteers to lead this effort. In the meantime, we’ll raise concerns with the Oslo project team about the lack of reviews for openstackdocstheme changes. The next meeting of the OpenStack Technical Committee is today (2024-11-12) on OFTC's #openstack-tc channel. You can find the agenda on the Meeting wiki [8]. I hope you'll be able to join us. === Governance Proposals === ==== Open for Review ==== Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] Superuser compilation of PTG summaries: https://superuser.openinfra.dev/articles/october-2024-openinfra-ptg-summary/ [3] OpenInfra Days CFP: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/me… [4] TC Meeting Video Recording, 2024-11-05: https://youtu.be/o_w3OutGEfM [5] TC Meeting IRC Log 2024-11-05: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-05-18.04.log.html [6] OpenStack VMware Migration Group: https://www.openstack.org/vmware-migration-to-openstack/ [7] VMware Migration to OpenStack White Paper: https://www.openstack.org/vmware-migration-to-openstack-white-paper [8] TC Meeting Agenda, 2024-11-12: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

[openstack-ansible] Victoria - enable neutron plugin Subnet_dns_publish_fixed_ip
by Rambo Rambo 11 Nov '24

11 Nov '24

Hi Team I am running openstack-ansible Victoria release and would need to activate automatic dns records creation when a new instance is created. So, I am looking for the way to enable neutron plugin Subnet_dns_publish_fixed_ip. After which I can enable dns_publish_fixed_ip in the subnet properties Regards Rambo

2 3

[telemetry] Event Pipeline Config and Use Cases
by Fernandes Figueiredo, Alex 11 Nov '24

11 Nov '24

Hello Openstack Telemetry Team, I'm currently working on re-enabling the Openstack Telemetry services for the StarlingX Openstack (STX-O) project. Openstack telemetry was integrated into STX-O long time ago, but it has been disabled for the past few years. Since the last time telemetry was used by STX-O, it seems that important updates happened, including: * Ceilometer API deprecation * Ceilometer Collector deprecation * Panko (event storage) deprecation My questions are mainly related to the last one (Panko deprecation): * Since Panko is deprecated and there is no actual replacement for event storage [1], is there any other possible telemetry use case to justify keeping the event pipeline enabled and generating data? * Are there any aodh alarming mechanisms that relies on events generated by Ceilometer event pipelines? If so, which "sinks.publishers" should be used to ensure the integration between the Ceilometer event pipeline and the aodh alarm service? * If event pipeline is enabled, what are the publishers currently supported for events? * The current event pipeline configuration (event_pipeline.yaml) in STX-O defines the "sinks.publishers" as "gnocchi://". However, based on Ceilometer doc, it looks like Gnocchi only supports samples/meters, it doesn't support events. Is my interpretation of the documentation correct or does Gnocchi support events and can be really used as a publisher for the event pipeline? [1] OpenInfra summit Vancouver 23 - Openstack Observability revamped<https://youtu.be/AhQJEBR5BUg?si=JbBuLG2iaaDZW7Iu&t=1590> Thank you in advance for any possible help! I'm looking forward to getting Openstack telemetry up and running on StarlingX Openstack. Best regards, Alex Figueirêdo.

3 4

[nova][neutron] Persistent PCI topology for network interfaces
by jan＠uponu.com 11 Nov '24

11 Nov '24

Hi, we have come came across the issue that some appliances rely on the PCI topology to assign network interfaces and not the mac address. This is the a problem, because the PCI topology changes, when an instance (the libvirt config) is recreated. Is there a way to make the network interface order persistent in the PCI topology and if not can someone give me a hint on the steps that would need to be done in nova? I think this could be an important feature in general. Thanks Jan ------ E-Mail: jan(a)uponu.com

2 1

[neutron] Bug deputy report (week starting on November 4)
by Elvira Garcia Ruiz 11 Nov '24

11 Nov '24

Hi folks! Here you have the list of bugs reported for Neutron during last week (04 Nov - 10 Nov) Many new RFEs this week. High ------ - https://bugs.launchpad.net/neutron/+bug/2086740 Deadlock when metadata agent starts Fix by Jakub https://review.opendev.org/c/openstack/neutron/+/934188 [Merged] - https://bugs.launchpad.net/neutron/+bug/2087824 random router related test failures since wsgi switch Unassigned Medium ----------- - https://bugs.launchpad.net/neutron/+bug/2087822 When security group rules have the same normalized CIDR, neutron-ovn-db-sync-util has a wrong log message. Unassigned - https://bugs.launchpad.net/neutron/+bug/2086787 Neutron fails with postgres on agents Unassigned Low ------ - https://bugs.launchpad.net/neutron/+bug/208678 ovn_db_revision_check is not required anymore Patch proposed by Sharat Sharma https://review.opendev.org/c/openstack/neutron/+/934241 Wishlist ---------- - https://bugs.launchpad.net/neutron/+bug/2086776 [RFE] Configurable Agent Termination on OVS Restart Unassigned - Reporter has updated the RFE description - https://bugs.launchpad.net/neutron/+bug/2086794 [RFE] L3 Agent Readiness Status for HA Routers Unassigned - https://bugs.launchpad.net/neutron/+bug/2086799 [RFE] OVS Agent Synchronization Status for Container Environments Unassigned - https://bugs.launchpad.net/neutron/+bug/2087541 [RFE] Add OVN Ganesha Agent to setup path for VM connectivity to NFS-Ganesha Assigned to Amir Nikpour Best regards, Elvira

1 0

[openstacksdk] Flakey unit test makes assumptions about the speed of API results
by Michael Still 09 Nov '24

09 Nov '24

So I've spent a day or so debugging an occasionally flakey unit test that I see in the py3 test suite for openstacksdk. I'm at the point that I want to check that I actually understand what it's doing before proposing a fix. Specifically, openstack.tests.unit.test_stats.TestStats.test_servers_no_detail periodically fails to find a statsd value: self.assert_reported_stat( 'openstack.api.compute.GET.servers.200', value='0', kind='ms' ) This assertion is effectively saying that a GET to the servers compute API should return a HTTP 200, and take 0 milliseconds to execute. The unit test is failing because sometimes it doesn't take 0 milliseconds. In my debugging for example, it takes 4 milliseconds in my environment every now and then: Considering key openstack.api.compute.GET.servers.200, value 4.000000|ms Value 4.000000, kind ms Value 0 != 4.000000 Am I understanding this correctly? It seems to me that perhaps we should be asserting the call took less than a threshold (5 milliseconds) instead of just insisting it always be instantaneous? Michael

1 0

[neutron][ptg] 2025.1 Epoxy PTG summary
by Brian Haley 08 Nov '24

08 Nov '24

Hi all, First, sorry for taking so long to get this out, was traveling the week after the PTG. Second, thanks for all that attended the meetings, we had a good turnout for Neutron, the Ironic cross-team and some eventlet discussions that were important to the team. For complete notes see the etherpad [0] but a summary is below. And please comment if you feel I missed anything. Thank you, -Brian ### Dalmatian retrospective ### Everyone felt we were able to improve our CI stability, despite a few spikes during the cycle. We were also able to improve the OVN parity gap with ML2/OVS, and were able to move OVN to supporting WSGI (thanks Rodolfo!) Lack of developers and reviewers continues to be an issue, and we have struggled to reduce our bug backlog significantly. ### Eventlet deprecation ### We have started to identify and document changes needed to deprecate eventlet [1]. We will start creating bugs to track all this work. ### Spring cleanup ### At the beginning of every cycle, we always try and cleanup the code base, for example, removing deprecated and dead code, doc updates, job consolidation. We have made progress on those already. We had a discussion on two items that have been marked experimental in the code - Linuxbridge support and IPv6 prefix delegation - neither of which is well maintained or tested in our CI. I will send a follow-up on next steps for these in the coming week. ### Code base modernization ### As python 3.9 is now the minimal version, we are going through the steps of running pyupgrade/autopep8 on both the neutron and neutron-lib trees to adopt any new conventions. In the process we've been fixing a number of pylint warnings that is still ongoing. The mypy job is now voting, and we have started adding type hints to the code (thanks Ihar!). ### Ironic cross-project ### The high-level discussion point was on Neutron better supporting baremetal (of course). There are some issues with trunk ports as well as external network Single Source of Truth (to help better integrate Neutron with DC networking). An RFE will be proposed for either creating a pre-commit hook to help validate a VNI is okay to use, or to be able to modify the VNI range via the API, as it is currently fixed in the conf file. The other topic was around implementing an additional DHCP backend, as dnsmasq has not been actively maintained, and there was at least one issue Ironic was seeing with it crashing. The consensus was to start supporting Kea, an RFE will be created. See etherpad for further information. ### ML2/OVS future ### As two of the major distros have moved to make OVN their default, there has been less of an emphasis on ML2/OVS development over the past few cycles, and as such, there are an increasing number of bugs in the backlog. There are still a large number of clouds running ML2/OVS as seen in the 2024 User Survey [2], so there is some risk there. As a team we wanted to make sure we communicated this concern, and also wanted to make it known that users doing new deployments, or migrating from older ones (eg Linuxbridge) strongly consider using OVN. We will attempt to give any help we can in this area as migration and deployment issues arise. ### Distributed metadata ### This has been ongoing work from Liu to better support distributed metadata in ML2/OVS, similar to what is done in OVN. Patches have been updated accordingly after discussion. ### Performance/scale testing ### Some recent bugs have shown there are some places where Neutron has DB scaling issues. Any we've found so far have been fixed, but there is a need to do some targeted testing to see if there are others. We also discussed changing our rally testing to try and better detect such issues. My take away was we should be vigilant when any DB change is proposed to make sure it doesn't regress things, and to start looking at places we don't use 'selectin' to see if a simple change would help. ### Gate stability ### Similar to the weekly Neutron CI meeting, we discusses issues in the gate, from things related to the WSGI switch, scenario and fullstack failures. Please see the etherpad for specific bug links. ### RFE discussions ### On-demand topic to discuss any in-progress or proposed RFE changes, please see etherpad for more information. [0] https://etherpad.opendev.org/p/oct2024-ptg-neutron [1] https://etherpad.opendev.org/p/neutron-eventlet-deprecation [2] https://www.openstack.org/analytics

1 0

[ironic/neutron] Using neutron to bind a specific port on an ironic node
by Michael Sherman 08 Nov '24

08 Nov '24

So, my question is: Is there a way for an end-user, launching an instance with nova, to specify which ironic port neutron should use? For context, a subset of our ironic nodes have relatively heterogenous network interfaces, but are configured as being attached to the same physical networks. Examples include nodes with both Broadcom and Mellanox 25g interfaces attached to the same switch, but also nodes with both 1g, 25g, and 100g NICs attached to different switches, but with the same vlans available. In our configuration we’re using Neutron with openvswitch, networking-baremetal, and networking-generic-switch. From initial observations, it seems like if we *don’t* specify, neutron allocates ports in the same order that they’re returned from “baremetal port list”. What we’d like to be able to do is let our users send some API call to bind neutron port X based on some property of the ironic port, ether UUID, tags for speed or vendor, or something else we’re able to define, but it’s not clear to me what information is even available to neutron to use in say, a binding profile? Any pointers would be appreciated. Thanks! -Mike Sherman

1 0

[release] Release countdown for week R-20
by Thierry Carrez 08 Nov '24

08 Nov '24

Development Focus ----------------- The Epoxy-1 milestone is next week, on November 14th! Project team plans for the 2025.1 Epoxy cycle should now be solidified. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library which had changes but has not been otherwise released since the 2024.2 Dalmatian release. PTL's or release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well, by posting a -1. If we do not hear anything at all by the end of the week, we will assume things are OK to proceed. NB: If one of your libraries is still releasing 0.x versions, start thinking about when it will be appropriate to do a 1.0 version. The version number does signal the state, real or perceived, of the library, so we strongly encourage going to a full major version once things are in a good and usable state. Upcoming Deadlines & Dates -------------------------- Epoxy-1 milestone: November 14th, 2024 2025.1 Epoxy final release: April 2nd, 2025 -- Thierry Carrez (ttx)

1 0

[kolla][OSA][release] 2024.2 Dalmatian cycle-trailing release deadline
by Elõd Illés 08 Nov '24

08 Nov '24

Hello teams with deliverables following the cycle-trailing release model! This is just a reminder to wrap up your trailing deliverables for 2024.2 Dalmatian. The release deadline is *December 5th, 2024.* [1]. If things are ready sooner than that though, all the better for our downstream consumers. Some of them have been delivered already, so let's list here the ones that still need a release before the deadline: kayobe openstack-ansible-roles openstack-ansible [1] https://releases.openstack.org/epoxy/schedule.html#e-cycle-trail Thanks in advance, Előd Illés irc: elodilles @ #openstack-release

1 0

[all] CFP for OpenInfra Days North America Open for One More week!
by Jeremy Stanley 07 Nov '24

07 Nov '24

For those of you who may not be subscribed to the OpenInfra Foundation mailing list[*], just a heads up that we have another week until the OpenInfra Days North America planning committee will begin reviewing talks, so the CFP deadline has been extended to November 15. If you want to present in Pasadena, California this March, get your proposal in ASAP![**] [*] https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/me… [**] https://forms.gle/81QV3SFV6ixSdsBq8 -- Jeremy Stanley

1 0

[neutron] Drivers meeting canceled Friday, November 8th
by Brian Haley 07 Nov '24

07 Nov '24

Hello Neutrinos: I'm going to cancel tomorrow's drivers meeting as there is no agenda and I will be offline. I will triage incoming ones next week so plan to meet next Friday. Have a nice weekend! -Brian

1 0

[Cinder][Ceph] Volumetria errada no cluster ceph
by lfsilva＠binario.cloud 07 Nov '24

07 Nov '24

Hello, I'm experiencing an issue with the volume usage of the Ceph cluster, which is currently used in OpenStack for volumes. I am working with Ceph version Octopus (15.2.17). When I run the ceph df command, I get this output: --- RAW STORAGE --- CLASS SIZE AVAIL USED RAW USED %RAW USED hdd 120 TiB 33 TiB 87 TiB 87 TiB 72.72 TOTAL 120 TiB 33 TiB 87 TiB 87 TiB 72.72 --- POOLS --- POOL ID PGS STORED OBJECTS USED %USED MAX AVAIL device_health_metrics 1 32 69 MiB 101 137 MiB 0 11 TiB images 2 256 1.3 TiB 166.90k 2.5 TiB 10.72 11 TiB vms 3 64 574 KiB 21 2.5 MiB 0 11 TiB volumes 4 2048 41 TiB 3.94M 82 TiB 79.42 11 TiB backups 5 1024 407 GiB 111.85k 818 GiB 3.63 11 TiB And when I run the rbd du -p volumes command, I get this total: NAME PROVISIONED USED <TOTAL> 14 TiB 8.3 TiB This "volumes" pool is currently set to replica 2, and mirroring is not enabled. I have checked for any locked or deleted snapshots but found none. I also ran a ceph osd pool deep-scrub volumes, but it didn't resolve the issue. Has anyone encountered this problem before? Could someone provide assistance?

3 21

[kolla] Vacation - kevko
by Michal Arbet 07 Nov '24

07 Nov '24

Hello, I would like to inform you that I will be on vacation from November 7th, 2024, to November 23rd, 2024. Thank you, Michal Arbet Michal Arbet Openstack Engineer Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic +420 604 228 897 michal.arbet(a)ultimum.io *https://ultimum.io <https://ultimum.io/>* LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter <https://twitter.com/ultimumtech> | Facebook <https://www.facebook.com/ultimumtechnologies/timeline>

1 0

Pointers to FOSDEM devrooms CfPs
by Felix Kronlage-Dammers 07 Nov '24

07 Nov '24

Hi, in the last couple weeks the Call for Participations for the various FOSDEM[1] devrooms have been published. In case not everyone knows FOSDEM: FOSDEM is _the_ yearly European free and open source software event where people from around the globe meet and discuss. Aside from the main track FOSDEM consists mainly of many devrooms which focus on certain topics. I wanted to quickly highlight a few of the relevant devrooms for the OpenStack audience and would like to encourage to submit talks :) An overview of the devrooms can be seen here: https://fosdem.org/2025/schedule/ The CfP for the room is linked from the details page of each devroom. Virtualization and Cloud Infrastructure: https://lists.fosdem.org/pipermail/fosdem/2024q4/003574.html Confidential Computing: https://falder.org/fosdem25-cfp Software Defined Storage: https://lists.fosdem.org/pipermail/fosdem/2024q4/003563.html Monitoring and Observability: https://docs.google.com/document/d/1KQmNqzN3cO29ZZUSpd83REvJY_QhbvxIYr4IRBr… The SCS projects organizes, together with the german ZenDiS and the french DINUM, a room with the topic ‘Government Collaboration’. As part of the room we would love to receive proposals that cover: - Standardization and certification of digital sovereign solutions and infrastructure - Automating deployment and operations of such solutions on sovereign infrastructure - Examples and success stories of digital sovereign cloud computing by the public sector The CfP for this room is here: https://lists.fosdem.org/pipermail/fosdem/2024q4/003608.html Am sure there are more relevant devrooms :) An easy overview is also the mailing list archive: https://lists.fosdem.org/pipermail/fosdem/2024q4/thread.html regards felix [1] <url: https://fosdem.org/2025/>

1 0

Question about Package Inclusion and User Creation in Kolla Base Image
by Amir Hossein Ahmadi 06 Nov '24

06 Nov '24

Hello OpenStack Community, I'm currently reading through the OpenStack Kolla source code, and I noticed a few design choices in the kolla/base image that I’d like to understand better: - The kolla/base image includes a number of packages (such as Apache) that don’t appear to be necessary for most containers. - Additionally, user and group accounts for all services are created in the base image, even though they aren’t used by all containers. Could someone explain the reasoning behind including these components in the base image? Is there any documentation available that provides insight into the design approach for Kolla images? Thank you in advance for your help and guidance. Best regards, Amir Hossein, Burna innovation Bridge

1 0

[publiccloud-sig] Reminder: next meeting November 6th (tomorrow) - 0800 UTC
by Felix Kronlage-Dammers 05 Nov '24

05 Nov '24

Hi everyone, This Wednesday, tomorrow, (November 6th) the next meeting of the publiccloud-sig is going to happen. We’re going to do a quick re-cap from PTG as well as the ALASCA Summit that happened last week. Furthermore I’d like to see wether we can spark some activity around the discoverability service that we’ve been thinking and discussing about in the past. We’ll be meeting at 0800 UTC here: https://meetpad.opendev.org/publiccloud-sig <https://meetpad.opendev.org/publiccloud-sig> See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG <https://wiki.openstack.org/wiki/PublicCloudSIG> See you tomorrow! felix -- Felix Kronlage-Dammers Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

debugging eventlet services in vscode.
by Sean Mooney 05 Nov '24

05 Nov '24

hi folks i recently had to debug a problematic interaction between eventlet and a native thread in watcher. while i probably could have printed my way though that eventually i decided to try and get debugging working in vscode instead. in this case i happened to be looking at watcher but this should apply to any eventlet service installed with devstack. in my .vscode directoy i created a launch.json that looks like this. { // Use IntelliSense to learn about possible attributes. // Hover to view descriptions of existing attributes. // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387 "version": "0.2.0", "configurations": [ { "name": "Python Debugger: watcher-applier", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-applier", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } }, { "name": "Python Debugger: watcher-decision-engine", "type": "debugpy", "request": "launch", "program": "/opt/stack/data/venv/bin/watcher-decision-engine", "console": "integratedTerminal", "args": [ "--config-file", "/etc/watcher/watcher.conf" ], "gevent": true, "env" : { "PYTEST_CURRENT_TEST": "true" } } ] } there are a few odd things about this config. first the program i am launching is the console script run in the devstack@* systemd service file. you can find what that is using systemctl show devstack@<name> to make this work without conflict do `sudo systemctl stop devstack@<name>` first. after your done debugging you can just start the systemd service again. second when using vscode to view the source of a project select the devstack global virtual env as the project virtual env. third im telling vscode to enable gevent support. now this will raise a warning/error as gevent is not installed but you can ignore that. for those of you not familiar with gevent its a fork of eventlet that actually has much better integration with debuggers due to its wider usage outside of openstack. eventlet and gevent both use greenlet and the eventlet implementation is close enough that when you enable gevent debugging it mostly works. forth to fix the "mostly" part, specifically the fact that dajngo will explode on import, we set "PYTEST_CURRENT_TEST": "true" in the terminal env. PYTEST_CURRENT_TEST is an undocumented environment variable intended only for unit testing that is check by eventlet https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… when we set that to true we skip this `isinstance(o, rlock_type)` check https://github.com/eventlet/eventlet/blob/8637820f468268ffb0b8504561ea4772d… which crashes the debugpy debugger because it tries to load setting form the danjgo module which get loaded into the python interpreter as a side effect of debugpy. i have not tried this on other project yet but it enables me to single step debug, run to breakpoints and many of the normal debug action like using the debug console to inspect the current function scope. all of the above worked without any code change to watcher or running in a specal debug mode in the service code. we occasionally get questions on this list about how to debug openstack services and i usually respond with "its possible to do, is often fragile and need a lot of context to do right" in this case it seam to work reasonable ok in the limited usage i have made of it so far. given the current incantation is pretty simple and non invasive i decided to share this in case others find it useful. keep in mind however most service are distributed systems so if you leave it stopped on a break point for 5 mins while you inspect things, what ever you were originally doing will probably time out and it might fail in odd ways but for local development that may still be fine. please don't try this in production. i have not tried this with a wsgi app. that probably will work fine with the eventlet wsgi web-server via the console scripts i.e. nova-api binary but probably wont work for uwsig/Apache. regards sean

2 1

[cinder][PTG] 2025.1 Epoxy PTG summary
by Jon Bernard 05 Nov '24

05 Nov '24

Hello everyone, thanks for a very productive PTG for the Epoxy cycle. Below is a brief summary of the highlights. The raw etherpad notes are here [1] and both the summary and transcribed etherpad notes are captured in our wiki here [2]. # Retrospective We spent the first few minutes discussing the state the Dalmatian release, what went well, not so well, and how we can improve in the Epoxy release cycle. The most notable point raised is that we have two less cores this cycle. This adds additional strain to an already sizable review queue. We will be looking to cultivate and elevate any qualified contributors if the opportunity arises. # NetApp We covered 6 different topics related to NetApp's feature development plans for the Epoxy cycle in an effort to make sure we're all aligned. Below is a summary of the full set: * Certificate-based authentication Right now only basic auth is implemented, which requires a username and password. There is a strong desire to add support for certificate-based auth. Discussions covered possible approaches and a few helpful links to existing implementations that can be used as a reference. * HA Active-Active support for NVMe driver This will allow users to configure NVMe driver backends in cinder clustered environments. We covered the failover process and which methods needed to be implemented. The related use cases were discussed as well as testing scenarios. We do not have replication and failover coverage in our upstream gate, so special care is needed in testing and verification on the developer side at the moment. Mention of Rally for testing as a possibility. * SVM scoping support for NVMe Desire to add support to the NVMe driver to SVM scoping. At the moment the driver fails to initialize. Work is ongoing to add full support for this feature. Nothing concerning was raised, looking forward to this for Epoxy. * ASA r2 support ASA r2 systems use storage units and consistency groups to simplify storage management and data protection. The REST API is different, and some features are restricted. We'll see how this unfolds in this cycle, good to have this on the radar as we go forward. * Replication - Snapmirror Sync Support Synchronous mirroring achieves a recovery point objective (RPO) of zero lost data by having a copy of important data available if a disaster happens on one of the two storage arrays. The copy is identical to production data at every moment because each time a write is done to the primary volume, a write is done to the secondary volume. The host does not receive an acknowledgment that the write was successful until the secondary volume is successfully updated with the changes that were made on the primary volume. * Metrocluster Support Metro cluster is a clustering solution with synchronous replication to deliver continuous availability, immediately duplicating all of the mission-critical data on a transaction-by-transaction basis. MetroCluster configurations enhance the built-in high availability and non disruptive operations of NetApp hardware and ONTAP storage software, providing an additional layer of protection for the entire storage and host environment. A few changes beyond driver-specific updates were discussed, no blockers. # Handling MYPY Updates We support mypy in our gate jobs, but having the version unpinned causes occasional breakages. We pin hacking and similar tools, and we agreed that mypy should be included. The caveat being we need to remember to periodically unpin and update these version numbers to stay current. # New volume driver LVM+Clone Jan Horstmann proposed a derivative of our LVM driver to add support for using device mapper clone target for transparently migrating volumes. This would all an admin to leverage local storage for performance and still be able to live migrate instances across compute hosts. A POC/Demo is hopefully coming soon, no fundamental blockers were identified, very interested to see how this comes together. # Reviews for backup service It is very difficult to get reviews for cinder-backup related changes. Tobias asked if any cores can focus some time on cinder-backup specifically (an potential backports). We agreed that mention of the backup review queue and review dashboard at the weekly meeting may help raise awareness. # New Location API adoption Rajat proposed a patch that adds the call for the `add_image_location` method which triggers the new location API workflow that is more secure and robust than the old location workflow. This call is made when glance is using cinder as a backend and we want to perform an optimized "upload volume to image" operation. Additional patches are up that add support for testing this feature. We may still need some input from the Glance team, to be followed up on in our weekly meeting. # Adding `service` query parameter to AZ API Tobias proposed an API change, adding a query parameter to an existing API. Specifically, adding `service` query parameter to os-availability-zone API. The patch is thought to be complete and ready for review. # Adding a backup summary API Tobias proposed adding a backup summary API (same way that is available for volumes). He also found a volume summary API bug when implementing this feature. The patch is ready for review. # Horizon Feature Gap We had the privilege to meet with the Horizon team and discuss their efforts on closing existing feature gaps. Several specifics were discussed and clarity was achieved. Concerns were raised over possible incompatibility issues between the openstack client and the SDK. Further, Horizon updates microversion only when necessary and value was identified in reviewing microversions to see if any additional functionality can be leveraged. # Optimize RBD update from Cinder to Glance Currently the path to upload a cinder volume to glance image is not optimized when cinder and glance are both using RBD. We can leverage RBD layering (or copy-on-write cloning) to complete the operation in seconds irrespective of the volume size (since no actual copy is happening). There are 3 features implemented that were needed to support this feature: 1. RBD clone dependency deletion: We have landed changes in glance and cinder that handle the RBD dependency chains so dependency from volumes to images pools shouldn't be an issue 2. Service role: Currently the attachment delete API already uses service role + service token and we can use similar changes for this feature 3. New location APIs: This new glance feature eliminates the security risk as we will be calculating the image hash even in this optimized path (in background so our operation performance won't be affected) * Previously several blockers, appear to be all resolved now * Significant performance increase * Glance -> Cinder already supported * Fallback to unoptimized codepath * Related, recent patches to improve RBD cinder/glance, need to be considered/reviewed Eric has committed to help with the reviews. # State of volume replication Some time was spent discussing the current state of replication. We do not have testing coverage in tempest, so the current state of any particular driver implementation is difficult to know. An idea was raised to poll vendors for use cases and testing results. There is a general lack of clarity for this feature across projects and what our overall disaster recovery solution is. # Image Encryption (Glance Cross-Project) The goal with this effort is to standardize encrypted images between Glance, Cinder and Nova, in an effort to make them consistent and accessible to users. Here is a recap: 1. We merged a patchset in os-brick so far that extends key handling to support more types of Barbican keys for image encryption in addition to Cinder's existing implementation (which strictly uses binascii.hexlify() conversion). 2. Patchset for Glance has been written to add Secret Consumer API from Barbican for encrypted images and to handle the new standardized format. 3. Patchset for Cinder has been written, making it able to produce and use the new standardized format + additionally support qcow2+LUKS in addition to raw LUKS format as well as non-hexlified keys from Barbican. After the initial discussion with Glance, we agreed on *not* introducing any new container_format or disk_format and instead handle this with `os_encrypt_*` properties instead (e.g. os_encrypt_format=luks); this is how the patchsets are currently implemented. However, due to the recent CVEs around image format parsing and conversion, valid concerns were raised that this approach might lead to new vulnerabilities due to potential conflicts and ambiguities between container_format, disk_format and the encryption attributes. We need to agree on a new way of handling the classification of encrypted images in the metadata. We have at least two cases to consider: 1. Raw LUKS (as currently produced by Cinder from encrypted volumes) 2. qcow2+LUKS (as produced by Nova for Ephemeral Storage); both could also be produced by the user (see the gist link in the reference section). # Volume type metadata progress We received a progress update on the volume type metadata effort. Consensus that the direction is sound. A new API endpoint in addition to changes to the response of volume type show / create will require a new microversion. The tempest test still need to be adjusted. Patches need review. # Status of OpenAPI effort Very little traction on initial patches, needs help getting reviews for the first set of patches. Patches also cleanup technical debt and fix several bugs, would be very good to have. The next set of patches is more meaningful, but being held until initial reviews are available. # Removal of uWSGI scripts (stephenfin) The patch to remove uWSGI scripts has been up and needs review. Relatively small patch. Update: this has been reviewed and merged. # Eventlet A dedicated etherpad was created to help document current status. Cinder is still in the investigatory phase. Not only do we use eventlet directly, some drivers such as RBD use C libraries that spawn native threads. We must take care in how we handle these cases. Our goal is only preparatory changes in Epoxy (because it is a SLURP release), and hold big changes for F (which most people won't upgrade to, so will give us more time to detect & fix issues) [1] https://etherpad.opendev.org/p/epoxy-ptg-cinder [2] https://wiki.openstack.org/wiki/CinderEpoxyPTGSummary -- Jon

1 0

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-21)
by Goutham Pacha Ravi 05 Nov '24

05 Nov '24

Hello Stackers, Thanks to all moderators and volunteers for compiling virtual PTG notes and sharing summaries through this mailing list. A lot of exciting work seems to be tracked through the action items from these summaries. I hope you were able to review the TC's PTG summary [1]. Next week is milestone-1 of the 2025.1 "Epoxy" release cycle [2]. This is typically a bug-targeting milestone for project teams. The following is a summary of this past week's happenings within the OpenStack Technical Committee. === Weekly Meeting === During the past week, the OpenStack Technical Committee met over IRC [3]. We followed up on action items from weeks preceding the Project Teams Gathering. We finalized PTL and DPL appointments for the "mistral" and "watcher" project teams. We spent some time reviewing action items related to translating documentation and user strings within OpenStack services. Automated jobs posting translation updates from Zanata are broken due to incompatibility with newer Python versions. Migration to Weblate remains a priority and the i18n team may require more support to complete this migration. We're collaborating over IRC to identify the needs and next steps. The TC also discussed progress on the community goal to migrate integration testing to Python 3.12 and to the latest LTS version of the operating system used in DevStack test jobs. Like me, you may have wondered why we’re combining these efforts. The reason is that it allows us to have a simpler test matrix despite the number of Python versions and OS distributions that OpenStack software is tested with. While we maintain unit test jobs for each supported Python version, it is impractical to duplicate each integration job for every OS distribution available. Therefore, we choose the latest LTS (e.g., Ubuntu 24.04 LTS / "Noble Numbat"), which includes the upper bound of the supported Python versions (Python 3.12). Additionally, each project will have at least one test job for the minimum supported Python version (Python 3.9), so you'll see a test job for the previous LTS release of the OS that will begin in this release and continue running as it transitions to a "stable" release branch. The next meeting of the OpenStack Technical Committee is on 2024-11-05 (today) at 1800 UTC. This is our monthly video meeting too! This meeting will occur simultaneously on Video (Zoom) and IRC (OFTC's #openstack-tc channel). Allison Price (aprice) and Jimmy McArthur (jimmymcarthur) from the OpenStack VMware Migration Working Group [4] will be joining us to discuss the working group's efforts. Meeting information, including other agenda items, is on the meeting wiki [5]. I hope you'll be able to join us. Please remember that you can always propose items to the TC's weekly meeting agenda. === Governance Proposals === ==== Merged ==== Add Watcher DPL for Epoxy | https://review.opendev.org/c/openstack/governance/+/933018 Adding Axel Vanzaghi as PTL for Mistral | https://review.opendev.org/c/openstack/governance/+/927962 ==== Open for Review ==== Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/931254 === How to Contact the TC === You can reach the TC in several ways: 1. **Email:** Send an email with the tag [tc] on this mailing list. 2. **IRC:** Use the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. 3. **Weekly Meeting:** Join us every Tuesday at 1800 UTC [5]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" TC PTG Summary: https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [3] TC Meeting IRC Log 2024-10-29: https://meetings.opendev.org/meetings/tc/2024/tc.2024-10-29-18.00.log.html [4] OpenStack VMware Migration Group: https://www.openstack.org/vmware-migration-to-openstack/ [5] TC Meeting Agenda, 2024-11-05: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting

1 0

Re: GPU PCI passthrough woes.
by Andy Speagle 05 Nov '24

05 Nov '24

Aye, that seems to have done it. Strange that this wasn't necessary for my old 1080 cards. Thanks! On Wed, 2024-10-30 at 08:12 +0000, Alexander Dibbo - STFC UKRI wrote: > ⚠️CAUTION⚠️ > This email originated outside of Toyon's network. Do not download > attachments or click links unless you know the content is safe. > > Hi Andy, > > We've seen many issues with pci passthrough and I suspect that the > l40s will have a seen similar issues with the a100s > > Here is an alias string that works for our a100s: > `alias = {"vendor_id": "10de", "product_id": "20f1", "device_type": > "type-PF", "name": "nvidia-tesla-a100-pcie-vga"}` > > I believe it is the `device_type` that will be your issue > > Regards > > Alexander Dibbo – Cloud Architect / Cloud Operations Group Leader > For STFC Cloud Documentation visit > https://urldefense.us/v3/__https://stfc.atlassian.net/wiki/spaces/CLOUDKB/o… > To raise a support ticket with the cloud team please email > cloud-support(a)stfc.ac.uk > To receive notifications about the service please subscribe to our > mailing list at: > https://urldefense.us/v3/__https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=S… > To receive fast notifications or to discuss usage of the cloud please > join our Slack: > https://urldefense.us/v3/__https://stfc-cloud.slack.com/__;!!A4KLn-w15EM4BR… > > -----Original Message----- > From: Andy Speagle <aspeagle(a)toyon.com> > Sent: 30 October 2024 02:15 > To: openstack-discuss(a)lists.openstack.org > Subject: GPU PCI passthrough woes. > > Hey Folks, > > I could use a little assistance getting GPU passthrough working. I > had > this working already for one flavor of nvidia gpu... and I've added > some hosts with a much newer gpu... > > I've updated the pci_alias and pci_passthrough variables and those > seem > to be getting set properly in nova.conf > > passthrough_whitelist = [{"vendor_id":"10de", > "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] > > alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} > alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": > "10de"} > > I believe I have all of the iommu stuff configured and have the pci- > stub module entries... dmesg output shows that the GPUs are being > claimed by the stub module. > > $ openstack flavor show t1.small_gpu_l40s > +----------------------------+--------------------------------------+ > > Field | Value | > +----------------------------+--------------------------------------+ > > OS-FLV-DISABLED:disabled | False | > > OS-FLV-EXT-DATA:ephemeral | 0 | > > access_project_ids | None | > > description | None | > > disk | 0 | > > id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | > > name | t1.small_gpu_l40s | > > os-flavor-access:is_public | True | > > properties | pci_passthrough:alias='gpu-l40s:1' | > > ram | 2048 | > > rxtx_factor | 1.0 | > > swap | 0 | > > vcpus | 1 | > +----------------------------+--------------------------------------+ > > Yet... I can't seem to get an instance to run using that new > flavor... > keeps complaining about there not being enough hosts available. > > Fault: > code: 500 > created: 2024-10-30T02:04:33Z > message: "No valid host was found. There are not enough hosts > available." > details: | > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 1580, in schedule_and_build_instances > host_lists = self._schedule_instances(context, > request_specs[0], > File "/usr/lib/python3/dist- > packages/nova/conductor/manager.py", > line 940, in _schedule_instances > host_lists = self.query_client.select_destinations( > File "/usr/lib/python3/dist- > packages/nova/scheduler/client/query.py", line 41, in > select_destinations > return self.scheduler_rpcapi.select_destinations(context, > spec_obj, > File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", > line 160, in select_destinations > return cctxt.call(ctxt, 'select_destinations', **msg_args) > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/client.py", line 189, in call > result = self.transport._send( > File "/usr/lib/python3/dist- > packages/oslo_messaging/transport.py", line 123, in _send > return self._driver.send(target, ctxt, message, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send > return self._send(target, ctxt, message, wait_for_reply, > timeout, > File "/usr/lib/python3/dist- > packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send > raise result > nova.exception_Remote.NoValidHost_Remote: No valid host was > found. > There are not enough hosts available. > > Traceback (most recent call last): > File "/usr/lib/python3/dist- > packages/oslo_messaging/rpc/server.py", line 241, in inner > return func(*args, **kwargs) > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 223, in select_destinations > selections = self._select_destinations( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 250, in _select_destinations > selections = self._schedule( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 416, in _schedule > self._ensure_sufficient_hosts( > > File "/usr/lib/python3/dist- > packages/nova/scheduler/manager.py", > line 455, in _ensure_sufficient_hosts > raise exception.NoValidHost(reason=reason) > > nova.exception.NoValidHost: No valid host was found. There are > not > enough hosts available. > > Any clues on how/where to dig into this more to see what might be > missing? Thanks. > > -- > Andy Speagle > -- Andy Speagle Sr. Site Reliability Engineer Toyon Research Corporation 316.617.2431

3 2

Is HPET recommended for x86 instance?
by Jiatong Shen 05 Nov '24

05 Nov '24

Hello Community Experts, I found that to enable the HPET timer for x86 guests, the operator should set hw_time_hpet on the given image. By default, HPET is disabled. My question is what are the pros and cons for using HPET? Thanks. -- Best Regards, Jiatong Shen

2 3

[cloudbase-init] This project doesn't seem to be well managed.
by dogyun kim 04 Nov '24

04 Nov '24

Hi, I wanted to contribute to that project, so we created a CR. https://review.opendev.org/c/x/cloudbase-init/+/933629 But no one managed this project, so i can't get review. If you have an administrator, I would like to ask you to confirm it. Thanks

2 1

[cloudbase-init] This project doesn't seem to be well managed.
by 김도균 04 Nov '24

04 Nov '24

3 2

[keystone] Weekly Meeting Canceled 06-Nov-2024
by Dave Wilde 04 Nov '24

04 Nov '24

Hello all, Due to some scheduling conflicts I have to cancel this weeks Keystone meeting. Please feel free to add to the agenda [0] or reach out in #openstack-keystone if you need anything from us. Thanks, /Dave 0: https://etherpad.opendev.org/p/keystone-weekly-meeting

1 0

[neutron] Bug deputy report (week starting on October 28)
by Bernard Cafarelli 04 Nov '24

04 Nov '24

Hey neutrinos, here is the list of bugs reported during 2024 week 44 While this was Halloween week, the number of reported bugs was not too scary, and bugs have patches or assignee High * [OVN] Revision number registers must be filtered by resource ID and type - https://bugs.launchpad.net/neutron/+bug/2085946 Patch by Rodolfo https://review.opendev.org/c/openstack/neutron/+/933752 W+1 Medium * ´openstack vpn service show´ command doesnt return external_v4_ip and external_v6_ip - https://bugs.launchpad.net/python-neutronclient/+bug/2086144 CLI issue for VPNaaS, Lajos looking into it * [ovn-octavia-provider] Load balancer enable/disable has a reverse effect - https://bugs.launchpad.net/neutron/+bug/2086194 Patch proposed https://review.opendev.org/c/openstack/ovn-octavia-provider/+/933879 * [OVN] The security group create command is not creating the SG rules revision number registers - https://bugs.launchpad.net/neutron/+bug/2086205 Patch proposed https://review.opendev.org/c/openstack/neutron/+/933969 Low * [neutron-tempest-plugin] Trunk tests are running out of VLAN IDs when creating a VLAN network - https://bugs.launchpad.net/neutron/+bug/2086602 Patch proposed https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/934027 Wishlist * Reboot breaks VPN connection (OVN) - https://bugs.launchpad.net/neutron/+bug/2086189 Marked as wishlist for VPNaaS, as this is for redundancy on reboot -- Bernard Cafarelli

1 0

[openstack-ansible][ptg] Epoxy Project Team Gathering summary for OpenStack-Ansible
by Dmitriy Rabotyagov 04 Nov '24

04 Nov '24

Hi folks, Despite low participation in the PTG this time, we still had quite productive discussions around a variety of topics. You can find some summary on our future plans below: * We agreed to proceed with upgrading RabbitMQ to version 4.0 for Dalmatian. This means that HA (replicated) Queues won't be available anymore so Caracal will be the only release having support for both HA Queues and Quorum Queues. Deployments are expected to accomplish their migration to Quorum Queues (or just Classic Queues v2) on Caracal before proceeding further. * Implementing unified limits At this point of time, support for unified limits across services is not clear enough for us as well as how from operator perspective a mix of unified limits and classic quota will look like, as these 2 options are very distinct from each other. While one use case for their implementation would be vGPUs, we need an interested party for implementing them at this point. * Reviving Watcher support We have retired the Watcher role in Stein. With the project itself getting more traction these days, we agreed to proceed with reviving the role code to re-add support for the Watcher. * uWSGI and eventlet replacement A big topic throughout this PTG was replacement of eventlet with more native async approaches. However, uWSGI has no support for any of them right now. Since uWSGI is also in a maintenance mode, there is no expectations for adding async support to the project either. With that we need to start looking at alternatives, like uvicorn, which is already used in Skyline role. With that we potentially want to re-purpose the uWSGI role to be more generic at serving WSGI, meaning we will look into implementing service selection (uwsgi/gunicorn/uvicorn/etc) and renaming the role to address the more generic nature of it. * PKI role improvements This was another big topic we have discussed this time. There are multiple issues we need to address with the role. It could be split into multiple sub-topics: - Improving CA private key storage method on the deploy host to be able to store them encrypted with ansible-vault/sops or some other lookup mechanism - We need to implement better way for creating and storing private keys for generated certificates as transferring them from the deploy host is not really a best practice. - Adding support for hashicorp vault to store it in an encrypted manner. With that we should also add support to have CA in hashi-vault directly [1] and issuing certs from there. - Improve certs rollout process to allow short-term certificates to be issued and rotated regularly. This includes dynamically reloading libvirt config to address VNC consoles and reducing runtime for certificate rotation. * Would be great to have finally another round on ProxySQL efforts * Document of making user_secrets not a plain-texted file. Also address upgrade script failure in case user_secrets contain ansible-vault/sops/etc rather than plain text. * Document 2 ways of doing "pretty endpoints" - domain based and subpath based. * We need to work closely on deprecation of gather_subset starting with ansible-core 2.18, as there is no straightforward replacement. * Some repositories are missing deb822_repository migration, ie: frrouting, zun, qdrouterd [1] https://developer.hashicorp.com/vault/tutorials/secrets-management/pki-engi…

1 0

[neutron][ovn-octavia-provider][octavia][ovn] Octavia OVN DBs sync cmd
by Rico Lin 04 Nov '24

04 Nov '24

Hi all I would like to raise attention to the patch set that introduces Octavia OVN DBs sync cmd. patches: https://review.opendev.org/q/topic:%22add-ovn-db-sync-utils%22 With command `octavia-ovn-db-sync-util {--debug}` It should allows us to do the OVN DB sync similar to what `neutron-ovn-db-sync-util` does. And thanks to Fernando Royo, the patches are small and earlier to review in focused functions. Would be great if we could have some more reviews for it. *Rico Lin*

2 1

[tacker][ptg] 2025.1 Epoxy vPTG summary
by Yasufumi Ogawa 01 Nov '24

01 Nov '24

Hi tacker team, Thank you for joining the vPTG in the last week. I'd like to share the summary of our two days discussion for the proposals on the etherpad[1]. ## [recap] Revise FTs(yasufum) We have revised functional test from Bobcat and it's a recap session for looking back the remained issues. All the issues are going to be fixed in Epoxy. * Revise directory structure * Names of functions, classes and files too long for maintenance to be done this release * Combine scenarios into one to reduce the number of tests if possible. In process * Reduce setting up infra for the tests to be done this release * Revising scenarios and reducing setups are going to be started in this release. ## Plan for Improvement FT in Epoxy release (yasuum) We discussed the items about strategies for improving functional tests. To complete the goal of this proposal, we will take several steps to focus on dominant ones. (1) Divide time consuming tests, (2) Move tests less frequently updated to non-voting (3) Drop tests almost not updated from zuul (4) Rename each test scinarios and revise directory structure. We agreed with this proposal, but which of tests will be divided or dropped was not decided in the discussion. So, continue the discussion in the next IRC meeting. ## Add option to enable/disable Cinder volume recreation in v1 Heal request (koba) In the current v1 Heal request, the related Cinder Volume for the specified VDU is recreated although it can be specified not to recreate it. To support it in v1, Clients specify a flag in the additional_params to indicate whether to recreate the volume or not when making a Heal request. * Tacker conductor checks this flag to decide whether to mark the storage resources when executing STACK_UPDATE. Heal process works by recreating the resources marked "mark-unhealthy" during STACK_UPDATE. *Consider ensuring version compatibility and consistency with v2 operations. * Default behavior will be the same as now (recreating the Cinde Volume). * The name of the flag parameter in the additonal_params can be configured by the admin in tacker.conf. ## Add default_secret_key Option to [vim_keys] for Multi-Master Tacker Deployment (koba) ## Enhancement of the Ansible Driver (sample mgmt driver) (koba) Tacker currently supports only a single version of Ansible. As a result, when tenants include Playbooks in the VNF-Package that only work with specific versions of Ansible, there may be compatibility problems with the Playbooks. Tacker administrators must modify the Ansible Driver’s source code directly if they want to enforce options related to ansible-playbook execution, such as specifying log storage locations or custom callback plugins. These issues lack flexibility for both tenants and administrators. We agreed the proposed approaches for supporting several versions of ansible, but still not sure it's appropriate to define all the supported versions in tacker.conf and select which version will be applied by specifying it in the definition file in VNF package. It going to be discussed the way of selecting version in the next meeting. ## Remove legacy engine facade (yasufum) It's one of the community goal, so we'll fix it in this release. ## Remove sphinxcontrib-*diag from your (docs) dependencies (yasufum) It's similar to the previous item, and fix it in this release. ## Ensuring Tacker Compatibility with TST and SOL Specifications (Renu Rani) Agreed to the approach proposed on the etherpad. * Addressing the Gap Between TST and SOL Specifications: Raise bugs on TST and request the TST team to update schemas and variables according to SOL specifications. * Support of Compliance Tests for Tacker V2 a. Resolving Issues with Interconnected Test Cases in Tacker: Update Tacker's code to ensure compatibility with the interconnected test cases in v4.5.1-dev. b. Supporting Tacker for TST Release-4: Implement new Tacker compliance test cases for TST Release-4. c. Validating Compliance Test Cases with Tacker V2: Seek the Tacker team's opinion on whether to use Release 3 or Release 4 for validation. ## Enhancing Tacker Testing Coverage in the OpenStack Helm Check Pipeline (Shivam Shukla) To further enhance Tacker testing within the OpenStack Helm Check pipeline by adding more testcases to ensure thorough verification of Tacker deployments. Although the test scenarios were proposed, but we were not sure all the scenario are really needed because some of them are existing scenarios. We agreed to accept the proposal itself, but still not all the scenario should be introduced. So, continue to discuss for that. 1. VIM management (create, update and delete VIMs) 2. VNFD management (create and delete VNF packages) 3. VNF Instance management (create, update and delete VNF instances) 4. VNF LCM operation (instantiate, terminate, heal and scale) 5. Subscription management (create and delete subscription for VNFs) [1] https://etherpad.opendev.org/p/oct2024-ptg-tacker

1 0

[ironic] Bare Metal SIG
by Julia Kreger 01 Nov '24

01 Nov '24

Greetings folks, During this past week's PTG, the Ironic community participants reached consensus to wrap up the Bare Metal SIG. There are many reasons, but the bulk of the reasoning revolves around spreading word and collaborating outward as opposed to focusing on our own vertical of interaction. Having discussions in different contexts helps bring strength to discussions and we encourage contributors to engage in adjacent community and group discussions where relevant then build connections between communities. Communities, when working together, are always stronger when we build these sorts of connections or bonds. We hope to continue having meetups with operators as we always have, even if it may not bear the Bare Metal SIG name. As always, operators of any bare metal provisioning software are welcome to join our discussions at any time in OFTC IRC #openstack-ironic or right here on the mailing list. Among the accomplishments of the Baremetal SIG, was a whitepaper[0] devoted to the Ironic use case. Special thanks goes to Chris Hoge, previously with the Open Infra Foundation who helped drive us to the finish line on that effort. Beyond the whitepaper, and likely even harder was to bring people together in person on multiple occasions! A special thank goes to Arne Wiebalck who championed the Bare Metal SIG and helped steward several meetups including the most recent meetup at CERN in late Spring of this year. The success of that event speaks volumes about our community and the power of collaboration! Many thanks also goes to everyone who helped with the videos we have posted to youtube[1]! Without everyone pitching in, we wouldn't have been able to achieve success! -Julia, on behalf of the Ironic team. [0]: https://www.openstack.org/use-cases/bare-metal/how-ironic-delivers-abstract… [1]: https://www.youtube.com/playlist?list=PLKqaoAnDyfgoBFAjUvZGjKXQjogWZBLL_

1 0

[devstack][nova][cinder][ironic][glance][swift][neutron][all] Deprecating/removing non-uWSGI deployment mechanisms?
by Stephen Finucane 01 Nov '24

01 Nov '24

o/ tl;dr: I'd like to remove the ability to deploy services under mod_wsgi from DevStack. I'd also like to remove the ability to deploy services under eventlet, where this feature is not currently being tested, and deprecate it where it is. I took a foray through DevStack last week and realised that, depending on the service in question, there are currently 3 possible ways an API can be deployed by DevStack: * Eventlet server * Apache + uWSGI * Apache + mod_wsgi In addition, there are a number of overlapping config options that could be set to modify which of these was in use: * ENABLE_HTTPD_MOD_WSGI_SERVICES * WSGI_MODE * Service-specific options like NOVA_USE_MOD_WSGI, CINDER_USE_MOD_WSGI etc. Potential confusion aside, there are two issues I see with this. Firstly, and most obviously, we are working on eventlet removal, which will necessarily involve eventlet server removal [1]. In addition, there is a goal [2][3] to remove use of '.wsgi' scripts in favour of Python module paths. Quite a few services have migrated to this already, and those that haven't are going to need to do so by time pip 25.0 comes out (since that removes the deprecated legacy editable install flow [4] meaning these files will not be generated by pbr and DevStack deployments will start failing...you have been warned). However, mod_wsgi need .wsgi scripts. I didn't realise we cared about mod_wsgi in DevStack still before last week, so to complete the wsgi script -> module migration for services that can be deployed with mod_wsgi (e.g. Keystone), we're going to have to either (a) package a .wsgi script in DevStack or (b) remove the ability to deploy with mod_wsgi. My immediate thought on this was to simply remove all non-uWSGI deployment paths immediately and deprecate the config options for removal to force DevStack plugins to adapt. However, some services still want to test eventlet server for their API while they stabilise support for standard WSGI-based deployments (neutron), while others explicitly rely on evenlet to generate a single standalone binary (ironic, glance, ...). That means we can't just go remove everything now and must instead take a phased approach. I would therefore like to propose we do the following: * Remove the ability to deploy services under mod_wsgi from DevStack and deprecate/remove the WSGI_MODE flag. To be clear, other deployment tools can continue to use mod_wsgi but DevStack won't. * Entirely remove the ability to deploy under eventlet from services that are not currently testing this. This includes Nova and Cinder. * Invert the default for services that are testing eventlet, so that they test with uWSGI by default now and jobs will need to opt-in to eventlet rather than opt-in to uWSGI. I think this just affects Neutron. * Optionally, and likely only one Neutron is no longer testing eventlet (next cycle?), deprecate ENABLE_HTTPD_MOD_WSGI_SERVICES and {SERVICE}_USE_MOD_WSGI config options entirely and replace it with {SERVICE}_STANDALONE options like Glance has. This will affect Ironic and maybe more. I have proposed a series that will do much of this. The WIP patches indicate the things that I am less clear about, such as Swift's readiness for deploying under uWSGI. I would be grateful if folks from the individual service teams could take a look and let me know what they think: https://review.opendev.org/c/openstack/devstack/+/932193/1 Cheers, Stephen [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [2] https://review.opendev.org/c/openstack/governance/+/902807 [3] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [4] https://ichard26.github.io/blog/2024/08/whats-new-in-pip-24.2/

6 10

[openstack helm] [Cinder] [Ceph] keyring not getting decode when doing cinder-storage-init
by daniel890723＠gmail.com 01 Nov '24

01 Nov '24

Hello today i was testing to deploy openstack with helm in my kubernetes cluster and i encounter an error when trying to deploy cinder Error: UPGRADE FAILED: post-upgrade hooks failed: 1 error occurred: * job cinder-storage-init failed: BackoffLimitExceeded when i print the log of the cinder-storage-init pod it shows 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: error parsing file /etc/ceph/ceph.client.admin.keyring: error setting modifier for [client.admin] type=key val=: Malformed input [buffer:3] 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 auth: failed to load /etc/ceph/ceph.client.admin.keyring: (5) Input/output error 2024-10-30T03:22:14.303+0000 7efc1353d640 -1 monclient: keyring not found [errno 5] RADOS I/O error (error connecting to the cluster) and the keyring secret is use looks like this: apiVersion: v1 data: key: AQDtmCFnh3STCBAAGKQIyIGuZjnmX3L4mPCQXw== kind: Secret metadata: name: pvc-ceph-client-key namespace: openstack after checking secret and the rook-ceph cluster works fine i change the template and makes the pod not executing the command so i can look inside the pod and it's keyrings the keyring looks like this: [client.admin] key = �!g�tȁ�f9�_r��_ so apparently the keyring secert or when it mount into the pod got some problem and change the key into the normal one fix the problem is there a reason why it cause this ? because the secret and ceph-cluster are all works fine

2 4