November 2024 - openstack-discuss

[all][dev][ops][tc] Bridging the gap between community and contributing orgs
by Jeremy Stanley 08 Dec '25

08 Dec '25

The tl;dr on this is that I'm opening a broad community discussion on how we can, collectively through improved communication, better bring together our established contributors with casual and prospective contributors who struggle to find successful patterns of contribution, so that we might all support each other in order to benefit OpenStack as a whole. Read on for details... OpenStack's TC has requested that foundation staff, when in conversation with representatives of member organizations, encourage participation in the project and collect feedback on any related challenges those organizations encounter in their attempts to do so. In order to better understand the feedback and brainstorm achievable recommendations for the broader community[*], Community Managers on the foundation staff started to work with a small focus group of established contributors with a solid understanding of what successful patterns of contribution look like. It also came to light that established contributors experience many of the same sorts of challenges, even more so if they don't have the luxury of focusing full-time upstream, and so success often comes down to knowing how to effectively navigate those challenges. The community's goal with this exercise is to improve everyone's experience, and your help is needed to do that! The most common themes reported by those struggling to contribute are not related to tooling or workflow confusion, but instead seem to come down to basic communication challenges. Ideas so far to address these gaps include: * Review bandwidth - Ensure contributors and companies understand the importance and benefits of code review - Incentivize and credit meaningful "+1" reviews more - Understand that the volume of changes and limited reviewer bandwidth often means proposals aren't reviewed quickly (or at all), and their proponents may need some additional tenacity and engagement with the team to get their work noticed - Utilize "review dashboards" as a way to find stale changes which have fallen through the cracks * Review strategy and etiquette - Be clear about the meaning of votes on changes, especially negative ones, and set explicit timelines for things like "procedural -2" or WIP blocks when their approval needs to be delayed - Focus on comments and requests that affect whether or not the change can be merged after the fixes are submitted, rather than requesting trivial adjustments to an unsuitable change which has more fundamental problems - Reciprocate with reviews of changes from new reviewers you see leaving insubstantial reviews, demonstrating to them what deeper and more meaningful reviewing looks like * Mentoring newcomers - Established contributors with sufficient bandwidth can help mentoring newcomers, potential core reviewers and new leaders - Mentoring can happen as part of internship programs as well as by helping newcomers determined to become active contributors; both ways should be embraced and utilized What challenges are you facing? And, how would you improve them? Two weeks from now, we'll have a forum session at the OpenInfra Summit Asia in Suwon[**] where our community can refine these possible approaches and integrate others, as well as bring attention to them and promote their application. I've also proposed a similar forum session for OpenInfra Days North America in Indianapolis a month after that, where we can hopefully continue this conversation. The goal for this multi-stage effort is to improve the contribution experience for established participants, casual contributors and newcomers alike. While it's easy to jump to blaming in these situations, it will be far more productive if we can focus on the challenges and experiences that we would each like to remove or improve for ourselves and others in our community. As we find ways together to improve our efficiency, we can reduce the load on all contributors and lower the barriers for people to join and participate. [*] https://etherpad.opendev.org/p/r.2205024e55689bccb82c20c960853cb5 [**] https://2024.openinfraasia.org/a/schedule#view=calendar&title=Bridging%20th… -- Jeremy Stanley

2 14

[openstack helm] [Glance] Glance showing OSError: unable to receive chunked part when uploading image
by daniel890723＠gmail.com 08 Nov '25

08 Nov '25

Hello i'm testing using Openstack-helm to install all component but after the installation complete i can't upload the OS image through both openstack CLI or Horizon it stuck at Queueing and when i checking Glance Api log it shows 2024-11-07 07:31:19.362 7 INFO glance.api.v2.image_data [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. 2024-11-07 07:31:19.400 7 ERROR glance.api.v2.image_data [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Failed to upload image data due to internal error: OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi [None req-2bd28444-196c-42e9-94ef-18443b58310e b39b42e27f8646acad1303711f1dc8b8 e4ab1a6192b9486da698efa311086c53 - - default default] Caught error: unable to receive chunked part: OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi Traceback (most recent call last): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1302, in __call__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1345, in dispatch 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return method(*args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 415, in wrapped 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/api/v2/image_data.py", line 299, in upload 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi with excutils.save_and_reraise_exception(): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.force_reraise() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise self.value 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/api/v2/image_data.py", line 162, in upload 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/notifier.py", line 492, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi with excutils.save_and_reraise_exception(): 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.force_reraise() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise self.value 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/notifier.py", line 443, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/quota/__init__.py", line 322, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/location.py", line 596, in set_data 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/location.py", line 487, in _upload_to_store 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/multi_backend.py", line 397, in add_with_multihash 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/multi_backend.py", line 479, in store_add_to_backend_with_multihash 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/driver.py", line 294, in add_adapter 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/capabilities.py", line 176, in op_checker 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/_drivers/filesystem.py", line 764, in add 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi raise errors.get(e.errno, e) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/_drivers/filesystem.py", line 746, in add 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi for buf in utils.chunkreadable(image_file, 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance_store/common/utils.py", line 73, in chunkiter 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi chunk = fp.read(chunk_size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 294, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi result = self.data.read(i) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/utils.py", line 121, in readfn 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi result = fd.read(*args) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/format_inspector.py", line 956, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi chunk = self._source.read(size) 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi File "/var/lib/openstack/lib/python3.10/site-packages/glance/common/wsgi.py", line 1045, in read 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi data = uwsgi.chunked_read() 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi OSError: unable to receive chunked part 2024-11-07 07:31:19.426 7 ERROR glance.common.wsgi Ceph Cluster Status: NAME DATADIRHOSTPATH MONCOUNT AGE PHASE MESSAGE HEALTH EXTERNAL FSID ceph /var/lib/rook 3 82m Ready Cluster created successfully HEALTH_OK 5cf8b9bb-13e1-4c31-b47d-9e7d92c3c05d Ceph osd status: ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP META AVAIL %USE VAR PGS STATUS TYPE NAME -1 0.35156 - 360 GiB 3.3 GiB 493 MiB 0 B 2.8 GiB 357 GiB 0.91 1.00 - root default -5 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 964 MiB 119 GiB 0.92 1.01 - host instance-lu-1 2 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 964 MiB 119 GiB 0.92 1.01 195 up osd.2 -3 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 - host instance-lu-2 0 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 195 up osd.0 -7 0.11719 - 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 - host instance-lu-3 1 hdd 0.11719 1.00000 120 GiB 1.1 GiB 164 MiB 0 B 951 MiB 119 GiB 0.91 1.00 197 up osd.1 TOTAL 360 GiB 3.3 GiB 493 MiB 0 B 2.8 GiB 357 GiB 0.91 MIN/MAX VAR: 1.00/1.01 STDDEV: 0.01 now i'm able to create Empty Volume using horizon(can't create volume using default cirros image ) and i can start an Instance without volume and mount the empty volume in(Ceph also can detect when i use that volume) is there something i need to set when starting glance? because i'm following the Openstack-helm installation document to install openstack with helm

5 8

[openstack][kolla-ansible]about group node config overwrite
by Nguyễn Hữu Khôi 30 Oct '25

30 Oct '25

Hello Koller. I want to ask if we can overwrite config for a group of nodes. For example, compute01 >> compute10 will have the same nova configuration compute11 >> compute20 will have the same nova configuration. It will be very nice if we support this way. Thank you. Regards Nguyen Huu Khoi

4 7

[i18n][infra][horizon] Zanata translation job failures after Python 3.8 deprecation
by Ian Y. Choi 09 Jul '25

09 Jul '25

Hi all, During last week's PTG, the I18n SIG identified Zanata translation job failures for Horizon (and likely related plugins). The root cause appears to be Python 3.8 deprecation on Horizon [1]. Failed jobs have been observed on Zuul [2], and translation proposals were made on Gerrit [3]. Since translation sync on Horizon stable branches is functioning correctly, the error message seems to be related to Python 3.8 deprecation: ERROR: Cannot install -r doc/requirements.txt (line 3) due to conflicting package version dependencies. To address this, I'd like to raise two points: 1/ To resolve the issue, would you support updating translation sync jobs with Bionic-based Zuul jobs by either upgrading to Jammy or modifying the Bionic image to support Python >3.8? 2/ At least me from I18n SIG acknowledges that recognizing these translation job errors was delayed. Moving forward, would it be beneficial to implement periodic (e.g., monthly) reporting on translation job successes and failures? Thank you, /Ian [1] https://docs.openstack.org/releasenotes/horizon/2024.2.html#upgrade-notes [2] https://zuul.opendev.org/t/openstack/builds?job_name=propose-translation-up… [3] https://review.opendev.org/q/owner:proposal-bot+project:openstack/horizon+b…

2 3

Problems with the Placement service
by Ron Gage 25 Jun '25

25 Jun '25

5 7

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

[oslo] python-consul2 is unmaintained
by Takashi Kajinami 21 Apr '25

21 Apr '25

Hello, I recently noticed that python-consul2 library[1], which is used by the consul coordination driver in tooz, hasn't been updated for 3+ years. Looking at issues in the project, there is an indication that the library is not compatible with ACL API in consul >= 1.11[2]. There is another fork created[3] and this includes fixes to support new API endpoints. Although the driver in tooz does not directly use the feature, I think it's better to discuss our plan to mitigate risks caused by the unmaintained library. As far as I can think of, there are three options now 1. Deprecate consul driver and remove it 2. Keep using python-consul2 now. At least basic functionality is proven to work by our functional tests (Note that the tests do not cover all configuration patterns) 3. Replace python-consul2 by py-consul Before we start the overall discussions I'd like to ask a few questions to the wider audience, as our inputs. - Is anyone using the consul coordination driver now ? - Does anyone have experience with python-consul2 and/or py-consul ? Note that this discussion affects the http messaging driver[4] currently proposed, because it also requires client library to communicate with Consul. Actually I noticed the situation when I noticed that the requirement check job is failing because the library is not part of global requirements list... Thank you, Takashi Kajinami [1] https://github.com/poppyred/python-consul2 [2] https://github.com/poppyred/python-consul2/issues/42 [3] https://github.com/criteo-forks/py-consul [4] https://review.opendev.org/c/openstack/oslo.messaging/+/912499 -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 3

[octavia] Purging ERROR'ed amphora and load balancers stuck in transitional states
by Paul Browne 11 Apr '25

11 Apr '25

Hello Octavia users, For a while I've been trying to find a systematic way to remove Octavia LBs+Amphora that have become stuck in ERROR or transitional PENDING_* states ; $ openstack loadbalancer amphora list -f value | grep ERROR a2323f14-3aaa-418a-8249-111aaa9c21fe 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR MASTER 10.8.0.242 192.168.3.247 e5b236ba-e7ee-4ed7-9f58-57ce7a408489 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR BACKUP 10.8.0.190 192.168.3.247 6b556f28-93c9-49dd-b6ee-4379288e7957 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR MASTER 10.8.0.39 192.168.3.126 c669db5d-8686-4d5c-9e95-e02030b34301 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR BACKUP 10.8.0.174 192.168.3.126 $ openstack loadbalancer list -f value | grep -vi active 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe k8s-clusterapi-cluster-default-ci-6386871107-kube-upgrade-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.247 PENDING_UPDATE amphora d5e402fe-2c4b-49af-a700-532cb408cee5 k8s-clusterapi-cluster-default-ci-6386871107-latest-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.126 PENDING_DELETE amphora These resources are marked immutable and so cannot be failed over or deleted; $ openstack loadbalancer amphora failover a2323f14-3aaa-418a-8249-111aaa9c21fe Load Balancer 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe is immutable and cannot be updated. (HTTP 409) (Request-ID: req-6e66c4e8-c3d3-4549-a03c-367017c8c8b3) $ openstack loadbalancer failover 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe Invalid state PENDING_UPDATE of loadbalancer resource 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe (HTTP 409) (Request-ID: req-84b44212-e7a8-4101-a16f-18c774c0577e) The backing Nova instances for these Amphora do seem to exist and be in good working order. Is there any API way to purge these out of Octavia's service state, or would (very careful) DB hackery be required here? Many thanks, Paul Browne ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

4 6

GPU PCI passthrough woes.
by Andy Speagle 10 Apr '25

10 Apr '25

Hey Folks, I could use a little assistance getting GPU passthrough working. I had this working already for one flavor of nvidia gpu... and I've added some hosts with a much newer gpu... I've updated the pci_alias and pci_passthrough variables and those seem to be getting set properly in nova.conf passthrough_whitelist = [{"vendor_id":"10de", "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": "10de"} I believe I have all of the iommu stuff configured and have the pci- stub module entries... dmesg output shows that the GPUs are being claimed by the stub module. $ openstack flavor show t1.small_gpu_l40s +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 0 | | id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | | name | t1.small_gpu_l40s | | os-flavor-access:is_public | True | | properties | pci_passthrough:alias='gpu-l40s:1' | | ram | 2048 | | rxtx_factor | 1.0 | | swap | 0 | | vcpus | 1 | +----------------------------+--------------------------------------+ Yet... I can't seem to get an instance to run using that new flavor... keeps complaining about there not being enough hosts available. Fault: code: 500 created: 2024-10-30T02:04:33Z message: "No valid host was found. There are not enough hosts available." details: | Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1580, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 940, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist- packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist- packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Any clues on how/where to dig into this more to see what might be missing? Thanks. -- Andy Speagle

3 2

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 17 Mar '25

17 Mar '25

Hello Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [Call Trace] ./stack.sh:1126:start_keystone /opt/stack/devstack/lib/keystone:549:die [ERROR] /opt/stack/devstack/lib/keystone:549 keystone did not start Error on exit World dumping... see /opt/stack/logs/worlddump-2024-11-18-150808.txt for details ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) Thanks in advance, Ishan

3 5

[all][tc][ptl][qa] Migrating upstream CI/CD jobs to Ubuntu Noble (24.04)
by Ghanshyam Mann 14 Mar '25

14 Mar '25

Hello Everyone, As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. This is the etherpad which has the migration plan and will be used for the tracking: - https://etherpad.opendev.org/p/migrate-to-noble I have prepared the change(WIP) for base jobs migration on Noble (openstack tox jobs, devstack, and tempest) which will migrate most of the projects jobs. Tempest change[3] include dependency on openstack tox, devstack base jobs changes so testing project gate using the the tempest change will test the complete set. Refer to the etherpad for the complete migration plan/steps. * Deadline: Nov 29th 2024 (R18) is the deadline to test/fix the projects jobs and to merge the base jobs running on Noble (openstck tox, devstack, and tempest). [1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubunt… [3] https://review.opendev.org/c/openstack/tempest/+/932156 -gmann

2 9

[kolla] [nova] API flags ignored in Wallaby
by Albert Braden 12 Mar '25

12 Mar '25

After rebuilding our Train clusters on Wallaby we are seeing some API issues. It appears that flags that work in Train are still documented in Wallaby, but are ignored. Do we have something setup wrong, or are we encountering bugs? The first thing is the “locked” list: Train: openstack server list --all-projects --locked --os-compute-api-version 2.73 <empty list because nothing is locked> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> Wallaby (locked flag is ignored): openstack server list --all-projects --locked --os-compute-api-version 2.73 <list of every VM in the cluster> openstack server list --all-projects --unlocked --os-compute-api-version 2.73 <list of every VM in the cluster> The second problem is the hypervisor list. We can’t get “long” output in Wallaby: Train: openstack hypervisor list --long | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | Wallaby (long flag is ignored): openstack hypervisor list --long | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |

6 14

[tc][all][monasca ] Retiring the Monasca project
by Ghanshyam Mann 16 Feb '25

16 Feb '25

Hello everyone, As you might know, the Monasca project is not active and was marked 'inactive' since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next step is to retire the project. I'm sending this email to propose the Monasca retirement and based on the response, I will proceed next. [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 -gmann

6 17

[nova] How to gain approval for a new feature on unavailable hardware
by taketani.ryo＠fujitsu.com 14 Feb '25

14 Feb '25

To Whom It May Concern, We are considering to propose and develop a new feature on OpenStack Nova for building Arm Confidential Computing Architecture (Arm CCA)-encrypted instances. I have some questions for doing it. * While specification of Arm CCA is already open(https://developer.arm.com/documentation/den0125/0300/Overview), No Arm CCA-enabled hardware is currently available to our knowledge. Therefore, we plan to conduct testing in an internal environment. * Would a proposal for a feature utilizing publicly unavailable hardware be acceptable to OpenStack? Alternatively, in this case, what steps or preparations are required for approval? * Should the blueprint and specification be submitted simultaneously? * The specification is currently incomplete. Building Arm CCA instances requires libvirt APIs, similar to AMD SEV instances(https://blueprints.launchpad.net/nova/+spec/amd-sev-libvirt-suppo…, but specification of these are not yet determined publicly. We are considering submitting the blueprint first to inform the community of our ongoing development efforts. Regards

3 6

[ironic]About the OS supported by Diskimage-builder
by Junya Noguchi (Fujitsu) 03 Feb '25

03 Feb '25

To whom it may concern. My name is Junya Noguchi. I have some questions about the supported OS for Diskimage-builder operation. Please respond. References https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html Does it mean that the OS that says "work but receive only limited testing and support" works but is not supported? References https://docs.openstack.org/diskimage-builder/latest/ This document describes RHEL and other operating systems that are not supported in the previous question. Are these operating systems supported? Also, are there any plans to support more operating systems in the future? Best regards, Junya Noguchi

5 7

[glance][ops] proposal to remove the glance-scrubber
by Brian Rosmaita 20 Jan '25

20 Jan '25

Hello Glance users, The Glance project team wants to determine whether anyone is using the glance-scrubber utility [0]. If not, we are thinking of deprecating it this cycle to be removed in the appropriate future cycle. If you have not previously heard of the glance-scrubber, that's most likely because it's only useful for particular deployment configurations of which yours is not one. See the doc [0] for details if you are curious. Our suspicion is that the glance-scrubber may have outlived its usefulness for modern OpenStack configurations. (The last time I remember it being discussed at a PTG was during the snowstorm in Dublin, which I believe was the Rocky development cycle). On the other hand, the team also has some ideas for improving the scrubber, but there's no point doing that if it's not actually being used in the first place. So if removal of the glance-scrubber would negatively affect you, please respond to this email so we can get some data points. Thanks! [0] https://docs.openstack.org/glance/latest/cli/glancescrubber.html

2 2

[cinder][kubernetes] pvc stuck in reserved state
by Eugen Block 10 Jan '25

10 Jan '25

Hi, we have a couple of k8s clusters in our openstack environment (Victoria) and are facing some problems when pods get moved to different worker nodes. The PVCs get stuck in "reserved" state, and they only get attached after resetting the volumes to "available" state. First I thought it was due to the missing multiattach volume type which seems to work fine in our customer's cluster. I also found the "duplicate connector" error messages in the nova-compute.log: > Unable to update attachment.(Invalid volume: duplicate connectors > detected on volume But after adding the multiattach volume type and retyping the affected volumes, the PVCs still get stuck in "reserved" state while I don't see any error messages on the hypervisors anymore. So it seems that something has changed/improved, but now I don't have a trace where to look next. Could this be a cinder-csi driver issue or something? Our cinder backend is ceph (Pacific). Not sure which information could help so please ask for anything I can provide. Any pointers are highly appreciated! Thanks! Eugen

1 2

Failing at TASK [glance : Copy over multiple ceph configs for Glance]
by Karera Tony 04 Jan '25

04 Jan '25

Hello Team, I am deploying Openstack release 2024.02 with External Ceph. However I am encountering the error below. But it is failing here TASK [glance : Copy over multiple ceph configs for Glance] Please assist Regards Tony Karera

2 1

ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version
by Swogat Pradhan 24 Dec '24

24 Dec '24

Hi, I am currently trying to deploy openstack wallaby using tripleo arch. I created the network jinja templates, ran the following commands also: #openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml # openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml # openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml and used the environment files in the openstack overcloud deploy command: (undercloud) [stack@hkg2director ~]$ cat deploy.sh #!/bin/bash THT=/usr/share/openstack-tripleo-heat-templates/ CNF=/home/stack/ openstack overcloud deploy --templates $THT \ -r $CNF/templates/roles_data.yaml \ -n $CNF/workplace/custom_network_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e $CNF/templates/node-info.yaml \ -e $CNF/templates/scheduler-hints.yaml \ -e $CNF/workplace/networks-deployed-environment.yaml \ -e $CNF/workplace/vip-deployed-environment.yaml \ -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ -e $CNF/workplace/custom-net-bond-with-vlans.yaml Now when i run the ./deploy.sh script i encounter an error stating: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command. 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1 Can someone tell me where the mistake is? With regards, Swogat Pradhan

4 4

[all] Installation guide for openSUSE and SUSE Linux Enterprise
by Takashi Kajinami 23 Dec '24

23 Dec '24

Hello, As you may know, we have been maintaining installation guide for three linux distributions below. - oepnSUSE and SUSE Linux Enterprise - Red Hat Enterprise Linux and CentOS - Ubuntu However, as far as I know, the packages for openSUSE and SLES haven't been built/updated for some time. Also openSUSE was removed from tested runtimes in wallaby and hasn't been tested. So I doulbt that users can use these operating systems to install OpenStack nowadays. Can/Should we remove the guide for openSUSE/SLES Now ? I'm not very sure if we can maintain the guide for these with no details available nowadays. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 1

[ops][largescale-sig] Join the Large Scale OpenStack virtual meetup with Rackspace !
by Thierry Carrez 09 Dec '24

09 Dec '24

Hi everyone, This cycle the Large Scale SIG is organizing virtual meetups, where operators of large-scale OpenStack deployments come to discuss their experiences, challenges and solutions with fellow operators. On Thursday, December 5, at 16 UTC (10am CT, 5pm Paris), we'll have our first one. The special guest for this meetup will be Kevin Carter, operator of the Rackspace Cloud. It will last one hour and will happen through a video meeting. To join us, please fill this form and we'll send you the link for the event the day before! https://forms.gle/92o1DTavHwivN8tn6 Hoping to see you there. -- Thierry Carrez (ttx)

4 6

[kolla-ansible][nova] Refresh cell cache in nova scheduler
by Satish Patel 04 Dec '24

04 Dec '24

Folks, What is wrong here, I do have the following hostname resolvable from all the nodes even if they are in the /etc/hosts file, Why is the following task saying UNREACHABLE and from where its trying to reach ? PLAY [Refresh nova scheduler cell cache] ***************************************************************************************************************************************************** TASK [nova : Refresh cell cache in nova scheduler] ******************************************************************************************************************************************* fatal: [os2-bos-ctrl01]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl01\". Make sure this host can be reached over ssh: ", "unreachable": true} fatal: [os2-bos-ctrl02]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl02\". Make sure this host can be reached over ssh: ", "unreachable": true} fatal: [os2-bos-ctrl03]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"os2-bos-ctrl03\". Make sure this host can be reached over ssh: ", "unreachable": true}

4 5

[octavia] [lbaas] [ovs] Is octavia sr-iov supporting tunnels?
by yardalgedal＠gmail.com 04 Dec '24

04 Dec '24

Hi, Openstack Team, Is new octavia sr-iov feature supporting vxlan tunnels like in ovs or it can work only with provider networks (vlan)?

3 2

[kayobe] Running seed on ansible control host
by Olaf Trygve Berglihn 04 Dec '24

04 Dec '24

Dear list, Can someone explain how to configure kayobe to set up the seed hypervisor and seed vm on the ansible control host (where you run 'kayobe seed hypervisor host configure' and 'kayobe seed vm provision')? Thanks! -Olaf

2 5

[kayobe] [bifrost] Enabling and Deploying compute node
by alidot-subs Mail 01 Dec '24

01 Dec '24

Hi Everyone, Its be good experience to deploy Kayobe in all in one mode and see cirros image getting deployed and acesssible. Moving forward for multi-node deployment of baremetal machines. i tried to enable bifrost bit i doesn't seems to et enable by setting configurations in bifrost.yaml. What it takes to enable and use Bifrost using kayobe ? It may sound basic question. Thank you Ali

2 3

[cinder] What is the "official" name for Cinder in the service catalog?
by Stephen Finucane 30 Nov '24

30 Nov '24

o/ I've proposed a series of patches to "clean up" the service catalog that DevStack creates in a deployment. Currently, DevStack creates two endpoints for Cinder: one called 'cinderv3' of type 'volumev3' and another called 'cinder' of type 'block-storage'. For example: { "services": [ { "name": "cinderv3", "description": "Cinder Volume Service V3", "id": "65460706a9864e71926f13042f526aeb", "type": "volumev3", "enabled": true, "links": { "self": "http://10.0.108.50/identity/v3/services/65460706a9864e71926f13042f526aeb" } }, { "name": "cinder", "description": "Cinder Volume Service", "id": "cd406a0e296d47dca1e481e992851bb2", "type": "block-storage", "enabled": true, "links": { "self": "http://10.0.108.50/identity/v3/services/cd406a0e296d47dca1e481e992851bb2" } }, // ... ], "links": { "next": null, "self": "http://10.0.108.50/identity/v3/services", "previous": null } } They both ultimately point to the same API, the Cinder v3 API, meaning one of these endpoints is redundant. My assumption was 'block-storage' was the correct one, since that's the official service type [1], and that 'cinder' was the appropriate name since it mirrors what we use for other services (nova, neutron, keystone, glance, ...). However, tkajinam raised the point [2] that the Install Guide currently recommends use of the 'volumev3' type and that projects like Nova and Glance have an '[cinder] catalog_info' that defaults to 'volumev3'. If so, is 'block-storage' really the official service type, or is it 'volumev3'. If it is, should we be changing the Install Guide and the various config option defaults? Cheers, Stephen [1] https://specs.openstack.org/openstack/service-types-authority/

6 18

[docs][arch] Architecture Guide, ch. Control Plane Design
by just.me.on.reddit＠protonmail.com 29 Nov '24

29 Nov '24

I am sorry for proprietary tag - I hope it helps to narrow down the scope of this mail. Tags in order generic to more specific. Hello, If I only understand the term "cloud controller" properly these are those OpenStack services which collectively control whole cloud, inclusively those services exposed to cloud user. Cloud control can run on single hosting node, or can be distributed to a group of nodes. Please correct me if I am wrong. My role is on cloud operative side (a learner to become OS user). Architecture Guide, chapter as in this mail subject states following "It’s also possible to use virtual machines for all or some of the services that the cloud controller manages, such as the message queuing. In this guide, we assume that all services are running directly on the cloud controller." It is hard to continue the lecture of this Guide due to unclear description at following points. 1. reg. "all services are running directly on the cloud controller" Can't actually understand this statement. The "on" suggests cloud controller be hosting services in mind. Furthermore, cloud controller includes only those services acting under the goal of cloud control. Service whose major purpose is to provide functionality to cloud user won't, I suppose, be part of cloud controller. Hence, this statement might be more generic as it should be. 2. reg. "use virtual machines for all or some of the services that the cloud controller manages, such as the message queuing" Will cloud controller really mange message queuing instead of include comprise it? Which use of virtual machine exactly is meant here * placing services to virtual machine during O.S. deployment to hosting IT * the IT hosting OpenStack-based cloud? Should the former bullet apply, wasn't is said in some other Guide, abstraction level rather high, that use of virtual machines as cloud host is recommended only for test and evaluation purposes?

1 0

[openstack-ansible] : GPU computes on openstack-ansible
by Rambo Rambo 29 Nov '24

29 Nov '24

Hi Team Openstack-Ansible Openstack Victoria Release Ubuntu 20.04 I need to add computes having GPU on them. So I need recommendations on the procedure to add them and also what all configurations through openstack-ansible are required for nova_api, nova_scheduler on control node nova_api containers and compute node nova_compute configurations? Also does anyone has experience using and installing openstack service : cyborg for GPU management. Regards Rambo

3 5

[heat][horizon][octavia][oslo][qa][skyline][takcer][telemetry][zaqar] Need attention for tomorrow (Nov 29th) CI migration to Ubutnu Noble
by Ghanshyam Mann 29 Nov '24

29 Nov '24

Hello Team, As you know, we have been trying very hard to test and fix the project CI for the Ubuntu Noble migration[1] and there are 9 projects that are still failing on Noble. Check the etherpad having testing change for me information on failures: - https://etherpad.opendev.org/p/migrate-to-noble The migration is planned for tomorrow (Nov 29th); request you to check/fix the failure before that; otherwise, it will break your project gate. [1] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… -gmann

1 0

[docs][kayobe] Architecture, terms used
by just.me.on.reddit＠protonmail.com 29 Nov '24

29 Nov '24

Dear Kayoube documentation, If possible please be bit more specific at few points - document Architecture, section Networks refers to bare metal overcloud hosts, however the chapter "Hosts in the System" presents only bare metal compute hosts, nor any other text passage clarifies the term "overcloud". This way it is hard to get full understanding of term "Overcloud out-of-band network". Furthermore, if you talk about an out-of-band stuff it gets hard to see what is out-of-band without term in-band be precisely clear.

1 3

[watcher][tc] watcher-drivers launchpad membership
by Sean Mooney 29 Nov '24

29 Nov '24

Hi folks o/ Following up form the PTG discussion https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L65 and the recently merged governance patch https://github.com/openstack/governance/commit/1e547ae39f9913dc9cfd9c755b24… I wanted to raise the topic of the watcher-drivers team https://launchpad.net/~watcher-drivers membership and ownership. during the TC sessions we discussed that that the <project>-coresec teams should be owned by the https://launchpad.net/~openstack-admins team https://etherpad.opendev.org/p/r.0f25532f564fb786a89cfe1de39b004b#L123 watcher does not currently have a sperate watcher-coresec team in launchpadd only a a watcher-drivers team i also think that the watcher driver team should be owned by the https://launchpad.net/~openstack-adminsteam. https://launchpad.net/~watcher-drivers is currently owned by https://launchpad.net/~david-tardivel and the only other member with admin rights is https://launchpad.net/~li-canwei2 Can i ask that one of the existing admin either grant my membership request and delegate admin rights to me or add the https://launchpad.net/~openstack-admins team as admins and make them the owner of the team? licanwei, david-tardivel, would either of ye be willing and able to make those changes in launchpad? i also think it would be prudent to add other active member (the current liaisons) as admins/moderators of the watcher-driver team but i can do that once i have access or https://launchpad.net/~openstack-adminscan. having all launchpad teams owned by https://launchpad.net/~openstack-adminsteam mitigates the risk of loosing access by the current admins becoming inactive, but it would be nice to not need to escalation to them for simple tasks like moderation of the group. that is why i want to add the other liaisons to the admins of the team. I am asking for this access for 2 reasons: Fist to add https://launchpad.net/~openstack-adminsas admins of the team and the launchpad project, Second so that i can create new launchpad series for current releases to do blueprint and bug tracking. at present because we do not have launchpad series fro 2025.1 we cannot properly track bugs/blueprints for the current cycle. In the future if there are no objections form the existing contributors i would also like to create a separate open bug team with less project wide permission so that anyone can contribute to bug triage without needing access to the drivers team. this is in lien with how other project like nova work for example https://launchpad.net/~nova-bugs i also think we should be creating a separate watcher-coresec team to allow dan as the security liaison and the vulnerability management team to properly manage security reports. once we have access to the watcher-drivers team we should be able to update the launchpad project to make that possible. Regards sean.

3 4

[watcher][tc][tempest] watcher gerrit group membership
by Sean Mooney 28 Nov '24

28 Nov '24

Hi everyone o/ As many will be aware, over the last month or two, I and some others have been looking to help maintain and extend the Watcher project. At the ptg, I proposed several topics to cover the administrivia of reviving the project. To bring it to a healthy state where features, bugs and general maintenance could be performed on an ongoing basis and avoid its retirement at the end of the 2025.1 cycle. One of the follow-up items to enable that was the extension of the watcher Gerrit groups to allow others to supplement the largely inactive core groups. https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L47 <https://etherpad.opendev.org/p/r.79644f8f3cd04686c4f45c10aa35d8ac#L47> To that end, I would like to ask the existing watcher core team and TC (or other community members :) ) to provide input on the following proposal. 0.) I want to state upfront that I do not want to take away review rights or diminish the experience and efforts of any of the past or present members of the watcher team. I am sending this proposal to enable the watcher project to function effectively and I am welcome to feedback or objections. 1.) To facilitate the changes in the rest of the proposal I would like to formally ask to be added to all of the existing watcher Gerrit groups * watcher-core https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47… <https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47…> * watcher-release https://review.opendev.org/admin/groups/b265e6b6457de71c7ecfc2a3c70f0c52cde… <https://review.opendev.org/admin/groups/b265e6b6457de71c7ecfc2a3c70f0c52cde…> * watcher-dashboard-core https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47… <https://review.opendev.org/admin/groups/77cd139bd57514e98e5e694a82f7e261b47…> * watcher-dashboard-release https://review.opendev.org/admin/groups/904be58ec9fa0e811166ce6144d6314cc7b… <https://review.opendev.org/admin/groups/904be58ec9fa0e811166ce6144d6314cc7b…> * watcher-specs-core https://review.opendev.org/admin/groups/747af616eb8748aee5b17d9493f9ebbc49e… <https://review.opendev.org/admin/groups/747af616eb8748aee5b17d9493f9ebbc49e…> * watcher-tempest-core https://review.opendev.org/admin/groups/09a91d8e24af9ce44b80062c4851a1d2fa3… <https://review.opendev.org/admin/groups/09a91d8e24af9ce44b80062c4851a1d2fa3…> 2.) watcher-release and watcher-dashboard-release Historically, there has been a benefit to having separate core teams for the master and the stable branches. As the velocity of projects has decreased and projects have become more mature the delineation between the maintainers of the master branch and the stable branches has lessened. I believe there is still merit in having the flexibility to add people directly to the release team independently from the core team so I am not proposing replacing the *-release teams with *-core. To that end, I would like to make the following changes to the membership of watcher-release and watcher-dashboard-release - add stable-maint to both release groups - add watcher-core to watcher-release, as we have done in other projects the ask to the watcher-core team is, if you do not have time to review stable branches, or do not feel comfortable with applying stable policy, then there is no expectation to review on the stable branches. - add watcher-dashboard-core to watcher-dashboard-release - me (sean-k-mooney as release-liaison) to facilitate backports and release activities and Gerrit membership admin. 3.) watcher-core and watcher-dashboard-core As it currently stands the core groups are relatively inactive https://tinyurl.com/bdz9k47n full url: https://openstack.biterg.io/app/dashboards?security_tenant=openstack#/view/…:'Gerrit%20Approvals%20panel%20by%20Bitergia',filters:!(('$state':(store:appState),meta:(alias:Bots,disabled:!f,index:gerrit,key:author_bot,negate:!t,params:(query:!t),type:phrase),query:(match:(author_bot:(query:!t,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:gerrit,key:project,negate:!f,params:(query:watcher),type:phrase),query:(match_phrase:(project:watcher)))),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),query:(language:lucene,query:''),timeRestore:!f,title:'Gerrit%20Approvals',viewMode:view <https://openstack.biterg.io/app/dashboards?security_tenant=openstack#/view/…:'Gerrit%20Approvals%20panel%20by%20Bitergia',filters:!(('$state':(store:appState),meta:(alias:Bots,disabled:!f,index:gerrit,key:author_bot,negate:!t,params:(query:!t),type:phrase),query:(match:(author_bot:(query:!t,type:phrase)))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:gerrit,key:project,negate:!f,params:(query:watcher),type:phrase),query:(match_phrase:(project:watcher)))),fullScreenMode:!f,options:(darkTheme:!f,hidePanelTitles:!f,useMargins:!t),query:(language:lucene,query:''),timeRestore:!f,title:'Gerrit%20Approvals',viewMode:view>) Dan Smith and Slawek Kaplonski were added to help with the retirement/gate issues on behalf of the TC While Dan has offered to help bootstrap the core team we need to be cognizant that both Dan and Slawek have other commitments. Chenker has previously expressed that they intend to review when time allows but they have limited time to do so. To that end, I would like to propose that I be added to both core teams, To have a health core team we will eventually need to have at least 3 active reviewers. Watcher will still be 1-2 reviews short of that, but I hope that can be addressed over the coming months. As others become active over the 2025.1 cycles I would like to revisit the membership and extend the core team after milestone 2 or more realistically after milestone 3 if others demonstrate interest and capacity to review. 4.) watcher-tempest-core This is a newly created group added by https://review.opendev.org/c/openstack/project-config/+/934357 <https://review.opendev.org/c/openstack/project-config/+/934357> This group intends to allow the review team for the tempest plugin to diverge from the main watcher core group. To enable effective review and maintenance of the tempest plugin I would like to propose the following membership changes. - tempest-core should be added as a fallback, to guide best practices, and review if they have time. - gmann has volunteered to help with this too so I propose they be added as a direct member - I would like to be added to have a second active reviewer available. Note, while I can provide some review input I have less knowledge of tempest in general as such I will rely on Gmann's guidance in this area. 5.) watcher-specs Several years ago Nova expanded the nova-specs core group to include nova-core there were a number of motivating factors but one of them was the large overlap between the two groups. another factor was the role of a spec core (focusing on the design constraints, upgrade impact, API impact ectra) is different from the implementation considerations(testing, code quality, performance) often the line between the two is subtle and blurred. To that end, I would like to propose the following changes. - first, I would like to be directly added to the specs core group - second, I would like to add the watcher-core group to watcher-specs. If this proposal is accepted I would like to try and move forward with this in the next week or so to unblock the pending review backlog. Regards sean PS many of the open reviews are cleanup i have submitted so this does not directly address review bandwidth for my personal patches but i think it still has value in general.

3 3

Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster
by Ishan Shanware (ishanwar) 28 Nov '24

28 Nov '24

For everyone’s reference From: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com> Date: Wednesday, 27 November 2024 at 10:01 PM To: Sven Kieske <kieske(a)osism.tech> Subject: Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster Hi Sven, Thanks for getting back to me. We have configured kolla to use 3 controllers in each cluster. Each controller has one rabbitmq container created by kolla. The process we follow move a controller out of rotation is as follows: 1. We first failover all the l3 and dhcp agents running on the controller to the 2 other controllers. 2. We then turn of the server and then delete any neutron agents on it. Currently the rabbitmq cluster only consists of only 2 containers. The problem specifically occurs during the reinstall of the rabbitmq container in the kolla deploy stage. Specifically, after it executes the checking rabbitmq containers role. It triggers a restart for the contains on all the blades. After this stage completed we observe that the masakari-engine disables all the nova-computes. Let me know if you need any other information for discussion Thanks Ishan From: Sven Kieske <kieske(a)osism.tech> Date: Tuesday, 26 November 2024 at 11:36 PM To: Ishan Shanware (ishanwar) <ishanwar(a)cisco.com>, openstack-discuss(a)lists.openstack.org <openstack-discuss(a)lists.openstack.org> Subject: Re: [kolla][oslo] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster Hi Ishan, my first question would be, how many controller nodes and specifically, rabbitmq nodes you are running inside your Openstack cluster? You should be following the general guidelines for running any raft consensus based distributed system and only run an odd number of systems, e.g. 3 or 5 control nodes. Can you confirm that this is the case in your setup? If you e.g. run a two node setup such errors are indeed expected. See also our production architecture guide: https://docs.openstack.org/kolla-ansible/latest/admin/production-architectu… > Control - Cloud controller nodes which host control services like APIs and databases. This group should have odd number of nodes for quorum. If you are running an odd number of control nodes and you're still facing this issue, I would be curious to know the rabbitmq cluster state before you add or remove a node, because this should theoretically just work. But maybe there is another issue with your cluster. HTH -- Sven Kieske Senior Cloud Engineer Mail: kieske(a)osism.tech Web: https://osism.tech OSISM GmbH / Talweg 8 / 75417 Mühlacker / Deutschland Geschäftsführer: Christian Berendt Unternehmenssitz: Mühlacker Amtsgericht: Stuttgart, HRB 756139

1 1

[watcher] Canceling Dec 5th weekly meeting
by Douglas Viroel 28 Nov '24

28 Nov '24

Hi all o/ Since most of our audience will be out next week, we agreed[1] on canceling watcher's weekly meeting[2], on Dec 5th. We will meet again in two weeks, on Dec 12th. [1] https://meetings.opendev.org/meetings/watcher/2024/watcher.2024-11-28-12.00… [2] https://meetings.opendev.org/#Watcher_Team_Meeting BR, -- Douglas Viroel - dviroel

1 0

[kolla-ansible] nova-cell can't not find nova.keyring file when deploy
by daniel890723＠gmail.com 28 Nov '24

28 Nov '24

hello Daniel here Previously i was testing openstack-helm but it doesn't work so i switch to kolla-ansible with all-in-one mode i was following the official guide and put the keyring file i generate to the config folder: (venv) root:~# nano /etc/kolla/config/nova/ ceph.client.nova.keyring ceph.conf localhost/ nova-compute.conf But when i try to deploy it show the error below TASK [nova-cell : Check nova keyring file] ***************************************************************************************************************** fatal: [localhost]: FAILED! => {"msg": "No file was found when using first_found."} Just in case i also put the keyring file in /config/localhost/nova Still shows the same error But when i put the cinder keyring file in nova folder like this: (venv) root@:~# nano /etc/kolla/config/nova/ ceph.client.cinder.keyring ceph.client.nova.keyring ceph.conf localhost/ nova-compute.conf It can find the cinder keyring file but this time the nova-compute container show error below: unable to find a keyring on /etc/ceph/ceph.client.nova.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory Did i do anything wrong or something makes it not able to detect the nova.keyring file P.S. Cinder and Glance works fine and Ceph can be used normally

2 5

[manila] Cancelling Nov 28th upstream weekly meeting
by Carlos Silva 28 Nov '24

28 Nov '24

Hello Zorillas, At least half of our audience tomorrow will not be online tomorrow, as it is thanksgiving in the US, so let's cancel the meeting. We'll resume the weekly meetings on Dec 5th. Regards, carloss

1 0

[ops][kolla-ansible][openstack-ansible] Deploying OpenStack using Ansible to LXC while project is used to grab for Kolla-Ansible
by just.me.on.reddit＠protonmail.com 27 Nov '24

27 Nov '24

Hi, I am learning OS because my imminent mission uses a cloud-based solution (cloud-app) which runs in OpenStack cloud. How do you estimate the probability for a cloud-app to feature built-in Docker dependency (runtime) in situation where the project of mentioned app appears to be used to apply Kolla Ansible in order to deploy OpenStack cloud? It means the cloud-app to depend at runtime on Docker-specific artefacts. 3rd-party cloud-app going to be in use will play a major role in my project. It is in responsibility of cloud-app operator to do provisioning of OpenStack cloud before mentioned app can be deployed to cloud. Recommendations found in cloud-app's documentation read to use Kolla Ansible in order to conduct mentioned provision. Myself learned that Kolla uses Docker. Docker use is that component which triggers me to ask this question. My own preference would be rather to deploy OpenStack services to LXC-containers. One can find in OpenStack web the project Ansible-based deployment to LXC containers. I wonder following if cloud-app developers could had placed in it any built-in Docker (eco-system) runtime dependency and they achieved it by the use of Ansible Kolla for OpenStack deployment. Considering my very flat knowledge of Docker (only familiarized with its purpose and very few exercises made in far past) there exists just hypothetical risk for cloud project in my focus to feature built-in docker eco-system dependency. Should such dependency be a fact (not just a risk) my aim to use OpenStack services in LXC-containers has good chance to fail. It is hard to contact 3rd-party cloud project developer regarding my questions. Question 2 Any common conclusions had been drawn so far if one of the two approaches be of remarkably higher suitableness to host OpenStack-based cloud services? * placing OpenStack-based cloud services to Docker-based containers * placing services of OpenStack-based cloud to Linux containers (LXC/LXD/...)

1 0

[os-vif][nova][neutron] Proposing Rodolfo as an os-vif core
by Sean Mooney 27 Nov '24

27 Nov '24

hi folks o/ I intended to send this last week but didn't find time till now. I'm sure many are familiar with Rodolfo form the work that he has done in neutron over the years. I have worked with them for a long time and over the years they have also contributed too and reviewed changes too os-vif. I think having a neutron core on the os-vif team would certainly help bubble up pain points form them and address such issue faster in os-vif going forward. Rodolfo is familiar with the code and with the upstream review process so i think they would be a good fit. If there are no objection to this ill bring this up again in next weeks nova team meeting and add rodoflo to the core list. reagrds sean.

2 2

[Octavia] Cancelling today's meeting
by Gregory Thiemonge 27 Nov '24

27 Nov '24

Hi Folks, Due to personal reasons, I have to cancel this week's meeting, sorry for the short notice, See you next week, Greg

1 0

Openstack ansible variables
by engineer2024 27 Nov '24

27 Nov '24

I want to print the values of all the ansible variables at the playbook and task level. If we execute os_glance.yml , I want to see the variable values of all the tasks/playbooks involved in this glance playbook. Is it also possible doing it from any one file ? Thanks elinux

1 0

[largescale-sig] Next meeting: November 27, 9utc
by Thierry Carrez 27 Nov '24

27 Nov '24

Hi everyone, The Large Scale SIG will be meeting this Wednesday in #openstack-operators on OFTC IRC, at 9UTC. You can doublecheck how that UTC time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20241127T09 Feel free to add topics to the agenda: https://etherpad.opendev.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[devstack][neutron] DNS Resolution Issue After Server Reboot with DevStack
by xuq.fnstxz＠fujitsu.com 27 Nov '24

27 Nov '24

Hi everyone, I am reaching out to seek your assistance regarding an issue I am experiencing with my DevStack installation. After successfully installing DevStack stable/2024.2 on ubuntu 22.04, I configured the network and set up DNS for private subnet. stack@controller:~$ openstack subnet list +--------------------------------------+---------------------+--------------------------------------+---------------------+ | ID | Name | Network | Subnet | +--------------------------------------+---------------------+--------------------------------------+---------------------+ | 2b6411d5-9c19-4a45-bd47-58dc748f5e00 | public-subnet | 45502b6a-773b-40cf-866a-e8b18397ffcb | 192.168.1.0/24 | | 69629170-2e0e-49c0-b741-61560558001c | ipv6-public-subnet | 45502b6a-773b-40cf-866a-e8b18397ffcb | 2001:db8::/64 | | c74bf062-b67a-45cb-8a50-73583498dfcd | ipv6-private-subnet | 8fb37e61-bd59-4624-a4be-1fc1841987d7 | fd3a:2f86:ca98::/64 | | f2e10d3d-f785-4cf9-8804-1969269623c2 | private-subnet | 8fb37e61-bd59-4624-a4be-1fc1841987d7 | 10.0.0.0/26 | +--------------------------------------+---------------------+--------------------------------------+---------------------+ stack@controller:~$ openstack subnet set --dns-nameserver 10.12.255.254 --dns-nameserver 192.168.1.95 f2e10d3d-f785-4cf9-8804-1969269623c2 10.12.255.254 is the real DNS server address and 192.168.1.95 is my server IP address. Initially, when I launched a VM instance, it was able to resolve the proxy and connect to the external network without any issues. jenkins@np0000000531:~$ curl opendev.org <a href="https://opendev.org/">Temporary Redirect</a>. However, after rebooting the server and launching the instance again, the instance is unable to resolve the proxy. jenkins@np0000000531:~$ curl opendev.org curl: (5) Could not resolve proxy: my.proxy.com I have verified that the systemd-resolved service is in an active state, but the logs show a large number of entries with the following message: Nov 13 11:53:51 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 10.12.255.254. Nov 13 11:54:01 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 192.168.1.95. Nov 13 11:54:01 np0000000531 systemd-resolved[568]: Using degraded feature set TCP instead of UDP for DNS server 10.12.255.254. Nov 13 11:54:11 np0000000531 systemd-resolved[568]: Using degraded feature set TCP instead of UDP for DNS server 192.168.1.95. Nov 13 11:54:21 np0000000531 systemd-resolved[568]: Using degraded feature set UDP instead of TCP for DNS server 10.12.255.254. I have checked the network and routing table of the VM instance, as well as the bridge settings on the server. Everything appears to be the same as it was before the reboot. Could you please help me identify the cause of this issue and suggest any potential solutions? If you need any additional information, please let me know, and I will be happy to provide it. Thank you for your time and assistance. Best regards, Inori.

2 3

[all] 2025 Open Infrastructure Foundation Individual Director nominations are open
by Jeremy Stanley 27 Nov '24

27 Nov '24

To those who aren't following the foundation mailing list, please be aware that the nomination period for the 2025 Open Infrastructure Foundation Board of Director Individual Member elections is open, and will close at 23:59 UTC on December 13. If you've considered taking on some additional leadership responsibility, this could be your opportunity! See Wes's post for more information: https://lists.openinfra.dev/archives/list/foundation@lists.openinfra.dev/th… -- Jeremy Stanley

1 0

[kolla] RabbitMQ issue Adding/Removing a Controller Node in an OpenStack Cluster
by Ishan Shanware (ishanwar) 26 Nov '24

26 Nov '24

Hi Kolla Community, I would like to understand the current guidelines for adding or removing a controller node from an OpenStack cluster. With the introduction of quorum queues, we’ve observed that adding a new controller node causes a loss of quorum within the cluster. Consequently, during the reinstallation of a controller node, Masakari disables all the nova-compute services. Is this behaviour expected with quorum queues, or could this be addressed differently? Additionally, I’ve noticed that as soon as the Kolla containers are restarted in the cluster, clients are unable to connect, causing the cluster to go down. Could you please share insights or recommendations on how to handle this scenario effectively? Looking forward to hearing your thoughts. Thanks, Ishan Cisco Systems

2 2

[keystone] Weekly meeting canceled 27-Nov-2024
by Dave Wilde 26 Nov '24

26 Nov '24

Hello, Due to the Thanksgiving holiday in the US I'm canceling the weekly meeting and reviewathon this week. Please reach out here or on IRC if you need anything. Thanks! /Dave

1 0

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

[openstack helm] [Ansible] Openstack-helm Ansible inventory.yaml missing some value
by daniel890723＠gmail.com 26 Nov '24

26 Nov '24

Hello today i'm testing installation with completely new enviroment i was following the guide from Openstack-helm document and install k8s cluster using ansible and deploy-env playbook but i encounter this error when setup ssh cluster TASK [deploy-env : Generate ssh key pair] ****************************************************************************************************************** skipping: [node-1] fatal: [primary]: FAILED! => {"msg": "Failed to change ownership of the temporary files Ansible (via chmod nor setfacl) needs to create despite connecting as a privileged user. Unprivileged become user would be unable to read the file."} and the log with -vvv output: debug2: mux_client_hello_exchange: master version 4 debug3: mux_client_forwards: request forwardings: 0 local, 0 remote debug3: mux_client_request_session: entering debug3: mux_client_request_alive: entering debug3: mux_client_request_alive: done pid = 114038 debug3: mux_client_request_session: session request sent debug1: mux_client_request_session: master session id: 2 chown: invalid user: ‘zuul’ debug3: mux_client_read_packet: read header failed: Broken pipe debug2: Received exit status from master 1 And by checking the playbook we can see the playbook is using "{{ client_ssh_user }}" on primary node and "{{ cluster_ssh_user }}" with cluster node which is not defined in inventory.yaml Causing it using default user zuul by adding client_ssh_user & cluster_ssh_user in ansible inventory it works normally Is this meant to be like this? or the document missing some variables in inventory.yaml

3 3

[tc][all] OpenStack Technical Committee Weekly Summary and Meeting Agenda (2025.1/R-18)
by Goutham Pacha Ravi 26 Nov '24

26 Nov '24

Hello Stackers, We're 18 weeks away from the coordinated release for 2025.1 ("Epoxy") [1]. We're beginning a period of low activity due to year-end holidays across the world. Last week, the TC didn't merge any new governance proposals; however, several are currently in progress. === Weekly Meeting === The TC met in OFTC's #openstack-tc IRC channel on 2024-11-19 [2]. We discussed a series of global requirements upper constraint bumps that were committed and the ensuing CI flakiness. We appreciated the effort to proactively update dependencies to their latest versions earlier in the release cycle so we can have more time to test and stabilize OpenStack software. Sometimes, these failures may interrupt other project priorities. The best way we can help is if project teams identify a relatively quick-running and stable integration job to run against changes to the global requirements repository. The TC and OpenDev administrators worked with Launchpad administrators to regain control of trackers belonging to the Watcher project team. It was noted that Launchpad trackers for any project should not be owned/administered by individuals. Instead, there must be a Launchpad team driving all trackers associated with an OpenStack initiative, and "openstack-admins" must own this team. This approach ensures the sustainability of project teams and protects projects from the impact of changing individual priorities. We also briefly discussed the "eventlet-removal" goal proposal. The goal document is being rewritten [3] to simplify what is being done and the timelines we're aiming for. I'd love for the community to take a look at this change and review it. The next meeting of the OpenStack Technical Committee is today, 2024-11-25, at 1800 UTC in OFTC's #openstack-tc channel. The agenda for this meeting is in our meeting wiki [4]. I hope you're able to join us. === Governance Proposals === ==== Open for Review ==== Add Cinder Huawei charm | https://review.opendev.org/c/openstack/governance/+/867588 Rework the initial goal proposal as suggested by people | https://review.opendev.org/c/openstack/governance/+/931254 Propose to select the eventlet-removal community goal | https://review.opendev.org/c/openstack/governance/+/934936 Resolve to adhere to non-biased language | https://review.opendev.org/c/openstack/governance/+/934907 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 Retire Murano/Senlin/Sahara OpenStack-Ansible roles | https://review.opendev.org/c/openstack/governance/+/935677 Add ansible-role-httpd repo to OSA-owned projects | https://review.opendev.org/c/openstack/governance/+/935694 ==== Merged ==== Fix doc job for pillow 11.0.0 | https://review.opendev.org/c/openstack/governance/+/935967 Add Ubuntu Noble migration goal tracking etherpad | https://review.opendev.org/c/openstack/governance/+/935461 === How to Contact the TC === You can reach the TC in several ways: - Email: Send an email with the tag [tc] on this mailing list. - IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC. - Weekly Meetings: Join us at our weekly meeting. The Technical Committee meets every week on Tuesdays at 1800 UTC [4]. Thank you very much for reading! On behalf of the OpenStack TC, Goutham Pacha Ravi (gouthamr) OpenStack TC Chair [1] 2025.1 "Epoxy" Release Schedule: https://releases.openstack.org/epoxy/schedule.html [2] TC Meeting IRC Log 2024-11-19: https://meetings.opendev.org/meetings/tc/2024/tc.2024-11-19-18.01.log.html [3] Eventlet removal Community goal: https://review.opendev.org/c/openstack/governance/+/931254 [4] TC Meeting Agenda, 2024-11-26: https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee

1 0

[docs][dev] replace_global_vars not working in project-deploy-guide
by Jakub Darmach 26 Nov '24

26 Nov '24

Hello everyone, I've been updating documentation in kolla-ansible and noticed that while replacing variable content works in https://docs.openstack.org/kolla-ansible it fails to replace the variable with proper content in https://docs.openstack.org/project-deploy-guide/kolla-ansible. If I understand correctly, it's being substituted with var having proper content in here: https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… Later on used in https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/co… And now for the example - |KOLLA_BRANCH_NAME| variable: Working substitution: https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#t… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… Failed to render content: https://docs.openstack.org/project-deploy-guide/kolla-ansible/latest/quicks… https://opendev.org/openstack/kolla-ansible/src/branch/master/doc/source/us… The var seems to be used the same in both examples, I suspect slightly different configuration on project end of the things. Can anyone advise where should I start looking? Best regards, Jakub

2 1