November 2023 - openstack-discuss

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat.| Openstack tripleo wallaby version
by Swogat Pradhan 24 Dec '24

24 Dec '24

Hi, I am currently trying to deploy openstack wallaby using tripleo arch. I created the network jinja templates, ran the following commands also: #openstack overcloud network provision --stack overcloud --output networks-deployed-environment.yaml custom_network_data.yaml # openstack overcloud network vip provision --stack overcloud --output vip-deployed-environment.yaml custom_vip_data.yaml # openstack overcloud node provision --stack overcloud --overcloud-ssh-key /home/stack/sshkey/id_rsa overcloud-baremetal-deploy.yaml and used the environment files in the openstack overcloud deploy command: (undercloud) [stack@hkg2director ~]$ cat deploy.sh #!/bin/bash THT=/usr/share/openstack-tripleo-heat-templates/ CNF=/home/stack/ openstack overcloud deploy --templates $THT \ -r $CNF/templates/roles_data.yaml \ -n $CNF/workplace/custom_network_data.yaml \ -e ~/containers-prepare-parameter.yaml \ -e $CNF/templates/node-info.yaml \ -e $CNF/templates/scheduler-hints.yaml \ -e $CNF/workplace/networks-deployed-environment.yaml \ -e $CNF/workplace/vip-deployed-environment.yaml \ -e $CNF/workplace/overcloud-baremetal-deployed.yaml \ -e $CNF/workplace/custom-net-bond-with-vlans.yaml Now when i run the ./deploy.sh script i encounter an error stating: ERROR openstack [-] Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command.: tripleoclient.exceptions.InvalidConfiguration: Resource OS::TripleO::Network::Ports::ControlPlaneVipPort maps to type OS::Neutron::Port and the Neutron service is not available when using ephemeral Heat. The generated environments from 'openstack overcloud baremetal provision' and 'openstack overcloud network provision' must be included with the deployment command. 2022-04-19 13:47:16.582 735924 INFO osc_lib.shell [-] END return value: 1 Can someone tell me where the mistake is? With regards, Swogat Pradhan

4 4

[kolla] [openeuler] CI jobs status
by Michał Nasiadka 26 Nov '24

26 Nov '24

Hello Koalas, OpenEuler jobs in master (to be Antelope) are failing for a long time. If no one is interested in maintaining OpenEuler support in Kolla-Ansible - I will propose dropping it’s support on Wednesday Kolla meeting. Best regards, Michal

4 3

Kolla Ansible on Ubuntu 20.04 - cloud-init & other network issues
by Tobias McNulty 10 Oct '24

10 Oct '24

Hi, I'm attempting to use Kolla Ansible 14.6.0 to deploy OpenStack Yoga on a small 3-node Ubuntu 20.04 cluster. The nodes have 128 GB RAM each, dual Xeon processors, and dual 10G Intel NICs. The NICs are connected to access ports on a 10G switch with separate VLANs for the local and external networks. All the playbooks run cleanly, but cloud-init is failing in the Ubuntu 20.04 and 22.04 VMs I attempt to boot. The VM images are unmodified from https://cloud-images.ubuntu.com/, and cloud-init works fine if I mount a second volume with user-data. The error is a timeout attempting to reach 169.254.169.254. This occurs both when booting a VM in an internal routed network and directly in an external network. I tried various neutron plugin agents (ovn, linuxbridge, and openvswitch both with and without firewall_driver = openvswitch <https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutro…>) first with a clean install of the entire OS each time, all with the same result. Running tcpdump looking for 169.254.169.254 shows nothing. As a possible clue, the virtual NICs are unable to pass any traffic (e.g., to reach an external DHCP server) unless I completely disable port security on the interface (even if the associated security group is wide open). But disabling port security does not fix cloud-init (not to mention I don't really want to disable port security). Are there any additional requirements related to deploying OpenStack with Kolla on Ubuntu 20.04? This is a fairly vanilla configuration using the multinode inventory as a starting point. I tried to follow the Quick Start <https://docs.openstack.org/kolla-ansible/yoga/user/quickstart.html> as closely as possible; the only material difference I see is that I'm using the same 3 nodes for control + compute. I am using MAAS so it's easy to get a clean OS install on all three nodes ahead of each attempt. I plan to try again with the standard (non-HWE) kernel just in case, but otherwise I am running out of ideas. In case of any additional clues, here are my globals.yml and inventory file, along with the playbook I'm using to configure the network, images, VMs, etc., after bootstrapping the cluster: https://gist.github.com/tobiasmcnulty/7dbbdbc67abc08cbb013bf5983852ed6 Thank you in advance for any advice! Cheers, Tobias

5 9

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[ptl][tc] OpenStack packages PyPi additional external maintainers audit & cleanup
by Ghanshyam Mann 17 May '24

17 May '24

Hi PTLs, As you might know or have seen for your project package on PyPi, OpenStack deliverables on PyPi have additional maintainers, For example, https://pypi.org/project/murano/, https://pypi.org/project/glance/ We should keep only 'openstackci' as a maintainer in PyPi so that releases of OpenStack deliverables can be managed in a single place. Otherwise, we might face the two sets of maintainers' places and packages might get released in PyPi by additional maintainers without the OpenStack project team knowing about it. One such case is in Horizon repo 'xstatic-font-awesome' where a new maintainer is added by an existing additional maintainer and this package was released without the Horizon team knowing about the changes and release. - https://github.com/openstack/xstatic-font-awesome/pull/2 To avoid the 'xstatic-font-awesome' case for other packages, TC discussed it in their weekly meetings[1] and agreed to audit all the OpenStack packages and then clean up the additional maintainers in PyPi (keep only 'openstackci' as maintainers). To help in this task, TC requests project PTL to perform the audit for their project's repo and add comments in the below etherpad. - https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup Thanks to knikolla to automate the listing of the OpenStack packages with additional maintainers in PyPi which you can find the result in output.txt at the bottom of this link. I have added the project list of who needs to check their repo in etherpad. - https://gist.github.com/knikolla/7303a65a5ddaa2be553fc6e54619a7a1 Please complete the audit for your project before March 15 so that TC can discuss the next step in vPTG. [1] https://meetings.opendev.org/meetings/tc/2023/tc.2023-01-11-16.00.log.html#… -gmann

18 27

[all][packaging][release][qa] pep-517 and pip 23
by smooney＠redhat.com 06 May '24

06 May '24

hi folks, TL;DR ----- this is a long email so the tl;dr is we should add pyproject.toml ``` [build-system] requires = ["pbr>=5.7.0", "setuptools>=36.6.0", "wheel"] build-backend = "pbr.build" ``` to all openstack projects to supprot pip 23.1+ Long version ------------ As some of you may know the ptg happend last week an done of hte topic raised there was a know limitation in devstack where GLOBAL_VENV=True does not work on centos/fedora derived distros. while i generally dont use centos/fedora derived distros for my daily work i have been using devstack for over 10 years at this point so when im told it broken i ocationlly will go fix it. In this case the issue is not with devstack (it often isnt) and is actully in our collective reliance on a deprecated functionality and how we package openstack today. Back story, for better or worse when openstack was created the landscape of python packaging was significantly different then it is today. To help corodinate our dependcies the openstack comunity went all in on requirements.txt and developed upper-constraits to manage our requirements and developed PBR (python build Reasonableness). both of those approches have served us well and likely will continue to do so for some time. Since then pyproject.toml has started to gain popularity and pep-517 (A build-system independent format for source trees) has been adotpted but so far openstack as a comunity has not embraced those changes. (although pbr already supprot pep-517 :) ). Until now this has not been a problem however with recent release of pip things are now breaking. Problem Context: - openstack does not provide pyporject.toml files - openstack project configure setuptools to use pbr via setup.py - setuptools support generating scripts form entry points(console_scripts) which we use for project binaries( nova- compute, neutron-server, ...) - openstack api services use a pbr extension know as wsgi_scripts which adds a similar functionality for generating wsgi application mains file - devstack due to coming changes in distro to block "sudo pip install" now installs openstack in a single by default on debian distros. on centos/fedora distros it currently almost functional (but not). - devstack installs in editable mode because that is how most developers use devstack for developemnt What changed: - pip 20.2 deprecated support for falling back to calling setup.py when pyproject.toml is not present - https://pip.pypa.io/en/stable/news/#v20-2 - https://github.com/pypa/pip/issues/8368 - at this point when pyproject.yaml is not called pip calls setup.py install [-e] <path to repo or package name> - setup.py invokes pbr to generate wsgi_scripts and other files - in pip 23.1 the fallback was removed https://pip.pypa.io/en/stable/news/#v23-1 - pip now builds a temp local wheel instead of calling setup.py install when pyproject.toml is not found - in editable mode (-e) the default way the wheel is built does not invoke pbr properly to generate the wsgi files - as a result the wsgi files are not installed to the venv bin dir and this causes devstack to fail to start keystone (or any other wsgi api) The fix, there are a few ways to approch this: - clamp pip (https://review.opendev.org/c/openstack/devstack/+/898845) this is a terible idea and it breaks debian - clamping pip below 23.1 works on centos for now however OVS on debian already requires pep 517 for installing - as such when that version of ovs (or other projects start enforcing that) hits centos it will break - add a minimal pyproject.toml to all openstack project - example for keystone https://review.opendev.org/c/openstack/keystone/+/899519 - this example jsut adds the minium pyproject.toml to enable pbr to be used as the build system - it does not remove supprot for setup.py but it will not be used by default going forward whit new version of pip - all config is still in setup.cfg - add pyproject.toml support but remove setup.py support https://review.opendev.org/c/opendev/bindep/+/816741 - this goes one step futrue and actuly nulls out setup.py - we could take this approach if we want to but its not required. The reason for this email is to get agreement on how to support pip 23.1+ My proposal would be to add pyproject.toml with the minimal config required to enable pip 23.2 but not drop the setup.py support in this cycle. i would suggest that we can deprecate installing via setup.py in caracal if that is desirable but not make that change until the next slurp to give time to projects and packages to catch up. if there are no objections to that i would like to ask what is the preferd way to create and submit the patches for the minimal pyproject.toml ``` [build-system] requires = ["pbr>=5.7.0", "setuptools>=36.6.0", "wheel"] build-backend = "pbr.build" ``` i can propose patches to the main project deployed by devstack by default. i.e. nova/neutron ectra but it might be better if someone form the core teams of each project could take on that task and chasing the reviews. if someone has a script to generate a patch for a set of repos that just add the above pyporject.toml then we can use that to keep a consistent topic/commit message across all patches. my proposal for testing this is two fold. in ci we can test the patches are functional by using a DNM patch to devstack to force enable the gloal venv on centos and depend on the set of patches, i will modify https://review.opendev.org/c/openstack/devstack/+/898845 to do that locally we can test that the patch is functional by doing the following substituting keystone for the relevant project. ``` git clone https://opendev.org/openstack/keystone cd keystone python3 -m venv .venv . .venv/bin/activate python3 -m pip install -U pip python3 -m pip install -e . ls .venv/bin/keystone* ``` the ls will not contain the wsgi script with the addition of the pyproject.toml it should. on debian sid i can reproduce the same issue the problem we are seeing on centos 9 now will start impacting other distos eventually so its good to fix this now. if i start proposing this to glance/nova/neutron/cinder/swift/horizon/placement (to get a default devstack working) can i get cores to review? can cores form each project nominate peopel to review or volenteer to do this work. i have created https://etherpad.opendev.org/p/pep-517-and-pip-23 to track this work i have some other work that i think we can do in parallel like addign pyporject.toml support to the pti requirements if we go this way but that i think we can discuss on a governacne patch to updated it. anyway this is a lot to digest so feedback is welcome. if someone else would like to drive this that is also welcome (i.e. tc/release team) regards sean.

10 27

Accessing virsh in openstack wallaby | tripleo
by Swogat Pradhan 24 Mar '24

24 Mar '24

Hi, Can someone please tell me how can i access the virsh utility on the compute node on openstack wallaby? Previously we used nova_libvirt but in wallaby there are multiple containers and i am unable to use virsh utility to edit my domain.xml file. List of containers: [root@dcn01-hci-2 ~]# podman ps | grep nova 10d981158842 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtlogd_wrapper e91667516afc 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtsecretd a4b07fe0e833 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtnodedevd d7e1db393e9c 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtstoraged 5308a171793e 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 2 months ago nova_virtqemud f490a6249ba1 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo kolla_start 2 months ago Up 6 weeks ago nova_virtproxyd ab0441e56957 172.25.201.68:8787/tripleomaster/openstack-nova-compute:current-tripleo kolla_start 2 months ago Up 2 months ago (healthy) nova_migration_target f35fbba5b690 172.25.201.68:8787/tripleomaster/openstack-nova-compute:current-tripleo kolla_start 2 months ago Up 6 weeks ago (unhealthy) nova_compute 82bc01993106 172.25.201.68:8787/tripleomaster/openstack-nova-libvirt:current-tripleo /usr/sbin/virtlog... 2 months ago Up 2 months ago Can someone please suggest how I can i edit an instance xml file in this setup? With regards, Swogat Pradhan

2 5

[Freezer] Is Freezer dead?
by pi3.14＠tuta.io 19 Mar '24

19 Mar '24

Hi folks, is Freezer still supported? I don't see any real development going on with it. Is it recommended to use in a production Cloud environment? Are there other free and open source alternatives to Freezer? Thanks Roby

8 19

[cinder][glance][nova][telemetry][monasca][oslo][vmware] Can we deprecate(and remove) VMWare support ?
by Takashi Kajinami 07 Mar '24

07 Mar '24

Hi all, As you might know, some of the OpenStack projects currently support usage of VMWare component. However recently the VMware virt driver in nova was marked experimental in 2023.2 release [1] and it has been discussed that the driver will be removed in 2023.2 release[2], after deprecation is backported to 2023.1. (Please someone from nova team correct me if I'm wrong) So my current question is; Does it make sense to deprecate all of the features to support VMWare component ? My understanding is that these features were added to be used along with VMWare virt driver in nova, but once the virt driver is removed this use case is voided. I'll list up the features within the discussion scope below. nova: VMWare virt driver (this is already experimental by [1]) cinder: VMware Storage Driver (vmdk) glance: VMWare Datastore (I proposed deprecating this in [3][4]) ceilometer: VMWare virt driver (this was deprecated by [5]) os-brick: VmdkConnector monasca-agent: vcenter collector (*1) oslo.vmware: the whole project NOTE: These features are found based on dependency on oslo.vmware but there can be a few not listed here. Outside of OpenStack's governance, there are two projects dependent on oslo.vmware, networking-vsphere and vmware-nsx but both of these look inactive. Because of SLURP, we have to complete deprecation during in 2024.1 release so that we can drop the feature right after 2024.1. If we fail then we have to maintain these features until 2025.1 (the features already deprecated can be removed earlier, though). So I'd like to get feedback and decision before deadline for 2024.1 Thank you, Takashi (*1) Monasca is now inactive so I doubt anyone would work on this topic before it is retired. [1] https://review.opendev.org/c/openstack/nova/+/863911 [2] https://review.opendev.org/c/openstack/nova/+/897017 [3] https://review.opendev.org/c/openstack/glance_store/+/893696 [4] https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… [5] https://review.opendev.org/c/openstack/ceilometer/+/894540

6 21

[Watcher] How use Watcher to scheduler Openstack resource without cpu_usage
by Nguyễn Hữu Khôi 06 Mar '24

06 Mar '24

Hello guys. I want to use watcher to scheduler openstack resources but some openstack versions are missing cpu_usage metric. How do we use Watcher without it? Thank you. Nguyen Huu Khoi

3 5

[openstack][skyline] Some works with skyline
by Nguyễn Hữu Khôi 26 Feb '24

26 Feb '24

Hello guys. I am working on Skyline. I can integrate my monitor tab on it. I will try to do something about it. If it works properly, I can contribute. It is a very nice project. I hope it will grow and grow in the future. Thanks Skyline Team. [image: image.png] Nguyen Huu Khoi

3 7

[neutron][vpnaas][ovn] Pushing VPNaaS support through OVN
by Mohammed Naser 12 Feb '24

12 Feb '24

Hi all! One of the biggest show stoppers for us for OVN is VPNaaS but I think it’s time that we’d like to be involved in getting it over the line, I want to thank Bodo who has shown a lot of patience in getting the code necessary, and it seems that the code works for them in production as well. I’d like to ask if possible for folks to review these two outstanding changes: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/847007 https://review.opendev.org/c/openstack/neutron-vpnaas/+/765353 I know Slawek had some concerns, so hoping that raising via the email list if Bodo can address those and over the next bit of time we can get this landed in, since it even has actual proper testing both in CI and with user feedback. Thanks! Mohammed

6 8

about magnum capi for production
by Nguyễn Hữu Khôi 07 Feb '24

07 Feb '24

Hello guys. Could you share with me some experience about using Magnum CAPI in production? https://github.com/vexxhost/magnum-cluster-api Thank you so much. Nguyen Huu Khoi

6 44

[all] SQLAlchemy 2.x support in Bobcat
by Thomas Goirand 02 Feb '24

02 Feb '24

Hi, As you may know, and to my great frustration, I'm not the maintainer of SQLAlchemy in Debian, even though OpenStack is the biggest consumer of it. The current maintainer insists that he wants to upload SQLA 2.x in Unstable, potentially breaking all of OpenStack. At the present moment, if I understand correctly, we're not there yet, and Bobcat doesn't have such a support. It would be ok for me, *IF* there are patches available on master, that I could backport to Bobcat and maintain in the debian/patches folder of each project. However, the biggest current annoyance, is that I have no idea where we are at. Are we close to such a support? Is there a list of patches to apply on top of Bobcat that is maintained somewhere? Please enlighten me... :) Cheers, Thomas Goirand (zigo)

5 10

[murano][heat][mistral][tc] Maintenance of yaql library
by Takashi Kajinami 21 Jan '24

21 Jan '24

Hello, The yaql library is currently maintained under murano's governance and is used by not only murano but a few other projects such as heat and mistral. In heat the library is essential since we introduced yaql function support. Looking at gerrit groups, I noticed the yaql library is maintained by a very small core group[1] and release group[2], but I'm afraid most of the people in these groups are no longer active in the community. Because heat is dependent on the library, I'm wondering if some of the heat cores can be added to these two groups so that we (at least I) can help maintaining the library. I've checked the changes merged into murano repo for these two years but iiuc all of these changes are gate fix. Also a dummy change[3] indicates that its CI is currently broken. If the murano project is no longer active then I'm also willing to take over yaql and bring it to heat's governance (of cause only if it's acceptable). I'll add the current PTL as cc, but please let me know if anyone has any opinion or concern. # I'll add mistral tag because the project is also dependent on yaql. # Also tc is added because this discussion is related to project governance. Thank you Takashi Kajinami [1] https://review.opendev.org/admin/groups/cba4f6fd54c3cc0d5706fc50b5ebd758860… [2] https://review.opendev.org/admin/groups/d5e894b34e5ab5a9fc2938a0d4e7bc76fd2… [3] https://review.opendev.org/c/openstack/murano/+/901888

3 7

[nova][release][tooz] Hold your rechecks
by Sylvain Bauza 10 Jan '24

10 Jan '24

Hey folks, FWIW, our grenade check on SLURP upgrades is 100% failing now [1] due to a behavioural change in the tooz library [2] that's pulled by Cinder and which was released [3] and delivered to upper-contraints yesterday [4] There are different possibilities in order to unblock the gate, but until one of them is agreed and merged, please avoid wasting our CI resources by asking to recheck for something which will fail. Thanks, -Sylvain (who will send an email once the job is back up) [1] https://zuul.opendev.org/t/openstack/builds?job_name=grenade-skip-level-alw… [2] https://review.opendev.org/c/openstack/tooz/+/891355 [3] https://review.opendev.org/c/openstack/releases/+/899966 [4] https://review.opendev.org/c/openstack/requirements/+/901131

4 6

[designate] Proposal to deprecate the agent framework and agent based backends
by Michael Johnson 09 Jan '24

09 Jan '24

TLDR: The Designate team would like to deprecate the backend agent framework and the agent based backends due to lack of development and design issues with the current implementation. The following backends would be deprecated: Bind9 (Agent), Denominator, Microsoft DNS (Agent), Djbdns (Agent), Gdnsd (Agent), and Knot2 (Agent). Designate includes many backend DNS server drivers[1], many of which are "native" (also known as xfr type backends) backend implementations. In addition to the "native" backends, Designate has an agent backend[2] that supports other backends via an agent process. To quote the agent backend documentation[2]: This backend uses an extension[3] of the DNS protocol itself to send management requests to the remote agent processes, where the requests will be actioned. The rpc traffic between designate and the agent is both unauthenticated and unencrypted. Do not run this traffic over unsecured networks. Here are the reasons we are proposing to deprecate the agent framework now: 1. The agent protocol used by Designate is using an "unassigned"[4][5] DNS opcode (14) that is causing problems with the dnspython library >= 2.3.0 which is now validating the opcode when building DNS messages. It is a bad practice to use "unassigned" values as they may be officially assigned at any time, likely with an incompatible message format. 2. The agent backends are not tested in the OpenStack jobs[1]. 3. Many of the agent backends have been marked as "Experimental" since 2016 with no additional contributions beyond general repository code maintenance. 4. The protocol between the Designate worker process and the agent process is unauthenticated and unencrypted. 5. We do not know of a development resource to rewrite the agent framework protocol to address issue #1 and #4 above. 6. The introduction of catalog zones[6] may eliminate the need for some of the agent based backend drivers. By marking the agent framework and agent based backends "deprecated" in the Antelope cycle, we would remove the code no earlier than the "C" release of OpenStack (per the OpenStack deprecation policy[7]). In the meantime, issue #1 has been worked around by overriding the dnspython opcode validation[7] as needed in the Designate code (similar to a monkey patch). This is not a sustainable long term solution. The following backend agent based drivers would be marked "deprecated" in addition to the agent framework itself: Bind9 (Agent) Denominator Microsoft DNS (Agent) Djbdns (Agent) Gdnsd (Agent) Knot2 (Agent) We plan to propose patches for the deprecation over the next week, but will not merge them until at least January 24th to allow time for comment from the community. If you have concerns about this deprecation plan or are interested in rewriting the agent framework protocol to address the above issues, please reply to this announcement. Michael [1] https://docs.openstack.org/designate/latest/admin/support-matrix.html [2] https://docs.openstack.org/designate/latest/admin/backends/agent.html [3] https://github.com/openstack/designate/blob/master/designate/backend/privat… [4] https://www.rfc-editor.org/rfc/rfc6895.html#section-2.2 [5] https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-pa… [6] https://www.ietf.org/archive/id/draft-ietf-dnsop-dns-catalog-zones-08.txt [7] https://docs.openstack.org/project-team-guide/deprecation.html [8] https://review.opendev.org/c/openstack/designate/+/870678

3 13

[TC][Monasca] Proposal to mark Monasca as an inactive project
by Sławek Kapłoński 05 Jan '24

05 Jan '24

Hi, I just proposed patch [1] to mark Monasca as inactive project. It was discussed during last TC meeting on 03.10.2023. Reasons for that are described in the commit message of the [1] but I will also put all of it here: previous cycle there wasn't almost any contributions to that project and most active contributors in last 180 days were Elod Illes and Dr. Jens Harbott who were fixing some Zuul configuration issues only. Here are detailed statistics about Monasca projects: Validating Gerrit... * There are 7 ready for review patches generated within 180 days * There are 2 not reviewed patches generated within 180 days * There are 5 merged patches generated within 180 days * Unreviewed patch rate for patches generated within 180 days is 28.0 % * Merged patch rate for patches generated within 180 days is 71.0 % * Here's top 10 owner for patches generated within 180 days (Name/Account_ID: Percentage): - Dr. Jens Harbott : 42.86% - Joel Capitao : 28.57% - Elod Illes : 28.57% Validate Zuul... Set buildsets fetch size to 500 * Repo: openstack/monasca-log-api gate job builds success rate: 82% * Repo: openstack/monasca-statsd gate job builds success rate: 95% * Repo: openstack/monasca-tempest-plugin gate job builds success rate: 81% * Repo: openstack/monasca-common gate job builds success rate: 84% * Repo: openstack/monasca-kibana-plugin gate job builds success rate: 83% * Repo: openstack/monasca-ceilometer gate job builds success rate: 100% * Repo: openstack/monasca-events-api gate job builds success rate: 83% * Repo: openstack/monasca-ui gate job builds success rate: 88% * Repo: openstack/monasca-specs gate job builds success rate: 100% * Repo: openstack/monasca-grafana-datasource gate job builds success rate: 100% * Repo: openstack/monasca-persister gate job builds success rate: 98% * Repo: openstack/monasca-notification gate job builds success rate: 93% * Repo: openstack/monasca-thresh gate job builds success rate: 100% * Repo: openstack/monasca-api gate job builds success rate: 76% * Repo: openstack/monasca-agent gate job builds success rate: 98% * Repo: openstack/python-monascaclient gate job builds success rate: 100% * Repo: openstack/monasca-transform gate job builds success rate: 100% What's next? According to the "Emerging and inactive projects" document [2] if there will be no volunteers who will step up and want to maintain this project before Milestone-2 of the 2024.1 cycle (week of Jan 08 2024) there will be no new release of Monasca in the 2024.1 cycle and TC will discuss if project should be retired. So if You are interested in keeping Monasca alive and active, please reach out to the TC to discuss that ASAP. Thx in advance. [1] https://review.opendev.org/c/openstack/governance/+/897520 [2] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… -- Slawek Kaplonski Principal Software Engineer Red Hat

8 14

[openstack][kolla-ansible] kolla-ansible stucked when reconfigure
by Nguyễn Hữu Khôi 22 Dec '23

22 Dec '23

Hello guys. I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause. Would you share some experiences on it because my solution causes instances on host to timeout some seconds. Thank you. Nguyen Huu Khoi

2 7

[kolla-ansible][zun] Intention to drop support for Zun for the 2023.1 and path forward
by Jan Gutter 21 Dec '23

21 Dec '23

Hi folks, In yesterday's Kolla IRC meeting we discussed the path forward for Zun [1] and came to the conclusion that it might be better for all involved to drop support for Zun for the 2023.1 release until it can be updated to support the necessary dependencies. More specific details at the end of the message. With this in mind, the plan is to start this process in a week from now, unless a better solution is presented. There is still some time for discussion, please don't hesitate! There are two paths forward after this: If Zun can be updated to support the dependencies during the maintenance timeframe of 2023.1, then support would be restored. If no support is available at the time Caracal releases, however, it will be considered for removal in kolla and kolla-ansible. What this means: Kolla-ansible operators currently running Zun should hold off migration to 2023.1. Any development help would of course be appreciated. Please coordinate with the Zun project to help with development. Details on the dependency problems: * Currently, Zun has an external dependency on Docker 20.x, and uses a feature that has been removed in later versions of Docker. * This Docker feature also relies on an older version of etcd (3.3) * The mitigation [2] is to pin to the old versions of Docker and etcd, but this is not possible with Debian Bookworm hosts. Why this is a problem in Kolla-ansible (and Kolla): * This can't easily be solved by vendoring in an older version of etcd: the migration path it introduces to operators would incur a significant amount of risk. * Host level involvement is also particularly painful, the host version of Docker needs to connect to the etcd service running in the containers. * Even though the containers can be kept buildable and the configuration and inventory settings could be maintained, a working setup is not testable in CI. Thanks very much for everyone's patience, contributions and time! [1]: https://meetings.opendev.org/meetings/kolla/2023/kolla.2023-09-13-13.00.log… [2]: https://bugs.launchpad.net/zun/+bug/2007142

2 2

[all][tc][ptls] Eventlet: Python 3.12, and other concerning discoveries
by Jay Faulkner 20 Dec '23

20 Dec '23

Hi all, For those who aren't aware, eventlet (https://github.com/eventlet/eventlet) is a library that almost every OpenStack project depends on and uses heavily. It's the primary method in most OpenStack project for achieving any form of parallelism via greenthreading. A couple of weeks ago it was brought to my attention, in the #openstack-tc IRC channel, that the eventlet library was broken on python 3.12 due to changes to the SSL stdlib in python 3.12. I enlisted a G-Research Open Source Software (GR-OSS) developer from outside the OpenStack community, Itamar, to help dig into what would be required to get python 3.12 support for eventlet, and asked him to do an overall health check on the project. The results were extremely concerning, and are why I'm emailing the list today. First of all, the original concern -- python 3.12 support -- is being worked on right now; there is a PR up to migrate the SSL module here https://github.com/eventlet/eventlet/pull/817 and a PR up here: https://github.com/eventlet/eventlet/pull/820 to begin migrating from nose3, a deprecated testing framework used by eventlet which does not support python 3.12. It's worth noting that in the past (2020) https://github.com/eventlet/eventlet/issues/638 there was resistance to removing nose for testing due to at-the-time-desired python2.7 support. Even if they are able to be merged, these PRs only scratch the surface of the total effort needed to get eventlet working on python 3.12. Furthermore, activity on the eventlet project is extremely slim right now. There have been no commits to the primary branch in nearly eight months and it's been a single maintainer project for an extremely long time. There are also indications there may be external factors impacting the maintainers' availability. This lack of redundancy on a core library that we have no clear migration path off of is concerning as well. As a large community that uses eventlet; I hope we can improve it. Now we get to the cherry on top -- code quality and existing bugs. When Itamar began evaluating the overall state of the project, he found that approximately 80% of the unit tests were passing. Some of the failing unit tests were valid failures. In fact, one of the failure cases -- around locks not being properly greened on upgrade -- explains an Ironic database migrations unit test failure which we spent several weeks off-and-on debugging last cycle. Along these lines; although eventlet has CI configured; those tests have never passed: https://github.com/eventlet/eventlet/actions/workflows/test.yaml?query=is%3… . I don't have a specific proposal for a solution. I do know that right now, OpenStack services are built "all-in" on eventlet, with no obvious or easy migration path off. At least in the short term, keeping eventlet running well is critical to keeping OpenStack running well. I'm sending this email to raise an alarm about this situation, and to hope that community members can work together to solve the problem. Thanks for reading and your help, Jay Faulkner OpenStack TC Chair

8 15

[zaqar][tc] master and 2023.2 unit tests are broken
by Takashi Kajinami 19 Dec '23

19 Dec '23

Hello, Today I happened to notice that unit tests in zaqar is broken because some tests requires base test class from oslo.db which was removed in 13.0.0 release[1]. The 13.0.0 release of oslo.db was made during 2023.2 cucle so these tests are also broken in stable/2023.2 branch. In addition, py311 job was made voting in master but is broken and we probably need to fix bindep.txt so that it works in Debian 12. I'm sending this mail so that people maintaining the project is aware of the situation. I'll add [tc] tag because if the CI is not fixed then zaqar may become a new candidate for inactive projects. Thank you, Takashi [1] https://review.opendev.org/c/openstack/oslo.db/+/874239

2 1

Cyborg nova reports mdev-capable resource is not available
by Karl Kloppenborg 08 Dec '23

08 Dec '23

Hi Cyborg Team! Karl from Helm Team. When creating a VM with the correct flavor, the mdev gets created by cyborg agent and I can see it in the nodedev-list --cap mdev. However Nova then fails with: nova.virt.libvirt.driver [<removed>- - default default] Searching for available mdevs... _get_existing_mdevs_not_assigned /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py :8357 2023-09-21 14:34:47.808 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Available mdevs at: set(). 2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] No available mdevs where found. Creating an new one... _allocate_mdevs /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driv er.py:8496 2023-09-21 14:34:47.809 1901814 DEBUG nova.virt.libvirt.driver [<removed> - - default default] Attempting to create new mdev... _create_new_mediated_device /var/lib/openstack/lib/python3.10/site-packages/nova/virt/libvirt/driver.py:8385 2023-09-21 14:34:48.455 1901814 INFO nova.virt.libvirt.driver [<removed> - - default default] Failed to create mdev. No free space found among the following devices: ['pci_0000_4b_03_1', … <truncated list>]. 2023-09-21 14:34:48.456 1901814 ERROR nova.compute.manager [<removed> - - default default] [instance: 2026e2a2-b17a-43ab-adcb-62a907f58b51] Instance failed to spawn: nova.exception.ComputeResourcesUnavailable: Insufficient compute resources: mdev-capable resource is not available. Once this happened, ARQ removes the mdev and cleans up. I’ve got Cyborg 2023.2 running and have a device profile like so: karl@Karls-Air ~ % openstack accelerator device profile show e2b07e11-fe69-4f33-83fc-0f9e38adb7ae +-------------+---------------------------------------------------------------------------+ | Field | Value | +-------------+---------------------------------------------------------------------------+ | created_at | 2023-09-21 13:30:05+00:00 | | updated_at | None | | uuid | e2b07e11-fe69-4f33-83fc-0f9e38adb7ae | | name | VGPU_A40-Q48 | | groups | [{'resources:VGPU': '1', 'trait:CUSTOM_NVIDIA_2235_A40_48Q': 'required'}] | | description | None | +-------------+---------------------------------------------------------------------------+ karl@Karls-Air ~ % I can see the allocation candidate: karl@Karls-Air ~ % openstack allocation candidate list --resource VGPU=1 | grep A40 | 41 | VGPU=1 | 229bf15f-5689-3d2c-b37b-5c8439ea6a71 | VGPU=0/1 | OWNER_CYBORG,CUSTOM_NVIDIA_2235_A40_48Q | karl@Karls-Air ~ % Am I missing something critical here? Because I cannot seem to figure this out… have I got a PCI address wrong, or something? Any help from the Cyborg or Nova teams would be fantastic. Thanks, Karl.

4 9