February 2025 - openstack-discuss

[manila] Removal of the Standalone CephFS NFS protocol helper in Epoxy
by Ashley Rodriguez 01 May '25

01 May '25

Hi zorillas! As per subject line, we will be deprecating the standalone CephFS NFS protocol helper in this cycle. This is following our deprecation warning we put out in 2024.1 / Caracal [1] [2]. We've supported using a clustered Ceph NFS service since the Zed release. We've also added upgrade helpers to migrate from a standalone NFS-Ganesha service to the ceph-orch-deployed clustered NFS service; this code was backported to stable/2023.1 (Antelope). Although the upgrade path is disruptive, there is a UX improvement in Manila to ease this migration through surfacing “preferred” export paths. Moving forward, deployments must have “cephfs_nfs_cluster_id” in your configuration, otherwise the driver will fail to start. Manila will not make any export changes to NFS-Ganesha, so upgrading manila without an NFS cluster will mean that only the control plane is not going to work; but, data plane will remain unaffected. To assist migration, admins can also configure "cephfs_ganesha_export_ips" (or alternatively, "cephfs_ganesha_server_ip") alongside "cephfs_nfs_cluster_id". Setting these options will allow the CephFS driver to report additional export paths. These additional export paths will have the "preferred" metadata key set to False. Best, Manila team [1] https://review.opendev.org/c/openstack/manila/+/912648 [2] https://docs.openstack.org/manila/2024.2/configuration/shared-file-systems/…

5 7

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

[oslo] python-consul2 is unmaintained
by Takashi Kajinami 21 Apr '25

21 Apr '25

Hello, I recently noticed that python-consul2 library[1], which is used by the consul coordination driver in tooz, hasn't been updated for 3+ years. Looking at issues in the project, there is an indication that the library is not compatible with ACL API in consul >= 1.11[2]. There is another fork created[3] and this includes fixes to support new API endpoints. Although the driver in tooz does not directly use the feature, I think it's better to discuss our plan to mitigate risks caused by the unmaintained library. As far as I can think of, there are three options now 1. Deprecate consul driver and remove it 2. Keep using python-consul2 now. At least basic functionality is proven to work by our functional tests (Note that the tests do not cover all configuration patterns) 3. Replace python-consul2 by py-consul Before we start the overall discussions I'd like to ask a few questions to the wider audience, as our inputs. - Is anyone using the consul coordination driver now ? - Does anyone have experience with python-consul2 and/or py-consul ? Note that this discussion affects the http messaging driver[4] currently proposed, because it also requires client library to communicate with Consul. Actually I noticed the situation when I noticed that the requirement check job is failing because the library is not part of global requirements list... Thank you, Takashi Kajinami [1] https://github.com/poppyred/python-consul2 [2] https://github.com/poppyred/python-consul2/issues/42 [3] https://github.com/criteo-forks/py-consul [4] https://review.opendev.org/c/openstack/oslo.messaging/+/912499 -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 3

[octavia] Purging ERROR'ed amphora and load balancers stuck in transitional states
by Paul Browne 11 Apr '25

11 Apr '25

Hello Octavia users, For a while I've been trying to find a systematic way to remove Octavia LBs+Amphora that have become stuck in ERROR or transitional PENDING_* states ; $ openstack loadbalancer amphora list -f value | grep ERROR a2323f14-3aaa-418a-8249-111aaa9c21fe 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR MASTER 10.8.0.242 192.168.3.247 e5b236ba-e7ee-4ed7-9f58-57ce7a408489 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe ERROR BACKUP 10.8.0.190 192.168.3.247 6b556f28-93c9-49dd-b6ee-4379288e7957 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR MASTER 10.8.0.39 192.168.3.126 c669db5d-8686-4d5c-9e95-e02030b34301 d5e402fe-2c4b-49af-a700-532cb408cee5 ERROR BACKUP 10.8.0.174 192.168.3.126 $ openstack loadbalancer list -f value | grep -vi active 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe k8s-clusterapi-cluster-default-ci-6386871107-kube-upgrade-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.247 PENDING_UPDATE amphora d5e402fe-2c4b-49af-a700-532cb408cee5 k8s-clusterapi-cluster-default-ci-6386871107-latest-kubeapi 3a06571936a0424bb40bc5c672c4ccb1 192.168.3.126 PENDING_DELETE amphora These resources are marked immutable and so cannot be failed over or deleted; $ openstack loadbalancer amphora failover a2323f14-3aaa-418a-8249-111aaa9c21fe Load Balancer 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe is immutable and cannot be updated. (HTTP 409) (Request-ID: req-6e66c4e8-c3d3-4549-a03c-367017c8c8b3) $ openstack loadbalancer failover 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe Invalid state PENDING_UPDATE of loadbalancer resource 1fa7bd54-f60c-420c-94f3-d4c02f03d4fe (HTTP 409) (Request-ID: req-84b44212-e7a8-4101-a16f-18c774c0577e) The backing Nova instances for these Amphora do seem to exist and be in good working order. Is there any API way to purge these out of Octavia's service state, or would (very careful) DB hackery be required here? Many thanks, Paul Browne ******************* Paul Browne Research Computing Platforms University Information Services Roger Needham Building JJ Thompson Avenue University of Cambridge Cambridge United Kingdom E-Mail: pfb29(a)cam.ac.uk<mailto:pfb29@cam.ac.uk> Tel: 0044-1223-746548 *******************

4 6

GPU PCI passthrough woes.
by Andy Speagle 09 Apr '25

09 Apr '25

Hey Folks, I could use a little assistance getting GPU passthrough working. I had this working already for one flavor of nvidia gpu... and I've added some hosts with a much newer gpu... I've updated the pci_alias and pci_passthrough variables and those seem to be getting set properly in nova.conf passthrough_whitelist = [{"vendor_id":"10de", "product_id":"1b06"},{"vendor_id":"10de", "product_id":"26b9"}] alias = {"name": "gpu", "product_id": "1b06", "vendor_id": "10de"} alias = {"name": "gpu-l40s", "product_id": "26b9", "vendor_id": "10de"} I believe I have all of the iommu stuff configured and have the pci- stub module entries... dmesg output shows that the GPUs are being claimed by the stub module. $ openstack flavor show t1.small_gpu_l40s +----------------------------+--------------------------------------+ | Field | Value | +----------------------------+--------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | description | None | | disk | 0 | | id | af70c94e-0026-4a39-bc1e-dfb93b286a54 | | name | t1.small_gpu_l40s | | os-flavor-access:is_public | True | | properties | pci_passthrough:alias='gpu-l40s:1' | | ram | 2048 | | rxtx_factor | 1.0 | | swap | 0 | | vcpus | 1 | +----------------------------+--------------------------------------+ Yet... I can't seem to get an instance to run using that new flavor... keeps complaining about there not being enough hosts available. Fault: code: 500 created: 2024-10-30T02:04:33Z message: "No valid host was found. There are not enough hosts available." details: | Traceback (most recent call last): File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1580, in schedule_and_build_instances host_lists = self._schedule_instances(context, request_specs[0], File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 940, in _schedule_instances host_lists = self.query_client.select_destinations( File "/usr/lib/python3/dist- packages/nova/scheduler/client/query.py", line 41, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj, File "/usr/lib/python3/dist-packages/nova/scheduler/rpcapi.py", line 160, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/client.py", line 189, in call result = self.transport._send( File "/usr/lib/python3/dist- packages/oslo_messaging/transport.py", line 123, in _send return self._driver.send(target, ctxt, message, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 689, in send return self._send(target, ctxt, message, wait_for_reply, timeout, File "/usr/lib/python3/dist- packages/oslo_messaging/_drivers/amqpdriver.py", line 681, in _send raise result nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. Traceback (most recent call last): File "/usr/lib/python3/dist- packages/oslo_messaging/rpc/server.py", line 241, in inner return func(*args, **kwargs) File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 223, in select_destinations selections = self._select_destinations( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 250, in _select_destinations selections = self._schedule( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 416, in _schedule self._ensure_sufficient_hosts( File "/usr/lib/python3/dist-packages/nova/scheduler/manager.py", line 455, in _ensure_sufficient_hosts raise exception.NoValidHost(reason=reason) nova.exception.NoValidHost: No valid host was found. There are not enough hosts available. Any clues on how/where to dig into this more to see what might be missing? Thanks. -- Andy Speagle

3 2

bug report
by Xavier Benjamin 04 Apr '25

04 Apr '25

whenever i try to connect to trove pts servers it would usually say fails to authenticate or something. Can you fix that please?

2 2

[openstack-helm] Merge openstack-helm and openstack-helm-infra git repos
by Vladimir Kozhukalov 31 Mar '25

31 Mar '25

Hi dear Openstack-Helmers, I would like to suggest merging the two project git repos: https://opendev.org/openstack/openstack-helm-infra.git https://opendev.org/openstack/openstack-helm.git At some point it seemed reasonable to have separate git repos for openstack and infrastructure charts. But now it looks like we are experiencing more cons than pros. When working on some features we are forced to prepare sets of interdependent patches. Some code can be deduplicated. Also when we publish charts we append two commit hashes as package metadata. Also we can reduce the number of test jobs if we have all chats in one repo. Other people also proposed this earlier. So if there are no objections let's make it done. I understand this is going to affect many of our users. Please express your opinions so we don't break any downstream. If we all agree about this, then I would say we can do this in the mid of March/2025 before 2025.2 PTG. The procedure is going to be smth. like 1. Freeze the infra repo and prepare retirement PSs for it 2. Merge openstack-helm-infra repo into openstack-helm ``` git clone https://opendev.org/openstack/openstack-helm.git cd openstack-helm git remote add infra https://opendev.org/openstack/openstack-helm-infra.git git fetch infra git checkout -b infra infra/master git checkout master git merge --allow-unrelated-histories infra # resolve not so many conflicts git review ``` 3. Accept merger PS 4. Update documentation 5. Accept infra retirement PSs (openstack-helm-infra and project-config) -- Best regards, Kozhukalov Vladimir

4 10

[mistral][heat][watcher][aodh] croniter being abandoned
by Thierry Carrez 22 Mar '25

22 Mar '25

Hi everyone, It was brought to my attention that the croniter maintainer has decided to abandon his project, due to not wanting to deal with his perceived exposure to the EU Cyber-Resilience act (CRA). https://pypi.org/project/croniter/#disclaimer The maintainer indicates that you should no longer use it and it may be "unpublished" any time after March 15, 2025. Croniter is being used in at least Mistral, Heat, Watcher, and Aodh as a dependency. While this is clearly an overreaction (the CRA only applies to "economic operators" putting "products with digital elements" on the market "in the course of a commercial activity"), we should probably prepare to migrate away from this dependency, or maintain a fork of it. -- Thierry Carrez (ttx)

5 8

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 16 Mar '25

16 Mar '25

Hello Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [Call Trace] ./stack.sh:1126:start_keystone /opt/stack/devstack/lib/keystone:549:die [ERROR] /opt/stack/devstack/lib/keystone:549 keystone did not start Error on exit World dumping... see /opt/stack/logs/worlddump-2024-11-18-150808.txt for details ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) Thanks in advance, Ishan

3 5