March 2025 - openstack-discuss - lists.openstack.org

[ovs] [neutron] Egress traffic is not reaching table 71
by yardalgedal＠gmail.com 18 Mar '25

18 Mar '25

Hi, all! We are experimenting with bluefield-2 and neutron-openvswith-agent on it and have a big problem – all our egress traffic (from stateless point of view) are not leaving the bluefield. Symptoms are the next: the baremetal to which bluefield-2 is connected successfully responds on ping, but does not return any response for TCP traffic and can not send anything from inside. We have some security group with fully allowed egress rules. +--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+----------------------+ | ID | IP Protocol | Ethertype | IP Range | Port Range | Direction | Remote Security Group | Remote Address Group | +--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+----------------------+ | 14604165-e0c1-4bce-ae21-d20bf4165fa2 | None | IPv6 | ::/0 | | ingress | None | None | | 16f3f44f-1c9a-48f1-bcd0-7ff8df0cd30a | None | IPv4 | 0.0.0.0/0 | | ingress | None | None | | c7054cde-72ce-4dc8-a4e0-361df199204e | None | IPv6 | ::/0 | | egress | None | None | | d8bc8f82-f181-443d-a0ca-67cbae78b47f | None | IPv4 | 0.0.0.0/0 | | egress | None | None | +--------------------------------------+-------------+-----------+-----------+------------+-----------+-----------------------+----------------------+ and port to which it is assigned has port_security_enabled: True (if we are disabling port security, there is no problem anymore, it is actual only for enabled port security) we have disabled DVR and VxLAN tunnel here, we have native openvswitch firewall also [agent] tunnel_types = l2_population = true arp_responder = true extensions = qos baremetal_smartnic = True [securitygroup] firewall_driver = openvswitch [ovs] bridge_mappings = br-ex:br-int datapath_type = system ovsdb_connection = tcp:127.0.0.1:6640 #local_ip = tunnel_bridge = of_inactivity_probe = 60 ovsdb_timeout = 60 when we're sending some traffic, all flows in table 71 (BASE_EGRESS_TABLE) has n_packets = 0, so they are not reaching it. in table 60 (TRANSIENT_TABLE), which should send traffic to the table 71, on flow which should do it we can see n_packets=0 as well cookie=0x6cc97f12f173f538, duration=2086.129s, table=60, n_packets=0, n_bytes=0, priority=100,in_port=pf0hpf actions=load:0x2b->NXM_NX_REG5[],load:0x1->NXM_NX_REG6[],resubmit(,71) at the same time, the counter for NORMAL flow is growing, so packets are probably going to here (not sure cookie=0x6cc97f12f173f538, duration=2126.547s, table=60, n_packets=139340, n_bytes=6817502, priority=3 actions=NORMAL on trace we can see that the packet was directly sent to port 1 instead of going via the table ovs-appctl ofproto/trace br-int 'dl_src=a0:88:c2:eb:15:4e,dl_dst=00:1c:73:00:57:ff,ip,nw_dst=DST_IP,nw_src=SRC_IP,in_port=43,nw_ttl=64,nw_proto=6,tp_dst=80' Flow: tcp,in_port=43,vlan_tci=0x0000,dl_src=a0:88:c2:eb:15:4e,dl_dst=00:1c:73:00:57:ff,nw_src=SRC_IP,nw_dst=DST_IP,nw_tos=0,nw_ecn=0,nw_ttl=64,nw_frag=no,tp_src=0,tp_dst=80,tcp_flags=0 bridge("br-int") ---------------- 0. priority 0, cookie 0x699a137a0a73d6a8 NORMAL -> forwarding to learned port bridge("br-int") ---------------- 0. in_port=5,dl_vlan=1, priority 4, cookie 0x699a137a0a73d6a8 set_field:4297->vlan_vid NORMAL -> forwarding to learned port Final flow: unchanged Megaflow: recirc_id=0,eth,ip,in_port=43,dl_src=a0:88:c2:eb:15:4e,dl_dst=00:1c:73:00:57:ff,nw_frag=no Datapath actions: push_vlan(vid=201,pcp=0),1 any ideas why it happens or any proposals for future debugging? Thank you in advance!

1 0

Controller node openstack LXC containers clashing on IP subnets
by Sarah Thompson 17 Mar '25

17 Mar '25

Hi all, I've got much further now, and am now tracking down networking issues. I'm pretty sure everything is now installing, but I'm seeing a systematic issue. My test network that I'm running the openstack instances on is 10.0.1.x -- the VMs all have fixed IP addresses, can talk to each other, etc, nothing weird going on. haproxy is installing and running, but throwing 503 errors. Digging into this, it seems that there are some issues with the network configs of at least some of the LXC containers. The one I'm seeing that's preventing Ansible from completing the infrastructure setup is the repo server container. If I attach to the container, the keepalive comes back correctly. Externally to the container, the HTTP connection is rejected. Looking into the reasons, it looks like there are 3 networks visible from inside the container: ens18: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.1.38 netmask 255.255.255.0 broadcast 10.0.1.255 inet6 fe80::216:3eff:feab:ab45 prefixlen 64 scopeid 0x20<link> ether 00:16:3e:ab:ab:45 txqueuelen 1000 (Ethernet) RX packets 96 bytes 6952 (6.9 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 16 bytes 1236 (1.2 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.0.3.113 netmask 255.255.255.0 broadcast 10.0.3.255 inet6 fe80::216:3eff:fe45:64d6 prefixlen 64 scopeid 0x20<link> ether 00:16:3e:45:64:d6 txqueuelen 1000 (Ethernet) RX packets 351 bytes 425285 (425.2 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 263 bytes 19114 (19.1 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1000 (Local Loopback) RX packets 2939 bytes 208134 (208.1 KB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 2939 bytes 208134 (208.1 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 The 10.0.1.38 address seems to be the problem. I think this is an internally routed subnet, not the actual physical subnet (note the lack of a gateway address, and 10.0.1.38 is definitely not being allocated via the DCHP server). Looking at some other containers, I'm seeing 10.0.2.x and 10.0.3.x there, so obviously this is being allocated on the fly either by lxc or the container(s). TL;DR: The internal 10.0.1.x is clashing with the physical 10.0.1.x network, which is almost certainly why the keepalive is failing. Does anyone have any idea how to fix the configuration to use some other CIDR block for this? I'd like to avoid the extreme pain of remapping my physical network (this is a way more complicated problem than a few test nodes unfortunotely!) Thank you in advance, Sarah Thompson -- [s]

3 4

PCI alias rtx is not defined
by Felipe Mogollon 17 Mar '25

17 Mar '25

Dear all, I am suffering a problem in my OpenStack servers. I have a 3 server configuration: - controller + compute - 2 x compute nodes The controller node has an Nvidia L4 configured in vGPU mode. One of the compute nodes has another Nvidia L4 in vGPU mode. Both servers are running fine, and I can deploy instances with vGPU capabilities. The problem comes with my 2nd compute node; that node has a Nvidia RTX4000 that can only work in passthrough mode. , I am pretty sure that in past times I managed to deploy this compute node with an Nvidia RTX 4000 in passthrough mode as a compute node of another OpenStack server configuration, but nowadays I am not able to use that GPU in passthrough mode and I don't know why: - I have configured IOMMU in compute node - I have configured nova.conf with pci alias and whitelist in compute node - I have configured nova.conf with pci alias and enabled filters in controller node - I have created a flavor and set up pci_passthrough:rtx:1 to it's configuration As far as I know, those are the steps to configure a GPU passthrough in a compute node in a Openstack multi-node configuration... But when I try to deploy a new instance I am always getting a "PCI alias rtx is not defined (HTTP 400)" error. Any ideas about what could be the problem? Is there any incompatibility with vGPU and passthrough working modes? Thanks in advance. Felipe -- [image: Vicomtech] <https://www.vicomtech.org> Industry Division Juan Felipe Mogollón Rodríguez Senior Researcher Digital Media and Communications fmogollon(a)vicomtech.org +(34) 943 30 92 30 The information contained in this electronic message is intended only for the personal and confidential use of the recipients. If you have received this e-mail by mistake, please, notify us and delete it. Avoid printing this message if it is not strictly necessary.

1 0

Routed provider networks
by Alvin Starr 17 Mar '25

17 Mar '25

I have a couple of small flat networks that I am using for a POC project. I am trying to use the routed provider networks features to create a single public net. I delete the old networks and try to install the following network and it looks like it works in that I see the 2 subnets as part of the single public-net. The trouble is that I cannot create a router with an interface in the public-net. Is there anything special that I need to do to create a router on the public-net? I changed the addresses so the subnets are likely wrong. I am installing with openstack-ansible using stable/2023.2 onto Centos-Stream. # Create the parent network with first physical segment PUBNET_ID=$(openstack network create --share \ --provider-physical-network pub204 \ --provider-network-type flat \ --external \ public-net \ -f value -c id ) FIRST_SEG=$(openstack network segment list --network ${PUBNET_ID} -f value -c ID) openstack network segment set --name segment-pub204 ${FIRST_SEG} # Add second physical segment to the network openstack network segment create \ --physical-network pub203 \ --network-type flat \ --network public-net \ segment-pub203 # Create subnet for pub203 segment openstack subnet create --network public-net \ --network-segment segment-pub203 \ --ip-version 4 \ --subnet-range 1.2.50.192/28 \ --allocation-pool start=1.2.50.196,end=1.2.50.206 \ --gateway 1.2.50.193 \ --dhcp \ public-subnet203 # Create subnet for pub204 segment openstack subnet create --network public-net \ --ip-version 4 \ --network-segment segment-pub204 \ --subnet-range 1.2.48.128/29 \ --allocation-pool start=1.2.48.132,end=1.2.48.134 \ --gateway 1.2.48.129 \ --dhcp \ public-subnet204 -- Alvin Starr || land: (647)478-6285 Netvel Inc. || home: (905)513-7688 alvin(a)netvel.net ||

1 0

[openstack-ansible]Playbooks failing after upgrading lxc containers from ubuntu18 to ubuntu20 due to python version mismatch.
by Danish Khan 17 Mar '25

17 Mar '25

Dear All, We are upgrading one of our openstack cluster from ussuri to wallaby and for this we are first upgrading OS of lxc container from ubuntu to 18 to 20(hosts OS is also upgraded to 20). We are destroying LXC containers(which don't form quorum like nova,neutron,heat,cinder,horizon) of one infra node and then creating it again. The new containers are getting created with Ubuntu 20 as the host OS is 20. Playbook for cinder, nova neutron were completed successfully for new containers of ubuntu 20. But Playbook is failing for heat, placement and memcached due to python version mismatch: fatal: [host5_heat_api_container-25f90e52]: FAILED! => {"attempts": 5, "changed": false, "cmd": ["/openstack/venvs/heat-21.2.5/bin/pip3", "install", "-U", "--constraint", "/openstack/venvs/heat-21.2.5/global-constraints.txt", "--constraint", "/openstack/venvs/heat-21.2.5/constraints.txt", "--pre", "--log", "/var/log/python_venv_build.log", "cryptography", "keystonemiddleware", "openstack-heat", "osprofiler", "pymemcache", "PyMySQL", "python-ceilometerclient", "python-glanceclient", "python-heatclient", "python-keystoneclient", "python-memcached", "python-neutronclient", "python-novaclient", "python-openstackclient", "python-swiftclient", "python-troveclient", "systemd-python"], "msg": "stdout: Ignoring python-saharaclient: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring python-saharaclient: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring python-saharaclient: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring oslo.limit: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring sphinxcontrib-actdiag: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring os-api-ref: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring oslo.concurrency: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring osprofiler: markers 'python_version == \"3.7\"' don't match your environment\nIgnoring os-resource-classes: markers 'python_version == \"2.7\"' don't match your environment\nIgnoring os-resource-classes: markers 'python_version == \"3.6\"' don't match your environment\nIgnoring os-resource-classes: markers I understand that 2.7 version is depricated but where should make changes to fix it? https://docs.openstack.org/releasenotes/python-heatclient/ussuri.html I tried updating few variable to fix python version like: /etc/ansible/roles/os_heat/defaults/main.yml #heat_venv_python_executable: "{{ openstack_venv_python_executable | default('python3') }}" updated it from python2 to python 3 Tried few more variables but that didn't work. Any idea how we can fix this version mismatch issue ? Regards, Danish

2 1

[Devstack] Trying to setup development environment on VM
by Ishan Shanware (ishanwar) 17 Mar '25

17 Mar '25

Hello Everyone, I am currently trying to setup in all-in-one development environment for OpenStack on a Debian VM. The VM has 16 GB RAM and 4 CPUs. However, I face an error while trying to run FORCE=yes ./stack.sh. I get an error that keystone does not start. This causes the entire process to fail. Requesting assistance in solving this issue. I am following the this guide: https://docs.openstack.org/devstack/latest/guides/single-machine.html PFA the error log and error message on the terminal. [Call Trace] ./stack.sh:1126:start_keystone /opt/stack/devstack/lib/keystone:549:die [ERROR] /opt/stack/devstack/lib/keystone:549 keystone did not start Error on exit World dumping... see /opt/stack/logs/worlddump-2024-11-18-150808.txt for details ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) Thanks in advance, Ishan

3 5

[neutron] Bug deputy report for week 11 of 2025
by Miguel Lavalle 16 Mar '25

16 Mar '25

I was the Neutron bug deputy last week. Following is a brief summary of the reported bugs: High bugs -------------- https://bugs.launchpad.net/neutron/+bug/2101839 [fullstack] Random errors during the execution of ``wait_until_true``. Assignee Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/943942 https://bugs.launchpad.net/neutron/+bug/2101857 ``neutron-tempest-plugin-ovn-ubuntu-jammy`` job unstable. No owner. Jobs neutron-tempest-plugin-*-ubuntu-jammy`` were made no voting: https://review.opendev.org/c/openstack/neutron-tempest-plugin/+/944119 Medium bugs ------------------ https://bugs.launchpad.net/neutron/+bug/2101840 ovs jobs randomly failing with guest vms not getting ip from dhcp. Assignee Lajos Katona https://bugs.launchpad.net/neutron/+bug/2101998 [OVN][Placement] Deleting a configured resource provider does not remove the Placement entry. Assignee Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/944053 https://bugs.launchpad.net/neutron/+bug/2102083 ``SingletonDecorator`` class to allow destroying instances. Assignee Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/944099 https://bugs.launchpad.net/neutron/+bug/2101871 Use of revision_number for router update on 2023.2 Bobcat cause error 500. Waiting for reproducer info from submitter Low bugs ------------- https://bugs.launchpad.net/neutron/+bug/2102095 [OVN][Placement] Change the message level when a Placement exception happens. Assignee Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/944101 RFEs ----- https://bugs.launchpad.net/neutron/+bug/2102090 [RFE][OVN][Placement] With Neutron API WSGI module, the initial chassis configuration is not executed. https://bugs.launchpad.net/neutron/+bug/2102165 [RFE] Forbid usage of certain IP addresses in subnet pools https://bugs.launchpad.net/neutron/+bug/2102184 [RFE] Project-Specific QoS Controls for Granular Resource Management Incomplete --------------- https://bugs.launchpad.net/neutron/+bug/2102044 Nova-API problem - cannot list instances anymore

1 0

[ci][all] Migrating doc job back to ubuntu jammy
by Takashi Kajinami 16 Mar '25

16 Mar '25

Hi, I noticed that doc job in taskflow repo was broken recently. Example: https://review.opendev.org/c/openstack/taskflow/+/944495?tab=change-view-ta… Digging into the failure, it seems the failure is caused by api doc build, trying to parse a module importing asyncore, because asyncore is no longer available in Python 3.12. IIUC Doc job was recently migrated from Ubuntu Jammy to Ubuntu Noble, which means that the default python version was bumped to 3.12. We are aware of the deprecation/removal of asyncore and already deprecated the implementation in taskflow. However deprecation was merged during 2024.2 cycle so we can't remove it until 2025.2 cycle to follow SLURP policy. There might be a way to drop the asyncore-related implementations from docs to unblock the doc job, I'm not sure if that is the right approach given the fact the implementation is deprecated but is still valid in older python versions. Thus, I wonder if we can keep running doc job in Jammy for 2025.1, which still supports Ubuntu Jammy, and then migrate the job again during 2025.2 . An alternative approach is to remove the asyncore implementation during this cycle, breaking SLURP policy, but I'm not sure if this can be the preferred option. Thank you, Takashi -- Takashi Kajinami irc: tkajinam github: https://github.com/kajinamit launchpad: https://launchpad.net/~kajinamit

2 4

Deploiment in 2025
by Albert Shih 16 Mar '25

16 Mar '25

Hi everyone Long time ago I deploy a openstack mostly manually, then I integrated in our deployment tool (puppet). Now I need to re-install everything. I'm searching the «best» method to do that. My deployment would be on bare-metal, I don't need/want a deployment of CEPH (I already have my ceph cluster). Just need the cinder to use the ceph. The main focus is the easy way to upgrade. I don't have a lot of hardware so I can afford to have a «test» cluster for upgrading. I'm thinking of kolla, what do you think ? I see they are now openstack-helm project, which If I understand correctly use a k8s to manage the deployment of openstack. Is that correct ? With those tools is they are any limitation about the result ? Or can I got the most the main features of openstack ? What would you recommend ? Regards. -- Albert SHIH 🦫 🐸 Observatoire de Paris France Heure locale/Local time: mer. 05 mars 2025 17:13:54 CET

8 16

[all][tc][ptl][qa] Migrating upstream CI/CD jobs to Ubuntu Noble (24.04)
by Ghanshyam Mann 14 Mar '25

14 Mar '25

Hello Everyone, As per the 2025.2 cycle testing runtime, we need to run upstream CI/CD jobs on Ubuntu Noble (24.04) [1]. For better tracking and speeding up this work, it is the community-wide goal for this cycle[2]. This is the etherpad which has the migration plan and will be used for the tracking: - https://etherpad.opendev.org/p/migrate-to-noble I have prepared the change(WIP) for base jobs migration on Noble (openstack tox jobs, devstack, and tempest) which will migrate most of the projects jobs. Tempest change[3] include dependency on openstack tox, devstack base jobs changes so testing project gate using the the tempest change will test the complete set. Refer to the etherpad for the complete migration plan/steps. * Deadline: Nov 29th 2024 (R18) is the deadline to test/fix the projects jobs and to merge the base jobs running on Noble (openstck tox, devstack, and tempest). [1] https://governance.openstack.org/tc/reference/runtimes/2025.1.html [2] https://governance.openstack.org/tc/goals/selected/migrate-ci-jobs-to-ubunt… [3] https://review.opendev.org/c/openstack/tempest/+/932156 -gmann

2 9