Hello Brian, Thanks for your answer. The thing is that I need Wallaby because of compatibility with the specific version of Openshift (v4.17) which I would like to deploy on top of Openstack. I also suspected that privsep is not configured properly but based on the line in the log: 2025-05-05 13:22:40.180 226859 INFO oslo.privsep.daemon [-] privsep process running with capabilities (eff/prm/inh): CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/CAP_DAC_OVERRIDE|CAP_DAC_READ_SEARCH|CAP_NET_ADMIN|CAP_SYS_ADMIN/none it is supposed to have SYS_ADMIN privileges which should allow for the modification of the interface. I also tried explicitly passing SYS_ADMIN privileges through [privsep] part of neutron.conf but the result was the same. No other operations are failing similarly. Best regards, Davorin

Hi, Correct the proccess is trying to delete IPv6 address. I got the thing to work by deleting IPv6 address from the interface: [root@overcloud-computesriov-0 ~]# ip -netns ovnmeta-fae7b699-3086-4213-bec7-9d91737c934c addr del fe80::ac69:a2ff:feb3:89c9/64 dev tapfae7b699-31 [root@overcloud-computesriov-0 ~]# ip a s 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: tapfae7b699-31@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether fa:16:3e:d0:d6:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 169.254.169.254/32 brd 169.254.169.254 scope global tapfae7b699-31 valid_lft forever preferred_lft forever inet 10.0.0.2/16 brd 10.0.255.255 scope global tapfae7b699-31 valid_lft forever preferred_lft forever Now the metadata service is accessible and it looks as it should. I will try to figure out what is making the IPv6 address attach to the interface by default since I installed Wallaby on CentOS 9 machines (it is compatible with CentOS8 which is EOL). I earlier had some problems because of this so I had to work around them. I will let you know what I find. Thank you very much! Br, Davorin

On 06/05/2025 14:45, davorin.mikulic@gmail.com wrote:

Also I checked versions of ip commands on the host (CentOS9) and the container (CentOS8) and there is indeed a mismatch as Sean suspected:

Host: [root@overcloud-computesriov-0 ~]# ip -V ip utility, iproute2-6.2.0, libbpf 1.4.0

Container: root@overcloud-computesriov-0 ~]# podman exec -it d6d9ac1a314d bash bash-4.4$ ip -V ip utility, iproute2-6.2.0, libbpf 0.5.0

What is interesting is that container adds the right ip address after I manually delete it with no problem. Also I can delete any ip manually from the container as well.

But process is unable to delete the undeeded IPv6 on creation of the VM nor DHCP and metadata IPv4 addresses after VM deletion. From the logs I can also see that it is unable to delete the veth pair after VM deletion.

I am guessing that add commands are more forgiving than add commands regarding the syntax.

If someone can think of any workaround for this please feel free to share. :)

the two some what obvious workaround the come to mind are as follows bind mount the host ip util into the container, that bascaily adding -v /usr/bin/ip:/usr/bin/ip to the podman command line or matching the contaiern and os version. i belive triple also has a script to allow specific comannd to be executed on the host and i tought it already used that for iptables so updating the ip command to use that wrapper may also be an option. i'm not sure how practical any of those options are for your deployment however. This would be a tripleo bug or a operator(human) one rather then a neturon bug as its related ot how its been packaged and deployed in your specific env. i belive downstream we allowed rhel 9 openstack container to be run on rhel 8 host but not the other way around. with that said i know for things like libvirt we built 2 version and required that the container and host os were the same i dont know which camp the ovn metadta container fell into(must align host and guest os or can differ with a wrapper to paper over the differnce). i also dont know exactly where that was implemtned in tripleo so i cant point you at an example.

Best regards, Davorin

Dear everyone, I have a working WA for the problem. There is a problem with pyroute2 0.6.6 version which is installed in the container. I upgraded the version to 0.6.11 and replaced the running container. I got it to work by editing the container image and replacing it as following: [root@overcloud-computesriov-16 new_ovn_metadata_agent_image]# cat Dockerfile FROM uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-neutron-metadata-agent-ovn:current-tripleo USER root # Just upgrade pyroute2 directly using existing pip RUN python3 -m pip install --no-cache-dir --upgrade "pyroute2==0.6.11" # (Optional) Switch back to original user USER neutron [root@overcloud-computesriov-16 new_ovn_metadata_agent_image]# podman build -t custom-neutron-metadata-agent . podman container run --name ovn_metadata_agent --cgroupns=host --conmon-pidfile /run/ovn_metadata_agent.pid --detach=true --env KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --env TRIPLEO_CONFIG_HASH=b0f0231085155addf0474368939de6e0 --healthcheck-command /openstack/healthcheck --label config_id=tripleo_step4 --label container_name=ovn_metadata_agent --label managed_by=tripleo_ansible --label "config_data={'cgroupns': 'host', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'TRIPLEO_CONFIG_HASH': 'b0f0231085155addf0474368939de6e0'}, 'healthcheck': {'test': '/openstack/healthcheck'}, 'image': 'uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-neutron-metadata-agent-ovn:current-tripleo', 'net': 'host', 'pid': 'host', 'privileged': True, 'restart': 'always', 'start_order': 1, 'volumes': ['/etc/hosts:/etc/hosts:ro', '/etc/localtime:/etc/localtime:ro', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '/dev/log:/dev/log', '/etc/puppet:/etc/puppet:ro', '/var/log/containers/neutron:/var/log/neutron:z', '/var/lib/kolla/config_files/ovn_metadata_agent.json:/var/lib/kolla/config_files/config.json:ro', '/var/lib/config-data/puppet-generated/neutron:/var/lib/kolla/config_files/src:ro', '/lib/modules:/lib/modules:ro', '/run/openvswitch:/run/openvswitch:shared,z', '/var/lib/neutron:/var/lib/neutron:shared,z', '/run/netns:/run/netns:shared', '/var/lib/neutron/kill_scripts:/etc/neutron/kill_scripts:shared,z', '/var/lib/neutron/ovn_metadata_haproxy_wrapper:/usr/local/bin/haproxy:ro', '/usr/sbin/ip:/usr/sbin/ip']}" --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/ovn_metadata_agent.log --network host --pid host --privileged=true --volume /etc/hosts:/etc/hosts:ro --volume /etc/localtime:/etc/localtime:ro --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /dev/log:/dev/log --volume /etc/puppet:/etc/puppet:ro --volume /var/log/containers/neutron:/var/log/neutron:z --volume /var/lib/kolla/config_files/ovn_metadata_agent.json:/var/lib/kolla/config_files/config.json:ro --volume /var/lib/config-data/puppet-generated/neutron:/var/lib/kolla/config_files/src:ro --volume /lib/modules:/lib/modules:ro --volume /run/openvswitch:/run/openvswitch:shared,z --volume /var/lib/neutron:/var/lib/neutron:shared,z --volume /run/netns:/run/netns:shared --volume /var/lib/neutron/kill_scripts:/etc/neutron/kill_scripts:shared,z --volume /var/lib/neutron/ovn_metadata_haproxy_wrapper:/usr/local/bin/haproxy:ro localhost/custom-neutron-metadata-agent:latest 245 podman container run --replace --name ovn_metadata_agent --cgroupns=host --conmon-pidfile /run/ovn_metadata_agent.pid --detach=true --env KOLLA_CONFIG_STRATEGY=COPY_ALWAYS --env TRIPLEO_CONFIG_HASH=b0f0231085155addf0474368939de6e0 --healthcheck-command /openstack/healthcheck --label config_id=tripleo_step4 --label container_name=ovn_metadata_agent --label managed_by=tripleo_ansible --label "config_data={'cgroupns': 'host', 'environment': {'KOLLA_CONFIG_STRATEGY': 'COPY_ALWAYS', 'TRIPLEO_CONFIG_HASH': 'b0f0231085155addf0474368939de6e0'}, 'healthcheck': {'test': '/openstack/healthcheck'}, 'image': 'uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-neutron-metadata-agent-ovn:current-tripleo', 'net': 'host', 'pid': 'host', 'privileged': True, 'restart': 'always', 'start_order': 1, 'volumes': ['/etc/hosts:/etc/hosts:ro', '/etc/localtime:/etc/localtime:ro', '/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '/dev/log:/dev/log', '/etc/puppet:/etc/puppet:ro', '/var/log/containers/neutron:/var/log/neutron:z', '/var/lib/kolla/config_files/ovn_metadata_agent.json:/var/lib/kolla/config_files/config.json:ro', '/var/lib/config-data/puppet-generated/neutron:/var/lib/kolla/config_files/src:ro', '/lib/modules:/lib/modules:ro', '/run/openvswitch:/run/openvswitch:shared,z', '/var/lib/neutron:/var/lib/neutron:shared,z', '/run/netns:/run/netns:shared', '/var/lib/neutron/kill_scripts:/etc/neutron/kill_scripts:shared,z', '/var/lib/neutron/ovn_metadata_haproxy_wrapper:/usr/local/bin/haproxy:ro', '/usr/sbin/ip:/usr/sbin/ip']}" --log-driver k8s-file --log-opt path=/var/log/containers/stdouts/ovn_metadata_agent.log --network host --pid host --privileged=true --volume /etc/hosts:/etc/hosts:ro --volume /etc/localtime:/etc/localtime:ro --volume /etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro --volume /etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro --volume /etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro --volume /etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro --volume /etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro --volume /dev/log:/dev/log --volume /etc/puppet:/etc/puppet:ro --volume /var/log/containers/neutron:/var/log/neutron:z --volume /var/lib/kolla/config_files/ovn_metadata_agent.json:/var/lib/kolla/config_files/config.json:ro --volume /var/lib/config-data/puppet-generated/neutron:/var/lib/kolla/config_files/src:ro --volume /lib/modules:/lib/modules:ro --volume /run/openvswitch:/run/openvswitch:shared,z --volume /var/lib/neutron:/var/lib/neutron:shared,z --volume /run/netns:/run/netns:shared --volume /var/lib/neutron/kill_scripts:/etc/neutron/kill_scripts:shared,z --volume /var/lib/neutron/ovn_metadata_haproxy_wrapper:/usr/local/bin/haproxy:ro localhost/custom-neutron-metadata-agent:latest Thanks everyone! Best regards, Davorin