[zun][kuryr][neutron] Missing vxlan ports in br-tun for Zun containers?
Hello stackers, I’m interested in using zun to launch containers and assign floating IPs via neutron to those containers. I am deploying zun, kuryr-libnetwork, and neutron with kolla-ansible on the Train release. I’ve configured neutron with one physical network and I’d like to use a VXLAN overlay for tenant networks. What works: - I can launch containers on a neutron tenant network, they start successfully, they get an IP and can reach each other if they’re co-located on a single host. - I can create all my neutron networks, routers, subnets, without (obvious) errors. - I can update security groups on the container and see the iptables rules updated appropriately. - I can directly create Docker networks using the kuryr driver/type. What doesn’t work: - I can’t see any vxlan ports on the br-tun OVS bridge - I can’t access the exposed container ports from the control/network node via the router netns - Because of that, I can’t assign floating IPs because NAT effectively won’t work to reach the containers The fact that there are no ports on br-tun is supicious, but I’m not sure how this is supposed to work. I don’t see anything weird in neutron-openvswitch-agent logs but those logs are quite noisy and I’m not sure what to look for. Has anybody deployed such a setup / are there limitations I should know about? Thank you! Jason Anderson DevOps Lead, Chameleon --- Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory jasonanderson@uchicago.edu
As usual, “rubber ducking” the openstack-discuss list yielded fruit. It turns out that I didn’t have the l2population mechanism driver enabled. I thought this was optional for some reason. It looks like enabling this and restarting the neutorn-openvswitch-agent has fixed connectivity! /Jason
On Apr 8, 2021, at 11:20 AM, Jason Anderson <jasonanderson@uchicago.edu> wrote:
Hello stackers,
I’m interested in using zun to launch containers and assign floating IPs via neutron to those containers. I am deploying zun, kuryr-libnetwork, and neutron with kolla-ansible on the Train release. I’ve configured neutron with one physical network and I’d like to use a VXLAN overlay for tenant networks.
What works: - I can launch containers on a neutron tenant network, they start successfully, they get an IP and can reach each other if they’re co-located on a single host. - I can create all my neutron networks, routers, subnets, without (obvious) errors. - I can update security groups on the container and see the iptables rules updated appropriately. - I can directly create Docker networks using the kuryr driver/type.
What doesn’t work: - I can’t see any vxlan ports on the br-tun OVS bridge - I can’t access the exposed container ports from the control/network node via the router netns - Because of that, I can’t assign floating IPs because NAT effectively won’t work to reach the containers
The fact that there are no ports on br-tun is supicious, but I’m not sure how this is supposed to work. I don’t see anything weird in neutron-openvswitch-agent logs but those logs are quite noisy and I’m not sure what to look for.
Has anybody deployed such a setup / are there limitations I should know about?
Thank you!
Jason Anderson
DevOps Lead, Chameleon
---
Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory jasonanderson@uchicago.edu
Hi, Dnia czwartek, 8 kwietnia 2021 19:00:19 CEST Jason Anderson pisze:
As usual, “rubber ducking” the openstack-discuss list yielded fruit. It turns out that I didn’t have the l2population mechanism driver enabled. I thought this was optional for some reason. It looks like enabling this and restarting the neutorn-openvswitch-agent has fixed connectivity!
L2pop should be optional. It's required only when DVR is used. But if You don't want to use it You should disable it on both agent and server's side. In such case neutron-openvswitcht-agent should establish vxlan tunnels to all other nodes just after start of the agent, during first rpc_loop iteration: https://github.com/openstack/neutron/blob/bdd661d21898d573ef39448316860aa4c6...
/Jason
On Apr 8, 2021, at 11:20 AM, Jason Anderson <jasonanderson@uchicago.edu> wrote:
Hello stackers,
I’m interested in using zun to launch containers and assign floating IPs via neutron to those containers. I am deploying zun, kuryr-libnetwork, and neutron with kolla-ansible on the Train release. I’ve configured neutron with one physical network and I’d like to use a VXLAN overlay for tenant networks.
What works: - I can launch containers on a neutron tenant network, they start successfully, they get an IP and can reach each other if they’re co-located on a single host. - I can create all my neutron networks, routers, subnets, without (obvious) errors. - I can update security groups on the container and see the iptables rules updated appropriately. - I can directly create Docker networks using the kuryr driver/type.
What doesn’t work: - I can’t see any vxlan ports on the br-tun OVS bridge - I can’t access the exposed container ports from the control/network node via the router netns - Because of that, I can’t assign floating IPs because NAT effectively won’t work to reach the containers
The fact that there are no ports on br-tun is supicious, but I’m not sure how this is supposed to work. I don’t see anything weird in neutron-openvswitch-agent logs but those logs are quite noisy and I’m not sure what to look for.
Has anybody deployed such a setup / are there limitations I should know about?
Thank you!
Jason Anderson
DevOps Lead, Chameleon
---
Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory jasonanderson@uchicago.edu
-- Slawek Kaplonski Principal Software Engineer Red Hat
participants (2)
-
Jason Anderson
-
Slawek Kaplonski