[neutron] OpenStack compute nodes L3 agent reconfiguration `dvr` mode to `dvr_no_external`
Dear upstream OpenStack colleagues, we recently identified a lot of allocated "network:floatingip_agent_gateway ports" as we were running in pure dvr / dvr_snat modes (compute/network nodes) as also discussed in thread [1].
As we want to reduce amount of "network:floatingip_agent_gateway ports", our approach is to migrate majority of compute nodes from L3 agent `dvr` to `dvr_no_external` mode. Although we have success with empty nodes where we can see the node working as expected so north/south traffic now goes via network nodes on the non-empty compute nodes we struggle.
The reconfiguration steps we are taking: 1. our maintenance per each compute node (still in `dvr` mode) 1.1. we first identify all VMs with FIP addresses, keep track of the FIPs and connected ports 1.2. we disconnect each FIP from port 1.3. all VMs on the compute node does not have FIPs 1.4. compute node L3 agent configuration is switched `dvr` -> `dvr_no_external` 1.5 delete `hypervisor network:floatingip_agent_gateway` port 1.6. re-attach all tracked FIPs 1.7. test FIP traffic
At the step 1.7. we can see FIPs are not accessible after L3 agent re-configuration. Revert of the L3 agent configuration into `dvr` mode helps to get back the FIP connectivity.
Our questions are:
1. Principally, can we get rid of `hypervisor network:floatingip_agent_gateway` ports by switching L3 agent to dvr_no_external mode? Can you think of a better way? 2. Would you recommend other steps how to migrate drom `dvr` to `dvr_no_external`? [2] Is it necesary to restart whole neutron / OVS or reconfigured L3 agent is just enough?
Hope for your reply. ;-)
Kind Regards, František
[1] https://lists.openstack.org/pipermail/openstack-dev/2016-June/096384.html [2] https://opendev.org/openstack/neutron/src/branch/master/releasenotes/notes/d...
Hi,
Dnia piątek, 13 września 2024 15:11:12 CEST frantisek.reznicek.szn@gmail.com pisze:
Dear upstream OpenStack colleagues, we recently identified a lot of allocated "network:floatingip_agent_gateway ports" as we were running in pure dvr / dvr_snat modes (compute/network nodes) as also discussed in thread [1].
As we want to reduce amount of "network:floatingip_agent_gateway ports", our approach is to migrate majority of compute nodes from L3 agent `dvr` to `dvr_no_external` mode. Although we have success with empty nodes where we can see the node working as expected so north/south traffic now goes via network nodes on the non-empty compute nodes we struggle.
The reconfiguration steps we are taking:
- our maintenance per each compute node (still in `dvr` mode)
1.1. we first identify all VMs with FIP addresses, keep track of the FIPs and connected ports 1.2. we disconnect each FIP from port 1.3. all VMs on the compute node does not have FIPs 1.4. compute node L3 agent configuration is switched `dvr` -> `dvr_no_external` 1.5 delete `hypervisor network:floatingip_agent_gateway` port 1.6. re-attach all tracked FIPs 1.7. test FIP traffic
At the step 1.7. we can see FIPs are not accessible after L3 agent re-configuration. Revert of the L3 agent configuration into `dvr` mode helps to get back the FIP connectivity.
You don't have connectivity because You set agent into the mode where it don't have external connectivity. That's why it not works for you :)
Our questions are:
- Principally, can we get rid of `hypervisor network:floatingip_agent_gateway` ports by switching L3 agent to dvr_no_external mode? Can you think of a better way?
Using ML2/ovs with DVR requires to use one such IP address per compute node per external network. You can't avoid that. You can configure some 'special' subnet in the network to use IPs from that subnet for that purpose. See https://docs.openstack.org/neutron/latest/admin/config-service-subnets.html You can also e.g. migrate to ML2/OVN backend which don't have this limitation.
- Would you recommend other steps how to migrate drom `dvr` to `dvr_no_external`? [2] Is it necesary to restart whole neutron / OVS or reconfigured L3 agent is just enough?
Hope for your reply. ;-)
Kind Regards, František
[1] https://lists.openstack.org/pipermail/openstack-dev/2016-June/096384.html [2] https://opendev.org/openstack/neutron/src/branch/master/releasenotes/notes/d...
Hi Slavek, thank you for the reply.
Dne 13. 09. 24 v 17:14 Sławek Kapłoński napsal(a):
...
At the step 1.7. we can see FIPs are not accessible after L3 agent
re-configuration. Revert of the L3 agent configuration into `dvr` mode helps to get back the FIP connectivity.
You don't have connectivity because You set agent into the mode where it don't have external connectivity. That's why it not works for you :)
This is correct, but I thought FIP traffic should be routed via network node then (when switched to dvr_no_external), isn't that correct?
Our questions are:
- Principally, can we get rid of `hypervisor
network:floatingip_agent_gateway` ports by switching L3 agent to dvr_no_external mode? Can you think of a better way?
Using ML2/ovs with DVR requires to use one such IP address per compute node per external network. You can't avoid that. You can configure some 'special' subnet in the network to use IPs from that subnet for that purpose. See https://docs.openstack.org/neutron/latest/admin/config-service-subnets.html
You can also e.g. migrate to ML2/OVN backend which don't have this limitation.
Thank you for your guidance here. I was looking into service subnets already. In my case I have single ostack network entity with two subnets as described below [10]. There are two physical external provider /24 networks on same VLAN id (same ostack segment).
To move `hypervisor network:floatingip_agent_gateway` ports from two physical external provider /24 networks I believe: * I cannot use internal ostack vxlan network, it has to be same segment i.e. provider network vlan on same VLAN id. is that correct? * This service subnet can have internal addresses for instance 10.0.0.0/16. Correct? * This service subnet has to be externally routed and NATed so traffic can get to the internet and back. Correct? * How many addresses I would need? I assume max. number of ostack compute nodes correct?
Now let's assume I have another couple of external networks which are smaller like ipv4 /25 or /26, those (each) has separate network which has single subnet. Do I need to create for those also specific service subnets?
Thanks for your response in advance.
Kind Regards, František
[10]
[freznicek@lenovo-t14 dvr_to_dvr_no_external-20240913_175901 0]$ openstack network show external-ipv4-general-public +---------------------------+----------------------------------------------------------------------------+ | Field | Value | +---------------------------+----------------------------------------------------------------------------+ | admin_state_up | UP | | availability_zone_hints | nova | | availability_zones | nova | | dns_domain | None | | id | 95e346fd-a52f-4498-84aa-23f2da323429 | | is_default | False | | is_vlan_transparent | None | | l2_adjacency | True | | mtu | 9000 | | name | external-ipv4-general-public | | port_security_enabled | True | | project_id | 2139f9e4d92e4a2ba77b781e01d6d3b0 | | provider:network_type | vlan | | provider:physical_network | provider | | provider:segmentation_id | 716 | | qos_policy_id | None | | revision_number | 36 | | router:external | External | | segments | None | | shared | False | | status | ACTIVE | | subnets | 51299ee0-ac11-49a9-b773-dde916e20a5f, bcd6cc41-1238-4925-b597-aa6c1929685b | | tags | | | tenant_id | 2139f9e4d92e4a2ba77b781e01d6d3b0 | +---------------------------+----------------------------------------------------------------------------+
[freznicek@lenovo-t14 dvr_to_dvr_no_external-20240913_175901 2]$ openstack subnet show 51299ee0-ac11-49a9-b773-dde916e20a5f +----------------------+--------------------------------------------+ | Field | Value | +----------------------+--------------------------------------------+ | allocation_pools | 147.251.245.3-147.251.245.254 | | cidr | 147.251.245.0/24 | | dns_nameservers | ..., 8.8.8.8 | | dns_publish_fixed_ip | None | | enable_dhcp | False | | gateway_ip | 147.251.245.1 | | host_routes | | | id | 51299ee0-ac11-49a9-b773-dde916e20a5f | | ip_version | 4 | | name | external-ipv4-general-public-147-251-245-0 | | network_id | 95e346fd-a52f-4498-84aa-23f2da323429 | | project_id | 2139f9e4d92e4a2ba77b781e01d6d3b0 | | revision_number | 0 | | segment_id | None | | service_types | | | subnetpool_id | None | +----------------------+--------------------------------------------+ [freznicek@lenovo-t14 dvr_to_dvr_no_external-20240913_175901 0]$ openstack subnet show bcd6cc41-1238-4925-b597-aa6c1929685b +----------------------+--------------------------------------------+ | Field | Value | +----------------------+--------------------------------------------+ | allocation_pools | 147.251.255.2-147.251.255.254 | | cidr | 147.251.255.0/24 | | description | | | dns_nameservers | ..., 8.8.8.8 | | dns_publish_fixed_ip | None | | enable_dhcp | False | | gateway_ip | 147.251.255.1 | | host_routes | | | id | bcd6cc41-1238-4925-b597-aa6c1929685b | | ip_version | 4 | | ipv6_address_mode | None | | ipv6_ra_mode | None | | name | external-ipv4-general-public-147-251-255-0 | | network_id | 95e346fd-a52f-4498-84aa-23f2da323429 | | project_id | 2139f9e4d92e4a2ba77b781e01d6d3b0 | | revision_number | 4 | | segment_id | None | | service_types | | | subnetpool_id | None | +----------------------+--------------------------------------------+
participants (3)
-
frantisek.reznicek.szn＠gmail.com
-
František Řezníček
-
Sławek Kapłoński