[octavia] Blocking a specific network for the VIP of LBs
Hi there! In our setup, we offer to our customers, on top of floating IPS, access to an OpenStack internal network that contains public IPs. The idea is that they can, using this network, spawn VMs with a public IP directly on the NIC of their VMs. The issue is, some customers also use this network for the VIP of their LBs. While this seems to work, actually, instead of reserving a single public IP address, it reserves 3 IPs. Not a big deal for us, as they are billed internally, though it's rather bad for our customers. On top of that, whenever one of the subnet of this network becomes full, it becomes impossible to failover amphoras spawned in this subnet. Each time one runs into ERROR, it's kind of a bad situation where we have no way to fix, except contacting the customer and ask them to fix their setup. So we advise our customers to spawn a port first, use that port for the LB, and associate a floating IP address to the original port. This is described in our doc here: https://docs.infomaniak.cloud/network/loadbalancer_usecase_simple_http/ However, as you may know, customers never read the docs. And therefore, we'd like to forbid the use of that shared OpenStack internal network for the VIP of LBs. But as much as possible, this isn't a feature of Octavia. Is it? I'd like to implement the feature. What's the best way to do so? Would a new "forbidden_networks" directive would do, and then have a check at the API level? Or is there a better way using the API policy? Also, wouldn't it be possible that Octavia just reuse the ports that it currently hold, rather than asking new ports for the failover? If so, any clue where the code that does that could be fixed? I'd love to have this fixed AND the forbid network feature too... Cheers, Thomas Goirand (zigo)
Hello, Interesting, I just identified the same in our cloud last week where I found user(s) using the shared network (ofcourse against the recommendation in documentation). I would be interested as well in blocking a Neutron network marked as external (is_router_external) to be blocked for use VIP network. Best regards /Tobias
On 11 Feb 2025, at 15:26, Thomas Goirand <zigo@debian.org> wrote:
[You don't often get email from zigo@debian.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Hi there!
In our setup, we offer to our customers, on top of floating IPS, access to an OpenStack internal network that contains public IPs. The idea is that they can, using this network, spawn VMs with a public IP directly on the NIC of their VMs.
The issue is, some customers also use this network for the VIP of their LBs. While this seems to work, actually, instead of reserving a single public IP address, it reserves 3 IPs. Not a big deal for us, as they are billed internally, though it's rather bad for our customers. On top of that, whenever one of the subnet of this network becomes full, it becomes impossible to failover amphoras spawned in this subnet. Each time one runs into ERROR, it's kind of a bad situation where we have no way to fix, except contacting the customer and ask them to fix their setup.
So we advise our customers to spawn a port first, use that port for the LB, and associate a floating IP address to the original port. This is described in our doc here:
https://docs.infomaniak.cloud/network/loadbalancer_usecase_simple_http/
However, as you may know, customers never read the docs. And therefore, we'd like to forbid the use of that shared OpenStack internal network for the VIP of LBs. But as much as possible, this isn't a feature of Octavia. Is it?
I'd like to implement the feature. What's the best way to do so? Would a new "forbidden_networks" directive would do, and then have a check at the API level? Or is there a better way using the API policy?
Also, wouldn't it be possible that Octavia just reuse the ports that it currently hold, rather than asking new ports for the failover? If so, any clue where the code that does that could be fixed? I'd love to have this fixed AND the forbid network feature too...
Cheers,
Thomas Goirand (zigo)
Hi Thomas, On 2/11/25 9:26 AM, Thomas Goirand wrote:
Hi there!
In our setup, we offer to our customers, on top of floating IPS, access to an OpenStack internal network that contains public IPs. The idea is that they can, using this network, spawn VMs with a public IP directly on the NIC of their VMs.
The issue is, some customers also use this network for the VIP of their LBs. While this seems to work, actually, instead of reserving a single public IP address, it reserves 3 IPs. Not a big deal for us, as they are billed internally, though it's rather bad for our customers. On top of that, whenever one of the subnet of this network becomes full, it becomes impossible to failover amphoras spawned in this subnet. Each time one runs into ERROR, it's kind of a bad situation where we have no way to fix, except contacting the customer and ask them to fix their setup.
So we advise our customers to spawn a port first, use that port for the LB, and associate a floating IP address to the original port. This is described in our doc here:
https://docs.infomaniak.cloud/network/loadbalancer_usecase_simple_http/
However, as you may know, customers never read the docs. And therefore, we'd like to forbid the use of that shared OpenStack internal network for the VIP of LBs. But as much as possible, this isn't a feature of Octavia. Is it?
I'd like to implement the feature. What's the best way to do so? Would a new "forbidden_networks" directive would do, and then have a check at the API level? Or is there a better way using the API policy?
So you can create a network in Neutron as an admin and share it via RBAC, I'm just not sure if it would help in your case, but it's something you could play with: https://docs.openstack.org/neutron/latest/admin/config-rbac.html The other thing that came to mind was using subnet service types, but it would only work in the case where you pre-allocate the port (which your customers don't do). The premise is that if you specify a device owner for a port, it will be used to filter which subnet it gets an IP from. https://docs.openstack.org/neutron/latest/admin/config-service-subnets.html $ openstack network create my-test-net $ openstack subnet create --network my-test-net --subnet-range 10.0.1.0/24 vm-subnet $ openstack subnet create --network my-test-net --subnet-range 10.0.2.0/24 --service-type 'compute:lb-subnet' lb-subnet I think you could optionally specify --service-type 'compute:nova' on vm-subnet. Then, when you go to create a port: $ openstack port create --network my-test-net --device-owner 'compute:lb-subnet' my-lb-port That will force the port IP to be in that subnet range. It's typically used for infrastructure device ports, but I don't see why it couldn't be used here as well. Of course it's been a while since I've played with this, and none of the commands here were tested in a live environment by me. We're always open to new proposals, but you might first want to hear from someone in Octavia to see if they have a solution. -Brian
Also, wouldn't it be possible that Octavia just reuse the ports that it currently hold, rather than asking new ports for the failover? If so, any clue where the code that does that could be fixed? I'd love to have this fixed AND the forbid network feature too...
Cheers,
Thomas Goirand (zigo)
On Tue, Feb 11, 2025 at 3:27 PM Thomas Goirand <zigo@debian.org> wrote:
Hi there!
In our setup, we offer to our customers, on top of floating IPS, access to an OpenStack internal network that contains public IPs. The idea is that they can, using this network, spawn VMs with a public IP directly on the NIC of their VMs.
The issue is, some customers also use this network for the VIP of their LBs. While this seems to work, actually, instead of reserving a single public IP address, it reserves 3 IPs. Not a big deal for us, as they are billed internally, though it's rather bad for our customers. On top of that, whenever one of the subnet of this network becomes full, it becomes impossible to failover amphoras spawned in this subnet. Each time one runs into ERROR, it's kind of a bad situation where we have no way to fix, except contacting the customer and ask them to fix their setup.
So we advise our customers to spawn a port first, use that port for the LB, and associate a floating IP address to the original port. This is described in our doc here:
https://docs.infomaniak.cloud/network/loadbalancer_usecase_simple_http/
However, as you may know, customers never read the docs. And therefore, we'd like to forbid the use of that shared OpenStack internal network for the VIP of LBs. But as much as possible, this isn't a feature of Octavia. Is it?
I'd like to implement the feature. What's the best way to do so? Would a new "forbidden_networks" directive would do, and then have a check at the API level? Or is there a better way using the API policy?
For the moment, Octavia doesn't support such a feature, I would take a look at the neutron RBAC to check if something can be achieved there. Octavia provides a 'valid_vip_networks' setting in the config file: https://docs.openstack.org/octavia/latest/configuration/configref.html#netwo... We could have a setting (deny_vip_networks) that does the opposite validation.
Also, wouldn't it be possible that Octavia just reuse the ports that it currently hold, rather than asking new ports for the failover? If so, any clue where the code that does that could be fixed? I'd love to have this fixed AND the forbid network feature too...
During a failover, the new amphora is created but the old amphora still exists and processes incoming requests until the new one is fully functional. So we need to have 2 ports at the same time. Reusing the existing port would lead to a long interruption of the dataplane.
Cheers,
Thomas Goirand (zigo)
participants (4)
-
Brian Haley
-
Gregory Thiemonge
-
Thomas Goirand
-
Tobias Urdin - Binero IT