[neutron] Static route not added in namespace using DVR on Wallaby
Hi, We're in this situation where we are going to move some instances from one openstack cluster to another. After this process, we want our instances on the new openstack cluster to keep the same floating IPs but also to be able to communicate with some instances that are in the same public IP range on the first cluster. To accomplish this, we want to add static routes like 'X.X.X.X/32 via Y.Y.Y.Y'. However, we're using DVR and when we add the static routes, they do not show up anywhere in any of the namespaces. Is there a different way to add static routes on DVR instead of using openstack router add route ? -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
Hi, W dniu pią, 6 maj 2022 o 14:14:47 -0400 użytkownik J-P Methot <jp.methot@planethoster.info> napisał:
Hi,
We're in this situation where we are going to move some instances from one openstack cluster to another. After this process, we want our instances on the new openstack cluster to keep the same floating IPs but also to be able to communicate with some instances that are in the same public IP range on the first cluster.
To accomplish this, we want to add static routes like 'X.X.X.X/32 via Y.Y.Y.Y'. However, we're using DVR and when we add the static routes, they do not show up anywhere in any of the namespaces. Is there a different way to add static routes on DVR instead of using openstack router add route ?
No, there is no other way to add static routes to the dvr router. I don't have any DVR deployment now to check it but IIRC route should be added in the qrouter namespace in the compute nodes where router exists. If it's not there please check logs of the l3-agent on those hosts, maybe there are some errors there.
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
-- Slawek Kaplonski Principal Software Engineer Red Hat
I tested this on my own DVR test environment with a random static route and I'm getting the same results as on production. Here's what I get in the logs : 2022-05-09 17:28:50.018 691 INFO neutron.agent.l3.agent [-] Starting processing update 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3, priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time elapsed: 0.001 2022-05-09 17:28:50.019 691 INFO neutron.agent.l3.agent [-] Starting router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3, priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time elapsed: 0.002 2022-05-09 17:28:51.640 691 INFO neutron.agent.l3.agent [-] Finished a router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, update_id 9e112de1-f538-4a41-9526-152aa3937129. Time elapsed: 1.622 As you can see, there was an attempt at updating the router and it did return as successful. However, there was no new route added in the router or floating ip namespace. No error either. On 5/6/22 14:40, Slawek Kaplonski wrote:
Hi,
W dniu pią, 6 maj 2022 o 14:14:47 -0400 użytkownik J-P Methot <jp.methot@planethoster.info> napisał:
Hi,
We're in this situation where we are going to move some instances from one openstack cluster to another. After this process, we want our instances on the new openstack cluster to keep the same floating IPs but also to be able to communicate with some instances that are in the same public IP range on the first cluster.
To accomplish this, we want to add static routes like 'X.X.X.X/32 via Y.Y.Y.Y'. However, we're using DVR and when we add the static routes, they do not show up anywhere in any of the namespaces. Is there a different way to add static routes on DVR instead of using openstack router add route ?
No, there is no other way to add static routes to the dvr router. I don't have any DVR deployment now to check it but IIRC route should be added in the qrouter namespace in the compute nodes where router exists. If it's not there please check logs of the l3-agent on those hosts, maybe there are some errors there.
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. -- Slawek Kaplonski Principal Software Engineer Red Hat
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
Hi, W dniu pon, 9 maj 2022 o 13:49:22 -0400 użytkownik J-P Methot <jp.methot@planethoster.info> napisał:
I tested this on my own DVR test environment with a random static route and I'm getting the same results as on production. Here's what I get in the logs :
2022-05-09 17:28:50.018 691 INFO neutron.agent.l3.agent [-] Starting processing update 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3, priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time elapsed: 0.001 2022-05-09 17:28:50.019 691 INFO neutron.agent.l3.agent [-] Starting router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3, priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time elapsed: 0.002 2022-05-09 17:28:51.640 691 INFO neutron.agent.l3.agent [-] Finished a router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, update_id 9e112de1-f538-4a41-9526-152aa3937129. Time elapsed: 1.622
As you can see, there was an attempt at updating the router and it did return as successful. However, there was no new route added in the router or floating ip namespace. No error either.
Can You do the same with debug logs enabled?
On 5/6/22 14:40, Slawek Kaplonski wrote:
Hi,
W dniu pią, 6 maj 2022 o 14:14:47 -0400 użytkownik J-P Methot <jp.methot@planethoster.info> <mailto:jp.methot@planethoster.info> napisał:
Hi,
We're in this situation where we are going to move some instances from one openstack cluster to another. After this process, we want our instances on the new openstack cluster to keep the same floating IPs but also to be able to communicate with some instances that are in the same public IP range on the first cluster.
To accomplish this, we want to add static routes like 'X.X.X.X/32 via Y.Y.Y.Y'. However, we're using DVR and when we add the static routes, they do not show up anywhere in any of the namespaces. Is there a different way to add static routes on DVR instead of using openstack router add route ?
No, there is no other way to add static routes to the dvr router. I don't have any DVR deployment now to check it but IIRC route should be added in the qrouter namespace in the compute nodes where router exists. If it's not there please check logs of the l3-agent on those hosts, maybe there are some errors there.
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
-- Slawek Kaplonski Principal Software Engineer Red Hat -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hi, I got the debug logs. They were a bit too long so I put them in a txt file. Please tell me if you'd prefer a pastebin instead. On 5/10/22 02:38, Slawek Kaplonski wrote:
Hi,
W dniu pon, 9 maj 2022 o 13:49:22 -0400 użytkownik J-P Methot <jp.methot@planethoster.info> napisał:
I tested this on my own DVR test environment with a random static route and I'm getting the same results as on production. Here's what I get in the logs :
2022-05-09 17:28:50.018 691 INFO neutron.agent.l3.agent [-] Starting processing update 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3, priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time elapsed: 0.001 2022-05-09 17:28:50.019 691 INFO neutron.agent.l3.agent [-] Starting router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3, priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time elapsed: 0.002 2022-05-09 17:28:51.640 691 INFO neutron.agent.l3.agent [-] Finished a router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, update_id 9e112de1-f538-4a41-9526-152aa3937129. Time elapsed: 1.622
As you can see, there was an attempt at updating the router and it did return as successful. However, there was no new route added in the router or floating ip namespace. No error either.
Can You do the same with debug logs enabled?
On 5/6/22 14:40, Slawek Kaplonski wrote:
Hi,
W dniu pią, 6 maj 2022 o 14:14:47 -0400 użytkownik J-P Methot <jp.methot@planethoster.info> napisał:
Hi,
We're in this situation where we are going to move some instances from one openstack cluster to another. After this process, we want our instances on the new openstack cluster to keep the same floating IPs but also to be able to communicate with some instances that are in the same public IP range on the first cluster.
To accomplish this, we want to add static routes like 'X.X.X.X/32 via Y.Y.Y.Y'. However, we're using DVR and when we add the static routes, they do not show up anywhere in any of the namespaces. Is there a different way to add static routes on DVR instead of using openstack router add route ?
No, there is no other way to add static routes to the dvr router. I don't have any DVR deployment now to check it but IIRC route should be added in the qrouter namespace in the compute nodes where router exists. If it's not there please check logs of the l3-agent on those hosts, maybe there are some errors there.
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. -- Slawek Kaplonski Principal Software Engineer Red Hat -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
Hi, Dnia czwartek, 12 maja 2022 19:23:15 CEST J-P Methot pisze:
Hi,
I got the debug logs. They were a bit too long so I put them in a txt file. Please tell me if you'd prefer a pastebin instead.
On 5/10/22 02:38, Slawek Kaplonski wrote:
Hi,
W dniu pon, 9 maj 2022 o 13:49:22 -0400 użytkownik J-P Methot <jp.methot@planethoster.info> napisał:
I tested this on my own DVR test environment with a random static route and I'm getting the same results as on production. Here's what I get in the logs :
2022-05-09 17:28:50.018 691 INFO neutron.agent.l3.agent [-] Starting processing update 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3, priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time elapsed: 0.001 2022-05-09 17:28:50.019 691 INFO neutron.agent.l3.agent [-] Starting router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3, priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time elapsed: 0.002 2022-05-09 17:28:51.640 691 INFO neutron.agent.l3.agent [-] Finished a router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, update_id 9e112de1-f538-4a41-9526-152aa3937129. Time elapsed: 1.622
As you can see, there was an attempt at updating the router and it did return as successful. However, there was no new route added in the router or floating ip namespace. No error either.
Can You do the same with debug logs enabled?
On 5/6/22 14:40, Slawek Kaplonski wrote:
Hi,
W dniu pią, 6 maj 2022 o 14:14:47 -0400 użytkownik J-P Methot <jp.methot@planethoster.info> napisał:
Hi,
We're in this situation where we are going to move some instances from one openstack cluster to another. After this process, we want our instances on the new openstack cluster to keep the same floating IPs but also to be able to communicate with some instances that are in the same public IP range on the first cluster.
To accomplish this, we want to add static routes like 'X.X.X.X/32 via Y.Y.Y.Y'. However, we're using DVR and when we add the static routes, they do not show up anywhere in any of the namespaces. Is there a different way to add static routes on DVR instead of using openstack router add route ?
No, there is no other way to add static routes to the dvr router. I don't have any DVR deployment now to check it but IIRC route should be added in the qrouter namespace in the compute nodes where router exists. If it's not there please check logs of the l3-agent on those hosts, maybe there are some errors there.
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc. -- Slawek Kaplonski Principal Software Engineer Red Hat -- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
-- Slawek Kaplonski Principal Software Engineer Red Hat
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
I just tested it today on my local env and everything works fine for me. When I added extra route to some external IP address it was added in snat-XXX namespace, When I tested dvr router only with private networks, extra route was added in the qrouter-XXX namespaces. Also, we have scenario test https://github.com/openstack/neutron-tempest-plugin/blob/6dcc0e81b5f3c656181091025f351eb479cdde21/neutron_tempest_plugin/scenario/test_connectivity.py#L73[1] which is creating such extra routes and uses them to connect between VMs. And this test is running fine AFAIK in our CI. -- Slawek Kaplonski Principal Software Engineer Red Hat -------- [1] https://github.com/openstack/neutron-tempest-plugin/blob/6dcc0e81b5f3c656181...
Is your local environment on Wallaby? I might consider upgrading if it's more recent than that. We use Kolla to deploy so it might also play into this. On 5/17/22 08:59, Slawek Kaplonski wrote:
Hi,
Dnia czwartek, 12 maja 2022 19:23:15 CEST J-P Methot pisze:
Hi,
I got the debug logs. They were a bit too long so I put them in a txt
file. Please tell me if you'd prefer a pastebin instead.
On 5/10/22 02:38, Slawek Kaplonski wrote:
Hi,
W dniu pon, 9 maj 2022 o 13:49:22 -0400 użytkownik J-P Methot
<jp.methot@planethoster.info> napisał:
I tested this on my own DVR test environment with a random static
route and I'm getting the same results as on production. Here's what
I get in the logs :
2022-05-09 17:28:50.018 691 INFO neutron.agent.l3.agent [-] Starting
processing update 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3,
priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time
elapsed: 0.001
2022-05-09 17:28:50.019 691 INFO neutron.agent.l3.agent [-] Starting
router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3,
priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time
elapsed: 0.002
2022-05-09 17:28:51.640 691 INFO neutron.agent.l3.agent [-] Finished
a router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, update_id
9e112de1-f538-4a41-9526-152aa3937129. Time elapsed: 1.622
As you can see, there was an attempt at updating the router and it
did return as successful. However, there was no new route added in
the router or floating ip namespace. No error either.
Can You do the same with debug logs enabled?
On 5/6/22 14:40, Slawek Kaplonski wrote:
Hi,
W dniu pią, 6 maj 2022 o 14:14:47 -0400 użytkownik J-P Methot
<jp.methot@planethoster.info> napisał:
Hi,
We're in this situation where we are going to move some instances
from one openstack cluster to another. After this process, we want
our instances on the new openstack cluster to keep the same
floating IPs but also to be able to communicate with some instances
that are in the same public IP range on the first cluster.
To accomplish this, we want to add static routes like 'X.X.X.X/32
via Y.Y.Y.Y'. However, we're using DVR and when we add the static
routes, they do not show up anywhere in any of the namespaces. Is
there a different way to add static routes on DVR instead of using
openstack router add route ?
No, there is no other way to add static routes to the dvr router. I
don't have any DVR deployment now to check it but IIRC route should
be added in the qrouter namespace in the compute nodes where router
exists. If it's not there please check logs of the l3-agent on those
hosts, maybe there are some errors there.
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
--
Slawek Kaplonski
Principal Software Engineer
Red Hat
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
--
Slawek Kaplonski
Principal Software Engineer
Red Hat
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
I just tested it today on my local env and everything works fine for me. When I added extra route to some external IP address it was added in snat-XXX namespace,
When I tested dvr router only with private networks, extra route was added in the qrouter-XXX namespaces.
Also, we have scenario test https://github.com/openstack/neutron-tempest-plugin/blob/6dcc0e81b5f3c656181... which is creating such extra routes and uses them to connect between VMs. And this test is running fine AFAIK in our CI.
--
Slawek Kaplonski
Principal Software Engineer
Red Hat
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
Hi, Dnia wtorek, 17 maja 2022 17:32:39 CEST J-P Methot pisze:
Is your local environment on Wallaby? I might consider upgrading if it's more recent than that. We use Kolla to deploy so it might also play into this.
No, I was testing on master (as I usually do in my dev env)
On 5/17/22 08:59, Slawek Kaplonski wrote:
Hi,
Dnia czwartek, 12 maja 2022 19:23:15 CEST J-P Methot pisze:
Hi,
I got the debug logs. They were a bit too long so I put them in a txt
file. Please tell me if you'd prefer a pastebin instead.
On 5/10/22 02:38, Slawek Kaplonski wrote:
Hi,
W dniu pon, 9 maj 2022 o 13:49:22 -0400 użytkownik J-P Methot
<jp.methot@planethoster.info> napisał:
I tested this on my own DVR test environment with a random static
route and I'm getting the same results as on production. Here's what
I get in the logs :
2022-05-09 17:28:50.018 691 INFO neutron.agent.l3.agent [-] Starting
processing update 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3,
priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time
elapsed: 0.001
2022-05-09 17:28:50.019 691 INFO neutron.agent.l3.agent [-] Starting
router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, action 3,
priority 1, update_id 9e112de1-f538-4a41-9526-152aa3937129. Wait time
elapsed: 0.002
2022-05-09 17:28:51.640 691 INFO neutron.agent.l3.agent [-] Finished
a router update for 41fcd10b-7db5-45d9-b23c-e22f34c45eec, update_id
9e112de1-f538-4a41-9526-152aa3937129. Time elapsed: 1.622
As you can see, there was an attempt at updating the router and it
did return as successful. However, there was no new route added in
the router or floating ip namespace. No error either.
Can You do the same with debug logs enabled?
On 5/6/22 14:40, Slawek Kaplonski wrote:
Hi,
W dniu pią, 6 maj 2022 o 14:14:47 -0400 użytkownik J-P Methot
<jp.methot@planethoster.info> napisał:
>
> Hi,
>
> We're in this situation where we are going to move some instances
> from one openstack cluster to another. After this process, we want
> our instances on the new openstack cluster to keep the same
> floating IPs but also to be able to communicate with some instances
> that are in the same public IP range on the first cluster.
>
> To accomplish this, we want to add static routes like 'X.X.X.X/32
> via Y.Y.Y.Y'. However, we're using DVR and when we add the static
> routes, they do not show up anywhere in any of the namespaces. Is
> there a different way to add static routes on DVR instead of using
> openstack router add route ?
>
No, there is no other way to add static routes to the dvr router. I
don't have any DVR deployment now to check it but IIRC route should
be added in the qrouter namespace in the compute nodes where router
exists. If it's not there please check logs of the l3-agent on those
hosts, maybe there are some errors there.
> --
> Jean-Philippe Méthot
> Senior Openstack system administrator
> Administrateur système Openstack sénior
> PlanetHoster inc.
--
Slawek Kaplonski
Principal Software Engineer
Red Hat
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
--
Slawek Kaplonski
Principal Software Engineer
Red Hat
--
Jean-Philippe Méthot
Senior Openstack system administrator
Administrateur système Openstack sénior
PlanetHoster inc.
I just tested it today on my local env and everything works fine for me. When I added extra route to some external IP address it was added in snat-XXX namespace,
When I tested dvr router only with private networks, extra route was added in the qrouter-XXX namespaces.
Also, we have scenario test https://github.com/openstack/neutron-tempest-plugin/blob/6dcc0e81b5f3c656181... which is creating such extra routes and uses them to connect between VMs. And this test is running fine AFAIK in our CI.
--
Slawek Kaplonski
Principal Software Engineer
Red Hat
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
-- Slawek Kaplonski Principal Software Engineer Red Hat
On Fri, May 6, 2022 at 11:23 AM J-P Methot <jp.methot@planethoster.info> wrote:
Hi,
We're in this situation where we are going to move some instances from one openstack cluster to another. After this process, we want our instances on the new openstack cluster to keep the same floating IPs but also to be able to communicate with some instances that are in the same public IP range on the first cluster.
To accomplish this, we want to add static routes like 'X.X.X.X/32 via Y.Y.Y.Y'. However, we're using DVR and when we add the static routes, they do not show up anywhere in any of the namespaces. Is there a different way to add static routes on DVR instead of using openstack router add route ?
-- Jean-Philippe Méthot Senior Openstack system administrator Administrateur système Openstack sénior PlanetHoster inc.
I don’t think that there is an automatic way to do what you are trying to do using static routes.
One way to approach this would be to use dynamic routing to advertise the availability of the /32 route. I just described an approach to this using BGP in another thread on this list with the subject “[ops][octavia][neutron] Distributed Virtual Routing: Floating IPs attached to virtual IP addresses are assigned on network nodes”. There are several BGP implementations which support this functionality, I believe Neutron Dynamic Routing for OVS works for Wallaby, Juniper Contrail (and probably the Open Source version Tungsten Fabric), and others. The implementation I describe is for OVN, but the same approach could be adapted to other Neutron drivers. -Dan Sneddon -- Dan Sneddon | Senior Principal Software Engineer dsneddon@redhat.com | redhat.com/cloud dsneddon:irc | @dxs:twitter
participants (3)
-
Dan Sneddon
-
J-P Methot
-
Slawek Kaplonski