Regarding Floating IP is existing Setup
hi Team, We got a issue in Xena release, where we set the environment in Ubuntu Platform, But later we get some issues in Floating IP not reachable. In a Network node, not all router namespace are Impacted and only few of them get affected, So we can not comment Network node has a issue. The L3 agent where the Router is tied up, Worked just fine, as other routers work Fine. and the one having issue in Floating IP, if i unassigned and assigned it starts working most of the time. Any thoughts on this Regards Adivya Singh
Hi, this sounds very familiar to me, I had to deal with something similar a couple of times in a heavily used cluster with 2 control nodes. What does your setup look like, is it a HA setup? I would start checking the DHCP and L3 agents. After increasing dhcp_agents_per_network to 2 in neutron.conf and restarting the services this didn't occur again (yet). This would impact floating IPs as well, sometimes I had to disable and enable the affected router(s). If you only have one control node a different approach is necessary. Do you see a high load on the control node? Zitat von Adivya Singh <adivya1.singh@gmail.com>:
hi Team,
We got a issue in Xena release, where we set the environment in Ubuntu Platform, But later we get some issues in Floating IP not reachable.
In a Network node, not all router namespace are Impacted and only few of them get affected, So we can not comment Network node has a issue.
The L3 agent where the Router is tied up, Worked just fine, as other routers work Fine.
and the one having issue in Floating IP, if i unassigned and assigned it starts working most of the time.
Any thoughts on this
Regards Adivya Singh
hi Eugen, The current setup is 3 controller nodes, The Load is not much on each controller and the number of DHCP agent is always set to 2 as per the standard in the neutron.conf, The L3 agent seems to be stables as other router namespace works fine under it, Only few router Namespace get affected under the agent. Most of the template having issue , Have all instance having FLoating IP, a Stack with a single floating IP have chance of issue very less Regards Adivya Singh On Tue, Jun 21, 2022 at 1:18 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
this sounds very familiar to me, I had to deal with something similar a couple of times in a heavily used cluster with 2 control nodes. What does your setup look like, is it a HA setup? I would start checking the DHCP and L3 agents. After increasing dhcp_agents_per_network to 2 in neutron.conf and restarting the services this didn't occur again (yet). This would impact floating IPs as well, sometimes I had to disable and enable the affected router(s). If you only have one control node a different approach is necessary. Do you see a high load on the control node?
Zitat von Adivya Singh <adivya1.singh@gmail.com>:
hi Team,
We got a issue in Xena release, where we set the environment in Ubuntu Platform, But later we get some issues in Floating IP not reachable.
In a Network node, not all router namespace are Impacted and only few of them get affected, So we can not comment Network node has a issue.
The L3 agent where the Router is tied up, Worked just fine, as other routers work Fine.
and the one having issue in Floating IP, if i unassigned and assigned it starts working most of the time.
Any thoughts on this
Regards Adivya Singh
Do you have ha routers enabled?
Most of the template having issue , Have all instance having FLoating IP, a Stack with a single floating IP have chance of issue very less
I don't quite understand that sentence, could you please clarify? Zitat von Adivya Singh <adivya1.singh@gmail.com>:
hi Eugen,
The current setup is 3 controller nodes, The Load is not much on each controller and the number of DHCP agent is always set to 2 as per the standard in the neutron.conf, The L3 agent seems to be stables as other router namespace works fine under it, Only few router Namespace get affected under the agent.
Most of the template having issue , Have all instance having FLoating IP, a Stack with a single floating IP have chance of issue very less
Regards Adivya Singh
On Tue, Jun 21, 2022 at 1:18 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
this sounds very familiar to me, I had to deal with something similar a couple of times in a heavily used cluster with 2 control nodes. What does your setup look like, is it a HA setup? I would start checking the DHCP and L3 agents. After increasing dhcp_agents_per_network to 2 in neutron.conf and restarting the services this didn't occur again (yet). This would impact floating IPs as well, sometimes I had to disable and enable the affected router(s). If you only have one control node a different approach is necessary. Do you see a high load on the control node?
Zitat von Adivya Singh <adivya1.singh@gmail.com>:
hi Team,
We got a issue in Xena release, where we set the environment in Ubuntu Platform, But later we get some issues in Floating IP not reachable.
In a Network node, not all router namespace are Impacted and only few of them get affected, So we can not comment Network node has a issue.
The L3 agent where the Router is tied up, Worked just fine, as other routers work Fine.
and the one having issue in Floating IP, if i unassigned and assigned it starts working most of the time.
Any thoughts on this
Regards Adivya Singh
Hi, Dnia wtorek, 21 czerwca 2022 13:55:51 CEST Adivya Singh pisze:
hi Eugen,
The current setup is 3 controller nodes, The Load is not much on each controller and the number of DHCP agent is always set to 2 as per the standard in the neutron.conf, The L3 agent seems to be stables as other router namespace works fine under it, Only few router Namespace get affected under the agent.
Is it that problem happens for new floating IPs or for the FIPs which were working fine and then suddenly stopped working? If the latter, was there any action which triggered the issue to happen? Is there e.g. only one FIP broken in the router or maybe when it happens, then all FIPs which uses same router are broken? Can You also try to analyze with e.g. tcpdump where traffic is dropped exactly? You can check http://kaplonski.pl/blog/neutron-where-is-my-packet-2/ for some more detailed description how traffic should go from the external network to Your instance.
Most of the template having issue , Have all instance having FLoating IP, a Stack with a single floating IP have chance of issue very less
Regards Adivya Singh
On Tue, Jun 21, 2022 at 1:18 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
this sounds very familiar to me, I had to deal with something similar a couple of times in a heavily used cluster with 2 control nodes. What does your setup look like, is it a HA setup? I would start checking the DHCP and L3 agents. After increasing dhcp_agents_per_network to 2 in neutron.conf and restarting the services this didn't occur again (yet). This would impact floating IPs as well, sometimes I had to disable and enable the affected router(s). If you only have one control node a different approach is necessary. Do you see a high load on the control node?
Zitat von Adivya Singh <adivya1.singh@gmail.com>:
hi Team,
We got a issue in Xena release, where we set the environment in Ubuntu Platform, But later we get some issues in Floating IP not reachable.
In a Network node, not all router namespace are Impacted and only few of them get affected, So we can not comment Network node has a issue.
The L3 agent where the Router is tied up, Worked just fine, as other routers work Fine.
and the one having issue in Floating IP, if i unassigned and assigned it starts working most of the time.
Any thoughts on this
Regards Adivya Singh
-- Slawek Kaplonski Principal Software Engineer Red Hat
Hi, Thanks for the advice and the link, What i saw when i do testing using tcpdump was "ARP" was not working, and it is not able to associate the FLoating IP with the MAC address of the interface in the VM, When i do the associate and disassociate the VM , it works fine But the Router NameSpace got changed. Regards Adivya Singh On Thu, Jun 23, 2022 at 1:22 PM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Dnia wtorek, 21 czerwca 2022 13:55:51 CEST Adivya Singh pisze:
hi Eugen,
The current setup is 3 controller nodes, The Load is not much on each controller and the number of DHCP agent is always set to 2 as per the standard in the neutron.conf, The L3 agent seems to be stables as other router namespace works fine under it, Only few router Namespace get affected under the agent.
Is it that problem happens for new floating IPs or for the FIPs which were working fine and then suddenly stopped working? If the latter, was there any action which triggered the issue to happen? Is there e.g. only one FIP broken in the router or maybe when it happens, then all FIPs which uses same router are broken?
Can You also try to analyze with e.g. tcpdump where traffic is dropped exactly? You can check http://kaplonski.pl/blog/neutron-where-is-my-packet-2/ for some more detailed description how traffic should go from the external network to Your instance.
Most of the template having issue , Have all instance having FLoating
IP, a
Stack with a single floating IP have chance of issue very less
Regards Adivya Singh
On Tue, Jun 21, 2022 at 1:18 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
this sounds very familiar to me, I had to deal with something similar a couple of times in a heavily used cluster with 2 control nodes. What does your setup look like, is it a HA setup? I would start checking the DHCP and L3 agents. After increasing dhcp_agents_per_network to 2 in neutron.conf and restarting the services this didn't occur again (yet). This would impact floating IPs as well, sometimes I had to disable and enable the affected router(s). If you only have one control node a different approach is necessary. Do you see a high load on the control node?
Zitat von Adivya Singh <adivya1.singh@gmail.com>:
hi Team,
We got a issue in Xena release, where we set the environment in Ubuntu Platform, But later we get some issues in Floating IP not reachable.
In a Network node, not all router namespace are Impacted and only few of them get affected, So we can not comment Network node has a issue.
The L3 agent where the Router is tied up, Worked just fine, as other routers work Fine.
and the one having issue in Floating IP, if i unassigned and assigned it starts working most of the time.
Any thoughts on this
Regards Adivya Singh
-- Slawek Kaplonski Principal Software Engineer Red Hat
hi Slawek, it happens with a given router namespace at a time Regards Adivya Singh On Fri, Jun 24, 2022 at 10:13 PM Adivya Singh <adivya1.singh@gmail.com> wrote:
Hi,
Thanks for the advice and the link,
What i saw when i do testing using tcpdump was "ARP" was not working, and it is not able to associate the FLoating IP with the MAC address of the interface in the VM, When i do the associate and disassociate the VM , it works fine
But the Router NameSpace got changed.
Regards Adivya Singh
On Thu, Jun 23, 2022 at 1:22 PM Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Dnia wtorek, 21 czerwca 2022 13:55:51 CEST Adivya Singh pisze:
hi Eugen,
The current setup is 3 controller nodes, The Load is not much on each controller and the number of DHCP agent is always set to 2 as per the standard in the neutron.conf, The L3 agent seems to be stables as other router namespace works fine under it, Only few router Namespace get affected under the agent.
Is it that problem happens for new floating IPs or for the FIPs which were working fine and then suddenly stopped working? If the latter, was there any action which triggered the issue to happen? Is there e.g. only one FIP broken in the router or maybe when it happens, then all FIPs which uses same router are broken?
Can You also try to analyze with e.g. tcpdump where traffic is dropped exactly? You can check http://kaplonski.pl/blog/neutron-where-is-my-packet-2/ for some more detailed description how traffic should go from the external network to Your instance.
Most of the template having issue , Have all instance having FLoating
IP, a
Stack with a single floating IP have chance of issue very less
Regards Adivya Singh
On Tue, Jun 21, 2022 at 1:18 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
this sounds very familiar to me, I had to deal with something similar a couple of times in a heavily used cluster with 2 control nodes. What does your setup look like, is it a HA setup? I would start checking the DHCP and L3 agents. After increasing dhcp_agents_per_network to 2 in neutron.conf and restarting the services this didn't occur again (yet). This would impact floating IPs as well, sometimes I had to disable and enable the affected router(s). If you only have one control node a different approach is necessary. Do you see a high load on the control node?
Zitat von Adivya Singh <adivya1.singh@gmail.com>:
hi Team,
We got a issue in Xena release, where we set the environment in Ubuntu Platform, But later we get some issues in Floating IP not reachable.
In a Network node, not all router namespace are Impacted and only few of them get affected, So we can not comment Network node has a issue.
The L3 agent where the Router is tied up, Worked just fine, as other routers work Fine.
and the one having issue in Floating IP, if i unassigned and assigned it starts working most of the time.
Any thoughts on this
Regards Adivya Singh
-- Slawek Kaplonski Principal Software Engineer Red Hat
participants (3)
-
Adivya Singh
-
Eugen Block
-
Slawek Kaplonski