Floating IP problem in HA OVN DVR with TripleO
Hi all, I deployed an environment with TripleO Ussuri with 3 HA Controllers and some Compute nodes with neutron-ovn-dvr-ha.yaml Instances have Internet access through routers with SNAT traffic (in this case traffic is routed via a controller node), and by assigning IP address directly from provider network (not having a router). But in case of assigning FIP from provider to an instance, VM Internet connection is lost. Here is the output of router nat lists, which seems OK: # ovn-nbctl lr-nat-list 587182a4-4d6b-41b0-9fd8-4c1be58811b0 TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat X.X.X.X 192.168.0.153 fa:16:3e:0a:86:4d e65bd8e9-5f95-4eb2-a316-97e86fbdb9b6 snat Y.Y.Y.Y 192.168.0.0/24 I replaced FIP with X.X.X.X and router IP with Y.Y.Y.Y When I remove * EXTERNAL_MAC* and *LOGICAL_PORT*, FIP works fine and as it has to be, but traffic routes from a Controller node and it won't be distributed anymore. Any idea or suggestion would be grateful. Regards, Reza
Hi, Maybe You hit this bug [1]. Please check what ovn version do You have and maybe update it if needed. On Mon, Sep 07, 2020 at 06:23:44PM +0430, Reza Bakhshayeshi wrote:
Hi all,
I deployed an environment with TripleO Ussuri with 3 HA Controllers and some Compute nodes with neutron-ovn-dvr-ha.yaml Instances have Internet access through routers with SNAT traffic (in this case traffic is routed via a controller node), and by assigning IP address directly from provider network (not having a router).
But in case of assigning FIP from provider to an instance, VM Internet connection is lost. Here is the output of router nat lists, which seems OK:
# ovn-nbctl lr-nat-list 587182a4-4d6b-41b0-9fd8-4c1be58811b0 TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat X.X.X.X 192.168.0.153 fa:16:3e:0a:86:4d e65bd8e9-5f95-4eb2-a316-97e86fbdb9b6 snat Y.Y.Y.Y 192.168.0.0/24
I replaced FIP with X.X.X.X and router IP with Y.Y.Y.Y
When I remove * EXTERNAL_MAC* and *LOGICAL_PORT*, FIP works fine and as it has to be, but traffic routes from a Controller node and it won't be distributed anymore.
Any idea or suggestion would be grateful. Regards, Reza
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1834433 -- Slawek Kaplonski Principal software engineer Red Hat
Hi Slawek, I'm using the latest CentOS 8 Ussuri OVN packages at: https://trunk.rdoproject.org/centos8-ussuri/deps/latest/x86_64/ On both Controller and Compute I get: # rpm -qa | grep ovn ovn-host-20.03.0-4.el8.x86_64 ovn-20.03.0-4.el8.x86_64 # yum info ovn Installed Packages Name : ovn Version : 20.03.0 Release : 4.el8 Architecture : x86_64 Size : 12 M Source : ovn-20.03.0-4.el8.src.rpm Repository : @System
From repo : delorean-ussuri-testing Summary : Open Virtual Network support URL : http://www.openvswitch.org/ License : ASL 2.0 and LGPLv2+ and SISSL
Do you suggest installing ovn manually from source on containers? ي On Tue, 8 Sep 2020 at 12:39, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Maybe You hit this bug [1]. Please check what ovn version do You have and maybe update it if needed.
On Mon, Sep 07, 2020 at 06:23:44PM +0430, Reza Bakhshayeshi wrote:
Hi all,
I deployed an environment with TripleO Ussuri with 3 HA Controllers and some Compute nodes with neutron-ovn-dvr-ha.yaml Instances have Internet access through routers with SNAT traffic (in this case traffic is routed via a controller node), and by assigning IP address directly from provider network (not having a router).
But in case of assigning FIP from provider to an instance, VM Internet connection is lost. Here is the output of router nat lists, which seems OK:
# ovn-nbctl lr-nat-list 587182a4-4d6b-41b0-9fd8-4c1be58811b0 TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat X.X.X.X 192.168.0.153 fa:16:3e:0a:86:4d e65bd8e9-5f95-4eb2-a316-97e86fbdb9b6 snat Y.Y.Y.Y 192.168.0.0/24
I replaced FIP with X.X.X.X and router IP with Y.Y.Y.Y
When I remove * EXTERNAL_MAC* and *LOGICAL_PORT*, FIP works fine and as it has to be, but traffic routes from a Controller node and it won't be distributed anymore.
Any idea or suggestion would be grateful. Regards, Reza
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1834433
-- Slawek Kaplonski Principal software engineer Red Hat
Hi Reza, Are you using 'geneve' tenant networks or 'vlan' ones? I am asking because with VLAN we have the following DVR issue [1] [1] Bug 1704596 - FIP traffix does not work on OVN-DVR setup when using VLAN tenant network type <https://bugzilla.redhat.com/show_bug.cgi?id=1704596> On Tue, Sep 8, 2020 at 2:04 PM Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hi Slawek,
I'm using the latest CentOS 8 Ussuri OVN packages at: https://trunk.rdoproject.org/centos8-ussuri/deps/latest/x86_64/
On both Controller and Compute I get:
# rpm -qa | grep ovn ovn-host-20.03.0-4.el8.x86_64 ovn-20.03.0-4.el8.x86_64
# yum info ovn Installed Packages Name : ovn Version : 20.03.0 Release : 4.el8 Architecture : x86_64 Size : 12 M Source : ovn-20.03.0-4.el8.src.rpm Repository : @System From repo : delorean-ussuri-testing Summary : Open Virtual Network support URL : http://www.openvswitch.org/ License : ASL 2.0 and LGPLv2+ and SISSL
Do you suggest installing ovn manually from source on containers? ي
On Tue, 8 Sep 2020 at 12:39, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Maybe You hit this bug [1]. Please check what ovn version do You have and maybe update it if needed.
Hi all,
I deployed an environment with TripleO Ussuri with 3 HA Controllers and some Compute nodes with neutron-ovn-dvr-ha.yaml Instances have Internet access through routers with SNAT traffic (in
On Mon, Sep 07, 2020 at 06:23:44PM +0430, Reza Bakhshayeshi wrote: this
case traffic is routed via a controller node), and by assigning IP address directly from provider network (not having a router).
But in case of assigning FIP from provider to an instance, VM Internet connection is lost. Here is the output of router nat lists, which seems OK:
# ovn-nbctl lr-nat-list 587182a4-4d6b-41b0-9fd8-4c1be58811b0 TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat X.X.X.X 192.168.0.153 fa:16:3e:0a:86:4d e65bd8e9-5f95-4eb2-a316-97e86fbdb9b6 snat Y.Y.Y.Y 192.168.0.0/24
I replaced FIP with X.X.X.X and router IP with Y.Y.Y.Y
When I remove * EXTERNAL_MAC* and *LOGICAL_PORT*, FIP works fine and as it has to be, but traffic routes from a Controller node and it won't be distributed anymore.
Any idea or suggestion would be grateful. Regards, Reza
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1834433
-- Slawek Kaplonski Principal software engineer Red Hat
-- ROMAN SAFRONOV SENIOR QE, OPENSTACK NETWORKING Red Hat <https://www.redhat.com/> Israel M: +972545433957 <https://red.ht/sig>
Hi Roman, I'm using 'geneve' for my tenant networks. By the way, by pinging 8.8.8.8 from an instance with FIP, tcpdump on its Compute node shows an ARP request for every lost ping. Is it normal behaviour? 21:13:04.808508 ARP, Request who-has dns.google tell X.X.X.X , length 28 21:13:05.808726 ARP, Request who-has dns.google tell X.X.X.X , length 28 21:13:06.808900 ARP, Request who-has dns.google tell X.X.X.X , length 28 . . . X.X.X.X if FIP of VM. On Tue, 8 Sep 2020 at 17:21, Roman Safronov <rsafrono@redhat.com> wrote:
Hi Reza,
Are you using 'geneve' tenant networks or 'vlan' ones? I am asking because with VLAN we have the following DVR issue [1]
[1] Bug 1704596 - FIP traffix does not work on OVN-DVR setup when using VLAN tenant network type <https://bugzilla.redhat.com/show_bug.cgi?id=1704596>
On Tue, Sep 8, 2020 at 2:04 PM Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hi Slawek,
I'm using the latest CentOS 8 Ussuri OVN packages at: https://trunk.rdoproject.org/centos8-ussuri/deps/latest/x86_64/
On both Controller and Compute I get:
# rpm -qa | grep ovn ovn-host-20.03.0-4.el8.x86_64 ovn-20.03.0-4.el8.x86_64
# yum info ovn Installed Packages Name : ovn Version : 20.03.0 Release : 4.el8 Architecture : x86_64 Size : 12 M Source : ovn-20.03.0-4.el8.src.rpm Repository : @System From repo : delorean-ussuri-testing Summary : Open Virtual Network support URL : http://www.openvswitch.org/ License : ASL 2.0 and LGPLv2+ and SISSL
Do you suggest installing ovn manually from source on containers? ي
On Tue, 8 Sep 2020 at 12:39, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Maybe You hit this bug [1]. Please check what ovn version do You have and maybe update it if needed.
Hi all,
I deployed an environment with TripleO Ussuri with 3 HA Controllers and some Compute nodes with neutron-ovn-dvr-ha.yaml Instances have Internet access through routers with SNAT traffic (in
On Mon, Sep 07, 2020 at 06:23:44PM +0430, Reza Bakhshayeshi wrote: this
case traffic is routed via a controller node), and by assigning IP address directly from provider network (not having a router).
But in case of assigning FIP from provider to an instance, VM Internet connection is lost. Here is the output of router nat lists, which seems OK:
# ovn-nbctl lr-nat-list 587182a4-4d6b-41b0-9fd8-4c1be58811b0 TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat X.X.X.X 192.168.0.153 fa:16:3e:0a:86:4d e65bd8e9-5f95-4eb2-a316-97e86fbdb9b6 snat Y.Y.Y.Y 192.168.0.0/24
I replaced FIP with X.X.X.X and router IP with Y.Y.Y.Y
When I remove * EXTERNAL_MAC* and *LOGICAL_PORT*, FIP works fine and as it has to be, but traffic routes from a Controller node and it won't be distributed anymore.
Any idea or suggestion would be grateful. Regards, Reza
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1834433
-- Slawek Kaplonski Principal software engineer Red Hat
--
ROMAN SAFRONOV
SENIOR QE, OPENSTACK NETWORKING
Red Hat <https://www.redhat.com/>
Israel
M: +972545433957
I'm using 'geneve' for my tenant networks.
By the way, by pinging 8.8.8.8 from an instance with FIP, tcpdump on its Compute node shows an ARP request for every lost ping. Is it normal behaviour?
21:13:04.808508 ARP, Request who-has dns.google tell X.X.X.X , length 28 21:13:05.808726 ARP, Request who-has dns.google tell X.X.X.X , length 28 21:13:06.808900 ARP, Request who-has dns.google tell X.X.X.X , length 28 . . . X.X.X.X if FIP of VM.
If so, it looks like the bug <https://bugzilla.redhat.com/show_bug.cgi?id=1834433> mentioned above. On Tue, Sep 8, 2020 at 7:51 PM Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hi Roman,
I'm using 'geneve' for my tenant networks.
By the way, by pinging 8.8.8.8 from an instance with FIP, tcpdump on its Compute node shows an ARP request for every lost ping. Is it normal behaviour?
21:13:04.808508 ARP, Request who-has dns.google tell X.X.X.X , length 28 21:13:05.808726 ARP, Request who-has dns.google tell X.X.X.X , length 28 21:13:06.808900 ARP, Request who-has dns.google tell X.X.X.X , length 28 . . . X.X.X.X if FIP of VM.
On Tue, 8 Sep 2020 at 17:21, Roman Safronov <rsafrono@redhat.com> wrote:
Hi Reza,
Are you using 'geneve' tenant networks or 'vlan' ones? I am asking because with VLAN we have the following DVR issue [1]
[1] Bug 1704596 - FIP traffix does not work on OVN-DVR setup when using VLAN tenant network type <https://bugzilla.redhat.com/show_bug.cgi?id=1704596>
On Tue, Sep 8, 2020 at 2:04 PM Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hi Slawek,
I'm using the latest CentOS 8 Ussuri OVN packages at: https://trunk.rdoproject.org/centos8-ussuri/deps/latest/x86_64/
On both Controller and Compute I get:
# rpm -qa | grep ovn ovn-host-20.03.0-4.el8.x86_64 ovn-20.03.0-4.el8.x86_64
# yum info ovn Installed Packages Name : ovn Version : 20.03.0 Release : 4.el8 Architecture : x86_64 Size : 12 M Source : ovn-20.03.0-4.el8.src.rpm Repository : @System From repo : delorean-ussuri-testing Summary : Open Virtual Network support URL : http://www.openvswitch.org/ License : ASL 2.0 and LGPLv2+ and SISSL
Do you suggest installing ovn manually from source on containers? ي
On Tue, 8 Sep 2020 at 12:39, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Maybe You hit this bug [1]. Please check what ovn version do You have and maybe update it if needed.
Hi all,
I deployed an environment with TripleO Ussuri with 3 HA Controllers and some Compute nodes with neutron-ovn-dvr-ha.yaml Instances have Internet access through routers with SNAT traffic (in
On Mon, Sep 07, 2020 at 06:23:44PM +0430, Reza Bakhshayeshi wrote: this
case traffic is routed via a controller node), and by assigning IP address directly from provider network (not having a router).
But in case of assigning FIP from provider to an instance, VM Internet connection is lost. Here is the output of router nat lists, which seems OK:
# ovn-nbctl lr-nat-list 587182a4-4d6b-41b0-9fd8-4c1be58811b0 TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT dnat_and_snat X.X.X.X 192.168.0.153 fa:16:3e:0a:86:4d e65bd8e9-5f95-4eb2-a316-97e86fbdb9b6 snat Y.Y.Y.Y 192.168.0.0/24
I replaced FIP with X.X.X.X and router IP with Y.Y.Y.Y
When I remove * EXTERNAL_MAC* and *LOGICAL_PORT*, FIP works fine and as it has to be, but traffic routes from a Controller node and it won't be distributed anymore.
Any idea or suggestion would be grateful. Regards, Reza
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1834433
-- Slawek Kaplonski Principal software engineer Red Hat
--
ROMAN SAFRONOV
SENIOR QE, OPENSTACK NETWORKING
Red Hat <https://www.redhat.com/>
Israel
M: +972545433957
Hi Reza, Here is a related bug: https://bugs.launchpad.net/bugs/1881041 I had to use ovn/ovs 2.13 builds from cbs to overcome this issue ( https://cbs.centos.org/koji/buildinfo?buildID=30482) Regards, Michal On Tue, 8 Sep 2020 at 18:52, Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hi Roman,
I'm using 'geneve' for my tenant networks.
By the way, by pinging 8.8.8.8 from an instance with FIP, tcpdump on its Compute node shows an ARP request for every lost ping. Is it normal behaviour?
21:13:04.808508 ARP, Request who-has dns.google tell
X.X.X.X
, length 28 21:13:05.808726 ARP, Request who-has dns.google tell
X.X.X.X
, length 28 21:13:06.808900 ARP, Request who-has dns.google tell
X.X.X.X
, length 28 . . . X.X.X.X if FIP of VM.
On Tue, 8 Sep 2020 at 17:21, Roman Safronov <rsafrono@redhat.com> wrote:
Hi Reza,
Are you using 'geneve' tenant networks or 'vlan' ones? I am asking because with VLAN we have the following DVR issue [1]
[1] Bug 1704596 - FIP traffix does not work on OVN-DVR setup when using VLAN tenant network type <https://bugzilla.redhat.com/show_bug.cgi?id=1704596>
On Tue, Sep 8, 2020 at 2:04 PM Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hi Slawek,
I'm using the latest CentOS 8 Ussuri OVN packages at: https://trunk.rdoproject.org/centos8-ussuri/deps/latest/x86_64/
On both Controller and Compute I get:
# rpm -qa | grep ovn ovn-host-20.03.0-4.el8.x86_64 ovn-20.03.0-4.el8.x86_64
# yum info ovn Installed Packages Name : ovn Version : 20.03.0 Release : 4.el8 Architecture : x86_64 Size : 12 M Source : ovn-20.03.0-4.el8.src.rpm Repository : @System From repo : delorean-ussuri-testing Summary : Open Virtual Network support URL : http://www.openvswitch.org/ License : ASL 2.0 and LGPLv2+ and SISSL
Do you suggest installing ovn manually from source on containers? ي
On Tue, 8 Sep 2020 at 12:39, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Maybe You hit this bug [1]. Please check what ovn version do You have and maybe
update it if needed.
On Mon, Sep 07, 2020 at 06:23:44PM +0430, Reza Bakhshayeshi wrote:
Hi all,
I deployed an environment with TripleO Ussuri with 3 HA Controllers and
some Compute nodes with neutron-ovn-dvr-ha.yaml
Instances have Internet access through routers with SNAT traffic (in this
case traffic is routed via a controller node), and by assigning IP address
directly from provider network (not having a router).
But in case of assigning FIP from provider to an instance, VM Internet
connection is lost.
Here is the output of router nat lists, which seems OK:
# ovn-nbctl lr-nat-list 587182a4-4d6b-41b0-9fd8-4c1be58811b0
TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP
EXTERNAL_MAC LOGICAL_PORT
dnat_and_snat X.X.X.X 192.168.0.153
fa:16:3e:0a:86:4d e65bd8e9-5f95-4eb2-a316-97e86fbdb9b6
snat Y.Y.Y.Y 192.168.0.0/24
I replaced FIP with X.X.X.X and router IP with Y.Y.Y.Y
When I remove * EXTERNAL_MAC* and *LOGICAL_PORT*, FIP works fine and as it
has to be, but traffic routes from a Controller node and it won't be
distributed anymore.
Any idea or suggestion would be grateful.
Regards,
Reza
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1834433
--
Slawek Kaplonski
Principal software engineer
Red Hat
--
ROMAN SAFRONOV
SENIOR QE, OPENSTACK NETWORKING
Red Hat <https://www.redhat.com/>
Israel
M: +972545433957
-- Michał Nasiadka mnasiadka@gmail.com
Hi all, Thanks a lot for your guidance. I didn't have such a problem in TripleO Stein. Do you think using OVN DVR in a production environment is a wise choice? Regards, Reza On Tue, 8 Sep 2020 at 21:42, Michał Nasiadka <mnasiadka@gmail.com> wrote:
Hi Reza,
Here is a related bug: https://bugs.launchpad.net/bugs/1881041
I had to use ovn/ovs 2.13 builds from cbs to overcome this issue ( https://cbs.centos.org/koji/buildinfo?buildID=30482)
Regards, Michal
On Tue, 8 Sep 2020 at 18:52, Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hi Roman,
I'm using 'geneve' for my tenant networks.
By the way, by pinging 8.8.8.8 from an instance with FIP, tcpdump on its Compute node shows an ARP request for every lost ping. Is it normal behaviour?
21:13:04.808508 ARP, Request who-has dns.google tell
X.X.X.X
, length 28 21:13:05.808726 ARP, Request who-has dns.google tell
X.X.X.X
, length 28 21:13:06.808900 ARP, Request who-has dns.google tell
X.X.X.X
, length 28 . . . X.X.X.X if FIP of VM.
On Tue, 8 Sep 2020 at 17:21, Roman Safronov <rsafrono@redhat.com> wrote:
Hi Reza,
Are you using 'geneve' tenant networks or 'vlan' ones? I am asking because with VLAN we have the following DVR issue [1]
[1] Bug 1704596 - FIP traffix does not work on OVN-DVR setup when using VLAN tenant network type <https://bugzilla.redhat.com/show_bug.cgi?id=1704596>
On Tue, Sep 8, 2020 at 2:04 PM Reza Bakhshayeshi <reza.b2008@gmail.com> wrote:
Hi Slawek,
I'm using the latest CentOS 8 Ussuri OVN packages at: https://trunk.rdoproject.org/centos8-ussuri/deps/latest/x86_64/
On both Controller and Compute I get:
# rpm -qa | grep ovn ovn-host-20.03.0-4.el8.x86_64 ovn-20.03.0-4.el8.x86_64
# yum info ovn Installed Packages Name : ovn Version : 20.03.0 Release : 4.el8 Architecture : x86_64 Size : 12 M Source : ovn-20.03.0-4.el8.src.rpm Repository : @System From repo : delorean-ussuri-testing Summary : Open Virtual Network support URL : http://www.openvswitch.org/ License : ASL 2.0 and LGPLv2+ and SISSL
Do you suggest installing ovn manually from source on containers? ي
On Tue, 8 Sep 2020 at 12:39, Slawek Kaplonski <skaplons@redhat.com> wrote:
Hi,
Maybe You hit this bug [1]. Please check what ovn version do You have and maybe
update it if needed.
On Mon, Sep 07, 2020 at 06:23:44PM +0430, Reza Bakhshayeshi wrote:
Hi all,
I deployed an environment with TripleO Ussuri with 3 HA Controllers and
some Compute nodes with neutron-ovn-dvr-ha.yaml
Instances have Internet access through routers with SNAT traffic (in this
case traffic is routed via a controller node), and by assigning IP address
directly from provider network (not having a router).
But in case of assigning FIP from provider to an instance, VM Internet
connection is lost.
Here is the output of router nat lists, which seems OK:
# ovn-nbctl lr-nat-list 587182a4-4d6b-41b0-9fd8-4c1be58811b0
TYPE EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP
EXTERNAL_MAC LOGICAL_PORT
dnat_and_snat X.X.X.X 192.168.0.153
fa:16:3e:0a:86:4d e65bd8e9-5f95-4eb2-a316-97e86fbdb9b6
snat Y.Y.Y.Y 192.168.0.0/24
I replaced FIP with X.X.X.X and router IP with Y.Y.Y.Y
When I remove * EXTERNAL_MAC* and *LOGICAL_PORT*, FIP works fine and as it
has to be, but traffic routes from a Controller node and it won't be
distributed anymore.
Any idea or suggestion would be grateful.
Regards,
Reza
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1834433
--
Slawek Kaplonski
Principal software engineer
Red Hat
--
ROMAN SAFRONOV
SENIOR QE, OPENSTACK NETWORKING
Red Hat <https://www.redhat.com/>
Israel
M: +972545433957
-- Michał Nasiadka mnasiadka@gmail.com
participants (4)
-
Michał Nasiadka
-
Reza Bakhshayeshi
-
Roman Safronov
-
Slawek Kaplonski