<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">Le 22 juin 2016 à 19:40, Assaf Muller <<a href="mailto:assaf@redhat.com" class="">assaf@redhat.com</a>> a écrit :</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On Wed, Jun 22, 2016 at 12:02 PM, fabrice grelaud</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""><</span><a href="mailto:fabrice.grelaud@u-bordeaux.fr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">fabrice.grelaud@u-bordeaux.fr</a><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">> wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class="">Le 22 juin 2016 à 17:35, fabrice grelaud <<a href="mailto:fabrice.grelaud@u-bordeaux.fr" class="">fabrice.grelaud@u-bordeaux.fr</a>> a<br class="">écrit :<br class=""><br class=""><br class="">Le 22 juin 2016 à 15:45, Assaf Muller <<a href="mailto:assaf@redhat.com" class="">assaf@redhat.com</a>> a écrit :<br class=""><br class="">On Wed, Jun 22, 2016 at 9:24 AM, fabrice grelaud<br class=""><<a href="mailto:fabrice.grelaud@u-bordeaux.fr" class="">fabrice.grelaud@u-bordeaux.fr</a>> wrote:<br class=""><br class="">Hi,<br class=""><br class="">we deployed our openstack infrastructure with your « exciting » project<br class="">openstack-ansible (mitaka 13.1.2) but we have some problems with L3HA after<br class="">create router.<br class=""><br class="">Our infra (closer to the doc):<br class="">3 controllers nodes (with bond0 (br-mgmt, br-storage), bond1 (br-vxlan,<br class="">br-vlan))<br class="">2 compute nodes (same for network)<br class=""><br class="">We create an external network (vlan type), an internal network (vxlan type)<br class="">and a router connected to both networks.<br class="">And when we launch an instance (cirros), we can’t receive an ip on the vm.<br class=""><br class="">We have:<br class=""><br class="">root@p-osinfra03-utility-container-783041da:~# neutron<br class="">l3-agent-list-hosting-router router-bim<br class="">+--------------------------------------+-----------------------------------------------+----------------+-------+----------+<br class="">| id | host<br class="">| admin_state_up | alive | ha_state |<br class="">+--------------------------------------+-----------------------------------------------+----------------+-------+----------+<br class="">| 3c7918e5-3ad6-4f82-a81b-700790e3c016 |<br class="">p-osinfra01-neutron-agents-container-f1ab9c14 | True | :-) |<br class="">active |<br class="">| f2bf385a-f210-4dbc-8d7d-4b7b845c09b0 |<br class="">p-osinfra02-neutron-agents-container-48142ffe | True | :-) |<br class="">active |<br class="">| 55350fac-16aa-488e-91fd-a7db38179c62 |<br class="">p-osinfra03-neutron-agents-container-2f6557f0 | True | :-) |<br class="">active |<br class="">+--------------------------------------+-----------------------------------------------+----------------+-------+—————+<br class=""><br class="">I know, i got a problem now because i should have :-) active, :-) standby,<br class="">:-) standby… Snif...<br class=""><br class="">root@p-osinfra01-neutron-agents-container-f1ab9c14:~# ip netns<br class="">qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6<br class="">qdhcp-0ba266fb-15c4-4566-ae88-92d4c8fd2036<br class=""><br class="">root@p-osinfra01-neutron-agents-container-f1ab9c14:~# ip netns exec<br class="">qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6 ip a sh<br class="">1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group<br class="">default<br class=""> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br class=""> inet 127.0.0.1/8 scope host lo<br class=""> valid_lft forever preferred_lft forever<br class=""> inet6 ::1/128 scope host<br class=""> valid_lft forever preferred_lft forever<br class="">2: ha-4a5f0287-91@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc<br class="">pfifo_fast state UP group default qlen 1000<br class=""> link/ether fa:16:3e:c2:67:a9 brd ff:ff:ff:ff:ff:ff<br class=""> inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-4a5f0287-91<br class=""> valid_lft forever preferred_lft forever<br class=""> inet 169.254.0.1/24 scope global ha-4a5f0287-91<br class=""> valid_lft forever preferred_lft forever<br class=""> inet6 fe80::f816:3eff:fec2:67a9/64 scope link<br class=""> valid_lft forever preferred_lft forever<br class="">3: qr-44804d69-88@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc<br class="">pfifo_fast state UP group default qlen 1000<br class=""> link/ether fa:16:3e:a5:8c:f2 brd ff:ff:ff:ff:ff:ff<br class=""> inet 192.168.100.254/24 scope global qr-44804d69-88<br class=""> valid_lft forever preferred_lft forever<br class=""> inet6 fe80::f816:3eff:fea5:8cf2/64 scope link<br class=""> valid_lft forever preferred_lft forever<br class="">4: qg-c5c7378e-1d@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br class="">pfifo_fast state UP group default qlen 1000<br class=""> link/ether fa:16:3e:b6:4c:97 brd ff:ff:ff:ff:ff:ff<br class=""> inet 147.210.240.11/23 scope global qg-c5c7378e-1d<br class=""> valid_lft forever preferred_lft forever<br class=""> inet 147.210.240.12/32 scope global qg-c5c7378e-1d<br class=""> valid_lft forever preferred_lft forever<br class=""> inet6 fe80::f816:3eff:feb6:4c97/64 scope link<br class=""> valid_lft forever preferred_lft forever<br class=""><br class="">Same result on infra02 and infra03, qr and qg interfaces have the same ip,<br class="">and ha interfaces the address 169.254.0.1.<br class=""><br class="">If we stop 2 neutron agent containers (p-osinfra02, p-osinfra03) and we<br class="">restart the first (p-osinfra01), we can reboot the instance and we got an<br class="">ip, a floating ip and we can access by ssh from internet to the vm. (Note:<br class="">after few time, we loss our connectivity too).<br class=""><br class="">But if we restart the two containers, we got a ha_state to « standby » until<br class="">the three become « active » and finally we have the problem again.<br class=""><br class="">The three routers on infra 01/02/03 are seen as master.<br class=""><br class="">If we ping from our instance to the router (internal network 192.168.100.4<br class="">to 192.168.100.254) we can see some ARP Request<br class="">ARP, Request who-has 192.168.100.254 tell 192.168.100.4, length 28<br class="">ARP, Request who-has 192.168.100.254 tell 192.168.100.4, length 28<br class="">ARP, Request who-has 192.168.100.254 tell 192.168.100.4, length 28<br class=""><br class="">And on the compute node we see all these frames on the various interfaces<br class="">tap / vxlan-89 / br-vxlan / bond1.vxlanvlan / bond1 / em2 but nothing back.<br class=""><br class="">We also have on ha interface, on each router, the VRRP communication<br class="">(heartbeat packets over a hidden project network that connects all ha<br class="">routers (vxlan 70) ) . Priori as normal, each router thinks to be master.<br class=""><br class="">root@p-osinfra01-neutron-agents-container-f1ab9c14:~# ip netns exec<br class="">qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6 tcpdump -nl -i ha-4a5f0287-91<br class="">tcpdump: verbose output suppressed, use -v or -vv for full protocol decode<br class="">listening on ha-4a5f0287-91, link-type EN10MB (Ethernet), capture size 65535<br class="">bytes<br class="">IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class=""><br class="">root@p-osinfra02-neutron-agents-container-48142ffe:~# ip netns exec<br class="">qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6 tcpdump -nt -i ha-4ee5f8d0-7f<br class="">tcpdump: verbose output suppressed, use -v or -vv for full protocol decode<br class="">listening on ha-4ee5f8d0-7f, link-type EN10MB (Ethernet), capture size 65535<br class="">bytes<br class="">IP 169.254.192.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">IP 169.254.192.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">IP 169.254.192.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">IP 169.254.192.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">IP 169.254.192.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class=""><br class=""><br class="">Are you seeing VRRP advertisements crossing nodes though? That tcpdump<br class="">only shows advertisements from the local node. If nodes aren't<br class="">receiving VRRP messages from other nodes, keepalived will declare<br class="">itself as master (As expected). Can you ping the 'ha' interface from<br class="">one router namespace to the other?<br class=""><br class=""><br class="">I stop the three neutron agent container.<br class="">Restart on infra01 then on infra02<br class=""><br class="">I can see VRRP frames from infra01 (169.254.192.1 -> 224.0.0.18) and receive<br class="">by infra02.<br class=""><br class="">root@p-osinfra02:~# tcpdump -nl -i em2 | grep 169.254<br class="">tcpdump: WARNING: em2: no IPv4 address assigned<br class="">tcpdump: verbose output suppressed, use -v or -vv for full protocol decode<br class="">listening on em2, link-type EN10MB (Ethernet), capture size 65535 bytes<br class="">IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class="">….<br class="">….<br class="">then i have<br class="">IP 169.254.192.3 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50,<br class="">authtype simple, intvl 2s, length 20<br class=""><br class="">No more 169.254.192.1 from infra01 but the IP of HA interface from router on<br class="">infra02.<br class=""><br class="">And no more VRRP advertisements cross the nodes.<br class="">On each infra node, we see VRRP advertisements from the node itself but<br class="">nothing from the other.<br class=""><br class="">And otherwise, i can ping ha interface from one router namespace to the<br class="">other:<br class="">root@p-osinfra01-neutron-agents-container-f1ab9c14:~# ip netns exec<br class="">qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6 ping 169.254.192.3<br class="">PING 169.254.192.3 (169.254.192.3) 56(84) bytes of data.<br class="">64 bytes from 169.254.192.3: icmp_seq=1 ttl=64 time=0.297 ms<br class="">64 bytes from 169.254.192.3: icmp_seq=2 ttl=64 time=0.239 ms<br class="">64 bytes from 169.254.192.3: icmp_seq=3 ttl=64 time=0.264 ms<br class=""><br class="">im’ going to test with other version of keepalived (current version here<br class="">1.2.7-1 ubuntu 14.04).<br class=""><br class="">Thanks to help<br class=""><br class=""><br class="">Note:<br class="">I said i can ping between ha interface but not for long time. At one point,<br class="">i can’t anymore… :-(<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">That's the problem. This becomes a normal Neutron troubleshooting: Why</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">can't one port ping the other? This might help:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="https://assafmuller.com/2015/08/31/neutron-troubleshooting/" class="">https://assafmuller.com/2015/08/31/neutron-troubleshooting/</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div>Hi,</div><div><br class=""></div><div>thanks for the link. I already had a look (more or less) and finally, i suspected a problem rather on switch side. (nexus)</div><div><br class=""></div><div>And after some investigation and tcpdump, we saw that packets to 239.1.1.1 are not forwarding by the switch.</div><div>In fact, « igmp snooping » is enabled by default on nexus switch causing the bad behaviour.</div><div><br class=""></div><div>We disable igmp snooping and That’s all folks !</div><div><br class=""></div><div>Cordially,</div><div><br class=""><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><br class=""><br class=""><br class=""><br class="">Someone could tell me if he has already encountered this problem ?<br class="">The infra and compute nodes are connected to a nexus 9000 switch.<br class=""><br class="">Thank you in advance for taking the time to study my request.<br class=""><br class="">Fabrice Grelaud<br class="">Université de Bordeaux<br class=""><br class=""><br class="">__________________________________________________________________________<br class="">OpenStack Development Mailing List (not for usage questions)<br class="">Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" class="">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class=""><br class=""><br class="">__________________________________________________________________________<br class="">OpenStack Development Mailing List (not for usage questions)<br class="">Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br class=""><br class=""><br class="">__________________________________________________________________________<br class="">OpenStack Development Mailing List (not for usage questions)<br class="">Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br class=""><br class=""><br class=""><br class="">__________________________________________________________________________<br class="">OpenStack Development Mailing List (not for usage questions)<br class="">Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br class=""><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">__________________________________________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">OpenStack Development Mailing List (not for usage questions)</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Unsubscribe:<span class="Apple-converted-space"> </span></span><a href="mailto:OpenStack-dev-request@lists.openstack.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">OpenStack-dev-request@lists.openstack.org</a><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">?subject:unsubscribe</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div></blockquote></div><br class=""></body></html>