<div dir="ltr">Version 1.2.13 is reliable. </div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 22, 2016 at 8:40 PM, Assaf Muller <span dir="ltr"><<a href="mailto:assaf@redhat.com" target="_blank">assaf@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Wed, Jun 22, 2016 at 12:02 PM, fabrice grelaud<br>
<<a href="mailto:fabrice.grelaud@u-bordeaux.fr">fabrice.grelaud@u-bordeaux.fr</a>> wrote:<br>
><br>
> Le 22 juin 2016 Ã 17:35, fabrice grelaud <<a href="mailto:fabrice.grelaud@u-bordeaux.fr">fabrice.grelaud@u-bordeaux.fr</a>> a<br>
> écrit :<br>
><br>
><br>
> Le 22 juin 2016 à 15:45, Assaf Muller <<a href="mailto:assaf@redhat.com">assaf@redhat.com</a>> a écrit :<br>
><br>
> On Wed, Jun 22, 2016 at 9:24 AM, fabrice grelaud<br>
> <<a href="mailto:fabrice.grelaud@u-bordeaux.fr">fabrice.grelaud@u-bordeaux.fr</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> we deployed our openstack infrastructure with your « exciting » project<br>
> openstack-ansible (mitaka 13.1.2) but we have some problems with L3HA after<br>
> create router.<br>
><br>
> Our infra (closer to the doc):<br>
> 3 controllers nodes (with bond0 (br-mgmt, br-storage), bond1 (br-vxlan,<br>
> br-vlan))<br>
> 2 compute nodes (same for network)<br>
><br>
> We create an external network (vlan type), an internal network (vxlan type)<br>
> and a router connected to both networks.<br>
> And when we launch an instance (cirros), we can’t receive an ip on the vm.<br>
><br>
> We have:<br>
><br>
> root@p-osinfra03-utility-container-783041da:~# neutron<br>
> l3-agent-list-hosting-router router-bim<br>
> +--------------------------------------+-----------------------------------------------+----------------+-------+----------+<br>
> | id                  | host<br>
> | admin_state_up | alive | ha_state |<br>
> +--------------------------------------+-----------------------------------------------+----------------+-------+----------+<br>
> | 3c7918e5-3ad6-4f82-a81b-700790e3c016 |<br>
> p-osinfra01-neutron-agents-container-f1ab9c14 | True      | :-)  |<br>
> active  |<br>
> | f2bf385a-f210-4dbc-8d7d-4b7b845c09b0 |<br>
> p-osinfra02-neutron-agents-container-48142ffe | True      | :-)  |<br>
> active  |<br>
> | 55350fac-16aa-488e-91fd-a7db38179c62 |<br>
> p-osinfra03-neutron-agents-container-2f6557f0 | True      | :-)  |<br>
> active  |<br>
> +--------------------------------------+-----------------------------------------------+----------------+-------+—————+<br>
><br>
> I know, i got a problem now because i should have :-) active, :-) standby,<br>
> :-) standby… Snif...<br>
><br>
> root@p-osinfra01-neutron-agents-container-f1ab9c14:~# ip netns<br>
> qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6<br>
> qdhcp-0ba266fb-15c4-4566-ae88-92d4c8fd2036<br>
><br>
> root@p-osinfra01-neutron-agents-container-f1ab9c14:~# ip netns exec<br>
> qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6 ip a sh<br>
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group<br>
> default<br>
>Â Â link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00<br>
>Â Â inet <a href="http://127.0.0.1/8" rel="noreferrer" target="_blank">127.0.0.1/8</a> scope host lo<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
>Â Â inet6 ::1/128 scope host<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
> 2: ha-4a5f0287-91@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc<br>
> pfifo_fast state UP group default qlen 1000<br>
>Â Â link/ether fa:16:3e:c2:67:a9 brd ff:ff:ff:ff:ff:ff<br>
>Â Â inet <a href="http://169.254.192.1/18" rel="noreferrer" target="_blank">169.254.192.1/18</a> brd 169.254.255.255 scope global ha-4a5f0287-91<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
>Â Â inet <a href="http://169.254.0.1/24" rel="noreferrer" target="_blank">169.254.0.1/24</a> scope global ha-4a5f0287-91<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
>Â Â inet6 fe80::f816:3eff:fec2:67a9/64 scope link<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
> 3: qr-44804d69-88@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc<br>
> pfifo_fast state UP group default qlen 1000<br>
>Â Â link/ether fa:16:3e:a5:8c:f2 brd ff:ff:ff:ff:ff:ff<br>
>Â Â inet <a href="http://192.168.100.254/24" rel="noreferrer" target="_blank">192.168.100.254/24</a> scope global qr-44804d69-88<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
>Â Â inet6 fe80::f816:3eff:fea5:8cf2/64 scope link<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
> 4: qg-c5c7378e-1d@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc<br>
> pfifo_fast state UP group default qlen 1000<br>
>Â Â link/ether fa:16:3e:b6:4c:97 brd ff:ff:ff:ff:ff:ff<br>
>Â Â inet <a href="http://147.210.240.11/23" rel="noreferrer" target="_blank">147.210.240.11/23</a> scope global qg-c5c7378e-1d<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
>Â Â inet <a href="http://147.210.240.12/32" rel="noreferrer" target="_blank">147.210.240.12/32</a> scope global qg-c5c7378e-1d<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
>Â Â inet6 fe80::f816:3eff:feb6:4c97/64 scope link<br>
>Â Â Â Â valid_lft forever preferred_lft forever<br>
><br>
> Same result on infra02 and infra03, qr and qg interfaces have the same ip,<br>
> and ha interfaces the address 169.254.0.1.<br>
><br>
> If we stop 2 neutron agent containers (p-osinfra02, p-osinfra03) and we<br>
> restart the first (p-osinfra01), we can reboot the instance and we got an<br>
> ip, a floating ip and we can access by ssh from internet to the vm. (Note:<br>
> after few time, we loss our connectivity too).<br>
><br>
> But if we restart the two containers, we got a ha_state to « standby » until<br>
> the three become « active » and finally we have the problem again.<br>
><br>
> The three routers on infra 01/02/03 are seen as master.<br>
><br>
> If we ping from our instance to the router (internal network 192.168.100.4<br>
> to 192.168.100.254) we can see some ARP Request<br>
> ARP, Request who-has 192.168.100.254 tell 192.168.100.4, length 28<br>
> ARP, Request who-has 192.168.100.254 tell 192.168.100.4, length 28<br>
> ARP, Request who-has 192.168.100.254 tell 192.168.100.4, length 28<br>
><br>
> And on the compute node we see all these frames on the various interfaces<br>
> tap / vxlan-89 / br-vxlan / bond1.vxlanvlan / bond1 / em2 but nothing back.<br>
><br>
> We also have on ha interface, on each router, the VRRP communication<br>
> (heartbeat packets over a hidden project network that connects all ha<br>
> routers (vxlan 70) ) . Priori as normal, each router thinks to be master.<br>
><br>
> root@p-osinfra01-neutron-agents-container-f1ab9c14:~# ip netns exec<br>
> qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6 tcpdump -nl -i ha-4a5f0287-91<br>
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode<br>
> listening on ha-4a5f0287-91, link-type EN10MB (Ethernet), capture size 65535<br>
> bytes<br>
> IP 169.254.192.1 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> IP 169.254.192.1 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> IP 169.254.192.1 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> IP 169.254.192.1 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
><br>
> root@p-osinfra02-neutron-agents-container-48142ffe:~# ip netns exec<br>
> qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6 tcpdump -nt -i ha-4ee5f8d0-7f<br>
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode<br>
> listening on ha-4ee5f8d0-7f, link-type EN10MB (Ethernet), capture size 65535<br>
> bytes<br>
> IP 169.254.192.3 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> IP 169.254.192.3 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> IP 169.254.192.3 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> IP 169.254.192.3 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> IP 169.254.192.3 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
><br>
><br>
> Are you seeing VRRP advertisements crossing nodes though? That tcpdump<br>
> only shows advertisements from the local node. If nodes aren't<br>
> receiving VRRP messages from other nodes, keepalived will declare<br>
> itself as master (As expected). Can you ping the 'ha' interface from<br>
> one router namespace to the other?<br>
><br>
><br>
> I stop the three neutron agent container.<br>
> Restart on infra01 then on infra02<br>
><br>
> I can see VRRP frames from infra01 (169.254.192.1 -> 224.0.0.18) and receive<br>
> by infra02.<br>
><br>
> root@p-osinfra02:~# tcpdump -nl -i em2 | grep 169.254<br>
> tcpdump: WARNING: em2: no IPv4 address assigned<br>
> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode<br>
> listening on em2, link-type EN10MB (Ethernet), capture size 65535 bytes<br>
> IP 169.254.192.1 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> IP 169.254.192.1 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
> ….<br>
> ….<br>
> then i have<br>
> IP 169.254.192.3 > <a href="http://224.0.0.18" rel="noreferrer" target="_blank">224.0.0.18</a>: VRRPv2, Advertisement, vrid 1, prio 50,<br>
> authtype simple, intvl 2s, length 20<br>
><br>
> No more 169.254.192.1 from infra01 but the IP of HA interface from router on<br>
> infra02.<br>
><br>
> And no more VRRP advertisements cross the nodes.<br>
> On each infra node, we see VRRP advertisements from the node itself but<br>
> nothing from the other.<br>
><br>
> And otherwise, i can ping ha interface from one router namespace to the<br>
> other:<br>
> root@p-osinfra01-neutron-agents-container-f1ab9c14:~# ip netns exec<br>
> qrouter-eeb2147a-5cc6-4b5e-b97c-07cfc141e8e6 ping 169.254.192.3<br>
> PING 169.254.192.3 (169.254.192.3) 56(84) bytes of data.<br>
> 64 bytes from <a href="http://169.254.192.3" rel="noreferrer" target="_blank">169.254.192.3</a>: icmp_seq=1 ttl=64 time=0.297 ms<br>
> 64 bytes from <a href="http://169.254.192.3" rel="noreferrer" target="_blank">169.254.192.3</a>: icmp_seq=2 ttl=64 time=0.239 ms<br>
> 64 bytes from <a href="http://169.254.192.3" rel="noreferrer" target="_blank">169.254.192.3</a>: icmp_seq=3 ttl=64 time=0.264 ms<br>
><br>
> im’ going to test with other version of keepalived (current version here<br>
> 1.2.7-1 ubuntu 14.04).<br>
><br>
> Thanks to help<br>
><br>
><br>
> Note:<br>
> I said i can ping between ha interface but not for long time. At one point,<br>
> i can’t anymore… :-(<br>
<br>
</div></div>That's the problem. This becomes a normal Neutron troubleshooting: Why<br>
can't one port ping the other? This might help:<br>
<a href="https://assafmuller.com/2015/08/31/neutron-troubleshooting/" rel="noreferrer" target="_blank">https://assafmuller.com/2015/08/31/neutron-troubleshooting/</a><br>
<div class="HOEnZb"><div class="h5"><br>
><br>
><br>
><br>
><br>
><br>
> Someone could tell me if he has already encountered this problem ?<br>
> The infra and compute nodes are connected to a nexus 9000 switch.<br>
><br>
> Thank you in advance for taking the time to study my request.<br>
><br>
> Fabrice Grelaud<br>
> Université de Bordeaux<br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Regards,<div>Ann Kamyshnikova</div><div>Mirantis, Inc</div></div></div>
</div>