Open Stack

On 12/13/2016 02:45 PM, Gustavo Randich wrote:
> Hi Openstackers,
>
> We have the folowing issue (using Mitaka / DVR / Xenial), perhaps someone can
> help ;)
>
> When our hosts boots up, the ARP cache population loop of L3 Agent is delaying
> the start of neutron-ns-metadata-proxy for around a minute -- see logs below;
> then, when nova-compute launches VMs, all of cloud-init runs fail with timeout
> when reading metadata

Hi Gustavo,

We had seen a similar slowdown with DVR and the ARP cache, see:

     https://bugs.launchpad.net/neutron/+bug/1511134
     https://review.openstack.org/#/c/239543/

We decided against that approach in favor of using privsep and the pyroute2 
library.  That adoption has not been as fast as we hoped, so it is probably time 
to re-visit this decision and possibly resurrect that change.

-Brian


> To workaround this, we've made a systemd unit on which nova-compute is
> dependent; this unit waits for ns-metadata-proxy process to appear, and only
> then nova-compute starts
>
> Curiously, in dvr_local_router.py, in _update_arp_entry function, there is a
> comment saying "# TODO(mrsmith): optimize the calls below for bulk calls"...
>
> By now we have a single virtual router with 170 VMs, but the number of VMs will
> grow, so my questions are
>
>     Should this be issue of concern?
>
>     Is there a better / faster / bulk way to execute those "ip neigh" commands?
>
>     Or simply, metadata proxy should launch before ARP cache population?
>
>
>
>
> PD: I've also seen (obviously) this ARP cache population in the L3 agent of
> Neutron Nodes, and I hope it does not affect / delay the HA failover
> mechanism... (didn't test yet)
>
>
>
>
> # journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-rootwrap
> /etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;
> PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap
> /etc/neutron/rootwrap.conf,,g' | head -25
>
> Dec 13 13:33:43 e71-host15 sudo[20157]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy
> --pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c-bf37-7d1827181228.pid
> --metadata_proxy_socket=/var/
> Dec 13 13:33:55 e71-host15 sudo[20309]:   ip -o netns list
> Dec 13 13:33:55 e71-host15 sudo[20315]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w net.ipv4.ip_forward=1
> Dec 13 13:33:55 e71-host15 sudo[20322]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
> net.ipv6.conf.all.forwarding=1
> Dec 13 13:33:56 e71-host15 sudo[20331]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show rfp-6149559f-f
> Dec 13 13:33:56 e71-host15 sudo[20336]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:33:56 e71-host15 sudo[20342]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:33:56 e71-host15 sudo[20345]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip addr show qr-24f3070a-d4 permanent
> Dec 13 13:33:56 e71-host15 sudo[20348]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route list dev qr-24f3070a-d4
> scope link
> Dec 13 13:33:56 e71-host15 sudo[20354]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -6 route list dev qr-24f3070a-d4
> scope link
> Dec 13 13:33:56 e71-host15 sudo[20357]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 arping -A -I qr-24f3070a-d4 -c 3 -w
> 4.5 10.96.0.1
> Dec 13 13:33:57 e71-host15 sudo[20368]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:33:57 e71-host15 sudo[20372]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.100
> lladdr fa:16:3e:1b:d6:cd nud permanent dev qr-24f3070a-d4
> Dec 13 13:33:57 e71-host15 sudo[20375]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:33:57 e71-host15 sudo[20378]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.101
> lladdr fa:16:3e:b4:12:28 nud permanent dev qr-24f3070a-d4
> Dec 13 13:33:58 e71-host15 sudo[20384]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:33:58 e71-host15 sudo[20387]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.102
> lladdr fa:16:3e:3f:bb:58 nud permanent dev qr-24f3070a-d4
> Dec 13 13:33:58 e71-host15 sudo[20390]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:33:58 e71-host15 sudo[20393]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.103
> lladdr fa:16:3e:5a:90:67 nud permanent dev qr-24f3070a-d4
> Dec 13 13:33:58 e71-host15 sudo[20399]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:33:58 e71-host15 sudo[20402]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.104
> lladdr fa:16:3e:ba:fc:f3 nud permanent dev qr-24f3070a-d4
> Dec 13 13:33:58 e71-host15 sudo[20405]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:33:59 e71-host15 sudo[20411]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.105
> lladdr fa:16:3e:0a:16:d1 nud permanent dev qr-24f3070a-d4
> ...
> ...
> ...
> # journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-rootwrap
> /etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;
> PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap
> /etc/neutron/rootwrap.conf,,g' | tail -25
>
> Dec 13 13:34:51 e71-host15 sudo[21771]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.1.83
> lladdr fa:16:3e:8d:d2:7d nud permanent dev qr-24f3070a-d4
> Dec 13 13:34:51 e71-host15 sudo[21777]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default via
> 10.96.0.2 dev qr-24f3070a-d4 table 174063617
> Dec 13 13:34:51 e71-host15 sudo[21780]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show
> Dec 13 13:34:52 e71-host15 sudo[21783]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
> net.ipv4.conf.qr-24f3070a-d4.send_redirects=0
> Dec 13 13:34:52 e71-host15 sudo[21786]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net -maxdepth 1
> -type l -printf %f
> Dec 13 13:34:52 e71-host15 sudo[21789]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
> net.ipv4.conf.all.send_redirects=0
> Dec 13 13:34:52 e71-host15 sudo[21792]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default via
> 10.96.0.2 dev qr-24f3070a-d4 table 174063617
> Dec 13 13:34:52 e71-host15 sudo[21795]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show
> Dec 13 13:34:52 e71-host15 sudo[21801]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
> net.ipv4.conf.qr-24f3070a-d4.send_redirects=0
> Dec 13 13:34:53 e71-host15 sudo[21804]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
> Dec 13 13:34:53 e71-host15 sudo[21807]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.2
> lladdr fa:16:3e:d9:34:cb nud permanent dev qr-24f3070a-d4
> Dec 13 13:34:53 e71-host15 sudo[21817]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net -maxdepth 1
> -type l -printf %f
> Dec 13 13:34:53 e71-host15 sudo[21822]:   ip netns exec
> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
> Dec 13 13:34:53 e71-host15 sudo[21827]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
> Dec 13 13:34:53 e71-host15 sudo[21833]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
> Dec 13 13:34:54 e71-host15 sudo[21836]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
> Dec 13 13:34:54 e71-host15 sudo[21839]:   ip netns exec
> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
> Dec 13 13:34:54 e71-host15 sudo[21844]:   ip netns exec
> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
> Dec 13 13:34:54 e71-host15 sudo[21849]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
> Dec 13 13:34:54 e71-host15 sudo[21852]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
> Dec 13 13:34:54 e71-host15 sudo[21855]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
> Dec 13 13:34:54 e71-host15 sudo[21861]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
> Dec 13 13:34:55 e71-host15 sudo[21864]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
> Dec 13 13:34:55 e71-host15 sudo[21869]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
> Dec 13 13:34:55 e71-host15 sudo[21877]:   ip netns exec
> qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy
> --pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c-bf37-7d1827181228.pid
> --metadata_proxy_socket=/var/lib/neutron/metadata_proxy
> --router_id=6149559f-fa54-493c-bf37-7d1827181228 --state_path=/var/lib/neutron
> --metadata_port=9697 --metadata_proxy_user=118 --metadata_proxy_group=120
> --debug --verbose --use-syslog --syslog-log-facility=LOG_LOCAL4
>
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>