<div dir="ltr">Thanks Brian for the link, it's exactly what I was looking for.<div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Dec 15, 2016 at 5:29 PM, Brian Haley <span dir="ltr"><<a href="mailto:brian.haley@hpe.com" target="_blank">brian.haley@hpe.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 12/13/2016 02:45 PM, Gustavo Randich wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Openstackers,<br>
<br>
We have the folowing issue (using Mitaka / DVR / Xenial), perhaps someone can<br>
help ;)<br>
<br>
When our hosts boots up, the ARP cache population loop of L3 Agent is delaying<br>
the start of neutron-ns-metadata-proxy for around a minute -- see logs below;<br>
then, when nova-compute launches VMs, all of cloud-init runs fail with timeout<br>
when reading metadata<br>
</blockquote>
<br></span>
Hi Gustavo,<br>
<br>
We had seen a similar slowdown with DVR and the ARP cache, see:<br>
<br>
<a href="https://bugs.launchpad.net/neutron/+bug/1511134" rel="noreferrer" target="_blank">https://bugs.launchpad.net/neu<wbr>tron/+bug/1511134</a><br>
<a href="https://review.openstack.org/#/c/239543/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/239543/</a><br>
<br>
We decided against that approach in favor of using privsep and the pyroute2 library. That adoption has not been as fast as we hoped, so it is probably time to re-visit this decision and possibly resurrect that change.<br>
<br>
-Brian<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
To workaround this, we've made a systemd unit on which nova-compute is<br>
dependent; this unit waits for ns-metadata-proxy process to appear, and only<br>
then nova-compute starts<br>
<br>
Curiously, in dvr_local_router.py, in _update_arp_entry function, there is a<br>
comment saying "# TODO(mrsmith): optimize the calls below for bulk calls"...<br>
<br>
By now we have a single virtual router with 170 VMs, but the number of VMs will<br>
grow, so my questions are<br>
<br>
Should this be issue of concern?<br>
<br>
Is there a better / faster / bulk way to execute those "ip neigh" commands?<br>
<br>
Or simply, metadata proxy should launch before ARP cache population?<br>
<br>
<br>
<br>
<br>
PD: I've also seen (obviously) this ARP cache population in the L3 agent of<br>
Neutron Nodes, and I hope it does not affect / delay the HA failover<br>
mechanism... (didn't test yet)<br>
<br>
<br>
<br>
<br>
# journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-root<wbr>wrap<br>
/etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;<br>
PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootw<wbr>rap<br>
/etc/neutron/rootwrap.conf,,g' | head -25<br>
<br>
Dec 13 13:33:43 e71-host15 sudo[20157]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 neutron-ns-metadata-proxy<br>
--pid_file=/var/lib/neutron/ex<wbr>ternal/pids/6149559f-fa54-493c<wbr>-bf37-7d1827181228.pid<br>
--metadata_proxy_socket=/var/<br>
Dec 13 13:33:55 e71-host15 sudo[20309]: ip -o netns list<br>
Dec 13 13:33:55 e71-host15 sudo[20315]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 sysctl -w net.ipv4.ip_forward=1<br>
Dec 13 13:33:55 e71-host15 sudo[20322]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 sysctl -w<br>
net.ipv6.conf.all.forwarding=1<br>
Dec 13 13:33:56 e71-host15 sudo[20331]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show rfp-6149559f-f<br>
Dec 13 13:33:56 e71-host15 sudo[20336]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:33:56 e71-host15 sudo[20342]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:33:56 e71-host15 sudo[20345]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip addr show qr-24f3070a-d4 permanent<br>
Dec 13 13:33:56 e71-host15 sudo[20348]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 route list dev qr-24f3070a-d4<br>
scope link<br>
Dec 13 13:33:56 e71-host15 sudo[20354]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -6 route list dev qr-24f3070a-d4<br>
scope link<br>
Dec 13 13:33:56 e71-host15 sudo[20357]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 arping -A -I qr-24f3070a-d4 -c 3 -w<br>
4.5 10.96.0.1<br>
Dec 13 13:33:57 e71-host15 sudo[20368]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:33:57 e71-host15 sudo[20372]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 neigh replace 10.96.0.100<br>
lladdr fa:16:3e:1b:d6:cd nud permanent dev qr-24f3070a-d4<br>
Dec 13 13:33:57 e71-host15 sudo[20375]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:33:57 e71-host15 sudo[20378]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 neigh replace 10.96.0.101<br>
lladdr fa:16:3e:b4:12:28 nud permanent dev qr-24f3070a-d4<br>
Dec 13 13:33:58 e71-host15 sudo[20384]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:33:58 e71-host15 sudo[20387]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 neigh replace 10.96.0.102<br>
lladdr fa:16:3e:3f:bb:58 nud permanent dev qr-24f3070a-d4<br>
Dec 13 13:33:58 e71-host15 sudo[20390]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:33:58 e71-host15 sudo[20393]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 neigh replace 10.96.0.103<br>
lladdr fa:16:3e:5a:90:67 nud permanent dev qr-24f3070a-d4<br>
Dec 13 13:33:58 e71-host15 sudo[20399]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:33:58 e71-host15 sudo[20402]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 neigh replace 10.96.0.104<br>
lladdr fa:16:3e:ba:fc:f3 nud permanent dev qr-24f3070a-d4<br>
Dec 13 13:33:58 e71-host15 sudo[20405]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:33:59 e71-host15 sudo[20411]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 neigh replace 10.96.0.105<br>
lladdr fa:16:3e:0a:16:d1 nud permanent dev qr-24f3070a-d4<br>
...<br>
...<br>
...<br>
# journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-root<wbr>wrap<br>
/etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;<br>
PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootw<wbr>rap<br>
/etc/neutron/rootwrap.conf,,g' | tail -25<br>
<br>
Dec 13 13:34:51 e71-host15 sudo[21771]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 neigh replace 10.96.1.83<br>
lladdr fa:16:3e:8d:d2:7d nud permanent dev qr-24f3070a-d4<br>
Dec 13 13:34:51 e71-host15 sudo[21777]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 route replace default via<br>
10.96.0.2 dev qr-24f3070a-d4 table 174063617<br>
Dec 13 13:34:51 e71-host15 sudo[21780]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 rule show<br>
Dec 13 13:34:52 e71-host15 sudo[21783]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 sysctl -w<br>
net.ipv4.conf.qr-24f3070a-d4.s<wbr>end_redirects=0<br>
Dec 13 13:34:52 e71-host15 sudo[21786]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 find /sys/class/net -maxdepth 1<br>
-type l -printf %f<br>
Dec 13 13:34:52 e71-host15 sudo[21789]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 sysctl -w<br>
net.ipv4.conf.all.send_redirec<wbr>ts=0<br>
Dec 13 13:34:52 e71-host15 sudo[21792]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 route replace default via<br>
10.96.0.2 dev qr-24f3070a-d4 table 174063617<br>
Dec 13 13:34:52 e71-host15 sudo[21795]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 rule show<br>
Dec 13 13:34:52 e71-host15 sudo[21801]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 sysctl -w<br>
net.ipv4.conf.qr-24f3070a-d4.s<wbr>end_redirects=0<br>
Dec 13 13:34:53 e71-host15 sudo[21804]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -o link show qr-24f3070a-d4<br>
Dec 13 13:34:53 e71-host15 sudo[21807]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip -4 neigh replace 10.96.0.2<br>
lladdr fa:16:3e:d9:34:cb nud permanent dev qr-24f3070a-d4<br>
Dec 13 13:34:53 e71-host15 sudo[21817]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 find /sys/class/net -maxdepth 1<br>
-type l -printf %f<br>
Dec 13 13:34:53 e71-host15 sudo[21822]: ip netns exec<br>
fip-8dd05891-6545-41df-9e89-7e<wbr>b099c80393 ip -o link show fpr-6149559f-f<br>
Dec 13 13:34:53 e71-host15 sudo[21827]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 iptables-save<br>
Dec 13 13:34:53 e71-host15 sudo[21833]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 iptables-restore -n<br>
Dec 13 13:34:54 e71-host15 sudo[21836]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip6tables-save<br>
Dec 13 13:34:54 e71-host15 sudo[21839]: ip netns exec<br>
fip-8dd05891-6545-41df-9e89-7e<wbr>b099c80393 ip -o link show fpr-6149559f-f<br>
Dec 13 13:34:54 e71-host15 sudo[21844]: ip netns exec<br>
fip-8dd05891-6545-41df-9e89-7e<wbr>b099c80393 ip -o link show fpr-6149559f-f<br>
Dec 13 13:34:54 e71-host15 sudo[21849]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 iptables-save<br>
Dec 13 13:34:54 e71-host15 sudo[21852]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 iptables-restore -n<br>
Dec 13 13:34:54 e71-host15 sudo[21855]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip6tables-save<br>
Dec 13 13:34:54 e71-host15 sudo[21861]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 iptables-save<br>
Dec 13 13:34:55 e71-host15 sudo[21864]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 iptables-restore -n<br>
Dec 13 13:34:55 e71-host15 sudo[21869]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 ip6tables-save<br>
Dec 13 13:34:55 e71-host15 sudo[21877]: ip netns exec<br>
qrouter-6149559f-fa54-493c-bf3<wbr>7-7d1827181228 neutron-ns-metadata-proxy<br>
--pid_file=/var/lib/neutron/ex<wbr>ternal/pids/6149559f-fa54-493c<wbr>-bf37-7d1827181228.pid<br>
--metadata_proxy_socket=/var/l<wbr>ib/neutron/metadata_proxy<br>
--router_id=6149559f-fa54-493c<wbr>-bf37-7d1827181228 --state_path=/var/lib/neutron<br>
--metadata_port=9697 --metadata_proxy_user=118 --metadata_proxy_group=120<br>
--debug --verbose --use-syslog --syslog-log-facility=LOG_LOCA<wbr>L4<br>
<br>
<br>
<br></div></div>
______________________________<wbr>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k</a><br>
<br>
</blockquote>
<br>
</blockquote></div><br></div>