Open Stack

Hi Openstackers,

We have the folowing issue (using Mitaka / DVR / Xenial), perhaps someone
can help ;)

When our hosts boots up, the ARP cache population loop of L3 Agent is
delaying the start of neutron-ns-metadata-proxy for around a minute -- see
logs below; then, when nova-compute launches VMs, all of cloud-init runs
fail with timeout when reading metadata

To workaround this, we've made a systemd unit on which nova-compute is
dependent; this unit waits for ns-metadata-proxy process to appear, and
only then nova-compute starts

Curiously, in dvr_local_router.py, in _update_arp_entry function, there is
a comment saying "# TODO(mrsmith): optimize the calls below for bulk
calls"...

By now we have a single virtual router with 170 VMs, but the number of VMs
will grow, so my questions are

Should this be issue of concern?

Is there a better / faster / bulk way to execute those "ip neigh" commands?

Or simply, metadata proxy should launch before ARP cache population?




PD: I've also seen (obviously) this ARP cache population in the L3 agent of
Neutron Nodes, and I hope it does not affect / delay the HA failover
mechanism... (didn't test yet)




# journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;
PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf,,g' | head -25

Dec 13 13:33:43 e71-host15 sudo[20157]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c-bf37-7d1827181228.pid
--metadata_proxy_socket=/var/
Dec 13 13:33:55 e71-host15 sudo[20309]:   ip -o netns list
Dec 13 13:33:55 e71-host15 sudo[20315]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w net.ipv4.ip_forward=1
Dec 13 13:33:55 e71-host15 sudo[20322]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
net.ipv6.conf.all.forwarding=1
Dec 13 13:33:56 e71-host15 sudo[20331]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show rfp-6149559f-f
Dec 13 13:33:56 e71-host15 sudo[20336]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:56 e71-host15 sudo[20342]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:56 e71-host15 sudo[20345]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip addr show qr-24f3070a-d4
permanent
Dec 13 13:33:56 e71-host15 sudo[20348]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route list dev
qr-24f3070a-d4 scope link
Dec 13 13:33:56 e71-host15 sudo[20354]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -6 route list dev
qr-24f3070a-d4 scope link
Dec 13 13:33:56 e71-host15 sudo[20357]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 arping -A -I qr-24f3070a-d4 -c
3 -w 4.5 10.96.0.1
Dec 13 13:33:57 e71-host15 sudo[20368]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:57 e71-host15 sudo[20372]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
10.96.0.100 lladdr fa:16:3e:1b:d6:cd nud permanent dev qr-24f3070a-d4
Dec 13 13:33:57 e71-host15 sudo[20375]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:57 e71-host15 sudo[20378]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
10.96.0.101 lladdr fa:16:3e:b4:12:28 nud permanent dev qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20384]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20387]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
10.96.0.102 lladdr fa:16:3e:3f:bb:58 nud permanent dev qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20390]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20393]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
10.96.0.103 lladdr fa:16:3e:5a:90:67 nud permanent dev qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20399]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20402]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
10.96.0.104 lladdr fa:16:3e:ba:fc:f3 nud permanent dev qr-24f3070a-d4
Dec 13 13:33:58 e71-host15 sudo[20405]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:33:59 e71-host15 sudo[20411]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
10.96.0.105 lladdr fa:16:3e:0a:16:d1 nud permanent dev qr-24f3070a-d4
...
...
...
# journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;
PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap
/etc/neutron/rootwrap.conf,,g' | tail -25

Dec 13 13:34:51 e71-host15 sudo[21771]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.1.83
lladdr fa:16:3e:8d:d2:7d nud permanent dev qr-24f3070a-d4
Dec 13 13:34:51 e71-host15 sudo[21777]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default
via 10.96.0.2 dev qr-24f3070a-d4 table 174063617
Dec 13 13:34:51 e71-host15 sudo[21780]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show
Dec 13 13:34:52 e71-host15 sudo[21783]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
net.ipv4.conf.qr-24f3070a-d4.send_redirects=0
Dec 13 13:34:52 e71-host15 sudo[21786]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net -maxdepth
1 -type l -printf %f
Dec 13 13:34:52 e71-host15 sudo[21789]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
net.ipv4.conf.all.send_redirects=0
Dec 13 13:34:52 e71-host15 sudo[21792]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default
via 10.96.0.2 dev qr-24f3070a-d4 table 174063617
Dec 13 13:34:52 e71-host15 sudo[21795]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show
Dec 13 13:34:52 e71-host15 sudo[21801]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
net.ipv4.conf.qr-24f3070a-d4.send_redirects=0
Dec 13 13:34:53 e71-host15 sudo[21804]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show qr-24f3070a-d4
Dec 13 13:34:53 e71-host15 sudo[21807]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace 10.96.0.2
lladdr fa:16:3e:d9:34:cb nud permanent dev qr-24f3070a-d4
Dec 13 13:34:53 e71-host15 sudo[21817]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net -maxdepth
1 -type l -printf %f
Dec 13 13:34:53 e71-host15 sudo[21822]:   ip netns exec
fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
Dec 13 13:34:53 e71-host15 sudo[21827]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
Dec 13 13:34:53 e71-host15 sudo[21833]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
Dec 13 13:34:54 e71-host15 sudo[21836]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
Dec 13 13:34:54 e71-host15 sudo[21839]:   ip netns exec
fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
Dec 13 13:34:54 e71-host15 sudo[21844]:   ip netns exec
fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
Dec 13 13:34:54 e71-host15 sudo[21849]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
Dec 13 13:34:54 e71-host15 sudo[21852]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
Dec 13 13:34:54 e71-host15 sudo[21855]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
Dec 13 13:34:54 e71-host15 sudo[21861]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
Dec 13 13:34:55 e71-host15 sudo[21864]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
Dec 13 13:34:55 e71-host15 sudo[21869]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
Dec 13 13:34:55 e71-host15 sudo[21877]:   ip netns exec
qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c-bf37-7d1827181228.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--router_id=6149559f-fa54-493c-bf37-7d1827181228
--state_path=/var/lib/neutron --metadata_port=9697
--metadata_proxy_user=118 --metadata_proxy_group=120 --debug --verbose
--use-syslog --syslog-log-facility=LOG_LOCAL4
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161213/cb7ea6ad/attachment.html>