Open Stack

Thanks Brian for the link, it's exactly what I was looking for.



On Thu, Dec 15, 2016 at 5:29 PM, Brian Haley <brian.haley at hpe.com> wrote:

> On 12/13/2016 02:45 PM, Gustavo Randich wrote:
>
>> Hi Openstackers,
>>
>> We have the folowing issue (using Mitaka / DVR / Xenial), perhaps someone
>> can
>> help ;)
>>
>> When our hosts boots up, the ARP cache population loop of L3 Agent is
>> delaying
>> the start of neutron-ns-metadata-proxy for around a minute -- see logs
>> below;
>> then, when nova-compute launches VMs, all of cloud-init runs fail with
>> timeout
>> when reading metadata
>>
>
> Hi Gustavo,
>
> We had seen a similar slowdown with DVR and the ARP cache, see:
>
>     https://bugs.launchpad.net/neutron/+bug/1511134
>     https://review.openstack.org/#/c/239543/
>
> We decided against that approach in favor of using privsep and the
> pyroute2 library.  That adoption has not been as fast as we hoped, so it is
> probably time to re-visit this decision and possibly resurrect that change.
>
> -Brian
>
>
> To workaround this, we've made a systemd unit on which nova-compute is
>> dependent; this unit waits for ns-metadata-proxy process to appear, and
>> only
>> then nova-compute starts
>>
>> Curiously, in dvr_local_router.py, in _update_arp_entry function, there
>> is a
>> comment saying "# TODO(mrsmith): optimize the calls below for bulk
>> calls"...
>>
>> By now we have a single virtual router with 170 VMs, but the number of
>> VMs will
>> grow, so my questions are
>>
>>     Should this be issue of concern?
>>
>>     Is there a better / faster / bulk way to execute those "ip neigh"
>> commands?
>>
>>     Or simply, metadata proxy should launch before ARP cache population?
>>
>>
>>
>>
>> PD: I've also seen (obviously) this ARP cache population in the L3 agent
>> of
>> Neutron Nodes, and I hope it does not affect / delay the HA failover
>> mechanism... (didn't test yet)
>>
>>
>>
>>
>> # journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-root
>> wrap
>> /etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;
>> PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap
>> /etc/neutron/rootwrap.conf,,g' | head -25
>>
>> Dec 13 13:33:43 e71-host15 sudo[20157]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy
>> --pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c
>> -bf37-7d1827181228.pid
>> --metadata_proxy_socket=/var/
>> Dec 13 13:33:55 e71-host15 sudo[20309]:   ip -o netns list
>> Dec 13 13:33:55 e71-host15 sudo[20315]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
>> net.ipv4.ip_forward=1
>> Dec 13 13:33:55 e71-host15 sudo[20322]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
>> net.ipv6.conf.all.forwarding=1
>> Dec 13 13:33:56 e71-host15 sudo[20331]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> rfp-6149559f-f
>> Dec 13 13:33:56 e71-host15 sudo[20336]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:33:56 e71-host15 sudo[20342]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:33:56 e71-host15 sudo[20345]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip addr show qr-24f3070a-d4
>> permanent
>> Dec 13 13:33:56 e71-host15 sudo[20348]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route list dev
>> qr-24f3070a-d4
>> scope link
>> Dec 13 13:33:56 e71-host15 sudo[20354]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -6 route list dev
>> qr-24f3070a-d4
>> scope link
>> Dec 13 13:33:56 e71-host15 sudo[20357]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 arping -A -I qr-24f3070a-d4
>> -c 3 -w
>> 4.5 10.96.0.1
>> Dec 13 13:33:57 e71-host15 sudo[20368]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:33:57 e71-host15 sudo[20372]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
>> 10.96.0.100
>> lladdr fa:16:3e:1b:d6:cd nud permanent dev qr-24f3070a-d4
>> Dec 13 13:33:57 e71-host15 sudo[20375]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:33:57 e71-host15 sudo[20378]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
>> 10.96.0.101
>> lladdr fa:16:3e:b4:12:28 nud permanent dev qr-24f3070a-d4
>> Dec 13 13:33:58 e71-host15 sudo[20384]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:33:58 e71-host15 sudo[20387]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
>> 10.96.0.102
>> lladdr fa:16:3e:3f:bb:58 nud permanent dev qr-24f3070a-d4
>> Dec 13 13:33:58 e71-host15 sudo[20390]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:33:58 e71-host15 sudo[20393]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
>> 10.96.0.103
>> lladdr fa:16:3e:5a:90:67 nud permanent dev qr-24f3070a-d4
>> Dec 13 13:33:58 e71-host15 sudo[20399]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:33:58 e71-host15 sudo[20402]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
>> 10.96.0.104
>> lladdr fa:16:3e:ba:fc:f3 nud permanent dev qr-24f3070a-d4
>> Dec 13 13:33:58 e71-host15 sudo[20405]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:33:59 e71-host15 sudo[20411]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
>> 10.96.0.105
>> lladdr fa:16:3e:0a:16:d1 nud permanent dev qr-24f3070a-d4
>> ...
>> ...
>> ...
>> # journalctl -u neutron-l3-agent | grep "COMMAND=/usr/bin/neutron-root
>> wrap
>> /etc/neutron/rootwrap.conf" | sed 's,neutron : TTY=unknown ;
>> PWD=/var/lib/neutron ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap
>> /etc/neutron/rootwrap.conf,,g' | tail -25
>>
>> Dec 13 13:34:51 e71-host15 sudo[21771]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
>> 10.96.1.83
>> lladdr fa:16:3e:8d:d2:7d nud permanent dev qr-24f3070a-d4
>> Dec 13 13:34:51 e71-host15 sudo[21777]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default
>> via
>> 10.96.0.2 dev qr-24f3070a-d4 table 174063617
>> Dec 13 13:34:51 e71-host15 sudo[21780]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show
>> Dec 13 13:34:52 e71-host15 sudo[21783]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
>> net.ipv4.conf.qr-24f3070a-d4.send_redirects=0
>> Dec 13 13:34:52 e71-host15 sudo[21786]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net
>> -maxdepth 1
>> -type l -printf %f
>> Dec 13 13:34:52 e71-host15 sudo[21789]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
>> net.ipv4.conf.all.send_redirects=0
>> Dec 13 13:34:52 e71-host15 sudo[21792]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 route replace default
>> via
>> 10.96.0.2 dev qr-24f3070a-d4 table 174063617
>> Dec 13 13:34:52 e71-host15 sudo[21795]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 rule show
>> Dec 13 13:34:52 e71-host15 sudo[21801]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 sysctl -w
>> net.ipv4.conf.qr-24f3070a-d4.send_redirects=0
>> Dec 13 13:34:53 e71-host15 sudo[21804]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -o link show
>> qr-24f3070a-d4
>> Dec 13 13:34:53 e71-host15 sudo[21807]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip -4 neigh replace
>> 10.96.0.2
>> lladdr fa:16:3e:d9:34:cb nud permanent dev qr-24f3070a-d4
>> Dec 13 13:34:53 e71-host15 sudo[21817]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 find /sys/class/net
>> -maxdepth 1
>> -type l -printf %f
>> Dec 13 13:34:53 e71-host15 sudo[21822]:   ip netns exec
>> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
>> Dec 13 13:34:53 e71-host15 sudo[21827]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
>> Dec 13 13:34:53 e71-host15 sudo[21833]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
>> Dec 13 13:34:54 e71-host15 sudo[21836]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
>> Dec 13 13:34:54 e71-host15 sudo[21839]:   ip netns exec
>> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
>> Dec 13 13:34:54 e71-host15 sudo[21844]:   ip netns exec
>> fip-8dd05891-6545-41df-9e89-7eb099c80393 ip -o link show fpr-6149559f-f
>> Dec 13 13:34:54 e71-host15 sudo[21849]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
>> Dec 13 13:34:54 e71-host15 sudo[21852]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
>> Dec 13 13:34:54 e71-host15 sudo[21855]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
>> Dec 13 13:34:54 e71-host15 sudo[21861]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-save
>> Dec 13 13:34:55 e71-host15 sudo[21864]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 iptables-restore -n
>> Dec 13 13:34:55 e71-host15 sudo[21869]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 ip6tables-save
>> Dec 13 13:34:55 e71-host15 sudo[21877]:   ip netns exec
>> qrouter-6149559f-fa54-493c-bf37-7d1827181228 neutron-ns-metadata-proxy
>> --pid_file=/var/lib/neutron/external/pids/6149559f-fa54-493c
>> -bf37-7d1827181228.pid
>> --metadata_proxy_socket=/var/lib/neutron/metadata_proxy
>> --router_id=6149559f-fa54-493c-bf37-7d1827181228
>> --state_path=/var/lib/neutron
>> --metadata_port=9697 --metadata_proxy_user=118 --metadata_proxy_group=120
>> --debug --verbose --use-syslog --syslog-log-facility=LOG_LOCAL4
>>
>>
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : http://lists.openstack.org/cgi
>> -bin/mailman/listinfo/openstack
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161215/281603e8/attachment.html>