<p dir="ltr">Is there a way to parallelize the period tasks? I wanted to go this route because I encountered cases where a bunch of routers would get scheduled to l3 agents and they would all hit the server nearly simultaneously with a sync routers task. </p>
<p dir="ltr">This could result in thousands of routers and their floating IPs being retrieved, which would result in tens of thousands of SQL queries. During this time, the agents would time out and have all their routers rescheduled, leading to a downward spiral of doom. </p>
<p dir="ltr">I spent a bunch of time optimizing the sync routers calls on the l3 side so it's hard to trigger this now, but I would be more comfortable if we didn't depend on sync routers taking less time than the agent down time. </p>
<p dir="ltr">If we can have the heartbeats always running, it should solve both issues. </p>
<div class="gmail_quote">On Jun 4, 2015 8:56 AM, "Salvatore Orlando" <<a href="mailto:sorlando@nicira.com">sorlando@nicira.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">One reason for not sending the heartbeat from a separate greenthread could be that the agent is already doing it [1].<div>The current proposed patch addresses the issue blindly - that is to say before declaring an agent dead let's wait for some more time because it could be stuck doing stuff. In that case I would probably make the multiplier (currently 2x) configurable.</div><div><br></div><div>The reason for which state report does not occur is probably that both it and the resync procedure are periodic tasks. If I got it right they're both executed as eventlet greenthreads but one at a time. Perhaps then adding an initial delay to the full sync task might ensure the first thing an agent does when it comes up is sending a heartbeat to the server?</div><div><br></div><div>On the other hand, while doing the initial full resync, is the agent able to process updates? If not perhaps it makes sense to have it down until it finishes synchronisation.</div><div><div><div><br></div><div>Salvatore</div><div><br></div><div>[1] <a href="http://git.openstack.org/cgit/openstack/neutron/tree/neutron/agent/l3/agent.py#n587" target="_blank">http://git.openstack.org/cgit/openstack/neutron/tree/neutron/agent/l3/agent.py#n587</a></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 4 June 2015 at 16:16, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Why don't we put the agent heartbeat into a separate greenthread on the agent so it continues to send updates even when it's busy processing changes?</p>
<div class="gmail_quote"><div><div>On Jun 4, 2015 2:56 AM, "Anna Kamyshnikova" <<a href="mailto:akamyshnikova@mirantis.com" target="_blank">akamyshnikova@mirantis.com</a>> wrote:<br type="attribution"></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr">Hi, neutrons!<div><br></div><div>Some time ago I discovered a bug for l3 agent rescheduling [1]. When there are a lot of resources and agent_down_time is not big enough neutron-server starts marking l3 agents as dead. The same issue has been discovered and fixed for DHCP-agents. I proposed a change similar to those that were done for DHCP-agents. [2]</div><div><br></div><div>There is no unified opinion on this bug and proposed change, so I want to ask developers whether it worth to continue work on this patch or not.</div><div><br></div><div>[1] - <a href="https://bugs.launchpad.net/neutron/+bug/1440761" target="_blank">https://bugs.launchpad.net/neutron/+bug/1440761</a> <br clear="all"><div>[2] - <a href="https://review.openstack.org/171592" target="_blank">https://review.openstack.org/171592</a></div><div><br></div>-- <br><div><div dir="ltr">Regards,<div>Ann Kamyshnikova</div><div>Mirantis, Inc</div></div></div>
</div></div>
<br></div></div>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div>