<div dir="ltr">><span style="font-size:12.8000001907349px">Timestamps are just one way (and likely the most primitive), using redis (or memcache) key/value and expiry are another (and letting memcache or redis expire using its own internal algorithms), using zookeeper ephemeral nodes[1] are another... The point being that its backend specific and tooz supports varying backends.</span><div><span style="font-size:12.8000001907349px"><br></span></div><div><span style="font-size:12.8000001907349px">Very cool. Is the backend completely transparent so a deployer could choose a service they are comfortable maintaining, or will that change the properties WRT to resiliency of state on node restarts, partitions, etc?</span></div><div><span style="font-size:12.8000001907349px"><br></span></div><div>The Nova implementation of Tooz seemed pretty straight-forward, although it looked like it had pluggable drivers for service management already. Before I dig into it much further I'll file a spec on the Neutron side to see if I can get some other cores onboard to do the review work if I push a change to tooz.</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Apr 12, 2015 at 9:38 AM, Joshua Harlow <span dir="ltr"><<a href="mailto:harlowja@outlook.com" target="_blank">harlowja@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">Kevin Benton wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
So IIUC tooz would be handling the liveness detection for the agents.<br>
That would be nice to get ride of that logic in Neutron and just<br>
register callbacks for rescheduling the dead.<br>
<br>
Where does it store that state, does it persist timestamps to the DB<br>
like Neutron does? If so, how would that scale better? If not, who does<br>
a given node ask to know if an agent is online or offline when making a<br>
scheduling decision?<br>
</blockquote>
<br></span>
Timestamps are just one way (and likely the most primitive), using redis (or memcache) key/value and expiry are another (and letting memcache or redis expire using its own internal algorithms), using zookeeper ephemeral nodes[1] are another... The point being that its backend specific and tooz supports varying backends.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
However, before (what I assume is) the large code change to implement<br>
tooz, I would like to quantify that the heartbeats are actually a<br>
bottleneck. When I was doing some profiling of them on the master branch<br>
a few months ago, processing a heartbeat took an order of magnitude less<br>
time (<50ms) than the 'sync routers' task of the l3 agent (~300ms). A<br>
few query optimizations might buy us a lot more headroom before we have<br>
to fall back to large refactors.<br>
</blockquote>
<br></span>
Sure, always good to avoid prematurely optimizing things...<br>
<br>
Although this is relevant for u I think anyway:<br>
<br>
<a href="https://review.openstack.org/#/c/138607/" target="_blank">https://review.openstack.org/#<u></u>/c/138607/</a> (same thing/nearly same in nova)...<br>
<br>
<a href="https://review.openstack.org/#/c/172502/" target="_blank">https://review.openstack.org/#<u></u>/c/172502/</a> (a WIP implementation of the latter).<br>
<br>
[1] <a href="https://zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html#Ephemeral+Nodes" target="_blank">https://zookeeper.apache.org/<u></u>doc/trunk/<u></u>zookeeperProgrammers.html#<u></u>Ephemeral+Nodes</a><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
Kevin Benton wrote:<br>
<br>
<br>
    One of the most common is the heartbeat from each agent. However, I<br>
    don't think we can't eliminate them because they are used to determine<br>
    if the agents are still alive for scheduling purposes. Did you have<br>
    something else in mind to determine if an agent is alive?<br>
<br>
<br>
Put each agent in a tooz[1] group; have each agent periodically<br>
heartbeat[2], have whoever needs to schedule read the active members of<br>
that group (or use [3] to get notified via a callback), profit...<br>
<br>
Pick from your favorite (supporting) driver at:<br>
<br>
</span><a href="http://docs.openstack.org/__developer/tooz/compatibility.__html" target="_blank">http://docs.openstack.org/__<u></u>developer/tooz/compatibility._<u></u>_html</a><br>
<<a href="http://docs.openstack.org/developer/tooz/compatibility.html" target="_blank">http://docs.openstack.org/<u></u>developer/tooz/compatibility.<u></u>html</a>><br>
<br>
[1]<br>
<a href="http://docs.openstack.org/__developer/tooz/compatibility.__html#grouping" target="_blank">http://docs.openstack.org/__<u></u>developer/tooz/compatibility._<u></u>_html#grouping</a><br>
<<a href="http://docs.openstack.org/developer/tooz/compatibility.html#grouping" target="_blank">http://docs.openstack.org/<u></u>developer/tooz/compatibility.<u></u>html#grouping</a>><br>
[2]<br>
<a href="https://github.com/openstack/__tooz/blob/0.13.1/tooz/__coordination.py#L315" target="_blank">https://github.com/openstack/_<u></u>_tooz/blob/0.13.1/tooz/__<u></u>coordination.py#L315</a><br>
<<a href="https://github.com/openstack/tooz/blob/0.13.1/tooz/coordination.py#L315" target="_blank">https://github.com/openstack/<u></u>tooz/blob/0.13.1/tooz/<u></u>coordination.py#L315</a>><br>
[3]<br>
<a href="http://docs.openstack.org/__developer/tooz/tutorial/group___membership.html#watching-__group-changes" target="_blank">http://docs.openstack.org/__<u></u>developer/tooz/tutorial/group_<u></u>__membership.html#watching-__<u></u>group-changes</a><br>
<<a href="http://docs.openstack.org/developer/tooz/tutorial/group_membership.html#watching-group-changes" target="_blank">http://docs.openstack.org/<u></u>developer/tooz/tutorial/group_<u></u>membership.html#watching-<u></u>group-changes</a>><br>
<br>
<br>
______________________________<u></u>______________________________<u></u>__________________<span class=""><br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br></span>
OpenStack-dev-request@lists.__<a href="http://openstack.org?subject:__unsubscribe" target="_blank"><u></u>openstack.org?subject:__<u></u>unsubscribe</a><br>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@<u></u>lists.openstack.org?subject:<u></u>unsubscribe</a>><br>
<a href="http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev" target="_blank">http://lists.openstack.org/__<u></u>cgi-bin/mailman/listinfo/__<u></u>openstack-dev</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a>><span class=""><br>
<br>
______________________________<u></u>______________________________<u></u>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.<u></u>openstack.org?subject:<u></u>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</span></blockquote><div class="HOEnZb"><div class="h5">
<br>
______________________________<u></u>______________________________<u></u>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.<u></u>openstack.org?subject:<u></u>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div>Kevin Benton</div></div>
</div>