<font size=2 face=" Arial">There is a ZK-backed driver in Nova service
heartbeat mechanism (</font><a href="https://blueprints.launchpad.net/nova/+spec/zk-service-heartbeat"><font size=2 color=blue face=" Arial"><u>https://blueprints.launchpad.net/nova/+spec/zk-service-heartbeat</u></font></a><font size=2 face=" Arial">)
-- would be interesting to know whether it is widely used (might be worth
asking at the general ML, or user groups). There have been also discussions
on using it for other purposes (some listed towards the bottom at </font><a href=https://wiki.openstack.org/wiki/NovaZooKeeperHeartbeat><font size=2 color=blue face=" Arial"><u>https://wiki.openstack.org/wiki/NovaZooKeeperHeartbeat</u></font></a><font size=2 face=" Arial">).
While I am not aware of any particular progress with implementing any of
them, I think they still make sense and could be useful.</font>
<br>
<br><font size=2 face=" Arial">Regards,</font>
<br><font size=2 face=" Arial">Alex</font>
<br>
<br>
<br>
<br>
<br><font size=1 color=#5f5f5f face="sans-serif">From:
</font><font size=1 face="sans-serif">Clint Byrum <clint@fewbar.com></font>
<br><font size=1 color=#5f5f5f face="sans-serif">To:
</font><font size=1 face="sans-serif">openstack-dev <openstack-dev@lists.openstack.org>,
</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Date:
</font><font size=1 face="sans-serif">30/10/2013 07:45 PM</font>
<br><font size=1 color=#5f5f5f face="sans-serif">Subject:
</font><font size=1 face="sans-serif">[openstack-dev]
[Heat] Locking and ZooKeeper - a space oddysey</font>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2>So, recently we've had quite a long thread in gerrit
regarding locking<br>
in Heat:<br>
<br>
</font></tt><a href=https://review.openstack.org/#/c/49440/><tt><font size=2>https://review.openstack.org/#/c/49440/</font></tt></a><tt><font size=2><br>
<br>
In the patch, there are two distributed lock drivers. One uses SQL,<br>
and suffers from all the problems you might imagine a SQL based locking<br>
system would. It is extremely hard to detect dead lock holders, so we<br>
end up with really long timeouts. The other is ZooKeeper.<br>
<br>
I'm on record as saying we're not using ZooKeeper. It is a little<br>
embarrassing to have taken such a position without really thinking things<br>
through. The main reason I feel this way though, is not because ZooKeeper<br>
wouldn't work for locking, but because I think locking is a mistake.<br>
<br>
The current multi-engine paradigm has a race condition. If you have a<br>
stack action going on, the state is held in the engine itself, and not<br>
in the database, so if another engine starts working on another action,<br>
they will conflict.<br>
<br>
The locking paradigm is meant to prevent this. But I think this is a<br>
huge mistake.<br>
<br>
The engine should store _all_ of its state in a distributed data store<br>
of some kind. Any engine should be aware of what is already happening<br>
with the stack from this state and act accordingly. That includes the<br>
engine currently working on actions. When viewed through this lense,<br>
to me, locking is a poor excuse for serializing the state of the engine<br>
scheduler.<br>
<br>
It feels like TaskFlow is the answer, with an eye for making sure<br>
TaskFlow can be made to work with distributed state. I am not well<br>
versed on TaskFlow's details though, so I may be wrong. It worries me<br>
that TaskFlow has existed a while and doesn't seem to be solving real<br>
problems, but maybe I'm wrong and it is actually in use already.<br>
<br>
Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper<br>
has some real advantages over using the database. But there is hesitance<br>
because it is not widely supported in OpenStack. What say you, OpenStack<br>
community? Should we keep ZooKeeper out of our.. zoo?<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.org<br>
</font></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><tt><font size=2>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></tt></a><tt><font size=2><br>
<br>
</font></tt>
<br>