<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>To me u just made state consistency be a lock by another name.</div>
<div><br>
</div>
<div>A lock protects a region of code from being mutually accessed, u can do the same with state consistency. In the end though u need some reliable transactional consistent storage (a database, zookeeper, other) to store that state (which itself is using locks).
So to me they are similar, state consistency is nice in that it becomes extremely obvious what the states and transitions are (and includes other very nice benefits, like having a log of what states occurred). The question to me becomes what happens to that
state consistency when its running in a distributed system, which all of openstack is running in. At that point u need a way to ensure multiple servers (going through various states) are not manipulating the same resources at the same time (delete volume from
cinder, while attaching it in nova). Those 2 separate services do not likely share the same state transitions (and will likely not as they become tightly coupled at that point). So then u need some type of coordination system to ensure the ordering of these
2 resource actions is done in a consistent manner. To me this starts to involve zookeeper (or something similar) since its in the end just distributed coordination (which is what these systems were designed for).</div>
<div><br>
</div>
<div>My 2 cents.</div>
<div><br>
</div>
<div>-Josh</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Georgy Okrokvertskhov <<a href="mailto:gokrokvertskhov@mirantis.com">gokrokvertskhov@mirantis.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, October 30, 2013 11:04 AM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Heat] Locking and ZooKeeper - a space oddysey<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Hi Clint,
<div><br>
</div>
<div>I think you rose a point here. We implemented distributed engine in Murano without locking mechanism by keeping state consistent on each step. We extracted this engine from Murano and plan to put it as a part of Mistral project for task management and
execution. Working Mistral implementation will appear during IceHouse development. We are working closely with taskflow team, so I think you can expect to have distributed task execution support in taskflow library natively or through Mistral.</div>
<div><br>
</div>
<div>I am not against ZooKeeper but I think that for OpenStack service it is better to use oslo library shared with other projects instead of adding some custom locking mechanism for one project.</div>
<div><br>
</div>
<div>Thanks</div>
<div>Georgy</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Oct 30, 2013 at 10:42 AM, Clint Byrum <span dir="ltr">
<<a href="mailto:clint@fewbar.com" target="_blank">clint@fewbar.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
So, recently we've had quite a long thread in gerrit regarding locking<br>
in Heat:<br>
<br>
<a href="https://review.openstack.org/#/c/49440/" target="_blank">https://review.openstack.org/#/c/49440/</a><br>
<br>
In the patch, there are two distributed lock drivers. One uses SQL,<br>
and suffers from all the problems you might imagine a SQL based locking<br>
system would. It is extremely hard to detect dead lock holders, so we<br>
end up with really long timeouts. The other is ZooKeeper.<br>
<br>
I'm on record as saying we're not using ZooKeeper. It is a little<br>
embarrassing to have taken such a position without really thinking things<br>
through. The main reason I feel this way though, is not because ZooKeeper<br>
wouldn't work for locking, but because I think locking is a mistake.<br>
<br>
The current multi-engine paradigm has a race condition. If you have a<br>
stack action going on, the state is held in the engine itself, and not<br>
in the database, so if another engine starts working on another action,<br>
they will conflict.<br>
<br>
The locking paradigm is meant to prevent this. But I think this is a<br>
huge mistake.<br>
<br>
The engine should store _all_ of its state in a distributed data store<br>
of some kind. Any engine should be aware of what is already happening<br>
with the stack from this state and act accordingly. That includes the<br>
engine currently working on actions. When viewed through this lense,<br>
to me, locking is a poor excuse for serializing the state of the engine<br>
scheduler.<br>
<br>
It feels like TaskFlow is the answer, with an eye for making sure<br>
TaskFlow can be made to work with distributed state. I am not well<br>
versed on TaskFlow's details though, so I may be wrong. It worries me<br>
that TaskFlow has existed a while and doesn't seem to be solving real<br>
problems, but maybe I'm wrong and it is actually in use already.<br>
<br>
Anyway, as a band-aid, we may _have_ to do locking. For that, ZooKeeper<br>
has some real advantages over using the database. But there is hesitance<br>
because it is not widely supported in OpenStack. What say you, OpenStack<br>
community? Should we keep ZooKeeper out of our.. zoo?<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
Georgy Okrokvertskhov<br>
Technical Program Manager,<br>
Cloud and Infrastructure Services,<br>
Mirantis<br>
<a href="http://www.mirantis.com/" target="_blank">http://www.mirantis.com</a><br>
Tel. +1 650 963 9828<br>
Mob. +1 650 996 3284<br>
</div>
</div>
</div>
</span>
</body>
</html>