<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 12pt;
font-family:΢ÈíÑźÚ
}
--></style></head>
<body class='hmmessage'><div dir='ltr'><div>Hi <span style="font-size: 12pt;">Anant,</span></div><div><br></div><div><span style="font-size: 12pt;">For the second option, if the leader engine fails, how to trigger a new leader election progress?</span></div><div><span style="font-size: 12pt;"><br></span></div><div>Best Regards,</div><div>Yingzhe Zeng<br><br><div>> To: openstack-dev@lists.openstack.org<br>> From: anant.patil@hpe.com<br>> Date: Wed, 30 Sep 2015 12:40:52 +0530<br>> Subject: [openstack-dev] [heat] Convergence: Detecting and handling worker failures<br>> <br>> Hi,<br>> <br>> One of remaining items in convergence is detecting and handling engine<br>> (the engine worker) failures, and here are my thoughts.<br>> <br>> Background: Since the work is distributed among heat engines, by some<br>> means heat needs to detect the failure and pick up the tasks from failed<br>> engine and re-distribute or run the task again.<br>> <br>> One of the simple way is to poll the DB to detect the liveliness by<br>> checking the table populated by heat-manage. Each engine records its<br>> presence periodically by updating current timestamp. All the engines<br>> will have a periodic task for checking the DB for liveliness of other<br>> engines. Each engine will check for timestamp updated by other engines<br>> and if it finds one which is older than the periodicity of timestamp<br>> updates, then it detects a failure. When this happens, the remaining<br>> engines, as and when they detect the failures, will try to acquire the<br>> lock for in-progress resources that were handled by the engine which<br>> died. They will then run the tasks to completion.<br>> <br>> Another option is to use a coordination library like the community owned<br>> tooz (http://docs.openstack.org/developer/tooz/) which supports<br>> distributed locking and leader election. We use it to elect a leader<br>> among heat engines and that will be responsible for running periodic<br>> tasks for checking state of each engine and distributing the tasks to<br>> other engines when one fails. The advantage, IMHO, will be simplified<br>> heat code. Also, we can move the timeout task to the leader which will<br>> run time out for all the stacks and sends signal for aborting operation<br>> when timeout happens. The downside: an external resource like<br>> Zookeper/memcached etc are needed for leader election.<br>> <br>> In the long run, IMO, using a library like tooz will be useful for heat.<br>> A lot of boiler plate needed for locking and running centralized tasks<br>> (such as timeout) will not be needed in heat. Given that we are moving<br>> towards distribution of tasks and horizontal scaling is preferred, it<br>> will be advantageous to use them.<br>> <br>> Please share your thoughts.<br>> <br>> - Anant<br>> <br>> <br>> <br>> __________________________________________________________________________<br>> OpenStack Development Mailing List (not for usage questions)<br>> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br></div></div> </div></body>
</html>