Open Stack

On Aug 2, 2015, at 7:40 PM, 王华 <wanghua.humble at gmail.com<mailto:wanghua.humble at gmail.com>> wrote:

Hi all,


As discussed in the Vancouver Summit, we are going to drop the bay lock implementation. Instead, each conductor will call Heat concurrently and rely on heat for concurrency control. However, I think we need an approach for state convergence from heat to magnum. Either periodic task [1] or heat notification [2] looks like a candidate.

[1] https://blueprints.launchpad.net/magnum/+spec/add-periodic-task
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-March/058898.html
--hongbin

If we use periodic task to sync state from heat to magnum, I think we should make periodic task a independent process and magnum-conductor only operates heat stack.

How to make periodic task high available？

1.We can run several periodic tasks.

2.Or we can use a leader selection mechanism to make only a periodic task running

and other periodic tasks waiting.

Shall we make periodic task a independent process? How to make periodic task high available?

Good question. The traditional solution for handling this in a distributed system is to hold a leader election. The elected leader is responsible for dispatching the job to a queue that one available worker will pick up and run. However, that may not actually be needed in our case. Consider the question:

What harm will come if all master nodes in the cluster perform the same periodic task?

One drawback is that more resources will be consumed than necessary (efficiency). As long as updates to the Magnum database are transactional, having concurrent updates to the same bay is actually not something we expect would result in data corruption. Worst case the same update would be processed multiple times. The advantage of using this approach is that we would not need to implement any form of leader selection. This would keep our implementation simpler, and less error prone.

We could still supervise each periodic task process so if they end up crashing on each node, they would be restarted. This is simple to do from a patent process that calls os.wait() on the child task.

Thoughts?

Adrian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150803/77c27e13/attachment.html>