Open Stack

Adrian,

If the reason to avoid leader election is because it is complicated and error prone, this argument may not be true. Leader election is complicated in a pure distributed system in which there is no centralized storage. However, Magnum has a centralized database, so it is possible to implement a very simple leader election algorithm. For example, we can let each conductor register itself in the DB and elect the first registered conductor to be the leader. How about that?

Best regards,
Hongbin

From: Adrian Otto [mailto:adrian.otto at rackspace.com]
Sent: August-03-15 12:49 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [Magnum]horizontal scalability


On Aug 2, 2015, at 7:40 PM, 王华 <wanghua.humble at gmail.com<mailto:wanghua.humble at gmail.com>> wrote:

Hi all,

As discussed in the Vancouver Summit, we are going to drop the bay lock implementation. Instead, each conductor will call Heat concurrently and rely on heat for concurrency control. However, I think we need an approach for state convergence from heat to magnum. Either periodic task [1] or heat notification [2] looks like a candidate.
[1] https://blueprints.launchpad.net/magnum/+spec/add-periodic-task
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-March/058898.html
--hongbin
If we use periodic task to sync state from heat to magnum, I think we should make periodic task a independent process and magnum-conductor only operates heat stack.
How to make periodic task high available？
1.We can run several periodic tasks.
2.Or we can use a leader selection mechanism to make only a periodic task running
and other periodic tasks waiting.
Shall we make periodic task a independent process? How to make periodic task high available?
Good question. The traditional solution for handling this in a distributed system is to hold a leader election. The elected leader is responsible for dispatching the job to a queue that one available worker will pick up and run. However, that may not actually be needed in our case. Consider the question:

What harm will come if all master nodes in the cluster perform the same periodic task?

One drawback is that more resources will be consumed than necessary (efficiency). As long as updates to the Magnum database are transactional, having concurrent updates to the same bay is actually not something we expect would result in data corruption. Worst case the same update would be processed multiple times. The advantage of using this approach is that we would not need to implement any form of leader selection. This would keep our implementation simpler, and less error prone.

We could still supervise each periodic task process so if they end up crashing on each node, they would be restarted. This is simple to do from a patent process that calls os.wait() on the child task.

Thoughts?

Adrian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150804/1d621be3/attachment.html>