[openstack-dev] [Fuel] Speed Up RabbitMQ Recovering

Bogdan Dobrelya bdobrelia at mirantis.com
Tue May 5 09:52:31 UTC 2015


On 05.05.2015 04:32, Andrew Beekhof wrote:
> 
> 
> [snip]
> 
> 
> Technically it calculates an ordered graph of actions that need to be performed for a set of related resources.
> You can see an example of the kinds of graphs it produces at:
> 
>    http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/s-config-testing-changes.html
> 
> There is a more complex one which includes promotion and demotion on the next page.
> 
> The number of actions that can run at any one time is therefor limited by
> - the value of batch-limit (the total number of in-flight actions)
> - the number of resources that do not have ordering constraints between them (eg. rsc{1,2,3} in the above example)  
> 
> So in the above example, if batch-limit >= 3, the monitor_0 actions will still all execute in parallel.
> If batch-limit == 2, one of them will be deferred until the others complete.
> 
> Processing of the graph stops the moment any action returns a value that was not expected.
> If that happens, we wait for currently in-flight actions to complete, re-calculate a new graph based on the new information and start again.
> 
> 
> First we do a non-recurring monitor (*_monitor_0) to check what state the resource is in.
> We can’t assume its off because a) we might have crashed, b) the admin might have accidentally configured it to start at boot or c) the admin may have asked us to re-check everything.
> 
> 
> Also important to know, the order of actions is:
> 
> 1. any necessary demotions
> 2. any necessary stops
> 3. any necessary starts
> 4. any necessary promotions
> 
> 

Thank you for explaining this, Andrew!

So, in the context of the given two example DB(MySQL) and
messaging(RabbitMQ) resources:

"The problem is that pacemaker can only promote a resource after it
detects the resource is started. During a full reassemble, in the first
transition batch, pacemaker starts all the resources including MySQL and
RabbitMQ. Pacemaker issues resource agent "start" invocation in parallel
and reaps the results.
For a multi-state resource agent like RabbitMQ, pacemaker needs the
start result reported in the first batch, then transition engine and
policy engine decide if it has to retry starting or promote, and put
this new transition job into a new batch."

So, for given example, it looks like we currently have:
_batch start_
...
3. DB, messaging resources start in a one batch
4. messaging resource promote blocked by the step 3 completion
_batch end_

Does this mean what an artificial constraints ordering between DB and
messaging could help them to get into the separate transition batches, like:

...
3. messaging multistate clone resource start
4. messaging multistate clone resource promote
_batch end_

_next batch start_
...
3. DB simple clone resource start

?

-- 
Best regards,
Bogdan Dobrelya,
Skype #bogdando_at_yahoo.com
Irc #bogdando



More information about the OpenStack-dev mailing list