Open Stack

On 28.04.2015 15:15, Bogdan Dobrelya wrote:
> 
> Hello, Zhou
> 
> 
> Yes, this is a known issue [0]. Note, there were many bugfixes, like
> [1],[2],[3], merged for MQ OCF script, so you may want to try to
> backport them as well by the following guide [4]
> 
> [0] https://bugs.launchpad.net/fuel/+bug/1432603
> [1] https://review.openstack.org/#/c/175460/
> [2] https://review.openstack.org/#/c/175457/
> [3] https://review.openstack.org/#/c/175371/
> [4] https://review.openstack.org/#/c/170476/
> 
> 
> Could you please elaborate the what is the same/different batches for MQ
> and DB? Note, there is a MQ clustering logic flow charts available here
> [5] and we're planning to release a dedicated technical bulletin for this.
> 
> [5] http://goo.gl/PPNrw7
> 
> 
> This is very interesting, thank you! I believe all commands for MySQL RA
> OCF script should be as well wrapped with timeout -SIGTERM or -SIGKILL
> as we did for MQ RA OCF. And there should no be any sleep calls. I
> created a bug for this [6].
> 
> [6] https://bugs.launchpad.net/fuel/+bug/1449542
> 
> 
> Yes, something like that. As I mentioned, there were several bug fixes
> in the 6.1 dev, and you can also check the MQ clustering flow charts.
> 
> after
> 
> Not exactly. There is no master in mirrored MQ cluster. We define the
> rabbit_hosts configuration option from Oslo.messaging. What ensures all
> queue masters will be spread around all of MQ nodes in a long run. And
> we use a master abstraction only for the Pacemaker RA clustering layer.
> Here, a "master" is the MQ node what joins the rest of the MQ nodes.
> 
> 
> We do erase the node master attribute in CIB for such cases. This should
> not bring problems into the master election logic.
> 
> 
> (Note, the RabbitMQ documentation mentions *queue* masters and slaves,
> which are not the case for the Pacemaker RA clustering abstraction layer.)
> 
> 
> We made an assumption what the node with the highest MQ uptime should
> know the most about recent cluster state, so other nodes must join it.
> RA OCF does not work with queue masters directly.
> 
> 
> The full MQ cluster reassemble logic is far from the perfect state,
> indeed. This might erase all mnesia files, hence any custom entities,
> like users or vhosts, would be removed as well. Note, we do not
> configure durable queues for Openstack so there is nothing to care about
> here - the full cluster downtime assumes there will be no AMQP messages
> stored at all.
> 
> 
> Yes, this option is only supported for newest RabbitMQ versions. But we
> definitely should look how this could help.
> 
> 
> Indeed, there are cases when MQ's autoheal can do nothing with existing
> partitions and remains partitioned for ever, for example:
> 
> Masters: [ node-1 ]
> Slaves: [ node-2 node-3 ]
> root at node-1:~# rabbitmqctl cluster_status
> Cluster status of node 'rabbit at node-1' ...
> [{nodes,[{disc,['rabbit at node-1','rabbit at node-2']}]},
> {running_nodes,['rabbit at node-1']},
> {cluster_name,<<"rabbit at node-2">>},
> {partitions,[]}]
> ...done.
> root at node-2:~# rabbitmqctl cluster_status
> Cluster status of node 'rabbit at node-2' ...
> [{nodes,[{disc,['rabbit at node-2']}]}]
> ...done.
> root at node-3:~# rabbitmqctl cluster_status
> Cluster status of node 'rabbit at node-3' ...
> [{nodes,[{disc,['rabbit at node-1','rabbit at node-2','rabbit at node-3']}]},
> {running_nodes,['rabbit at node-3']},
> {cluster_name,<<"rabbit at node-2">>},
> {partitions,[]}]

Sorry, here is the correct one [0] !

[0] http://pastebin.com/m3fDdMA6

> 
> So we should test the pause-minority value as well.
> But I strongly believe we should make MQ multi-state clone to support
> many masters, related bp [7]
> 
> [7]
> https://blueprints.launchpad.net/fuel/+spec/rabbitmq-pacemaker-multimaster-clone
> 
> 
> Well, we should not mess the queue masters and multi-clone master for MQ
> resource in the pacemaker.
> As I said, pacemaker RA has nothing to do with queue masters. And we
> introduced this "master" mostly in order to support the full cluster
> reassemble case - there must be a node promoted and other nodes should join.
> 
> 
> This is a very good point, thank you.
> 
> 
> Thank you for a thorough feedback! This was a really great job.
> 
> 
> 


-- 
Best regards,
Bogdan Dobrelya,
Skype #bogdando_at_yahoo.com
Irc #bogdando