[openstack-dev] [HA][RabbitMQ][messaging][Pacemaker][operators] Improved OCF resource agent for dynamic active-active mirrored clustering

Bogdan Dobrelya bdobrelia at mirantis.com
Fri Apr 8 14:50:51 UTC 2016


On 03/17/2016 10:01 AM, Bogdan Dobrelya wrote:
> On 03/17/2016 12:17 AM, Andrew Beekhof wrote:
>>
>>
>> On Tue, Feb 16, 2016 at 2:58 AM, Bogdan Dobrelya <bdobrelia at mirantis.com
>> <mailto:bdobrelia at mirantis.com>> wrote:
>>
>>     Hello!
>>     A quick status update inline:
>>
>>
>> [snip] 
>>
>>
>>     So, what's next?
>>
>>     - I'm open for merging both [5], [6] of the existing OCF RA solutions,
>>     as it was proposed by Andrew Beekhof. Let's make it happen.
>>
>>
>> Great :-)
>> Oyvind (CC'd) is the relevant contact from our side, should he talk to
>> you or someone else?
> 
> Yes, we should make a follow-up perhaps and define the plan for merging
> agents

An update!
The followup was very productive, the upstream OCF RAs merge plan is

1. The rabbitmq-server repo [0] will be the destination for the merged
OCF RA.
2. Fork it [1] for the merge process and enable a travis CI check for
incoming pull requests. Note that the check exists upstream as well but
has yet to be enabled properly. Also, we decided to run tests for *only*
a cluster assembled with the merged OCF RA, w/o other components like
OpenStack or DB cluster.
3. Update the travis CI check (see example job script [2]) to meet use
cases of the 2nd OCF RA, from the resource-agents repo, we want to merge
with. So the test case shall benefit all sides of the merge process.
4. Start submitting patches to the merged RA.
5. Once finished, submit it to the upstream source at step #1. Or do it
by every (stable) patch based as well.

Action items:
* Peter Lemenkov (petro) to try a dev env via vagrant scripts [3] as
well as check it against the (RHOS/resource-agents?) packages containing
the 2nd upstream OCF RA.
* Oyvind Albrigtsen (e-ddie) to investigate ASL/GPL licensing questions.
Hopefully no issues expected and the merged OCF RA should be licensed
with ASL v2.0.
* Bogdan Dobrelya (bogdando) to investigate destructive test cases for
the travis CI check as well. The plan is to use custom jepsen tests [4]
for that.

An update item #2:
I created the aforementioned jepsen test [4], to test a rabbit cluster
recovery after network partitions. The travis CI check may run it now as
well [5], although faily a little - It may time out, but works well for
my dev box :) How-to use can be found in the README [3].

[0]
https://github.com/rabbitmq/rabbitmq-server/blob/master/scripts/rabbitmq-server-ha.ocf
[1]
https://github.com/bogdando/rabbitmq-server/blob/master/scripts/rabbitmq-server-ha.ocf
[2]
https://github.com/bogdando/rabbitmq-server/blob/rabbit_ocf_ra_travis/scripts/travis_test_ocf_ra.sh
[3] https://github.com/bogdando/rabbitmq-cluster-ocf-vagrant
[4] https://github.com/bogdando/jepsen/tree/rabbit_pcmk/rabbitmq_ocf_pcmk
[5] https://travis-ci.org/bogdando/rabbitmq-server/jobs/121704254

> 
>>  
>>
>>
>>     - Would be nice to make Travis CI based gate to the upstream
>>     rabbitmq-server's HA OCF RA. As for now, it relies on Fuel CI gates and
>>     manual testing with atlas boxes.
>>
>>     - Please also consider Travis or a suchlike CI for the resource-agents'
>>     rabbit-cluster OCF RA as well.
>>
>>     [1] https://github.com/bogdando/rabbitmq-cluster-ocf-vagrant
>>     [2] https://github.com/bogdando/packer-atlas-example
>>     [3] https://hub.docker.com/r/bogdando/rabbitmq-cluster-ocf/
>>     [4] https://hub.docker.com/r/bogdando/rabbitmq-cluster-ocf-wily/
>>     [5]
>>     https://github.com/rabbitmq/rabbitmq-server/blob/master/scripts/rabbitmq-server-ha.ocf
>>     [6]
>>     https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/rabbitmq-cluster
>>
>>     >
>>     > I'm also planning to refer this official RabbitMQ cluster setup guide in
>>     > the OpenStack HA guide as well [2].
>>
>>     Done, see [7]
>>
>>     [7] http://docs.openstack.org/ha-guide/controller-ha-rabbitmq.html
>>
>>     >
>>     > PS. Original rabbitmq-users mail thread is here [3].
>>     > [openstack-operators] cross posted as well.
>>     >
>>     > [0] http://www.rabbitmq.com/pacemaker.html
>>     > [1] https://atlas.hashicorp.com/bogdando/boxes/rabbitmq-cluster-ocf
>>     > [2] https://bugs.launchpad.net/openstack-manuals/+bug/1497528
>>     > [3] https://groups.google.com/forum/#!topic/rabbitmq-users/BnoIQJb34Ao
>>     >
>>
>>
>>     --
>>     Best regards,
>>     Bogdan Dobrelya,
>>     Irc #bogdando
>>
>>     __________________________________________________________________________
>>     OpenStack Development Mailing List (not for usage questions)
>>     Unsubscribe:
>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 


-- 
Best regards,
Bogdan Dobrelya,
Irc #bogdando



More information about the OpenStack-dev mailing list