<div dir="ltr"><div><div>Hi crew,<br><br></div>Thank you for starting this topic. I've already performed the research and started blueprint. Since we changed our blueprint strategy, I made it in rst format and added it to Gerrit workflow. Feel free to participate.<br>
<br><a href="https://review.openstack.org/#/c/97191/" target="_blank">https://review.openstack.org/#/c/97191/</a><br><a href="http://docs-draft.openstack.org/91/97191/8/check/gate-fuel-specs-docs/a1e7c72/doc/build/html/specs/5.1/pacemaker-galera-resource-agent.html">http://docs-draft.openstack.org/91/97191/8/check/gate-fuel-specs-docs/a1e7c72/doc/build/html/specs/5.1/pacemaker-galera-resource-agent.html</a><br>
<br></div><div>It's still draft as some discussions/tests are still going on.<br></div><div><br>>PoC RA get GTID from sql (SHOW STATUS LIKE ‚wsrep_last_committed’) if
MySQL is running, in other case RA start mysqld with --wsrep-recover. I
>skipped grastate.dat because in all my test this file had commit_id set
to -1.<br><br></div><div>Percona way is more robust as they can restore the state even when CIB became corrupted and whole cluster went down (power outage).<br><br></div><div>~Sergii<br></div><div><br></div><div><br></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jun 2, 2014 at 3:09 PM, Bartosz Kupidura <span dir="ltr"><<a href="mailto:bkupidura@mirantis.com" target="_blank">bkupidura@mirantis.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Vladimir,<br>
<br>
<br>
Wiadomość napisana przez Vladimir Kuklin <<a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a>> w dniu 2 cze 2014, o godz. 13:49:<br>
<div class=""><br>
> Bartosz, if you look into what Percona guys are doing - you will see here: <a href="https://github.com/percona/percona-pacemaker-agents/blob/new_pxc_ra/agents/pxc_resource_agent#L516" target="_blank">https://github.com/percona/percona-pacemaker-agents/blob/new_pxc_ra/agents/pxc_resource_agent#L516</a> that they first try to use MySQL and then to get GTID from grastate.dat. Also, I am wondering if you are using cluster-wide attributes instead of node-attributes. If you use node-scoped attributes, then shadow/commit commands should not affect anything.<br>
<br>
</div>PoC RA get GTID from sql (SHOW STATUS LIKE ‚wsrep_last_committed’) if MySQL is running, in other case RA start mysqld with --wsrep-recover. I skipped grastate.dat because in all my test this file had commit_id set to -1.<br>
<br>
In PoC i use only node-attributes (crm_attribute --node $HOSTNAME --lifetime forever --name gtid --update $GTID).<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
><br>
> On Mon, Jun 2, 2014 at 2:34 PM, Bogdan Dobrelya <<a href="mailto:bdobrelia@mirantis.com">bdobrelia@mirantis.com</a>> wrote:<br>
> On 05/29/2014 02:06 PM, Bartosz Kupidura wrote:<br>
> > Hello,<br>
> ><br>
> ><br>
> > Wiadomość napisana przez Vladimir Kuklin <<a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a>> w dniu 29 maj 2014, o godz. 12:09:<br>
> ><br>
> >> may be the problem is that you are using liftetime crm attributes instead of 'reboot' ones. shadow/commit is used by us because we need transactional behaviour in some cases. if you turn crm_shadow off, then you will experience problems with multi-state resources and location/colocation/order constraints. so we need to find a way to make commits transactional. there are two ways:<br>
> >> 1) rewrite corosync providers to use crm_diff command and apply it instead of shadow commit that can swallow cluster attributes sometimes<br>
> ><br>
> > In PoC i removed all cs_commit/cs_shadow, and looks that everything is working. But as you says, this can lead to problems with more complicated deployments.<br>
> > This need to be verified.<br>
> ><br>
> >> 2) store 'reboot' attributes instead of lifetime ones<br>
> ><br>
> > I test with —lifetime forever and reboot. No difference for cs_commit/cs_shadow fail.<br>
> ><br>
> > Moreover we need method to store GTID permanent (to support whole cluster reboot).<br>
><br>
> Please note, GTID could always be fetched from the<br>
> /var/lib/mysql/grastate.dat at the galera node<br>
><br>
> > If we want to stick to cs_commit/cs_shadow, we need other method to store GTID than crm_attribute.<br>
><br>
> WE could use a modified ocf::pacemaker:SysInfo resource. We could put<br>
> GTID there and use it the similar way as I did for fencing PoC[0] (for<br>
> free space monitoring)<br>
><br>
> [0]<br>
> <a href="https://github.com/bogdando/fuel-library-1/blob/ha_fencing_WIP/deployment/puppet/cluster/manifests/fencing_primitives.pp#L41-L70" target="_blank">https://github.com/bogdando/fuel-library-1/blob/ha_fencing_WIP/deployment/puppet/cluster/manifests/fencing_primitives.pp#L41-L70</a><br>
><br>
> ><br>
> >><br>
> >><br>
> >><br>
> >> On Thu, May 29, 2014 at 12:42 PM, Bogdan Dobrelya <<a href="mailto:bdobrelia@mirantis.com">bdobrelia@mirantis.com</a>> wrote:<br>
> >> On 05/27/14 16:44, Bartosz Kupidura wrote:<br>
> >>> Hello,<br>
> >>> Responses inline.<br>
> >>><br>
> >>><br>
> >>> Wiadomość napisana przez Vladimir Kuklin <<a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a>> w dniu 27 maj 2014, o godz. 15:12:<br>
> >>><br>
> >>>> Hi, Bartosz<br>
> >>>><br>
> >>>> First of all, we are using openstack-dev for such discussions.<br>
> >>>><br>
> >>>> Second, there is also Percona's RA for Percona XtraDB Cluster, which looks like pretty similar, although it is written in Perl. May be we could derive something useful from it.<br>
> >>>><br>
> >>>> Next, if you are working on this stuff, let's make it as open for the community as possible. There is a blueprint for Galera OCF script: <a href="https://blueprints.launchpad.net/fuel/+spec/reliable-galera-ocf-script" target="_blank">https://blueprints.launchpad.net/fuel/+spec/reliable-galera-ocf-script</a>. It would be awesome if you wrote down the specification and sent newer galera ocf code change request to fuel-library gerrit.<br>
> >>><br>
> >>> Sure, I will update this blueprint.<br>
> >>> Change request in fuel-library: <a href="https://review.openstack.org/#/c/95764/" target="_blank">https://review.openstack.org/#/c/95764/</a><br>
> >><br>
> >> That is a really nice catch, Bartosz, thank you. I believe we should<br>
> >> review the new OCF script thoroughly and consider omitting<br>
> >> cs_commits/cs_shadows as well. What would be the downsides?<br>
> >><br>
> >>><br>
> >>>><br>
> >>>> Speaking of crm_attribute stuff. I am very surprised that you are saying that node attributes are altered by crm shadow commit. We are using similar approach in our scripts and have never faced this issue.<br>
> >>><br>
> >>> This is probably because you update crm_attribute very rarely. And with my approach GTID attribute is updated every 60s on every node (3 updates in 60s, in standard HA setup).<br>
> >>><br>
> >>> You can try to update any attribute in loop during deploying cluster to trigger fail with corosync diff.<br>
> >><br>
> >> It sounds reasonable and we should verify it.<br>
> >> I've updated the statuses for related bugs and attached them to the<br>
> >> aforementioned blueprint as well:<br>
> >> <a href="https://bugs.launchpad.net/fuel/+bug/1283062/comments/7" target="_blank">https://bugs.launchpad.net/fuel/+bug/1283062/comments/7</a><br>
> >> <a href="https://bugs.launchpad.net/fuel/+bug/1281592/comments/6" target="_blank">https://bugs.launchpad.net/fuel/+bug/1281592/comments/6</a><br>
> >><br>
> >><br>
> >>><br>
> >>>><br>
> >>>> Corosync 2.x support is in our roadmap, but we are not sure that we will use Corosync 2.x earlier than 6.x release series start.<br>
> >>><br>
> >>> Yeah, moreover corosync CMAP is not synced between cluster nodes (or maybe im doing something wrong?). So we need other solution for this...<br>
> >>><br>
> >><br>
> >> We should use CMAN for Corosync 1.x, perhaps.<br>
> >><br>
> >>>><br>
> >>>><br>
> >>>> On Tue, May 27, 2014 at 3:08 PM, Bartosz Kupidura <<a href="mailto:bkupidura@mirantis.com">bkupidura@mirantis.com</a>> wrote:<br>
> >>>> Hello guys!<br>
> >>>> I would like to start discussion on a new resource agent for galera/pacemaker.<br>
> >>>><br>
> >>>> Main features:<br>
> >>>> * Support cluster boostrap<br>
> >>>> * Support reboot any node in cluster<br>
> >>>> * Support reboot whole cluster<br>
> >>>> * To determine which node have latest DB version, we should use galera GTID (Global Transaction ID)<br>
> >>>> * Node with latest GTID is galera PC (primary component) in case of reelection<br>
> >>>> * Administrator can manually set node as PC<br>
> >>>><br>
> >>>> GTID:<br>
> >>>> * get GTID from mysqld --wsrep-recover or SQL query 'SHOW STATUS LIKE ‚wsrep_local_state_uuid''<br>
> >>>> * store GTID as crm_attribute for node (crm_attribute --node $HOSTNAME --lifetime $LIFETIME --name gtid --update $GTID)<br>
> >>>> * on every monitor/stop/start action update GTID for given node<br>
> >>>> * GTID can have 3 format:<br>
> >>>> - XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX:123 - standard cluster-id:commit-id<br>
> >>>> - XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX:-1 - standard non initialized cluster, 00000000-0000-0000-0000-000000000000:-1<br>
> >>>> - XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX:INF - commit-id manually set to INF, force RA to create new cluster, with master on given node<br>
> >>>><br>
> >>>> Check if reelection of PC is needed:<br>
> >>>> * (node is located in partition with quorum OR we have only 1 node configured in cluster) AND galera resource is not running on any node<br>
> >>>> * GTID is manually set to INF on given node<br>
> >>>><br>
> >>>> Check if given node is PC:<br>
> >>>> * have highest GTID in cluster, in case we have more than one node with „highest” GTID, we use CRC32 to choose proper PC.<br>
> >>>> * GTID is manually set to INF<br>
> >>>> * in case node with highest GTID will not come back after cluster reboot (for example disk failure) administrator should set GTID to INF on other node<br>
> >>>><br>
> >>>> I have almost ready RA: <a href="http://zynzel.spof.pl/mysql-wss" target="_blank">http://zynzel.spof.pl/mysql-wss</a><br>
> >>>><br>
> >>>> Tested with vanila centos galera/pacemaker/corosync - OK<br>
> >>>> Tested with Fuel 4.1 - Fail<br>
> >>>><br>
> >>>><br>
> >>>> Fuel 4.1 with that RA will not deploy correctly, because we use crm_attribute to store GTID, and in manifest we use cs_shadow/cs_commit for every pacemaker resource.<br>
> >>>> This lead to cs_commit problem with different configuration in shadow copy and running configuration (running config changed by RA).<br>
> >>>> "Could not commit shadow instance [..] to the CIB: Application of an update diff failed”<br>
> >>>><br>
> >>>> To solve this we can go in 2 ways:<br>
> >>>> 1) dont use cs_commit/cs_shadow in manifests<br>
> >>>> 2) store GTID in other way than crm_attribute<br>
> >>>><br>
> >>>> IMHO 2) is better (less invasive) and we can store GTID in corosync CMAP (<a href="http://www.polarhome.com/service/man/generic.php?qf=corosync-cmapctl" target="_blank">http://www.polarhome.com/service/man/generic.php?qf=corosync-cmapctl</a>), but this require corosync 2.X<br>
> >>>><br>
> >>>><br>
> >>>> --<br>
> >>>> Mailing list: <a href="https://launchpad.net/~fuel-dev" target="_blank">https://launchpad.net/~fuel-dev</a><br>
> >>>> Post to : <a href="mailto:fuel-dev@lists.launchpad.net">fuel-dev@lists.launchpad.net</a><br>
> >>>> Unsubscribe : <a href="https://launchpad.net/~fuel-dev" target="_blank">https://launchpad.net/~fuel-dev</a><br>
> >>>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> --<br>
> >>>> Yours Faithfully,<br>
> >>>> Vladimir Kuklin,<br>
> >>>> Fuel Library Tech Lead,<br>
> >>>> Mirantis, Inc.<br>
> >>>> <a href="tel:%2B7%20%28495%29%20640-49-04" value="+74956404904">+7 (495) 640-49-04</a><br>
> >>>> <a href="tel:%2B7%20%28926%29%20702-39-68" value="+79267023968">+7 (926) 702-39-68</a><br>
> >>>> Skype kuklinvv<br>
> >>>> 45bk3, Vorontsovskaya Str.<br>
> >>>> Moscow, Russia,<br>
> >>>> <a href="http://www.mirantis.com" target="_blank">www.mirantis.com</a><br>
> >>>> <a href="http://www.mirantis.ru" target="_blank">www.mirantis.ru</a><br>
> >>>> <a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a><br>
> >>><br>
> >>><br>
> >><br>
> >><br>
> >> --<br>
> >> Best regards,<br>
> >> Bogdan Dobrelya,<br>
> >> Skype #<a href="http://bogdando_at_yahoo.com" target="_blank">bogdando_at_yahoo.com</a><br>
> >> Irc #bogdando<br>
> >><br>
> >><br>
> >><br>
> >> --<br>
> >> Yours Faithfully,<br>
> >> Vladimir Kuklin,<br>
> >> Fuel Library Tech Lead,<br>
> >> Mirantis, Inc.<br>
> >> <a href="tel:%2B7%20%28495%29%20640-49-04" value="+74956404904">+7 (495) 640-49-04</a><br>
> >> <a href="tel:%2B7%20%28926%29%20702-39-68" value="+79267023968">+7 (926) 702-39-68</a><br>
> >> Skype kuklinvv<br>
> >> 45bk3, Vorontsovskaya Str.<br>
> >> Moscow, Russia,<br>
> >> <a href="http://www.mirantis.com" target="_blank">www.mirantis.com</a><br>
> >> <a href="http://www.mirantis.ru" target="_blank">www.mirantis.ru</a><br>
> >> <a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a><br>
> ><br>
> ><br>
> ><br>
><br>
><br>
> --<br>
> Best regards,<br>
> Bogdan Dobrelya,<br>
> Skype #<a href="http://bogdando_at_yahoo.com" target="_blank">bogdando_at_yahoo.com</a><br>
> Irc #bogdando<br>
><br>
><br>
><br>
> --<br>
> Yours Faithfully,<br>
> Vladimir Kuklin,<br>
> Fuel Library Tech Lead,<br>
> Mirantis, Inc.<br>
> <a href="tel:%2B7%20%28495%29%20640-49-04" value="+74956404904">+7 (495) 640-49-04</a><br>
> <a href="tel:%2B7%20%28926%29%20702-39-68" value="+79267023968">+7 (926) 702-39-68</a><br>
> Skype kuklinvv<br>
> 45bk3, Vorontsovskaya Str.<br>
> Moscow, Russia,<br>
> <a href="http://www.mirantis.com" target="_blank">www.mirantis.com</a><br>
> <a href="http://www.mirantis.ru" target="_blank">www.mirantis.ru</a><br>
> <a href="mailto:vkuklin@mirantis.com">vkuklin@mirantis.com</a><br>
<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>