<html>
<head>
<meta name="generator" content="Windows Mail 17.5.9600.20911">
<style type="text/css"><!--html { font-family: "Color Emoji", "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; }--></style><style data-externalstyle="true"><!--
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph {
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
}
p.MsoNormal, li.MsoNormal, div.MsoNormal {
margin:0in;
margin-bottom:.0001pt;
}
p.MsoListParagraphCxSpFirst, li.MsoListParagraphCxSpFirst, div.MsoListParagraphCxSpFirst,
p.MsoListParagraphCxSpMiddle, li.MsoListParagraphCxSpMiddle, div.MsoListParagraphCxSpMiddle,
p.MsoListParagraphCxSpLast, li.MsoListParagraphCxSpLast, div.MsoListParagraphCxSpLast {
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
line-height:115%;
}
--></style></head>
<body dir="ltr">
<div data-externalstyle="false" dir="ltr" style="font-family: 'Calibri', 'Segoe UI', 'Meiryo', 'Microsoft YaHei UI', 'Microsoft JhengHei UI', 'Malgun Gothic', 'sans-serif';font-size:12pt;"><div>Hi Roman.</div><div>That's interesting, although’s hard to believe (there is no slave lag in galera multi master). I can only suggest us to create another jepsen test to verify exactly scenario you describe. As well as other OpenStack specific patterns.<br></div><div data-signatureblock="true"><div><br></div><div>Regards,</div><div>Bogdan.</div><div><br></div></div><div style="padding-top: 5px; border-top-color: rgb(229, 229, 229); border-top-width: 1px; border-top-style: solid;"><div><font face=" 'Calibri', 'Segoe UI', 'Meiryo', 'Microsoft YaHei UI', 'Microsoft JhengHei UI', 'Malgun Gothic', 'sans-serif'" style='line-height: 15pt; letter-spacing: 0.02em; font-family: "Calibri", "Segoe UI", "Meiryo", "Microsoft YaHei UI", "Microsoft JhengHei UI", "Malgun Gothic", "sans-serif"; font-size: 12pt;'><b>Od:</b> <a href="mailto:rpodolyaka@mirantis.com" target="_parent">Roman Podoliaka</a><br><b>Wysłano:</b> piątek, 29 kwietnia 2016 21:04<br><b>Do:</b> <a href="mailto:openstack-dev@lists.openstack.org" target="_parent">OpenStack Development Mailing List (not for usage questions)</a><br><b>DW:</b> <a href="mailto:openstack-operators@lists.openstack.org" target="_parent">openstack-operators@lists.openstack.org</a></font></div></div><div><br></div><div dir=""><div id="readingPaneBodyContent">Hi Bogdan,<br><br>Thank you for sharing this! I'll need to familiarize myself with this<br>Jepsen thing, but overall it looks interesting.<br><br>As it turns out, we already run Galera in multi-writer mode in Fuel<br>unintentionally in the case, when the active MySQL node goes down,<br>HAProxy starts opening connections to a backup, then the active goes<br>up again, HAProxy starts opening connections to the original MySQL<br>node, but OpenStack services may still have connections opened to the<br>backup in their connection pools - so now you may have connections to<br>multiple MySQL nodes at the same time, exactly what you wanted to<br>avoid by using active/backup in the HAProxy configuration.<br><br>^ this actually leads to an interesting issue [1], when the DB state<br>committed on one node is not immediately available on another one.<br>Replication lag can be controlled via session variables [2], but that<br>does not always help: e.g. in [1] Nova first goes to Neutron to create<br>a new floating IP, gets 201 (and Neutron actually *commits* the DB<br>transaction) and then makes another REST API request to get a list of<br>floating IPs by address - the latter can be served by another<br>neutron-server, connected to another Galera node, which does not have<br>the latest state applied yet due to 'slave lag' - it can happen that<br>the list will be empty. Unfortunately, 'wsrep_sync_wait' can't help<br>here, as it's two different REST API requests, potentially served by<br>two different neutron-server instances.<br><br>Basically, you'd need to *always* wait for the latest state to be<br>applied before executing any queries, which Galera is trying to avoid<br>for performance reasons.<br><br>Thanks,<br>Roman<br><br>[1] https://bugs.launchpad.net/fuel/+bug/1529937<br>[2] http://galeracluster.com/2015/06/achieving-read-after-write-semantics-with-galera/<br><br>On Fri, Apr 22, 2016 at 10:42 AM, Bogdan Dobrelya<br><bdobrelia@mirantis.com> wrote:<br>> [crossposting to openstack-operators@lists.openstack.org]<br>><br>> Hello.<br>> I wrote this paper [0] to demonstrate an approach how we can leverage a<br>> Jepsen framework for QA/CI/CD pipeline for OpenStack projects like Oslo<br>> (DB) or Trove, Tooz DLM and perhaps for any integration projects which<br>> rely on distributed systems. Although all tests are yet to be finished,<br>> results are quite visible, so I better off share early for a review,<br>> discussion and comments.<br>><br>> I have similar tests done for the RabbitMQ OCF RA clusterers as well,<br>> although have yet wrote a report.<br>><br>> PS. I'm sorry for so many tags I placed in the topic header, should I've<br>> used just "all" :) ? Have a nice weekends and take care!<br>><br>> [0] https://goo.gl/VHyIIE<br>><br>> --<br>> Best regards,<br>> Bogdan Dobrelya,<br>> Irc #bogdando<br>><br>><br>><br>> __________________________________________________________________________<br>> OpenStack Development Mailing List (not for usage questions)<br>> Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br><br>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br></div></div></div>
</body>
</html>