<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cordia New";
panose-1:2 11 3 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cordia New";
panose-1:2 11 3 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I know there is some work in the XenAPI driver to make it resilient to these kinds of failures (to allow frequent updates of the nova code), and I think there were plans for the work to be reused in the Libvirt driver.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>AFAIK, in Essex and lower, bad things can happen if you don’t wait for all the tasks to finish. You may well be OK some of the time.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>It boils down to an issue of consuming the message from Rabbit but not completing the task, and not being able to recover from half completed tasks.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Hope that helps,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>John<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Igor Laskovy [mailto:igor.laskovy@gmail.com] <br><b>Sent:</b> 15 June 2012 11:31<br><b>To:</b> Christian Parpart<br><b>Cc:</b> John Garbutt; openstack-operators@lists.openstack.org; <,openstack@lists.launchpad.net>,<br><b>Subject:</b> Re: [Openstack-operators] Nova Controller HA issues<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p>I am using OpenStack for my little lab for a short time too))<o:p></o:p></p><p>Ok, you are right of course, but I meant a some another design when told about virtualization controller nodes.<o:p></o:p></p><p>It is can be only two dedicated hypetvisor with dedicated share/drbd between them. This hypervisors will be standalone, and not be part of nova. Than, maybe pacemaker or another tool can take availability function to restart VM to alive node when active will die.<o:p></o:p></p><p>Main question here - how worth can be if occurs controller nodes unexpected power off. In another word, when VM restart it will be in crash consisted state. <br>Will some nova services will loose here?<br>Will RabbiMQ loose some data here? (I am new to RabbitMQ too)<o:p></o:p></p><p>Igor Laskovy<br><a href="http://facebook.com/igor.laskovy">facebook.com/igor.laskovy</a><br>Kiev, Ukraine<o:p></o:p></p><div><p class=MsoNormal>On Jun 15, 2012 10:54 AM, "Christian Parpart" <<a href="mailto:trapni@gmail.com">trapni@gmail.com</a>> wrote:<o:p></o:p></p><p class=MsoNormal>Hey,<o:p></o:p></p><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>well, I said "I might be wrong" because I have no "clear" vision on how OpenStack works in<o:p></o:p></p></div><div><p class=MsoNormal>its deepest detail, however, I would not like to depend on a controller node that<o:p></o:p></p></div><div><p class=MsoNormal>is inside a virtual machine, controlled by compute nodes, that are controlled by the controller<o:p></o:p></p></div><div><p class=MsoNormal>node. This sounds quite like a chicken-and-egg problem.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>However, at the time of this writing, I think you'll have to have a working nova-scheduler process,<o:p></o:p></p></div><div><p class=MsoNormal>which is responsible on deciding on which compute node to spawn your VM (what else?),<o:p></o:p></p></div><div><p class=MsoNormal>and think about what you do when this (or all your controller-)VMs terribly die,<o:p></o:p></p></div><div><p class=MsoNormal>and you want to rebuild it, how do you plan to do this when your controller node is out-of-service?<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I in my case have put the controller services onto two compute nodes, and use Pacemaker<o:p></o:p></p></div><div><p class=MsoNormal>to switch between them, in case one node goes down, the other can take over (via shared service-IP).<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Again, these are my thoughts, and I am using OpenStack for just about a month now :-)<o:p></o:p></p></div><div><p class=MsoNormal>But I hope this helps a bit...<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Best regards,<o:p></o:p></p></div><div><p class=MsoNormal>Christian Parpart.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Fri, Jun 15, 2012 at 8:16 AM, Igor Laskovy <<a href="mailto:igor.laskovy@gmail.com" target="_blank">igor.laskovy@gmail.com</a>> wrote:<o:p></o:p></p><p>Why? Can you please clarify.<o:p></o:p></p><p><span style='color:#888888'>Igor Laskovy<br><a href="http://facebook.com/igor.laskovy" target="_blank">facebook.com/igor.laskovy</a><br>Kiev, Ukraine<o:p></o:p></span></p><div><div><div><p class=MsoNormal>On Jun 15, 2012 1:55 AM, "Christian Parpart" <<a href="mailto:trapni@gmail.com" target="_blank">trapni@gmail.com</a>> wrote:<o:p></o:p></p><p class=MsoNormal>I don't think putting the controller node completely into a VM is a good advice,<o:p></o:p></p><div><p class=MsoNormal>at least when speaking of nova-scheduler and nova-api (if central).<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I may be wrong, and if so, please correct me.<br><br>Christian.<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>On Thu, Jun 14, 2012 at 7:20 PM, Igor Laskovy <<a href="mailto:igor.laskovy@gmail.com" target="_blank">igor.laskovy@gmail.com</a>> wrote:<o:p></o:p></p><p class=MsoNormal>Hi, have any updates there?<br>Can anybody clarify what happens if controller nodes just going hard shutdown?<br><br>I thinking about solution with two hypervisors and putting controller<br>node in VM shared storage, which can be relaunched when active<br>hypervisor will die.<br>Any ideas, advise?<o:p></o:p></p><div><div><p class=MsoNormal><br><br>On Tue, Jun 12, 2012 at 3:52 PM, John Garbutt <<a href="mailto:John.Garbutt@citrix.com" target="_blank">John.Garbutt@citrix.com</a>> wrote:<br>> Sure, I get your point.<br>><br>> I think Florian is working on some docs to help on that.<br>><br>> Not sure how much has been done already.<br>><br>><br>><br>> Cheers,<br>><br>> John<br>><br>><br>><br>> From: Christian Parpart [mailto:<a href="mailto:trapni@gmail.com" target="_blank">trapni@gmail.com</a>]<br>> Sent: 12 June 2012 13:47<br>> To: John Garbutt<br>> Cc: <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>> Subject: Re: [Openstack-operators] Nova Controller HA issues<br>><br>><br>><br>> Hey, ya I also found this page, but didn't find it yet that helpful, it<br>> rather much sounds like a theoretical paper on<br>><br>> how they implemented it rather then telling me on how to actually make it<br>> happen (from the sysop point of view :-)<br>><br>><br>><br>> I hoped that someone had to face this already, since I really find it very<br>> unintuitive to realize, or need to wait until<br>><br>> I get more time to investigate dedicated. :-)<br>><br>><br>><br>> Regards,<br>><br>> Christian.<br>><br>> On Tue, Jun 12, 2012 at 12:52 PM, John Garbutt <<a href="mailto:John.Garbutt@citrix.com" target="_blank">John.Garbutt@citrix.com</a>><br>> wrote:<br>><br>> I thought Rabbit had a built in HA solution these days:<br>><br>> <a href="http://www.rabbitmq.com/ha.html" target="_blank">http://www.rabbitmq.com/ha.html</a><br>><br>><br>><br>> From: <a href="mailto:openstack-operators-bounces@lists.openstack.org" target="_blank">openstack-operators-bounces@lists.openstack.org</a><br>> [mailto:<a href="mailto:openstack-operators-bounces@lists.openstack.org" target="_blank">openstack-operators-bounces@lists.openstack.org</a>] On Behalf Of<br>> Christian Parpart<br>> Sent: 12 June 2012 09:59<br>> To: <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>> Subject: [Openstack-operators] Nova Controller HA issues<br>><br>><br>><br>> Hi all,<br>><br>><br>><br>> after spending the whole evening in making our cloud controller node highly<br>> available<br>><br>> using Corosync/Pacemaker, at which I am really proud about it, I am having<br>> just a few<br>><br>> problems left, and the one that freaks me out the most is rabbitmq-server.<br>><br>><br>><br>> That beast I just seem to find no good documenation on how to set<br>> rabbitmq-server up<br>><br>> properly for HA'ing.<br>><br>><br>><br>> Does anyone have ever tried to set a nova controller (including rabbitmq<br>> dependency) up for HAing?<br>><br>> If so, I'd be pleased to share experiences, especially to the latter part.<br>> :-)<br>><br>><br>><br>> Best regards,<br>><br>> Christian Parpart<br>><br>><br>><br>><o:p></o:p></p></div></div><p class=MsoNormal>> _______________________________________________<br>> Openstack-operators mailing list<br>> <a href="mailto:Openstack-operators@lists.openstack.org" target="_blank">Openstack-operators@lists.openstack.org</a><br>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>><br><span style='color:#888888'><br><br><br>--<br>Igor Laskovy<br>Kiev, Ukraine</span><o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p></div></div></div></div></div><p class=MsoNormal><o:p> </o:p></p></div></div></div></div></body></html>