Hi Matt,<br><br>we already provide this feature in our enterprise version of the StackOps Distro. It only works with KVM + libvirt, but it works like a breeze. Still, we can do it due to a dirty hack on libvirt configuration. It would be great if we could standardize how to implement nova-compute failover features in the future. <br>
<br>May be it would be a good idea to work on a blueprint on this for next summit. Sadly a small company like StackOps we cannot dedicate much efforts to it, but we can show how we do it and may be give it back to the community with some help of others.<br>
<br>Cheers<br>Diego<br> -- <br><span style="border-collapse:separate;font-family:Times"><span style="border-collapse:collapse;font-family:arial,sans-serif"><div style="font-size:13px" align="left"><div><font><span style="font-family:Arial" lang="ES">Diego Parrilla<br>
<a href="http://www.stackops.com/" title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com
www.garrigues.com" style="color:rgb(7,77,143)" target="_blank"><span title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com"></span></a></span></font><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><b>CEO</b><font size="1"><br>
</font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"></span></span><b><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><a href="http://www.stackops.com/" target="_blank"><b>www.stackops.com</b></a> | </font></b><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><font size="1"> <a href="mailto:diego.parrilla@stackops.com" target="_blank">diego.parrilla@stackops.com</a></font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span style="font-size:10pt" lang="EN-GB"></span></b></b></font></span></span><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><font size="1"> | </font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><font size="1">+34 649 94 43 29 | <a>skype:diegoparrilla</a></font></font></span></span><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><a href="http://www.stackops.com/" target="_blank"><b><br>
</b></a></font></div></div><div style="font-size:13px"><font color="#004438" face="Arial"><b><p><span style="border-collapse:separate;color:rgb(0,0,0);font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span style="font-size:10pt" lang="EN-GB"><img src="http://stackops.s3-external-3.amazonaws.com/STACKOPSLOGO-ICON.png"></span></b></b></font></span></span></span></span></p>
</b></font></div></span></span><div><br></div><br>
<br><br><div class="gmail_quote">On Sat, Sep 8, 2012 at 12:31 AM, Matt Joyce <span dir="ltr"><<a href="mailto:matt.joyce@cloudscaling.com" target="_blank">matt.joyce@cloudscaling.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I agree largely with Diego on this.<br><br>As far as nova-compute failure is concerned it really falls to the instance owner to detect whether or not their instance has failed and initiate the failover actions they wish to employ. The world of false positives is tricky. That being said, providing an SOP for failover of instances in the event of a fallen nova-compute host is absolutely in our interest to define and add to documentation.<span class="HOEnZb"><font color="#888888"><br>
<br>-Matt</font></span><div class="HOEnZb"><div class="h5"><br><br><div class="gmail_quote">On Thu, Sep 6, 2012 at 12:38 PM, Diego Parrilla Santamaría <span dir="ltr"><<a href="mailto:diego.parrilla.santamaria@gmail.com" target="_blank">diego.parrilla.santamaria@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>I think you are mixing live migration of VMs and compute node fail over. Live migration assumes your compute nodes are running healthy (or healthy enough) and you want to move VMs around in order to perform updates or fix problems on specific nodes. </div>
<div><br></div>Failure detection of a compute node seems to be easy, but it's not. There are a ton of events that could raise a false positive and the logic to filter these events and perform a recovery could lead to VM duplication: the same VM running in two compute nodes at the same time. That's the reason we let our customers to use our APIs to implement their own fail detection process and then trigger the fail over action.<div>
<br></div><div>Compute node failover means one or more compute nodes fail and you need to replace the failing compute nodes with some spare server you have around preconfigured in your datacenter. If the problem in the compute node is not complete, you might live migrate the VMs first, and then replace the compute node. This is not a complex task with Openstack. </div>
<div><br></div><div>But, if the failure in the compute node is complete, then you need to replace the compute node and recover the existing configuration of the node as your previous node. The state of your VMs are lost, but Openstack can spin them up again. The latter needs not only shared storage, but some logic to recover the OS configuration, nova-compute configuration, and the most important thing: libvirt configuration (if you are running KVM, of course).</div>
<div><br></div><div>Cheers,</div><div>Diego</div><div> -- <br><span style="border-collapse:separate;font-family:Times"><span style="border-collapse:collapse;font-family:arial,sans-serif"><div style="font-size:13px" align="left">
<div><font><span style="font-family:Arial" lang="ES">Diego Parrilla<br><a href="http://www.stackops.com/" title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com
www.garrigues.com" style="color:rgb(7,77,143)" target="_blank"><span title="file:///C:/Documents%20and%20Settings/carolina.capsir.per1/Application%20Data/Microsoft/Signatures/www.garrigues.com"></span></a></span></font><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><b>CEO</b><font size="1"><br>
</font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"></span></span><b><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><a href="http://www.stackops.com/" target="_blank"><b>www.stackops.com</b></a> | </font></b><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><font size="1"> <a href="mailto:diego.parrilla@stackops.com" target="_blank">diego.parrilla@stackops.com</a></font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span style="font-size:10pt" lang="EN-GB"></span></b></b></font></span></span><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><font size="1"> | </font></font><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><font size="1"><a href="tel:%2B34%20649%2094%2043%2029" value="+34649944329" target="_blank">+34 649 94 43 29</a> | <a>skype:diegoparrilla</a></font></font></span></span><font style="font-family:arial,helvetica,sans-serif" size="2" face="Arial"><a href="http://www.stackops.com/" target="_blank"><b><br>
</b></a></font></div></div><div style="font-size:13px"><font color="#004438" face="Arial"><b><p><span style="border-collapse:separate;font-size:medium;font-family:Times"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><span style="border-collapse:separate;font-family:Times;font-size:medium"><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font color="#004438" face="Arial"><b><b><span style="font-size:10pt" lang="EN-GB"><img src=""></span></b></b></font></span></span></span></span></p>
</b></font></div></span></span><div><div><div><br></div><br>
<br><br><div class="gmail_quote">On Thu, Sep 6, 2012 at 9:16 PM, Paul Walton <span dir="ltr"><<a href="mailto:paul.d.walton@gmail.com" target="_blank">paul.d.walton@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Well, I guess my real question next is, what happens when a VM is running on a compute node and that node simply fails for whatever reason? Does OpenStack have any way to detect a compute node failure, and attempt to migrate its VMs to another node? I realize that the real VM and the migrated VM may be out of sync with each other, but I'm assuming that a relatively recent version of the VM is still present on the distributed file system to facilitate migration.<br>
<br>Obviously, regular backups would be required for a real disaster recovery, but the VMs I'm talking about won't be changing much, and being a few minutes out of sync won't be a problem.<div>
<div><br><br><div class="gmail_quote">
On Thu, Sep 6, 2012 at 1:32 PM, Joe Topjian <span dir="ltr"><<a href="mailto:joe.topjian@cybera.ca" target="_blank">joe.topjian@cybera.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Paul,<div><br></div><div>OpenStack does not have anything like this yet. Anton Beloglazov has created a blueprint from some research work he's done that would provide such a feature:</div><div><br></div><div><a href="https://blueprints.launchpad.net/nova/+spec/dynamic-consolidation-of-virtual-machines" target="_blank">https://blueprints.launchpad.net/nova/+spec/dynamic-consolidation-of-virtual-machines</a></div>
<div><br></div><div>Thanks,</div><div>Joe<br><br><div class="gmail_quote"><div><div>On Thu, Sep 6, 2012 at 12:26 PM, Paul Walton <span dir="ltr"><<a href="mailto:paul.d.walton@gmail.com" target="_blank">paul.d.walton@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>Hello,<br><br>I have a quick question about live migration of VMs. I can see how to do this manually from the command line, i.e. if I want to move a VM from one compute node to another myself, but I was wondering if there is a way to have OpenStack do this automatically for me to balance the load across all the compute nodes dynamically?<br>
<br>Thanks<span><font color="#888888"><br clear="all"><br>-- <br><br>Paul Walton<br><br>University of Arkansas<br>College of Engineering<br>CSCE Technical Support Team<br>J.B. Hunt Building, Room 440<br>
</font></span><br></div></div>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br>Joe Topjian<div>Systems Administrator</div><div>Cybera Inc.</div><div><br></div><div><a href="http://www.cybera.ca" target="_blank">www.cybera.ca</a></div>
<div><br></div><div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">Big data is coming to Canada. Join the welcome wagon.</div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<b>Cyber Summit 2012</b></div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">October 1-3, Banff</div><div style="color:rgb(34,34,34);font-size:13px;font-family:arial,sans-serif">
<a href="http://www.cybera.ca/summit2012" style="color:rgb(17,85,204)" target="_blank">www.cybera.ca/summit2012</a></div></div><div><br></div><div><font color="#666666"><span>Cybera</span><span> is a not-for-profit organization that works to spur and support innovation, for the economic benefit of Alberta, through the use of cyberinfrastructure.</span></font></div>
<br>
</div>
</blockquote></div><br><br clear="all"><br>-- <br><br>Paul Walton<br><br>University of Arkansas<br>College of Engineering<br>CSCE Technical Support Team<br>J.B. Hunt Building, Room 440<br>
</div></div><br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br></div></div></div>
<br>_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br></blockquote></div><br>
</div></div></blockquote></div><br>