<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Thu, Aug 21, 2014 at 1:17 AM, Tim Bell <span dir="ltr"><<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="#0563C1" vlink="#954F72">
<div>
<p class="MsoNormal">Michael has been posting very informative blogs on the summary of the mid-cycle meetups for Nova. The one on the Nova Network to Neutron migration was of particular interest to me as it raises a number of potential impacts for the CERN
production cloud. The blog itself is at <a href="http://www.stillhq.com/openstack/juno/000014.html" target="_blank">
http://www.stillhq.com/openstack/juno/000014.html</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I would welcome suggestions from the community on the approach to take and areas that the nova/neutron team could review to limit the impact on the cloud users.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">For some background, CERN has been running nova-network in flat DHCP mode since our first Diablo deployment. We moved to production for our users in July last year and are currently supporting around 70,000 cores, 6 cells, 100s of projects
and thousands of VMs. Upgrades generally involve disabling the API layer while allowing running VMs to carry on without disruption. Within the time scale of the migration to Neutron (M release at the latest), these numbers are expected to double.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">For us, the concerns we have with the ‘cold’ approach would be on the user impact and operational risk of such a change. Specifically,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p><u></u><span>1.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>A big bang approach of shutting down the cloud, upgrade and the resuming the cloud would cause significant user disruption<u></u><u></u></p>
<p><u></u><span>2.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>The risks involved with a cloud of this size and the open source network drivers would be difficult to mitigate through testing and could lead to site wide downtime<u></u><u></u></p>
<p><u></u><span>3.<span style="font:7.0pt "Times New Roman"">
</span></span><u></u>Rebooting VMs may be possible to schedule in batches but would need to be staggered to keep availability levels<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Note, we are not looking to use Neutron features initially, just to find a functional equivalent of the flat DHCP network.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">We would appreciate suggestions on how we could achieve a smooth migration for the simple flat DHCP models.</p><span class="HOEnZb"><font color="#888888"><p class="MsoNormal"><u></u></p></font></span></div>
</div></blockquote></div><div class="gmail_extra"><br></div>Thanks for sending this Tim. Sorry for my slow reply, a day long meeting and some international travel got in the way. When we originally talked, I said I needed to understand more of the background to your need for a "zero downtime" upgrade. That said...</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Mark McClain and I discussed a possible plan for nova-network to neutron upgrades at the Ops Meetup today, and it seemed generally acceptable. It defines a "cold migration" as freezing the ability to create or destroy instances during the upgrade, and then requiring a short network outage for each instance in the cell.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">This is why I'm trying to understand the "no downtime" use case better. Is it literally no downtime, ever? Or is it a more simple "no simultaneous downtime for instances"?</div>
<div class="gmail_extra"><br></div><div class="gmail_extra">Michael<br clear="all"><div><br></div>-- <br>Rackspace Australia
</div></div>