<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">It would be great if each OpenStack component could provide a maintenance mode like this… there was some work being considered on
Cells <a href="https://blueprints.launchpad.net/nova/+spec/disable-child-cell-support">
https://blueprints.launchpad.net/nova/+spec/disable-child-cell-support</a> which would have allowed parts of Nova to indicate they were in maintenance.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Something generic would be very useful. Some operators have asked for ‘read-only’ modes also where query is OK but update is not
permitted.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Tim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Mike Scherbakov [mailto:mscherbakov@mirantis.com]
<br>
<b>Sent:</b> 09 September 2014 23:20<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [All] Maintenance mode in OpenStack during patching/upgrades<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">Sergii, Clint,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">to rephrase what you are saying - there are might be situations when our OpenStack API will not be responding, as simply services would be down for upgrade.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Do we want to support it somehow? For example, if we know that Nova is going to be down, can we respond with HTTP 503 with appropriate Retry-After time in header?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">The idea is not simply deny or hang requests from clients, but provide them "we are in maintenance mode, retry in X seconds"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">> <span style="font-size:9.5pt;font-family:"Arial","sans-serif"">Turbo Hipster was added to the gate</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial","sans-serif"">great idea, I think we should use it in Fuel too</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial","sans-serif"">> You probably would want 'nova host-servers-migrate <host>'</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:"Arial","sans-serif"">yeah for migrations - but as far as I understand, it doesn't help with disabling this host in scheduler - there is can be a chance that some workloads will be scheduled to the
host.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Sep 9, 2014 at 6:02 PM, Clint Byrum <<a href="mailto:clint@fewbar.com" target="_blank">clint@fewbar.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Excerpts from Mike Scherbakov's message of 2014-09-09 00:35:09 -0700:<br>
> Hi all,<br>
> please see below original email below from Dmitry. I've modified the<br>
> subject to bring larger audience to the issue.<br>
><br>
> I'd like to split the issue into two parts:<br>
><br>
> 1. Maintenance mode for OpenStack controllers in HA mode (HA-ed<br>
> Keystone, Glance, etc.)<br>
> 2. Maintenance mode for OpenStack computes/storage nodes (no HA)<br>
><br>
> For first category, we might not need to have maintenance mode at all. For<br>
> example, if we apply patching/upgrade one by one node to 3-node HA cluster,<br>
> 2 nodes will serve requests normally. Is that possible for our HA solutions<br>
> in Fuel, TripleO, other frameworks?<br>
<br>
You may have a broken cloud if you are pushing out an update that<br>
requires a new schema. Some services are better than others about<br>
handling old schemas, and can be upgraded before doing schema upgrades.<br>
But most of the time you have to do at least a brief downtime:<br>
<br>
* turn off DB accessing services<br>
* update code<br>
* run db migration<br>
* turn on DB accessing services<br>
<br>
It is for this very reason, I believe, that Turbo Hipster was added to<br>
the gate, so that deployers running against the upstream master branches<br>
can have a chance at performing these upgrades in a reasonable amount of<br>
time.<br>
<br>
><br>
> For second category, can not we simply do "nova-manage service disable...",<br>
> so scheduler will simply stop scheduling new workloads on particular host<br>
> which we want to do maintenance on?<br>
><br>
<br>
You probably would want 'nova host-servers-migrate <host>' at that<br>
point, assuming you have migration set up.<br>
<br>
<a href="http://docs.openstack.org/user-guide/content/novaclient_commands.html" target="_blank">http://docs.openstack.org/user-guide/content/novaclient_commands.html</a><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
> On Thu, Aug 28, 2014 at 6:44 PM, Dmitry Pyzhov <<a href="mailto:dpyzhov@mirantis.com">dpyzhov@mirantis.com</a>> wrote:<br>
><br>
> > All,<br>
> ><br>
> > I'm not sure if it deserves to be mentioned in our documentation, this<br>
> > seems to be a common practice. If an administrator wants to patch his<br>
> > environment, he should be prepared for a temporary downtime of OpenStack<br>
> > services. And he should plan to perform patching in advance: choose a time<br>
> > with minimal load and warn users about possible interruptions of service<br>
> > availability.<br>
> ><br>
> > Our current implementation of patching does not protect from downtime<br>
> > during the patching procedure. HA deployments seems to be more or less<br>
> > stable. But it looks like it is possible to schedule an action on a compute<br>
> > node and get an error because of service restart. Deployments with one<br>
> > controller... well, you won’t be able to use your cluster until the<br>
> > patching is finished. There is no way to get rid of downtime here.<br>
> ><br>
> > As I understand, we can get rid of possible issues with computes in HA.<br>
> > But it will require migration of instances and stopping of nova-compute<br>
> > service before patching. And it will make the overall patching procedure<br>
> > much longer. Do we want to investigate this process?<br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Mike Scherbakov<br>
#mihgen<o:p></o:p></p>
</div>
</div>
</div>
</div>
</body>
</html>