<div dir="ltr">Hi Corey,<div><br></div><div>What is the steps when we do zero downtime upgrades? Can you explain it in detail? Or is there any docs for it?</div><div><br></div><div>Best Regards</div><div>Wanghua</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Feb 27, 2016 at 9:26 PM, Corey O'Brien <span dir="ltr"><<a href="mailto:coreypobrien@gmail.com" target="_blank">coreypobrien@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">We discussed this at the midcycle as a part of zero downtime upgrades. In order to be able to do Magnum upgrades, we talked about directing all DB access through a service that understands the schema for both the current and previous versions of the database. Moving direct DB access into the API will make it harder to do zero downtime upgrades in the future. I think we should leave things the way they are today.</p><span class="HOEnZb"><font color="#888888">
<p dir="ltr">Corey</p></font></span><div class="HOEnZb"><div class="h5">
<br><div class="gmail_quote"><div dir="ltr">On Fri, Feb 26, 2016, 22:08 王华 <<a href="mailto:wanghua.humble@gmail.com" target="_blank">wanghua.humble@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>I want to allow magnum-api to access DB, so that magnum-api can work even if magnum-conductor is down. </div><div><br></div><div>Best Regards,</div><div>Wanghua </div></div><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 4, 2016 at 3:42 PM, 王华 <span dir="ltr"><<a href="mailto:wanghua.humble@gmail.com" target="_blank">wanghua.humble@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I think we should allow magnum-api to access DB directly like nova-api.<div><br></div><div>As describe in [1], nova may have many compute nodes and it may take an hour or a month to upgrade. But the number of magnum-api and magnum-conductor is limited, the upgrade of them is fast. They don't benefit from the method. We should upgrade them like the control services in nova and upgrade them together.</div><div><br></div><div><span style="color:rgb(210,216,222);font-family:Verdana,Helvetica,sans-serif;font-size:14px;line-height:18px;text-align:justify;background-color:rgb(71,76,82)">In this step, you will upgrade everything but the compute nodes. This means nova-api, nova-scheduler, nova-conductor, nova-consoleauth, nova-network, and nova-cert. In reality, this needs to be done fairly atomically. So, shut down all of the affected services, roll the new code, and start them back up. This will result in some downtime for your API, but in reality, it should be easy to quickly perform the swap. In later releases, we’ll reduce the pain felt here by eliminating the need for the control services to go together.</span><br></div><div><br></div><div>[1] <a href="http://www.danplanet.com/blog/2015/06/26/upgrading-nova-to-kilo-with-minimal-downtime/" target="_blank">http://www.danplanet.com/blog/2015/06/26/upgrading-nova-to-kilo-with-minimal-downtime/</a></div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 4, 2016 at 4:59 AM, Hongbin Lu <span dir="ltr"><<a href="mailto:hongbin.lu@huawei.com" target="_blank">hongbin.lu@huawei.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-CA" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I can clarify Eli’s question further.<u></u><u></u></span></p><span>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span lang="EN-US">1) is this by designed that we don't allow magnum-api to access DB directly
</span><span lang="ZH-CN" style="font-family:SimSun">？</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
</span><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yes, that is what it is. Actually, The magnum-api was allowed to access DB directly in before. After the indirection API patch landed [1], magnum-api starts
using magnum-conductor as a proxy to access DB. According to the inputs from oslo team, this design allows operators to take down either magnum-api or magnum-conductor to upgrade. This is not the same as nova-api, because nova-api, nova-scheduler, and nova-conductor
are assumed to be shutdown all together as an atomic unit.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I think we should make our own decision here. If we can pair magnum-api with magnum-conductor as a unit, we can remove the indirection API and allow both binaries
to access DB. This could mitigate the potential performance bottleneck of message queue. On the other hand, if we stay with the current design, we would allow magnum-api and magnum-conductor to scale independently. Thoughts?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">[1]
<a href="https://review.openstack.org/#/c/184791/" target="_blank">https://review.openstack.org/#/c/184791/</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hongbin<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Kumari, Madhuri [mailto:<a href="mailto:madhuri.kumari@intel.com" target="_blank">madhuri.kumari@intel.com</a>]
<br>
<b>Sent:</b> February-03-16 10:57 AM<span><br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
</span></span></p><div><div><b>Subject:</b> Re: [openstack-dev] [Magnum] API service won't work if conductor down?<u></u><u></u></div></div><p></p>
</div>
</div><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Corey the one you are talking about has changed to coe-service-*.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Eli, IMO we should display proper error message. M-api service should only have read permission.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Madhuri<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><a name="-1673912655_msg-f:1527295501943997483_15106837_-1268257133______replyseparator"></a><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Corey O'Brien [<a href="mailto:coreypobrien@gmail.com" target="_blank">mailto:coreypobrien@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, February 3, 2016 6:50 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<b>Subject:</b> Re: [openstack-dev] [Magnum] API service won't work if conductor down?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">The service-* commands aren't related to the magnum services (e.g. magnum-conductor). The service-* commands are for services on the bay that the user creates and deletes.<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Corey<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Wed, Feb 3, 2016 at 2:25 AM Eli Qiao <<a href="mailto:liyong.qiao@intel.com" target="_blank">liyong.qiao@intel.com</a>> wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">hi<br>
Whey I try to run magnum service-list to list all services (seems now we only have m-cond service), it m-cond is down(which means no conductor at all),<br>
API won't response and will return a timeout error.<br>
<br>
taget@taget-ThinkStation-P300:~/devstack$ magnum service-list<br>
ERROR: Timed out waiting for a reply to message ID fd1e9529f60f42bf8db903bbf75bbade (HTTP 500)<br>
<br>
And I debug more and compared with nova service-list, nova will give response and will tell the conductor is down.<br>
<br>
and deeper I get this in magnum-api boot up:<br>
<i><br>
# Enable object backporting via the conductor<br>
base.MagnumObject.indirection_api = base.MagnumObjectIndirectionAPI()</i><br>
<br>
so in magnum_service api code<br>
<br>
return objects.MagnumService.list(context, limit, marker, sort_key,<br>
sort_dir)<br>
<br>
will require to use magnum-conductor to access DB, but no magnum-conductor at all, then we get a 500 error.<br>
(nova-api doesn't specify <i>indirection_api so nova-api can access DB</i>)<br>
<br>
My question is:<br>
<br>
1) is this by designed that we don't allow magnum-api to access DB directly </span>
<span lang="ZH-CN" style="font-family:SimSun">？</span><span lang="EN-US"><br>
2) if 1) is by designed, then `magnum service-list` won't work, and the error message should be improved such as "magnum service is down , please check magnum conductor is alive"<br>
<br>
What do you think?<br>
<br>
P.S. I tested comment this line:<br>
<i># base.MagnumObject.indirection_api = base.MagnumObjectIndirectionAPI()</i><br>
magnum-api will response but failed to create bay(), which means api service have read access but can not write it at all since(all db write happened in conductor layer).<br>
<br>
<u></u><u></u></span></p>
<pre><span lang="EN-US">-- <u></u><u></u></span></pre>
<pre><span lang="EN-US">Best Regards, Eli(Li Yong)Qiao<u></u><u></u></span></pre>
<pre><span lang="EN-US">Intel OTC China<u></u><u></u></span></pre>
</div>
<p class="MsoNormal"><span lang="EN-US">__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><u></u><u></u></span></p>
</blockquote>
</div>
</div></div></div>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div>
</div></div><br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>