<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jul 23, 2014 at 7:33 PM, Tim Simpson <span dir="ltr"><<a href="mailto:tim.simpson@rackspace.com" target="_blank">tim.simpson@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt">
<div>To summarize, this is a conversation about the following LaunchPad bug: <a href="https://launchpad.net/bugs/1325512" style="font-family:'Times New Roman';font-size:16px;line-height:18.399999618530273px;text-align:justify;text-decoration:none" target="_blank"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);text-decoration:underline;vertical-align:baseline">https://launchpad.net/bugs/1325512</span></a></div>
<div>and Gerrit review: <a href="https://review.openstack.org/#/c/97194/6" target="_blank">https://review.openstack.org/#/c/97194/6</a></div>
<div><br>
</div>
<div><span style="font-size:10pt">You are saying the function "_service_is_active" in addition to polling the datastore service status also polls the status of the Nova resource. At first I thought this wasn't the case, however looking at your pull request
I was surprised to see on line 320 (</span><a href="https://review.openstack.org/#/c/97194/6/trove/taskmanager/models.py" style="font-size:10pt" target="_blank">https://review.openstack.org/#/c/97194/6/trove/taskmanager/models.py</a><span style="font-size:10pt">)
polls Nova using the "get" method (which I wish was called "refresh" as to me it sounds like a lazy-loader or something despite making a full GET request each time). </span></div>
<div>So moving this polling out of there into the two respective "create_server" methods as you have done is not only going to be useful for Heat and avoid the issue of calling Nova 99 times you describe but it will actually help operations teams to see more
clearly that the issue was with a server that didn't provision. We actually had an issue in Staging the other day that took us forever to figure out because the </div></div></div></blockquote><div><br></div><div>Agreed, i guess i would need to update bug-report to add more info about given issue, but i'm really glad to hear that proposed change would be useful. And i agree, that from operation/support team would be useful to track provisioning issues that has nothing common with Trove but tight to infrastructure.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt"><div>server wasn't provisioning, but before anything checked that it was ACTIVE the DNS code detected
the server had no ip address (never mind it was in a FAILED state) so the logs surfaced this as a DNS error. This change should help us avoid such issues.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Tim</div>
<div>
<div style="text-align:justify"><font face="Arial"><span style="font-size:15px;line-height:17.25px"><br>
</span></font></div>
<div style="text-align:justify"><font face="Arial"><span style="font-size:15px;line-height:17.25px"><br>
</span></font></div>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>From:</b> Denis Makogon [<a href="mailto:dmakogon@mirantis.com" target="_blank">dmakogon@mirantis.com</a>]<br>
<b>Sent:</b> Wednesday, July 23, 2014 7:30 AM<br>
<b>To:</b> OpenStack Development Mailing List<br>
<b>Subject:</b> [openstack-dev] [Trove] Guest prepare call polling mechanism issue<br>
</font><br>
</div><div><div class="h5">
<div></div>
<div>
<div dir="ltr">
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Hello, Stackers.<br>
</span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline"><br>
</span></p>
<p style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">I’d like to discuss guestagent prepare call polling mechanism
issue (see [1]).</span></p>
<br>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Let me first describe why this is actually an issue and
why it should be fixed. For those of you who is familiar with Trove knows that Trove can provision instances through Nova API and Heat API (see [2] and see [3]).</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">
</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">
</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">What’s the difference between this two ways (in
general)? The answer is simple:</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">- Heat-based provisioning method has polling mechanism
that verifies that stack provisioning was completed with successful state (see [4]) which means that all stack resources are in ACTIVE state.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">- Nova-based provisioning method doesn’t do any polling
(which is wrong, since instance can’t fail as fast as possible because Trove-taskmanager service doesn’t verify that launched server had reached ACTIVE state. That’s the issue #1 - compute instance state is unknown, but</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">
right after resources (deliverd by heat) already in ACTIVE states.<br>
</span></p>
<br>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Once one method [2] or [3] finished, taskmanager trying
to prepare data for guest (see [5]) and then it tries to send prepare call to guest (see [6]). Here comes issue #2 - polling mechanism does at least 100 API calls to Nova to define compute instance status.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Also taskmanager does almost the same amount of calls
to Trove backend to discover guest status which is totally normal.</span></p>
<br>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">
</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">So, here comes the question, why should i call
99 times Nova for the same value if the value asked for the first time was completely acceptable?</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">
</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">
</span><span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">There’s only one way to fix it. Since heat-based
provisioning delivers instance with status validation procedure, the same thing should be done for nova-base provisioning (we should extract compute instance status polling from guest prepare polling mechanism and integrate it into [2]) and leave only guest
status discovering in guest prepare polling mechanism.</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">
</span></p>
<br>
<br>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Benefits? Proposed fix will give an ability for fast-failing
for corrupted instances, it would reduce amount of redundant Nova API calls while attempting to discover guest status.</span></p>
<br>
<br>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-indent:36pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Proposed fix for this issue - [7].</span></p>
<br>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[1] -
</span><a href="https://launchpad.net/bugs/1325512" style="text-decoration:none" target="_blank"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">https://launchpad.net/bugs/1325512</span></a></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[2] -
</span><a href="https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L198-L215" style="text-decoration:none" target="_blank"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L198-L215</span></a></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[3] -
</span><a href="https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L190-L197" style="text-decoration:none" target="_blank"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L190-L197</span></a></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[4] -
</span><a href="https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L420-L429" style="text-decoration:none" target="_blank"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L420-L429</span></a></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[5] -
</span><a href="https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L217-L256" style="text-decoration:none" target="_blank"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L217-L256</span></a></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[6] -
</span><a href="https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L254-L266" style="text-decoration:none" target="_blank"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">https://github.com/openstack/trove/blob/master/trove/taskmanager/models.py#L254-L266</span></a></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">[7] -
</span><a href="https://review.openstack.org/#/c/97194/" style="text-decoration:none" target="_blank"><span style="font-size:15px;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline">https://review.openstack.org/#/c/97194/</span></a></p>
<br>
<br>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Thoughts?</span></p>
<br>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Best regards,</span></p>
<p dir="ltr" style="line-height:1.15;margin-top:0pt;margin-bottom:0pt;text-align:justify">
<span style="font-size:15px;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:normal;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline">Denis Makogon</span></p>
</div>
</div>
</div></div></div>
</div>
</div>
</div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>