<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>On May 22, 2013, at 7:01 AM, "Day, Phil" <<a href="mailto:philip.day@hp.com">philip.day@hp.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div lang="EN-GB" link="blue" vlink="purple" style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="WordSection1" style="page: WordSection1; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks Chris,<o:p></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Yep we do have the scheduler_host_subset_size set (I think that was one of our contributions), and we are getting the kick back from the compute nodes when the resource tracker kicks in, so all of that is working as it should do. I’ve been a bit wary of bumping the retry count too high as we’ve also seen a number of errors bouncing thought host due to other errors (such as the quantum port quota issue), but I might look at bumping it up a tad.</span></div></div></div></blockquote><div><br></div><div>Yeah. I think we need some better logic based on the type of exception that occurs. You may want to possibly retry forever (maybe not forever, but a lot more than 3 times -- although maybe forever is OK -- worst case you loop through all your hosts and they're all full :) if the exception comes from the resource tracker. It's doing its job properly and it's kicking back the message quickly. But other exceptions can occur later… possibly things like bad images, etc. You probably want to retry a few times just to make sure, but you don't want to retry very long because it's likely never going to succeed and you want to make sure don't have an instance sitting in 'building' forever.</div><div><br></div><div>As I said, I think this can get better with conductor. If we have conductor responsible for building, we can break up the process to make things easier. We can do things like:</div><div><br></div><div>[conductor]</div><div>def _get_host_from_scheduer():</div><div> [query scheduler and return a host we've not tried with this request]</div><div><br></div><div>def _allocate_networks():</div><div> return nwinfo</div><div><br></div><div>def _tell_compute_to_assign_instance():</div><div> [on the compute side, this does resource_tracker checking and assigns instance['host']]</div><div><br></div><div>def _tell_compute_to_build():</div><div> [compute downloads image and builds]</div><div>def _tell_compute_to_run_instance()</div><div><br></div><div>Then build instance logic could be:</div><div><br></div><div>with try_2_times():</div><div> _allocate_networks()</div><div>with try_2_times():</div><div> with try_forever():</div><div> _get_host_from_scheduler()</div><div> _tell_compute_to_assign_instance()</div><div> _tell_compute_to_build()</div><div> _tell_compute_to_run_instance()</div><div><br></div><div>Something like that if you know what I mean. This would happen to address one of the other issues with retries… they deallocate and reallocate networks every time you have to retry due to resource tracker failure.</div><div><br></div><blockquote type="cite"><div lang="EN-GB" link="blue" vlink="purple" style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="WordSection1" style="page: WordSection1; "><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I’m also thinking about a small mod to the scheduler_host_subset_size code that adds a scheduler_host_subset_offest, so that you could for example have one scheduler picking from the top 10 hosts, and another picking from 11-20. That won’t guarantee there’s never an overlap, but I think it would reduce it considerably. It would also mean that if you do loose a scheduler the most hosts that are no longer scheduled to becomes scheduler_host_subset_size.</span></div></div></div></blockquote><div><br></div><div>That's an interesting thought. Although I'd like to make things "just work" without having to explicitly configure something like this. :)</div><div><br></div><div>- Chris</div><div><br></div></div></body></html>