<div dir="ltr">Hi,<br><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr">On Fri, Apr 3, 2015 at 1:31 AM, Zane Bitter <span dir="ltr"><<a href="mailto:zbitter@redhat.com" target="_blank">zbitter@redhat.com</a>></span> wrote:<br></div></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">A few of us have been looking for a way to perform software updates to servers in a TripleO Heat/Puppet-based overcloud that avoids an impedance mismatch with Heat concepts and how Heat runs its workflow. As many talented TripleO-ers who have gone before can probably testify, that's surprisingly difficult to do, but we did come up with an idea that I think might work and which I'd like to get wider feedback on. For clarity, I'm speaking here in the context of the new overcloud-without-mergepy templates.<br>
<br>
The idea is that we create a SoftwareConfig that, when run, can update some software on the server. (The exact mechanism for the update is not important for this discussion; suffice to say that in principle it could be as simple as "[yum|apt-get] update".) The SoftwareConfig would have at least one input, though it need not do anything with the value.<br>
<br>
Then each server has that config deployed to it with a SoftwareDeployment at the time it is created. However, it is set to execute only on the UPDATE action. The value of (one of) the input(s) is obtained from a parameter.<br>
<br>
As a result, we can trigger the software update by simply changing the value of the input parameter, and the regular Heat dependency graph will be respected. The actual input value could be by convention a uuid, a timestamp, a random string, or just about anything so long as it changes.<br>
<br>
Here's a trivial example of what this deployment might look like:<br>
<br>
update_config:<br>
type: OS::Heat::SoftwareConfig<br>
properties:<br>
config: {get_file: do_sw_update.sh}<br>
inputs:<br>
- name: update_after_time<br>
description: Timestamp of the most recent update request<br>
<br>
update_deployment:<br>
type: OS::Heat::SoftwareDeployment<br>
properties:<br>
actions:<br>
- UPDATE<br>
config: {get_resource: update_config}<br>
server: {get_resource: my_server}<br>
input_values:<br>
update_after_time: {get_param: update_timestamp}<br>
<br>
<br>
(A possible future enhancement is that if you keep a mapping between previous input values and the system state after the corresponding update, you could even automatically handle rollbacks in the event the user decided to cancel the update.)<br>
<br>
And now we should be able to trigger an update to all of our servers, in the regular Heat dependency order, by simply (thanks to the fact that parameters now keep their previous values on stack updates unless they're explicitly changed) running a command like:<br>
<br>
heat stack-update my_overcloud -f $TMPL -P "update_timestamp=$(date)"<br>
<br>
(A future goal of Heat is to make specifying the template again optional too... I don't think that change landed yet, but in this case we can always obtain the template from Tuskar, so it's not so bad.)<br>
<br>
<br>
Astute readers may have noticed that this does not actually solve our problem. In reality groups of similar servers are deployed within ResourceGroups and there are no dependencies between the members. So, for example, all of the controller nodes would be updated in parallel, with the likely result that the overcloud could be unavailable for some time even if it is deployed with HA.<br>
<br>
The good news is that a solution to this problem is already implemented in Heat: rolling updates. For example, the controller node availability problem can be solved by setting a rolling update batch size of 1. The bad news is that rolling updates are implemented only for AutoscalingGroups, not ResourceGroups.<br></blockquote><div><br></div><div>It seems we should implement rolling_updates for ResourceGroup, and even more, rolling_create too. Just a couple of days ago we had a chat with Sahara's PTL, Sergey Lukjanov, and he was asking if there is a way to create a number of resources in batches, but with a single call. Sahara does not need autoscaling, so our idea was exactly that - rolling_create and rolling_update for ResourceGroups should solve this. Thus we have one more use case for that, I'm going to raise a spec (or bug?) soon.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
Accordingly, I propose that we switch the implementation of overcloud-without-mergepy from ResourceGroups to AutoscalingGroups. This would be a breaking change for overcloud updates (although no worse than the change from merge.py over to overcloud-without-mergepy), but that also means that there'll never be a better time than now to make it.<br>
<br>
I suspect that some folks (Tomas?) have possibly looked into this in the past... can anybody identify any potential obstacles to the change? Two candidates come to mind:<br>
<br>
1) The SoftwareDeployments (plural) resource type. I believe we carefully designed that to work with both ResourceGroup and AutoscalingGroup though.<br>
2) The elision feature (<a href="https://review.openstack.org/#/c/128365/" target="_blank">https://review.openstack.org/<u></u>#/c/128365/</a>). Steve, I think this was only implemented for ResourceGroup? An AutoscalingGroup version of this should be feasible though, or do we have better ideas for how to solve it in that context?<br>
<br>
cheers,<br>
Zane.<br>
<br>
______________________________<u></u>______________________________<u></u>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.<u></u>openstack.org?subject:<u></u>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote></div><br></div><div class="gmail_extra">Best regards,</div><div class="gmail_extra"><div><div class="gmail_signature"><div dir="ltr">Pavlo Shchelokovskyy<div>Software Engineer</div><div>Mirantis Inc</div><div><a href="http://www.mirantis.com/" target="_blank">www.mirantis.com</a></div></div></div></div><br></div></div>