<div dir="ltr"><div><div>Sorry to chime in but i will throw in another use case for Steven since is about the HA/ auto-scaling and i think it does match what i asked back in October.<br><br><a href="http://lists.openstack.org/pipermail/openstack-dev/2014-October/049375.html">http://lists.openstack.org/pipermail/openstack-dev/2014-October/049375.html</a><br><br></div>If you need more info let me know<br><br></div>Dani<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 2, 2015 at 10:59 AM, Huangtianhua <span dir="ltr"><<a href="mailto:huangtianhua@huawei.com" target="_blank">huangtianhua@huawei.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">If we replace a autoscaling group member, we can't make sure the attached resources keep the same, why not to call the evacuate or rebuild api of nova,<br>
just to add meters for ha(vm state or host state) in ceilometer, and then signal to HA resource(such as HARestarter)?<br>
<br>
-----邮件原件-----<br>
发件人: Steven Hardy [mailto:<a href="mailto:shardy@redhat.com">shardy@redhat.com</a>]<br>
发送时间: 2014年12月23日 2:21<br>
收件人: <a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a><br>
主题: [openstack-dev] [heat] Application level HA via Heat<br>
<div><div class="h5"><br>
Hi all,<br>
<br>
So, lately I've been having various discussions around $subject, and I know it's something several folks in our community are interested in, so I wanted to get some ideas I've been pondering out there for discussion.<br>
<br>
I'll start with a proposal of how we might replace HARestarter with AutoScaling group, then give some initial ideas of how we might evolve that into something capable of a sort-of active/active failover.<br>
<br>
1. HARestarter replacement.<br>
<br>
My position on HARestarter has long been that equivalent functionality should be available via AutoScalingGroups of size 1. Turns out that shouldn't be too hard to do:<br>
<br>
resources:<br>
server_group:<br>
type: OS::Heat::AutoScalingGroup<br>
properties:<br>
min_size: 1<br>
max_size: 1<br>
resource:<br>
type: ha_server.yaml<br>
<br>
server_replacement_policy:<br>
type: OS::Heat::ScalingPolicy<br>
properties:<br>
# FIXME: this adjustment_type doesn't exist yet<br>
adjustment_type: replace_oldest<br>
auto_scaling_group_id: {get_resource: server_group}<br>
scaling_adjustment: 1<br>
<br>
So, currently our ScalingPolicy resource can only support three adjustment types, all of which change the group capacity. AutoScalingGroup already supports batched replacements for rolling updates, so if we modify the interface to allow a signal to trigger replacement of a group member, then the snippet above should be logically equivalent to HARestarter AFAICT.<br>
<br>
The steps to do this should be:<br>
<br>
- Standardize the ScalingPolicy-AutoScaling group interface, so aynchronous adjustments (e.g signals) between the two resources don't use the "adjust" method.<br>
<br>
- Add an option to replace a member to the signal interface of AutoScalingGroup<br>
<br>
- Add the new "replace adjustment type to ScalingPolicy<br>
<br>
I posted a patch which implements the first step, and the second will be required for TripleO, e.g we should be doing it soon.<br>
<br>
<a href="https://review.openstack.org/#/c/143496/" target="_blank">https://review.openstack.org/#/c/143496/</a><br>
<a href="https://review.openstack.org/#/c/140781/" target="_blank">https://review.openstack.org/#/c/140781/</a><br>
<br>
2. A possible next step towards active/active HA failover<br>
<br>
The next part is the ability to notify before replacement that a scaling action is about to happen (just like we do for LoadBalancer resources<br>
already) and orchestrate some or all of the following:<br>
<br>
- Attempt to quiesce the currently active node (may be impossible if it's<br>
in a bad state)<br>
<br>
- Detach resources (e.g volumes primarily?) from the current active node,<br>
and attach them to the new active node<br>
<br>
- Run some config action to activate the new node (e.g run some config<br>
script to fsck and mount a volume, then start some application).<br>
<br>
The first step is possible by putting a SofwareConfig/SoftwareDeployment resource inside ha_server.yaml (using NO_SIGNAL so we don't fail if the node is too bricked to respond and specifying DELETE action so it only runs when we replace the resource).<br>
<br>
The third step is possible either via a script inside the box which polls for the volume attachment, or possibly via an update-only software config.<br>
<br>
The second step is the missing piece AFAICS.<br>
<br>
I've been wondering if we can do something inside a new heat resource, which knows what the current "active" member of an ASG is, and gets triggered on a "replace" signal to orchestrate e.g deleting and creating a VolumeAttachment resource to move a volume between servers.<br>
<br>
Something like:<br>
<br>
resources:<br>
server_group:<br>
type: OS::Heat::AutoScalingGroup<br>
properties:<br>
min_size: 2<br>
max_size: 2<br>
resource:<br>
type: ha_server.yaml<br>
<br>
server_failover_policy:<br>
type: OS::Heat::FailoverPolicy<br>
properties:<br>
auto_scaling_group_id: {get_resource: server_group}<br>
resource:<br>
type: OS::Cinder::VolumeAttachment<br>
properties:<br>
# FIXME: "refs" is a ResourceGroup interface not currently<br>
# available in AutoScalingGroup<br>
instance_uuid: {get_attr: [server_group, refs, 1]}<br>
<br>
server_replacement_policy:<br>
type: OS::Heat::ScalingPolicy<br>
properties:<br>
# FIXME: this adjustment_type doesn't exist yet<br>
adjustment_type: replace_oldest<br>
auto_scaling_policy_id: {get_resource: server_failover_policy}<br>
scaling_adjustment: 1<br>
<br>
By chaining policies like this we could trigger an update on the attachment resource (or a nested template via a provider resource containing many attachments or other resources) every time the ScalingPolicy is triggered.<br>
<br>
For the sake of clarity, I've not included the existing stuff like ceilometer alarm resources etc above, but hopefully it gets the idea accross so we can discuss further, what are peoples thoughts? I'm quite happy to iterate on the idea if folks have suggestions for a better interface etc :)<br>
<br>
One problem I see with the above approach is you'd have to trigger a failover after stack create to get the initial volume attached, still pondering ideas on how best to solve that..<br>
<br>
Thanks,<br>
<br>
Steve<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div>