Open Stack

On 04/02/14 20:34, Robert Collins wrote:
> On 5 February 2014 13:14, Zane Bitter <zbitter at redhat.com> wrote:
>
>
>> That's not a great example, because one DB server depends on the other,
>> forcing them into updating serially anyway.
>>
>> I have to say that even in general, this whole idea about applying update
>> policies to non-grouped resources doesn't make a whole lot of sense to me.
>> For non-grouped resources you control the resource definitions individually
>> - if you don't want them to update at a particular time, you have the option
>> of just not updating them.
>
> Well, I don't particularly like the idea of doing thousands of
> discrete heat stack-update calls, which would seem to be what you're
> proposing.

I'm not proposing you do it by hand if that's any help ;)

Ideally a workflow service would exist that could do the messy parts for 
you, but at the end of the day it's just a for-loop in your code. From 
what you say below, I think you started down the path of managing a lot 
of complexity yourself when you were forced to generate templates for 
server groups rather than use autoscaling. I think it would be better 
for _everyone_ for us to put resources into helping TripleO get off that 
path rather than it would for us to put resources into making it less 
inconvenient to stay on it.

> On groups: autoscale groups are a problem for secure minded
> deployments because every server has identical resources (today) and
> we very much want discrete credentials per server - at least this is
> my understanding of the reason we're not using scaling groups in
> TripleO.

OK, I wasn't aware that y'all are not using scaling groups. It sounds 
like this is the real problem we should be addressing, because everyone 
wants secure-minded deployments and nobody wants to have to manually 
define the configs for their 1000 all-but-identical servers. If we had a 
mechanism to ensure that every server in a scaling group could obtain 
its own credentials then it seems to me that the issue of whether to 
apply autoscaling-style rolling upgrades to manually-defined groups of 
resources becomes moot.

(Note: if anybody read that paragraph and started thinking "hey, we 
could make Turing-complete programmable template templates using the 
JSON equivalent of XSLT, please just stop right now kthx.)

>> Where you _do_ need it is for scaling groups where every server is based on
>> the same launch config, so you need a way to control the members
>> individually - by batching up operations (done), adding delays (done) or,
>> even better, notifications and callbacks.
>>
>> So it seems like doing 'rolling' updates for any random subset of resources
>> is effectively turning Heat into something of a poor-man's workflow service,
>> and IMHO that is probably a mistake.
>
> I mean to reply to the other thread, but here is just as good :) -
> heat as a way to describe the intended state, and heat takes care of
> transitions, is a brilliant model. It absolutely implies a bunch of
> workflows - the AWS update policy is probably the key example.

Absolutely. Orchestration works by building a workflow internally, which 
Heat then also executes. No disagreement there.

> Being able to gracefully, *automatically* work through a transition
> between two defined states, allowing the nodes in question to take
> care of their own needs along the way seems like a pretty core
> function to fit inside Heat itself. Its not at all the same as 'allow
> users to define abitrary workflows'.

That's fair and, I like to think, consistent with what I was suggesting 
below.

>> What we do need for all resources (not just scaling groups) is a way for the
>> user to say "for this particular resource, notify me when it has updated
>> (but, if possible, before we have taken any destructive actions on it), give
>> me a chance to test it and accept or reject the update". For example, when
>> you resize a server, give the user a chance to confirm or reject the change
>> at the VERIFY_RESIZE step (Trove requires this). Or when you replace a
>> server during an update, give the user a chance to test the new server and
>> either keep it (continue on and delete the old one) or not (roll back). Or
>> when you replace a server in a scaling group, notify the load balancer _or
>> some other thing_ (e.g. OpenShift broker node) that a replacement has been
>> created and wait for it to switch over to the new one before deleting the
>> old one. Or, of course, when you update a server to some new config, give
>> the user a chance to test it out and make sure it works before continuing
>> with the stack update. All of these use cases can, I think, be solved with a
>> single feature.
>>
>> The open questions for me are:
>> 1) How do we notify the user that it's time to check on a resource?
>> (Marconi?)
>
> This is the graceful update stuff I referred to in my mail to Clint -
> the proposal from hallway discussions in HK was to do this by
> notifying the server itself (that way we don't create a centralised
> point of fail). I can see though that in a general sense not all
> resources are servers. But - how about allowing to specify where to
> notify (and notifing is always by setting a value in metadata
> somewhere) - users can then pull that out themselves however they want
> to. Adding push notifications is orthogonal IMO - we'd like that for
> all metadata changes, for instance.

TBH I think I would be OK with it only working for servers. I can't 
think of a non-server case where this would be interesting (though if 
somebody else can, please speak now). I guess, say, a Trove instance is 
"server-like", but in that these kinds of notifications should be 
handled under the hood by Trove (using a Heat template containing an 
actual server) and not by the user anyway.

The idea of sending the notification to the server itself is an 
interesting one, and I can see it working well in many cases 
(particularly for a completely user-controlled client-server system like 
e.g. OpenShift). However, I can imagine some potential issues, 
especially in what I expect to be the most common case of a nunch of app 
servers behind a load balancer:

* Do we want to force the user to have an in-instance agent for managing 
the load balancer (I realise this is fine for TripleO, but in general...)?
* What if you're replacing a server and the old one is already dead for 
some reason? How do you get it out of the load balancer then?
* Do we want the server to contain credentials that would allow it to 
manipulate the load balancer?

Maybe the answer is that we continue to special-case load balancers, but 
this is already causing us problems (since the ecosystem has not yet 
standardised on the Neutron LBaaS) and it would be really nice if we 
could use a single mechanism for everything.

>> 2) How does the user ack/nack? (You're suggesting reusing WaitCondition, and
>> that makes sense to me.)
>
> The server would use a WaitCondition yes.
>
>> 3) How do we break up the operations so the notification occurs at the right
>> time? (With difficulty, but it should be do-able.)
>
> Just wrap the existing operations - if <should notify> then:
> notify-wait-do, otherwise just do.

Yeah, this is not an important discussion to have right here and now. 
Suffice it to say that there are some subtleties - e.g. the 
VERIFY_RESIZE thing happens in the _middle_ of an existing operation, so 
just wrapping the existing operations is, sadly, not sufficient. 
Something worth keeping in mind when it comes to implementation time.

>> 4) How does the user indicate for which resources they want to be notified?
>> (Inside an update_policy? Another new directive at the
>> type/properties/depends_on/update_policy level?)
>
> I would say per resource.

I agree (and, to be clear, all of the options listed were intended to be 
per-resource), but the exact syntax remains an open question.

cheers,
Zane.