Open Stack

Excerpts from Zane Bitter's message of 2014-04-24 14:23:38 -0700:
> On 23/04/14 20:45, Robert Collins wrote:
> > Hi, we've got this summit session planned -
> > http://summit.openstack.org/cfp/details/428 which is really about
> > https://etherpad.openstack.org/p/heat-workflow-vs-convergence
> >
> > We'd love feedback and questions - this is a significant amount of
> > work, but work I (and many others based on responses so far) believe
> > it is needed to really take Heat to users and ops teams.
> >
> > Right now we're looking for both high and low level design and input.
> >
> > If you're an operator/user/developer of/with/around heat - please take
> > a couple of minutes to look - feedback inline in the etherpad, or here
> > on the list - whatever suits you.
> >
> > The basic idea is:
> >   - no changes needed to the heat template language etc
> 
> +1 for this part, definitely :)
> 
> >   - take a holistic view and fix the system's emergent properties by
> > using a different baseline architecture within it
> >   - ???
> >   - profit!
> 
> Thanks for writing this up Rob. This is certainly a more ambitious scale 
> of application to deploy than we ever envisioned in the early days of 
> Heat ;) But I firmly believe that what is good for TripleO will be great 
> for the rest of our users too. All of the observed issues mentioned are 
> things we definitely want to address.
> 
> I have a few questions about the specific architecture being proposed. 
> It's not clear to me what you mean by "call-stack style" in referring to 
> the current paradigm. Maybe you could elaborate on how the current style 
> and the "convergence style" differ.
> 
> Specifically, I am not clear on whether 'convergence' means:
>   (a) Heat continues to respect the dependency graph but does not stop 
> after one traversal, instead repeatedly processing it until (and even 
> after) the stack is complete; or
>   (b) Heat ignores the dependency graph and just throws everything 
> against the wall, repeating until it has all stuck.
> 

I think (c). We still have the graph driving "what to do next" so that
the things are more likely to stick. Also we don't want to do 10,000
instance creations if the database they need isn't going to come
available.

But we decouple "I need to do something" from "The user asked for
something" by allowing the convergence engine to act on notifications
from the observer engine. In addition to allowing more automated actions,
it should allow us to use finer grained locking because no individual
task will need to depend on the whole graph or stack. If an operator
comes along and changes templates or parameters, we can still complete
our outdated action. Eventually convergence will arrive at a state which
matches the desired stack.

> I also have doubts about the principle "Users should only need to 
> intervene with a stack when there is no right action that Heat can take 
> to deliver the current template+parameters". That sounds good in theory, 
> but in practice it's very hard to know when there is a right action Heat 
> can take and when there isn't. e.g. There are innumerable ways to create 
> a template that can _never_ actually converge, and I don't believe 
> there's a general way we can detect that, only the hard way: one error 
> type at a time, for every single resource type. Offering users a way to 
> control how and when that happens allows them to make the best decisions 
> for their particular circumstances - and hopefully a future WFaaS like 
> Mistral will make it easy to set up continuous monitoring for those who 
> require it. (Not incidentally, it also gives cloud operators an 
> opportunity to charge their users in proportion to their actual 
> requirements.)
> 

There are some obvious times where there _is_ a clear automated answer
that does not require me to defer to a user's special workflow. 503 or
429 (I know, not ratified yet) status codes mean I should retry after
maybe backing off a bit. If I get an ERROR state on a nova VM, I should
retry a few times before giving up.

The point isn't that we have all the answers, it is that there are
plenty of places where where we do have good answers that will serve
most users well.

> > This can be constrasted with many other existing attempts to design
> > solutions which relied on keeping the basic internals of heat as-is
> > and just tweaking things - an approach we don't believe will work -
> > the issues arise from the current architecture, not the quality of the
> > code (which is fine).
> 
> Some of the ideas that have been proposed in the past:
> 
> - Moving execution of operations on individual resources to a 
> distributed execution system using taskflow. (This should address the 
> scalability issue.)

This is a superset of that. The same work that was going to be required
there, will be required for this. We can't be loading the whole stack
just to do a single operation on a single resource.

> - Updating the stored template in real time during stack updates - this 
> is happening in Juno btw. (This will solve the problem of inability to 
> ever recover from an update failure. In theory, it would also make it 
> possible to interrupt a running update and make changes.)

This obsoletes that. We don't need to keep track if we adopt a convergence
model. The template that the user has asked for, is the template we
converge on. The diff between that and reality dictates the changes we
need to make. Wherever we're at with the convergence step that was last
triggered can just be cancelled by the new one.

Deletes are handled by comparing what we have versus what the template
says we should have. The observer engine would never let go of a nova
instance until it has both been removed by a template update, _AND_
it has actually gone away. So there's no need to keep the old template
around, just the old record of what we are responsible for.

> - Implementing a 'stack converge' operation that the user can trigger to 
> compare the actual state of the stack with the model and bring it back 
> into spec.
> 

This command would simply kick off a refresh of the convergence engine.

> It would be interesting to see some analysis on exactly how these 
> existing attempts fall down in trying to fulfil the goals, as well as 
> the specific points at which the proposed implementation differs.
>

The massively coarse grained lock is a huge problem on update. Consider
a situation where 10,000 servers are managed via rolling updates in a
single stack. 5,000 are doing "roleA" and 5,000 are doing "roleB". Say I
update the image for roleA, and that will take 3 hours before it rolls
the whole thing out. I now have a 3 hour window where I cannot change
_ANYTHING_ in the parameters for the servers doing roleB.

Also by loading the whole stack we've allowed resources to bleed into
other resource. Currently to read Metadata for a single item that
entails _a lot_ of queries to the database because we end up having to
load the entire stack. We can't continue that as stacks grow in size.

> Depending on the answers to the above questions, this proposal could be 
> anything between a modest reworking of those existing ideas and a 
> complete re-imagining of the entire concept of Heat. I'd very much like 
> to find out where along that spectrum it lies :)
>

I think the concept stays the same. Users define their end goal, and we
push toward it. The states and actions even stay largely the same. This is
merely a change in the way Heat itself operates to hopefully be more
scalable and more robust.

> BTW, it appears that the schedule you're suggesting involves assigning a 
> bunch of people unfamiliar with the current code base and having them 
> complete a ground-up rearchitecting of the whole engine, all within the 
> Juno development cycle (about 3.5 months). This is simply not consistent 
> with reality as I have observed it up to this point.
> 

That's life. We'll get what we can done and if it does not land for Juno,
so be it. We can't get it done without starting somewhere. We'd be quite
enthused to have other Heat developers working on it, and I think the
newcomers we're bringing on would also be able to make a positive impact
in other parts of Heat as they come up to speed as well.