[openstack-dev] [heat][sahara][magnum][tripleo] Scaling nested stack validation

Thomas Herve therve at redhat.com
Thu Nov 24 16:21:45 UTC 2016

On Wed, Nov 23, 2016 at 11:58 PM, Zane Bitter <zbitter at redhat.com> wrote:
> We discussed $SUBJECT at the summit as one of the main performance problems
> that people are running into when trying to create very large autoscaling
> groups, as projects like Sahara, Magnum, TripleO, OpenShift are wont to do.
> Of course, as we all know, validation happens synchronously, so it's prone
> to causing RPC timeouts that mean a hard failure of the parent stack.
> First the good news - I just committed this patch:
> https://review.openstack.org/#/c/400961/
> which should mean from now on that resources with identical definitions will
> not all be validated, and instead we'll just validate one representative
> one. In theory this should mean that autoscaling groups should now validate
> in constant rather than linear time. If anyone from one of the affected
> projects is able to confirm this, then I'd be happy to backport the patch to
> stable/newton. It really is very simple.
> The bad news here is for users of ResourceGroups with %index% substitution
> (*cough*TripleO*cough*) - this makes each resource definition unique, so it
> won't benefit from this fix. (Adding this to my mental list of reasons why
> index substitution is bad.)
> I also investigated another issue, which is that since the fix for
> https://bugs.launchpad.net/heat/+bug/1388140 landed (in Kilo) I believe we
> are validating nested stacks multiple times (specifically, m times, where m
> is the stack's depth in the tree):
>   root                     child                    grandchild
>   create
>    -> validate ----------> validate --------------> validate
>    -> Resource.create ===> create
>                             -> validate ----------> validate
>                             -> Resource.create ===> create
>                                                      -> validate
> The only good news here is that ResourceGroup is smart enough to make sure
> that it generates a nested stack with at most 1 resource to validate when
> validate() is called. (However, when the nested stack is created, and thus
> validated, it is of course full-sized.) Autoscaling groups make no such
> allowances, but the patch above should actually have the same effect. (We
> can't get rid of the special case for ResourceGroup though, because of index
> substitution.)
> An obvious fix would be to disable validation - or, more specifically,
> validation of _resources_ - on create/update for stacks that have a non-null
> owner_id (i.e. nested stacks), so that we had something like:
>   root                     child                    grandchild
>   create
>    -> validate ----------> validate --------------> validate
>    -> Resource.create ===> create
>                             -> Resource.create ===> create
> That would eliminate the duplication/triplication/multiplication of
> validation. It would also mean that we'd cut out the expensive part of
> ResourceGroup validation with index substitution, leaving only the cheap
> part.
> One downside is that in the ResourceGroup/index substitution case we'd be
> creating resources whose definitions hadn't _ever_ been validated. I _think_
> that's safe, in the sense that you'd just hear about errors later, as
> opposed to everything falling over in a heap, but it's difficult to be
> certain. Hearing about problems late is also not ideal (since it may cause
> otherwise-healthy siblings to be cancelled), but I would guess that heavy
> users like TripleO developers would say that it's worth the tradeoff.
> However, one other thing about this bothers me. The part of validation that
> we're keeping:
>    -> validate ----------> validate --------------> validate
> involves loading all of the nested stacks in memory at once (i.e. the thing
> we were not supposed to be doing any more in Kilo, in favour of farming
> nested stacks out over RPC.) As we discovered when we found out we were
> doing the same thing with outputs[1], this is a bit like hanging out a giant
> "Kick Me" sign for the OOM Killer.
> That's mitigated quite a lot by my patch though... we'll load the whole
> autoscaling group stack in memory, but if its members are themselves nested
> stacks we'll load only one of them. So the scaling tendencies will hopefully
> be dominated by the complexity of your templates more than than the size of
> your deployment. ResourceGroup is in a better position, because its nested
> stack will actually have only one member, so the size shouldn't affect
> memory consumption at all during validation.
> Some options:
> 1) Chalk it up to an acceptable tradeoff
> 2) Add a single-member special case for autoscaling group validation
> 3) Farm out the nested validation over RPC
> 4) Both (2) & (3)
> 5) Some totally different arrangement of how nested stacks are validated

I think I'd like to see what difference 3 makes. Maybe then also do 2.
Again, we really need to have some reproducible big template that we
can use to make sure what we're doing is useful.


More information about the OpenStack-dev mailing list