Open Stack

On 11/10/16 02:24, Rabi Mishra wrote:
>
>             1.2
>             For large stacks, 1000 nodes, the final call to heat to
>             fetch the
>             IPs for all nodes takes 3 to 4 minutes. In heat, the stack
>             has status
>             CREATE_COMPLETE but magnum's state is updated when this long
>             final
>             call is done. Can we do better? Maybe fetch only the master
>             IPs or
>             get he IPs in chunks.
>
>
>
> We seem load the nested stacks in memory to retrieve their outputs. That
> would probably explain the
> behaviour above, where you load all the nested stacks for the nodes to
> fetch their ips. There is some
> work[1][2] happening atm to change that.
>
> [1] https://review.openstack.org/#/c/383839/
> <https://review.openstack.org/#/c/383839/>
> [2] https://review.openstack.org/#/c/384718
> <https://review.openstack.org/#/c/384718>

That's a problem, but in Magnum's case I believe the tree is pretty 
shallow. The real problem there is that we dynamically resolve intrinsic 
functions (like get_attr) in outputs - meaning that in the case 
mentioned we will make 1000 calls to Nova to get the IP address of each 
node, every time you show the stack.

This is nuts not only from a performance standpoint and also for simple 
consistency of data. I think we should move to storing the data at the 
end up a stack create/update, so that we can return it instantly when we 
show the stack. (I do worry about breaking users who may be relying on 
live updates to that data, but I think we *have* to make this change 
sometime.) With convergence we already have a lot of the code we'd need 
in place - we already know how to grab all the attributes we need at the 
time a resource is created/updated and store them to the SyncPoint, and 
we already have a SyncPoint for when the whole stack is complete. It's 
just a matter of finding a longer-term place to cache that data, since 
SyncPoints are pretty short-lived at the moment.

I definitely think we should discuss this in the session.

cheers,
Zane.