[openstack-dev] [heat] [magnum] Subjects to discuss during the summit
Zane Bitter
zbitter at redhat.com
Tue Oct 11 13:32:09 UTC 2016
On 11/10/16 02:24, Rabi Mishra wrote:
>
> 1.2
> For large stacks, 1000 nodes, the final call to heat to
> fetch the
> IPs for all nodes takes 3 to 4 minutes. In heat, the stack
> has status
> CREATE_COMPLETE but magnum's state is updated when this long
> final
> call is done. Can we do better? Maybe fetch only the master
> IPs or
> get he IPs in chunks.
>
>
>
> We seem load the nested stacks in memory to retrieve their outputs. That
> would probably explain the
> behaviour above, where you load all the nested stacks for the nodes to
> fetch their ips. There is some
> work[1][2] happening atm to change that.
>
> [1] https://review.openstack.org/#/c/383839/
> <https://review.openstack.org/#/c/383839/>
> [2] https://review.openstack.org/#/c/384718
> <https://review.openstack.org/#/c/384718>
That's a problem, but in Magnum's case I believe the tree is pretty
shallow. The real problem there is that we dynamically resolve intrinsic
functions (like get_attr) in outputs - meaning that in the case
mentioned we will make 1000 calls to Nova to get the IP address of each
node, every time you show the stack.
This is nuts not only from a performance standpoint and also for simple
consistency of data. I think we should move to storing the data at the
end up a stack create/update, so that we can return it instantly when we
show the stack. (I do worry about breaking users who may be relying on
live updates to that data, but I think we *have* to make this change
sometime.) With convergence we already have a lot of the code we'd need
in place - we already know how to grab all the attributes we need at the
time a resource is created/updated and store them to the SyncPoint, and
we already have a SyncPoint for when the whole stack is complete. It's
just a matter of finding a longer-term place to cache that data, since
SyncPoints are pretty short-lived at the moment.
I definitely think we should discuss this in the session.
cheers,
Zane.
More information about the OpenStack-dev
mailing list