Open Stack

In summary, we have decided to use the usage notification events emitted by heat (http://tinyurl.com/oultjm5). This should allow us to detect stack action completions, and errors (with reasons) which should be enough for a Bay implementation that does not need to poll. Stack timeouts can be passed to heat as parameters. This is not as elegant as what Angus and Zane suggested because it requires Magnum to access all RPC messages. With their proposed approach we could use a Zaqar queue that only emits the stack events relevant to that user/tenant. This conforms better to the principle of least privilege. Our preference for the decided approach is that it allows us tight integration with Heat using functionality that exists today. We admit that this implementation will be harder to debug than other options.

More remarks in-line.

On Mar 11, 2015, at 2:27 AM, Jay Lau <jay.lau.513 at gmail.com> wrote:

> 
> 
> 2015-03-10 23:21 GMT+08:00 Hongbin Lu <hongbin034 at gmail.com>:
> Hi Adrian,
> 
> On Mon, Mar 9, 2015 at 6:53 PM, Adrian Otto <adrian.otto at rackspace.com> wrote:
> Magnum Team,
> 
> In the following review, we have the start of a discussion about how to tackle bay status:
> 
> https://review.openstack.org/159546
> 
> I think a key issue here is that we are not subscribing to an event feed from Heat to tell us about each state transition, so we have a low degree of confidence that our state will match the actual state of the stack in real-time. At best, we have an eventually consistent state for Bay following a bay creation.
> 
> Here are some options for us to consider to solve this:
> 
> 1) Propose enhancements to Heat (or learn about existing features) to emit a set of notifications upon state changes to stack resources so the state can be mirrored in the Bay resource.
>  
> A drawback of this option is that it increases the difficulty of trouble-shooting. In my experience of using Heat (SoftwareDeployments in particular), Ironic and Trove, one of the most frequent errors I encountered is that the provisioning resources stayed in deploying state (never went to completed). The reason is that they were waiting a callback signal from the provisioning resource to indicate its completion, but the callback signal was blocked due to various reasons (e.g. incorrect firewall rules, incorrect configs, etc.). Troubling-shooting such problem is generally harder.
> I think that the "heat convergence" is working on the issues for your concern: https://wiki.openstack.org/wiki/Heat/Blueprints/Convergence  

Yes, that wold address part of the concern, but not all of it. Depending on the implementation of state exchange, and the configuration of the network, it may be possible to create an installation of the system that does not function at all due to network ACLs, and almost no clue to the implementer about why it does not work. To mitigate this concern, we would want an implementation that does not require a bi-directional network trust relationship between Heat and Magnum. Instead, implement it in a way that connections are only made from Magnum to Heat, so if the stack creation call succeeds, so can the status updates. Perhaps we can revisit this design discussion in a future iteration of our Bay status code.

Adrian

> 
> 2) Spawn a task to poll the Heat stack resource for state changes, and express them in the Bay status, and allow that task to exit once the stack reaches its terminal (completed) state.
> 
> 3) Don’t store any state in the Bay object, and simply query the heat stack for status as needed. 
> 
> Are each of these options viable? Are there other options to consider? What are the pro/con arguments for each?
> 
> Thanks,
> 
> Adrian
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> -- 
> Thanks,
> 
> Jay Lau (Guangya Liu)
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev