Hi, While testing magnum, a problem of upgrades came up - while work has been done to make kubernetes upgrades without interruption, operating system upgrades seem to be handled only partially. According to the documentation, two ways of upgrading system are available: - via specifying ostree_commit or ostree_remote labels in the cluster template used for upgrade - via specifying a new image in the cluster template used for upgrade The first one is specific to Fedora Atomic (and, while probably untested, seems to be mostly working with Fedora CoreOS) but it has some drawbacks. Firstly, due to base image staying the same we require this image for the life of the cluster, even if OS has already been upgraded. Secondly, using this method only upgrades existing instances and new instances (spawned via scaling cluster up) will not be upgraded. Thirdly, even if that is fixed I'm worried that at some point upgrading from old base image to some future ostree snapshot will fail (there is also cost associated with diff growing with each release). The second method, of specifying a new image in the cluster template used for upgrade, comes with an ugly warning about nodes not being drained properly before server rebuild (and it actually doesn't seem to be working anyway as the new image parameter is not being passed to the heat template on upgrade). This does however seem like a more valid approach in general. I'm not that familar with Heat, and the documentation of various OS::Heat::Software* resources seems inconclusive, but is there no way of executing some code before instance is rebuilt? If not, how are other projects and users handling this in general? -- Krzysztof Klimonda kklimonda@syntaxhighlighted.com