Open Stack

Hi Pradeep,

Firstly, as discussed on IRC, I echo all of bnemec's concerns, this is not
well aligned with our stable branch policy[1], or the stable-maint
direction towards "critical bugfixes only"[2], so if possible I'd rather we
figured out a general way to solve this problem that doesn't involve
invasive/risky feature backports.

[1] http://lists.openstack.org/pipermail/openstack-dev/2016-March/090855.html
[2] http://lists.openstack.org/pipermail/openstack-dev/2016-May/094440.html

On Mon, May 16, 2016 at 03:33:29PM -0400, James Slagle wrote:
> On Mon, May 16, 2016 at 10:34 AM, Pradeep Kilambi <prad at redhat.com> wrote:
> > Hi Everyone:
> >
> > I wanted to start a discussion around considering backporting Aodh to
> > stable/liberty for upgrades. We have been discussing quite a bit on whats
> > the best way for our users to upgrade ceilometer alarms to Aodh when moving
> > from liberty to mitaka. A quick refresh on what changed, In Mitaka,
> > ceilometer alarms were replaced by Aodh. So only way to get alarms
> > functionality is use aodh. Now when the user kicks off upgrades from liberty
> > to Mitaka, we want to make sure alarms continue to function as expected
> > during the process which could take multiple days. To accomplish this I
> > propose the following approach:
> >
> > * Backport Aodh functionality to stable/liberty. Note, Aodh functionality is
> > backwards compatible, so with Aodh running, ceilometer api and client will
> > redirect requests to Aodh api. So this should not impact existing users who
> > are using ceilometer api or client.
> >
> > * As part of Aodh deployed via heat stack update, ceilometer alarms services
> > will be replaced by openstack-aodh-*. This will be done by the puppet apply
> > as part of stack convergence phase.
> >
> > * Add checks in the Mitaka pre upgrade steps when overcloud install kicks
> > off to check and warn the user to update to liberty + aodh to ensure aodh is
> > running. This will ensure heat stack update is run and, if alarming is used,
> > Aodh is running as expected.
> >
> > The upgrade scenarios between various releases would work as follows:
> >
> > Liberty -> Mitaka
> >
> > * Upgrade starts with ceilometer alarms running
> > * A pre-flight check will kick in to make sure Liberty is upgraded to
> > liberty + aodh with stack update
> > * Run heat stack update to upgrade to aodh
> > * Now ceilometer alarms should be removed and Aodh should be running
> > * Proceed with mitaka upgrade
> > * End result, Aodh continue to run as expected
> >
> > Liberty + aodh -> Mitaka:
> >
> > * Upgrade starts with Aodh running
> > * A pre-flight check will kick in to make sure Liberty is upgraded to Aodh
> > with stack update
> > * Confirming Aodh is indeed running, proceed with Mitaka upgrade with Aodh
> > running
> > * End result, Aodh continue to be run as expected
> >
> >
> > This seems to be a good way to get the upgrades working for aodh. Other less
> > effective options I can think of are:
> >
> > 1. Let the Mitaka upgrade kick off and do "yum update" which replace aodh
> > during migration, alarm functionality will be down until puppet converge
> > runs and configures Aodh. This means alarms will be down during upgrade
> > which is not ideal.
> >
> > 2. During Mitaka upgrades, replace with Aodh and add a bash script that
> > fully configures Aodh and ensures aodh is functioning. This will involve
> > significant work and results in duplicating everything puppet does today.
> 
> How much duplication would this really be? Why would it have to be in bash?
> 
> Could it be:
> 
> Liberty -> Mitaka
> 
> * Upgrade starts with ceilometer alarms running
> * Add a new hook for the first step of Mitaka upgrade that does:
>   ** sets up mitaka repos
>   ** migrates from ceilometer alarms to aodh, can use puppet
>   ** ensures aodh is running
> * Proceed with rest of mitaka upgrade

+1, I was thinking the same thing - I also don't get why it has to be bash,
surely we could have a script that can apply a puppet manifest that uses our
existing puppet profile/module implementation to bring up aodh?

If we can figure out a clean way to do that, we could add a pre-upgrade
interface to composable services that allows the same thing, e.g implement
a supportable upgrade workflow that can be reused vs a one-off workaround.

> At most, it seems we'd have to surround the puppet apply with some
> pacemaker commands to possibly set maintenance mode and migrate
> constraints.
> 
> The puppet manifest itself would just be the includes and classes for aodh.

+1

> One complication might be that the aodh packages from Mitaka might
> pull in new deps that required updating other OpenStack services,
> which we wouldn't yet want to do. That is probably worth confirming
> though.

It seems like we should at least investigate this approach before going
ahead with the backport proposed - I'll -2 the backports pending further
discussion and investigation into this alternate approach.

Thanks,

Steve