Open Stack

On Tue, May 17, 2016 at 4:59 AM, Steven Hardy <shardy at redhat.com> wrote:
> Hi Pradeep,
>
> Firstly, as discussed on IRC, I echo all of bnemec's concerns, this is not
> well aligned with our stable branch policy[1], or the stable-maint
> direction towards "critical bugfixes only"[2], so if possible I'd rather we
> figured out a general way to solve this problem that doesn't involve
> invasive/risky feature backports.
>
> [1] http://lists.openstack.org/pipermail/openstack-dev/2016-March/090855.html
> [2] http://lists.openstack.org/pipermail/openstack-dev/2016-May/094440.html
>
> On Mon, May 16, 2016 at 03:33:29PM -0400, James Slagle wrote:
>> On Mon, May 16, 2016 at 10:34 AM, Pradeep Kilambi <prad at redhat.com> wrote:
>> > Hi Everyone:
>> >
>> > I wanted to start a discussion around considering backporting Aodh to
>> > stable/liberty for upgrades. We have been discussing quite a bit on whats
>> > the best way for our users to upgrade ceilometer alarms to Aodh when moving
>> > from liberty to mitaka. A quick refresh on what changed, In Mitaka,
>> > ceilometer alarms were replaced by Aodh. So only way to get alarms
>> > functionality is use aodh. Now when the user kicks off upgrades from liberty
>> > to Mitaka, we want to make sure alarms continue to function as expected
>> > during the process which could take multiple days. To accomplish this I
>> > propose the following approach:
>> >
>> > * Backport Aodh functionality to stable/liberty. Note, Aodh functionality is
>> > backwards compatible, so with Aodh running, ceilometer api and client will
>> > redirect requests to Aodh api. So this should not impact existing users who
>> > are using ceilometer api or client.
>> >
>> > * As part of Aodh deployed via heat stack update, ceilometer alarms services
>> > will be replaced by openstack-aodh-*. This will be done by the puppet apply
>> > as part of stack convergence phase.
>> >
>> > * Add checks in the Mitaka pre upgrade steps when overcloud install kicks
>> > off to check and warn the user to update to liberty + aodh to ensure aodh is
>> > running. This will ensure heat stack update is run and, if alarming is used,
>> > Aodh is running as expected.
>> >
>> > The upgrade scenarios between various releases would work as follows:
>> >
>> > Liberty -> Mitaka
>> >
>> > * Upgrade starts with ceilometer alarms running
>> > * A pre-flight check will kick in to make sure Liberty is upgraded to
>> > liberty + aodh with stack update
>> > * Run heat stack update to upgrade to aodh
>> > * Now ceilometer alarms should be removed and Aodh should be running
>> > * Proceed with mitaka upgrade
>> > * End result, Aodh continue to run as expected
>> >
>> > Liberty + aodh -> Mitaka:
>> >
>> > * Upgrade starts with Aodh running
>> > * A pre-flight check will kick in to make sure Liberty is upgraded to Aodh
>> > with stack update
>> > * Confirming Aodh is indeed running, proceed with Mitaka upgrade with Aodh
>> > running
>> > * End result, Aodh continue to be run as expected
>> >
>> >
>> > This seems to be a good way to get the upgrades working for aodh. Other less
>> > effective options I can think of are:
>> >
>> > 1. Let the Mitaka upgrade kick off and do "yum update" which replace aodh
>> > during migration, alarm functionality will be down until puppet converge
>> > runs and configures Aodh. This means alarms will be down during upgrade
>> > which is not ideal.
>> >
>> > 2. During Mitaka upgrades, replace with Aodh and add a bash script that
>> > fully configures Aodh and ensures aodh is functioning. This will involve
>> > significant work and results in duplicating everything puppet does today.
>>
>> How much duplication would this really be? Why would it have to be in bash?
>>
>> Could it be:
>>
>> Liberty -> Mitaka
>>
>> * Upgrade starts with ceilometer alarms running
>> * Add a new hook for the first step of Mitaka upgrade that does:
>>   ** sets up mitaka repos
>>   ** migrates from ceilometer alarms to aodh, can use puppet
>>   ** ensures aodh is running
>> * Proceed with rest of mitaka upgrade
>
> +1, I was thinking the same thing - I also don't get why it has to be bash,
> surely we could have a script that can apply a puppet manifest that uses our
> existing puppet profile/module implementation to bring up aodh?
>
> If we can figure out a clean way to do that, we could add a pre-upgrade
> interface to composable services that allows the same thing, e.g implement
> a supportable upgrade workflow that can be reused vs a one-off workaround.
>

Thanks Steve. I was under the impression we cannot run puppet at this
stage. Hence my suggestion to run bash or some script here, but if we
can find a way to easily wire the existing aodh puppet manifests into
the upgrade process and get aodh up and running then even better, we
dont have to duplicate what puppet gives us already and reuse that.


>> At most, it seems we'd have to surround the puppet apply with some
>> pacemaker commands to possibly set maintenance mode and migrate
>> constraints.
>>
>> The puppet manifest itself would just be the includes and classes for aodh.
>
> +1
>
>> One complication might be that the aodh packages from Mitaka might
>> pull in new deps that required updating other OpenStack services,
>> which we wouldn't yet want to do. That is probably worth confirming
>> though.
>
> It seems like we should at least investigate this approach before going
> ahead with the backport proposed - I'll -2 the backports pending further
> discussion and investigation into this alternate approach.
>

Makes sense to me. I understand the hesitation behind backports. I'm
happy to work with jistr and slagle to see if this is a viable
alaternative. If we can get this working without too much effort, i'm
all for dumping the backports and going with this.


Thanks,
~ Prad




> Thanks,
>
> Steve
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev