Open Stack

On Mon, May 16, 2016 at 12:25 PM, Ben Nemec <openstack at nemebean.com> wrote:

> From what I've seen of the IRC discussion it sounds like we're doing
> this, but can we at least agree that it is a bad way to handle service
> replacement?
>
> First, we are completely replacing a service on a minor upgrade, which
> even if it is a 100% compatible drop-in replacement may have
> implications for deployers around things like their monitoring setup.
> To me this is a pretty big violation of user expectations around what
> happens in a minor upgrade.
>

I completely understand the concern here and I fully agree that doing this
in a minor update is not something that stable branch policy agrees with,
even if its fully backward compatible. I'm open to suggestions.


>
> Second, if I'm understanding the problem we're trying to solve here, it
> means that on major upgrades we are leaving the controller services
> upgraded but unconfigured for the duration of the upgrade.  That means
> if something happens and pacemaker restarts a service then all hell may
> break loose.  I've actually run into this problem on undercloud upgrades
> because the package update restarts the services automatically, and if
> we haven't run puppet yet sometimes the old configs don't work with the
> new service.
>


This is exactly the problem. If we can come up with a way to handle this,
we dont need to backport. Would it be possible to run puppet after the
controller upgrade runs and then re-run after compute upgrades again?
Considering puppet runs should be idempotent?

 It would be good to prioritize handling this use case as sooner or later
we will run into this again.


>
> I understand that there are technical reasons things work the way they
> do today, but I also want to be on record that this is a problem to be
> solved, not a precedent to be followed in the future.
>
>
Thanks,
~ Prad


> On 05/16/2016 09:34 AM, Pradeep Kilambi wrote:
> > Hi Everyone:
> >
> > I wanted to start a discussion around considering backporting Aodh to
> > stable/liberty for upgrades. We have been discussing quite a bit on
> > whats the best way for our users to upgrade ceilometer alarms to Aodh
> > when moving from liberty to mitaka. A quick refresh on what changed, In
> > Mitaka, ceilometer alarms were replaced by Aodh. So only way to get
> > alarms functionality is use aodh. Now when the user kicks off upgrades
> > from liberty to Mitaka, we want to make sure alarms continue to function
> > as expected during the process which could take multiple days. To
> > accomplish this I propose the following approach:
> >
> > * Backport Aodh functionality to stable/liberty. Note, Aodh
> > functionality is backwards compatible, so with Aodh running, ceilometer
> > api and client will redirect requests to Aodh api. So this should not
> > impact existing users who are using ceilometer api or client.
> >
> > * As part of Aodh deployed via heat stack update, ceilometer alarms
> > services will be replaced by openstack-aodh-*. This will be done by the
> > puppet apply as part of stack convergence phase.
> >
> > * Add checks in the Mitaka pre upgrade steps when overcloud install
> > kicks off to check and warn the user to update to liberty + aodh to
> > ensure aodh is running. This will ensure heat stack update is run and,
> > if alarming is used, Aodh is running as expected.
> >
> > The upgrade scenarios between various releases would work as follows:
> >
> > *Liberty -> Mitaka
> > *
> > * Upgrade starts with ceilometer alarms running
> > * A pre-flight check will kick in to make sure Liberty is upgraded to
> > liberty + aodh with stack update
> > * Run heat stack update to upgrade to aodh
> > * Now ceilometer alarms should be removed and Aodh should be running
> > * Proceed with mitaka upgrade
> > * End result, Aodh continue to run as expected
> >
> > *Liberty + aodh -> Mitaka:
> > *
> > * Upgrade starts with Aodh running
> > * A pre-flight check will kick in to make sure Liberty is upgraded to
> > Aodh with stack update
> > * Confirming Aodh is indeed running, proceed with Mitaka upgrade with
> > Aodh running
> > * End result, Aodh continue to be run as expected
> >
> >
> > This seems to be a good way to get the upgrades working for aodh. Other
> > less effective options I can think of are:
> >
> > 1. Let the Mitaka upgrade kick off and do "yum update" which replace
> > aodh during migration, alarm functionality will be down until puppet
> > converge runs and configures Aodh. This means alarms will be down during
> > upgrade which is not ideal.
> >
> > 2. During Mitaka upgrades, replace with Aodh and add a bash script that
> > fully configures Aodh and ensures aodh is functioning. This will involve
> > significant work and results in duplicating everything puppet does today.
> >
> > If there are any suggestions please let me know. I'm open to any
> > approach that can save us time and effort to get this working.
> >
> > Otherwise if we can agree to consider Aodh backported to Liberty i think
> > this will save us time.
> >
> > Let me know what you guys think.
> >
> > Cheers,
> > - Prad
> >
> >
> >
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160516/1f051806/attachment.html>