Open Stack

I agree with Mike's concerns, and propose to make these limitations (4
weeks before FF for OS upgrades, 2 weeks for upgrades of key
dependencies -- RabbitMQ, MCollective, Puppet, MySQL, PostgreSQL,
anything else?) official for 10.0/Newton. 

For 9.0/Mitaka, it is too late to impose them, so we just have to be
very careful and conservative with this upgrade. First of all, we need
to have a green BVT before and after switching to the CentOS 7.2 repo
snapshot, so while I approved the spec, we can't move forward with this
until BVT is green again, and right now it's red:

https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/

If we get it back to green but it becomes red after the upgrade, you
must switch back to CentOS 7.1 *immediately*. If you are able to stick
to this plan, there is still time to complete the transition today
without requiring an FFE.

-- 
Dmitry Borodaenko


On Wed, Mar 02, 2016 at 05:53:53PM +0000, Mike Scherbakov wrote:
> Formally, we can merge it today. Historically, every update of OS caused us
> instability for some time: from days to a couple of month.
> Taking this into account and number of other exceptions requested, overall
> stability of code, my opinion would be to postpone this to 10.0.
> 
> Also, I'd suggest to change the process, and have freeze date for all OS
> updates no later than a month before official FF date. This will give us
> time to stabilize, and ensure that base on which all new code is being
> developed is stable when approaching FF.
> 
> I'd also propose to have freeze for major upgrades of 3rd party packages no
> later than 2 weeks before FF, which Fuel depends heavily upon. For
> instance, such will include RabbitMQ, MCollective, Puppet.
> 
> On Wed, Mar 2, 2016 at 7:34 AM Igor Marnat <imarnat at mirantis.com> wrote:
> 
> > Igor,
> > couple of points from my side.
> >
> > CentOS 7.2 will be getting updates for several more months, and we have
> > snapshots and all the mechanics in place to switch to the next version when
> > needed.
> >
> > Speaking of getting this update into 9.0, we actually don't need FFE, we
> > can merge remaining staff today. It has enough reviews, so if you add your
> > +1 today, we don't need FFE.
> >
> > https://review.openstack.org/#/c/280338/
> > https://review.fuel-infra.org/#/c/17400/
> >
> >
> >
> > Regards,
> > Igor Marnat
> >
> > On Wed, Mar 2, 2016 at 6:23 PM, Dmitry Teselkin <dteselkin at mirantis.com>
> > wrote:
> >
> >> Igor,
> >>
> >> Your statement about updates for 7.2 isn't correct - it will receive
> >> updates,  because it's the latest release ATM. There is *no* pinning inside
> >> ISO, and the only place where it was 8.0 were docker containers just
> >> because we had to workaround some issues. But there are no docker
> >> containers in 9.0, so that's not the case.
> >> The proposed solution to switch to CentOS-7.2 in fact is based on
> >> selecting the right snapshot with packages. There is no pinning in ISO (it
> >> was in earlier versions of the spec but was removed).
> >>
> >> On Wed, Mar 2, 2016 at 6:11 PM, Igor Kalnitsky <ikalnitsky at mirantis.com>
> >> wrote:
> >>
> >>> Dmitry, Igor,
> >>>
> >>> > Very important thing is that CentOS 7.1 which master node is based now
> >>> > don't get updates any longer.
> >>>
> >>> If you are using "fixed" release you must be ready that you won't get
> >>> any updates. So with CentOS 7.2 the problem still the same.
> >>>
> >>> However, let's wait for Fuel PTL decision. I only shared my POV:
> >>> that's not a critical feature, and taking into account the risks of
> >>> regression - I'd prefer to do not accept it in 9.0.
> >>>
> >>> Regards,
> >>> Igor
> >>>
> >>> On Wed, Mar 2, 2016 at 4:42 PM, Igor Marnat <imarnat at mirantis.com>
> >>> wrote:
> >>> > Igor,
> >>> > please note that this is pretty much not like update of master node
> >>> which we
> >>> > had in 8.0. This is minor _update_ of CentOS from 7.1 to 7.2 which team
> >>> > tested for more than 2 months already.
> >>> >
> >>> > We don't expect it to require any additional efforts from core or qa
> >>> team.
> >>> >
> >>> > Very important thing is that CentOS 7.1 which master node is based now
> >>> don't
> >>> > get updates any longer. Updates are only provided for CentOS 7.2.
> >>> >
> >>> > So we'll have to switch CentOS 7.1 to CentOS 7.2 anyways.
> >>> >
> >>> > We can do it now for more or less free, later in release cycle for
> >>> higher
> >>> > risk and QA efforts and after the release for 2x price because of
> >>> additional
> >>> > QA cycle we'll need to pass through.
> >>> >
> >>> >
> >>> >
> >>> > Regards,
> >>> > Igor Marnat
> >>> >
> >>> > On Wed, Mar 2, 2016 at 2:57 PM, Dmitry Teselkin <
> >>> dteselkin at mirantis.com>
> >>> > wrote:
> >>> >>
> >>> >> Hi Igor,
> >>> >>
> >>> >> Postponing this till Fuel 10 means we have to elaborate a plan to do
> >>> such
> >>> >> upgrade for Fuel 9 after the release - the underlying system will not
> >>> get
> >>> >> updated on it's own, and the security issues will not close
> >>> themselves. The
> >>> >> problem here is that such upgrade of deployed master node requires a
> >>> lot of
> >>> >> QA work also.
> >>> >>
> >>> >> Since we are not going to update package we build on our own (they
> >>> still
> >>> >> targeted 7.1) switching master node base to CentOS-72 is not that
> >>> dangerous
> >>> >> as it seems.
> >>> >>
> >>> >> On Wed, Mar 2, 2016 at 1:54 PM, Igor Kalnitsky <
> >>> ikalnitsky at mirantis.com>
> >>> >> wrote:
> >>> >>>
> >>> >>> Hey Dmitry,
> >>> >>>
> >>> >>> No offence, but I rather against that exception. We have too many
> >>> >>> things to do in Mitaka, and moving to CentOS 7.2 means
> >>> >>>
> >>> >>> * extra effort from core team
> >>> >>> * extra effort from qa team
> >>> >>>
> >>> >>> Moreover, it might block development by introducing unpredictable
> >>> >>> regressions. Remember 8.0? So I think it'd be better to reduce risk
> >>> of
> >>> >>> regressions that affects so many developers by postponing CentOS 7.2
> >>> >>> till Fuel 10.
> >>> >>>
> >>> >>> Thanks,
> >>> >>> Igor
> >>> >>>
> >>> >>>
> >>> >>> On Mon, Feb 29, 2016 at 7:13 PM, Dmitry Teselkin <
> >>> dteselkin at mirantis.com>
> >>> >>> wrote:
> >>> >>> > I'd like to ask for a feature freeze exception for switching to
> >>> >>> > CentOS-7.2
> >>> >>> > feature [0].
> >>> >>> >
> >>> >>> > CentOS-7.2 ISO's have been tested periodically since the beginning
> >>> of
> >>> >>> > the
> >>> >>> > year, and all major issues were addressed / fixed at the moment.
> >>> During
> >>> >>> > the
> >>> >>> > last weekend I've made 70 BVT runs to verify that the  solution
> >>> [2] for
> >>> >>> > the
> >>> >>> > last issue - e1000 transmit unit hangs works. And it works, 0
> >>> tests of
> >>> >>> > 35
> >>> >>> > failed [3].
> >>> >>> >
> >>> >>> > Benefits of switching to CentOS-7.2 are quite obvious - we will
> >>> return
> >>> >>> > to
> >>> >>> > latest supported CentOS release, will fix a lot of bugs / security
> >>> >>> > issues
> >>> >>> > [4] and will make further updates easier.
> >>> >>> >
> >>> >>> > Risk of regression still exists, but it's quite low, 35 successful
> >>> BVTs
> >>> >>> > can't be wrong.
> >>> >>> >
> >>> >>> > To finish that feature the following should be done:
> >>> >>> > * review and merge e1000 workaround [2]
> >>> >>> > * review and merge spec [0]
> >>> >>> > * review and merge request that switches build CI to CentOS-7.2 [5]
> >>> >>> >
> >>> >>> > I expect the last day it could be done is March, 4.
> >>> >>> >
> >>> >>> > [0] https://review.openstack.org/#/c/280338/
> >>> >>> > [1] https://bugs.launchpad.net/fuel/+bug/1526544
> >>> >>> > [2] https://review.openstack.org/#/c/285306/
> >>> >>> > [3]
> >>> https://etherpad.openstack.org/p/r.1c4cfee8185326d6922d6c9321404350
> >>> >>> > [4]
> >>> https://etherpad.openstack.org/p/r.a7fe0b575d891ed81206765fa5be6630
> >>> >>> > [5] https://review.fuel-infra.org/#/c/17400/
> >>> >>> >
> >>> >>> >
> >>> >>> > --
> >>> >>> > Thanks,
> >>> >>> > Dmitry Teselkin
> >>> >>> > Mirantis
> >>> >>> > http://www.mirantis.com
> >>> >>> >
> >>> >>> >
> >>> >>> >
> >>> __________________________________________________________________________
> >>> >>> > OpenStack Development Mailing List (not for usage questions)
> >>> >>> > Unsubscribe:
> >>> >>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>> >>> >
> >>> >>>
> >>> >>>
> >>> >>>
> >>> __________________________________________________________________________
> >>> >>> OpenStack Development Mailing List (not for usage questions)
> >>> >>> Unsubscribe:
> >>> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> --
> >>> >> Thanks,
> >>> >> Dmitry Teselkin
> >>> >> Mirantis
> >>> >> http://www.mirantis.com
> >>> >>
> >>> >>
> >>> __________________________________________________________________________
> >>> >> OpenStack Development Mailing List (not for usage questions)
> >>> >> Unsubscribe:
> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>> >>
> >>> >
> >>> >
> >>> >
> >>> __________________________________________________________________________
> >>> > OpenStack Development Mailing List (not for usage questions)
> >>> > Unsubscribe:
> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>> >
> >>>
> >>>
> >>> __________________________________________________________________________
> >>> OpenStack Development Mailing List (not for usage questions)
> >>> Unsubscribe:
> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Dmitry Teselkin
> >> Mirantis
> >> http://www.mirantis.com
> >>
> >> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> -- 
> Mike Scherbakov
> #mihgen

> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev