Open Stack

Thanks for the detailed explanation, very helpful!

Considering that this change is atomic and easily revertable, lets
proceed with the change, the sooner we do that the more time we'll have
to confirm that there is no impact and revert if necessary.

-- 
Dmitry Borodaenko

On Thu, Mar 03, 2016 at 03:40:22AM +0300, Aleksandra Fedorova wrote:
> Hi,
> 
> let me add some details about the change:
> 
> 1) There are two repositories used to build Fuel ISO: base OS
> repository [1], and mos repository [2], where we put Fuel dependencies
> and packages we rebuild due to certain version requirements.
> 
> The CentOS 7.2 feature is related to the upstream repo only. Packages
> like RabbitMQ, MCollective, Puppet, MySQL and PostgreSQL live in mos
> repository, which has higher priority then upstream.
> 
> I think we need to setup a separate discussion about our policy
> regarding these packages, but for now they are fixed and won't be
> updated by CentOS 7.2 switch.
> 
> 2) This change doesn't affect Fuel codebase.
> 
> The upstream mirror we use for ISO build is controlled by environment
> variable which is set via Jenkins [3] and can be changed anytime.
> 
> As we have daily snapshots of CentOS repository available at [4], in
> case of regression in upstream we can pin our builds to stable
> snapshot and work on the issue without blocking the main development
> flow.

Please make sure that the current snapshot of CentOS 7.1 is not rotated
away so that we don't loose the point we can revert to.

> 3) The "improve snapshotting" work item which is at the moment in
> progress, will prevent any possibility to "accidentally" migrate to
> CentOS 7.3, when it becomes available.
> Thus the only changes which we can fetch from upstream are changes
> which are published to updates/ component of CentOS 7.2 repo.
> 
> 
> As latest BVT on master is green
>    https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/69/
> I think we should proceed with Jenkins reconfiguration [5] and switch
> to latest snapshots by default.
> 
> [1] currently http://vault.centos.org/7.1.1503/
> [2] http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7-fuel/os/x86_64/
> [3] https://github.com/fuel-infra/jenkins-jobs/blob/76b5cdf1828b7db1957f7967180d20be099b0c63/common/scripts/all.sh#L84
> [4] http://mirror.fuel-infra.org/pkgs/
> [5] https://review.fuel-infra.org/#/c/17712/
> 
> On Wed, Mar 2, 2016 at 9:22 PM, Mike Scherbakov
> <mscherbakov at mirantis.com> wrote:
> > It is not just about BVT. I'd suggest to monitor situation overall,
> > including failures of system tests [1]. If we see regressions there, or some
> > test cases will start flapping (what is even worse), then we'd have to
> > revert back to CentOS 7.1.
> >
> > [1] https://github.com/openstack/fuel-qa
> >
> > On Wed, Mar 2, 2016 at 10:16 AM Dmitry Borodaenko <dborodaenko at mirantis.com>
> > wrote:
> >>
> >> I agree with Mike's concerns, and propose to make these limitations (4
> >> weeks before FF for OS upgrades, 2 weeks for upgrades of key
> >> dependencies -- RabbitMQ, MCollective, Puppet, MySQL, PostgreSQL,
> >> anything else?) official for 10.0/Newton.
> >>
> >> For 9.0/Mitaka, it is too late to impose them, so we just have to be
> >> very careful and conservative with this upgrade. First of all, we need
> >> to have a green BVT before and after switching to the CentOS 7.2 repo
> >> snapshot, so while I approved the spec, we can't move forward with this
> >> until BVT is green again, and right now it's red:
> >>
> >> https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/
> >>
> >> If we get it back to green but it becomes red after the upgrade, you
> >> must switch back to CentOS 7.1 *immediately*. If you are able to stick
> >> to this plan, there is still time to complete the transition today
> >> without requiring an FFE.
> >>
> >> --
> >> Dmitry Borodaenko
> >>
> >>
> >> On Wed, Mar 02, 2016 at 05:53:53PM +0000, Mike Scherbakov wrote:
> >> > Formally, we can merge it today. Historically, every update of OS caused
> >> > us
> >> > instability for some time: from days to a couple of month.
> >> > Taking this into account and number of other exceptions requested,
> >> > overall
> >> > stability of code, my opinion would be to postpone this to 10.0.
> >> >
> >> > Also, I'd suggest to change the process, and have freeze date for all OS
> >> > updates no later than a month before official FF date. This will give us
> >> > time to stabilize, and ensure that base on which all new code is being
> >> > developed is stable when approaching FF.
> >> >
> >> > I'd also propose to have freeze for major upgrades of 3rd party packages
> >> > no
> >> > later than 2 weeks before FF, which Fuel depends heavily upon. For
> >> > instance, such will include RabbitMQ, MCollective, Puppet.
> >> >
> >> > On Wed, Mar 2, 2016 at 7:34 AM Igor Marnat <imarnat at mirantis.com> wrote:
> >> >
> >> > > Igor,
> >> > > couple of points from my side.
> >> > >
> >> > > CentOS 7.2 will be getting updates for several more months, and we
> >> > > have
> >> > > snapshots and all the mechanics in place to switch to the next version
> >> > > when
> >> > > needed.
> >> > >
> >> > > Speaking of getting this update into 9.0, we actually don't need FFE,
> >> > > we
> >> > > can merge remaining staff today. It has enough reviews, so if you add
> >> > > your
> >> > > +1 today, we don't need FFE.
> >> > >
> >> > > https://review.openstack.org/#/c/280338/
> >> > > https://review.fuel-infra.org/#/c/17400/
> >> > >
> >> > >
> >> > >
> >> > > Regards,
> >> > > Igor Marnat
> >> > >
> >> > > On Wed, Mar 2, 2016 at 6:23 PM, Dmitry Teselkin
> >> > > <dteselkin at mirantis.com>
> >> > > wrote:
> >> > >
> >> > >> Igor,
> >> > >>
> >> > >> Your statement about updates for 7.2 isn't correct - it will receive
> >> > >> updates,  because it's the latest release ATM. There is *no* pinning
> >> > >> inside
> >> > >> ISO, and the only place where it was 8.0 were docker containers just
> >> > >> because we had to workaround some issues. But there are no docker
> >> > >> containers in 9.0, so that's not the case.
> >> > >> The proposed solution to switch to CentOS-7.2 in fact is based on
> >> > >> selecting the right snapshot with packages. There is no pinning in
> >> > >> ISO (it
> >> > >> was in earlier versions of the spec but was removed).
> >> > >>
> >> > >> On Wed, Mar 2, 2016 at 6:11 PM, Igor Kalnitsky
> >> > >> <ikalnitsky at mirantis.com>
> >> > >> wrote:
> >> > >>
> >> > >>> Dmitry, Igor,
> >> > >>>
> >> > >>> > Very important thing is that CentOS 7.1 which master node is based
> >> > >>> > now
> >> > >>> > don't get updates any longer.
> >> > >>>
> >> > >>> If you are using "fixed" release you must be ready that you won't
> >> > >>> get
> >> > >>> any updates. So with CentOS 7.2 the problem still the same.
> >> > >>>
> >> > >>> However, let's wait for Fuel PTL decision. I only shared my POV:
> >> > >>> that's not a critical feature, and taking into account the risks of
> >> > >>> regression - I'd prefer to do not accept it in 9.0.
> >> > >>>
> >> > >>> Regards,
> >> > >>> Igor
> >> > >>>
> >> > >>> On Wed, Mar 2, 2016 at 4:42 PM, Igor Marnat <imarnat at mirantis.com>
> >> > >>> wrote:
> >> > >>> > Igor,
> >> > >>> > please note that this is pretty much not like update of master
> >> > >>> > node
> >> > >>> which we
> >> > >>> > had in 8.0. This is minor _update_ of CentOS from 7.1 to 7.2 which
> >> > >>> > team
> >> > >>> > tested for more than 2 months already.
> >> > >>> >
> >> > >>> > We don't expect it to require any additional efforts from core or
> >> > >>> > qa
> >> > >>> team.
> >> > >>> >
> >> > >>> > Very important thing is that CentOS 7.1 which master node is based
> >> > >>> > now
> >> > >>> don't
> >> > >>> > get updates any longer. Updates are only provided for CentOS 7.2.
> >> > >>> >
> >> > >>> > So we'll have to switch CentOS 7.1 to CentOS 7.2 anyways.
> >> > >>> >
> >> > >>> > We can do it now for more or less free, later in release cycle for
> >> > >>> higher
> >> > >>> > risk and QA efforts and after the release for 2x price because of
> >> > >>> additional
> >> > >>> > QA cycle we'll need to pass through.
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>> > Regards,
> >> > >>> > Igor Marnat
> >> > >>> >
> >> > >>> > On Wed, Mar 2, 2016 at 2:57 PM, Dmitry Teselkin <
> >> > >>> dteselkin at mirantis.com>
> >> > >>> > wrote:
> >> > >>> >>
> >> > >>> >> Hi Igor,
> >> > >>> >>
> >> > >>> >> Postponing this till Fuel 10 means we have to elaborate a plan to
> >> > >>> >> do
> >> > >>> such
> >> > >>> >> upgrade for Fuel 9 after the release - the underlying system will
> >> > >>> >> not
> >> > >>> get
> >> > >>> >> updated on it's own, and the security issues will not close
> >> > >>> themselves. The
> >> > >>> >> problem here is that such upgrade of deployed master node
> >> > >>> >> requires a
> >> > >>> lot of
> >> > >>> >> QA work also.
> >> > >>> >>
> >> > >>> >> Since we are not going to update package we build on our own
> >> > >>> >> (they
> >> > >>> still
> >> > >>> >> targeted 7.1) switching master node base to CentOS-72 is not that
> >> > >>> dangerous
> >> > >>> >> as it seems.
> >> > >>> >>
> >> > >>> >> On Wed, Mar 2, 2016 at 1:54 PM, Igor Kalnitsky <
> >> > >>> ikalnitsky at mirantis.com>
> >> > >>> >> wrote:
> >> > >>> >>>
> >> > >>> >>> Hey Dmitry,
> >> > >>> >>>
> >> > >>> >>> No offence, but I rather against that exception. We have too
> >> > >>> >>> many
> >> > >>> >>> things to do in Mitaka, and moving to CentOS 7.2 means
> >> > >>> >>>
> >> > >>> >>> * extra effort from core team
> >> > >>> >>> * extra effort from qa team
> >> > >>> >>>
> >> > >>> >>> Moreover, it might block development by introducing
> >> > >>> >>> unpredictable
> >> > >>> >>> regressions. Remember 8.0? So I think it'd be better to reduce
> >> > >>> >>> risk
> >> > >>> of
> >> > >>> >>> regressions that affects so many developers by postponing CentOS
> >> > >>> >>> 7.2
> >> > >>> >>> till Fuel 10.
> >> > >>> >>>
> >> > >>> >>> Thanks,
> >> > >>> >>> Igor
> >> > >>> >>>
> >> > >>> >>>
> >> > >>> >>> On Mon, Feb 29, 2016 at 7:13 PM, Dmitry Teselkin <
> >> > >>> dteselkin at mirantis.com>
> >> > >>> >>> wrote:
> >> > >>> >>> > I'd like to ask for a feature freeze exception for switching
> >> > >>> >>> > to
> >> > >>> >>> > CentOS-7.2
> >> > >>> >>> > feature [0].
> >> > >>> >>> >
> >> > >>> >>> > CentOS-7.2 ISO's have been tested periodically since the
> >> > >>> >>> > beginning
> >> > >>> of
> >> > >>> >>> > the
> >> > >>> >>> > year, and all major issues were addressed / fixed at the
> >> > >>> >>> > moment.
> >> > >>> During
> >> > >>> >>> > the
> >> > >>> >>> > last weekend I've made 70 BVT runs to verify that the
> >> > >>> >>> > solution
> >> > >>> [2] for
> >> > >>> >>> > the
> >> > >>> >>> > last issue - e1000 transmit unit hangs works. And it works, 0
> >> > >>> tests of
> >> > >>> >>> > 35
> >> > >>> >>> > failed [3].
> >> > >>> >>> >
> >> > >>> >>> > Benefits of switching to CentOS-7.2 are quite obvious - we
> >> > >>> >>> > will
> >> > >>> return
> >> > >>> >>> > to
> >> > >>> >>> > latest supported CentOS release, will fix a lot of bugs /
> >> > >>> >>> > security
> >> > >>> >>> > issues
> >> > >>> >>> > [4] and will make further updates easier.
> >> > >>> >>> >
> >> > >>> >>> > Risk of regression still exists, but it's quite low, 35
> >> > >>> >>> > successful
> >> > >>> BVTs
> >> > >>> >>> > can't be wrong.
> >> > >>> >>> >
> >> > >>> >>> > To finish that feature the following should be done:
> >> > >>> >>> > * review and merge e1000 workaround [2]
> >> > >>> >>> > * review and merge spec [0]
> >> > >>> >>> > * review and merge request that switches build CI to
> >> > >>> >>> > CentOS-7.2 [5]
> >> > >>> >>> >
> >> > >>> >>> > I expect the last day it could be done is March, 4.
> >> > >>> >>> >
> >> > >>> >>> > [0] https://review.openstack.org/#/c/280338/
> >> > >>> >>> > [1] https://bugs.launchpad.net/fuel/+bug/1526544
> >> > >>> >>> > [2] https://review.openstack.org/#/c/285306/
> >> > >>> >>> > [3]
> >> > >>> https://etherpad.openstack.org/p/r.1c4cfee8185326d6922d6c9321404350
> >> > >>> >>> > [4]
> >> > >>> https://etherpad.openstack.org/p/r.a7fe0b575d891ed81206765fa5be6630
> >> > >>> >>> > [5] https://review.fuel-infra.org/#/c/17400/
> >> > >>> >>> >
> >> > >>> >>> >
> >> > >>> >>> > --
> >> > >>> >>> > Thanks,
> >> > >>> >>> > Dmitry Teselkin
> >> > >>> >>> > Mirantis
> >> > >>> >>> > http://www.mirantis.com
> >> > >>> >>> >
> >> > >>> >>> >
> >> > >>> >>> >
> >> > >>>
> >> > >>> __________________________________________________________________________
> >> > >>> >>> > OpenStack Development Mailing List (not for usage questions)
> >> > >>> >>> > Unsubscribe:
> >> > >>> >>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > >>> >>> >
> >> > >>> >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> > >>> >>> >
> >> > >>> >>>
> >> > >>> >>>
> >> > >>> >>>
> >> > >>>
> >> > >>> __________________________________________________________________________
> >> > >>> >>> OpenStack Development Mailing List (not for usage questions)
> >> > >>> >>> Unsubscribe:
> >> > >>> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > >>> >>>
> >> > >>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> > >>> >>
> >> > >>> >>
> >> > >>> >>
> >> > >>> >>
> >> > >>> >> --
> >> > >>> >> Thanks,
> >> > >>> >> Dmitry Teselkin
> >> > >>> >> Mirantis
> >> > >>> >> http://www.mirantis.com
> >> > >>> >>
> >> > >>> >>
> >> > >>>
> >> > >>> __________________________________________________________________________
> >> > >>> >> OpenStack Development Mailing List (not for usage questions)
> >> > >>> >> Unsubscribe:
> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > >>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> > >>> >>
> >> > >>> >
> >> > >>> >
> >> > >>> >
> >> > >>>
> >> > >>> __________________________________________________________________________
> >> > >>> > OpenStack Development Mailing List (not for usage questions)
> >> > >>> > Unsubscribe:
> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> > >>> >
> >> > >>>
> >> > >>>
> >> > >>>
> >> > >>> __________________________________________________________________________
> >> > >>> OpenStack Development Mailing List (not for usage questions)
> >> > >>> Unsubscribe:
> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> > >>>
> >> > >>
> >> > >>
> >> > >>
> >> > >> --
> >> > >> Thanks,
> >> > >> Dmitry Teselkin
> >> > >> Mirantis
> >> > >> http://www.mirantis.com
> >> > >>
> >> > >>
> >> > >> __________________________________________________________________________
> >> > >> OpenStack Development Mailing List (not for usage questions)
> >> > >> Unsubscribe:
> >> > >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> > >>
> >> > >>
> >> > >
> >> > > __________________________________________________________________________
> >> > > OpenStack Development Mailing List (not for usage questions)
> >> > > Unsubscribe:
> >> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> > >
> >> > --
> >> > Mike Scherbakov
> >> > #mihgen
> >>
> >> >
> >> > __________________________________________________________________________
> >> > OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe:
> >> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> > --
> > Mike Scherbakov
> > #mihgen
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> 
> 
> -- 
> Aleksandra Fedorova
> CI Team Lead
> bookwar
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev