Open Stack

Dmitry, Aleksandra,
thank you for help and support!

Regards,
Igor Marnat

On Fri, Mar 4, 2016 at 1:20 AM, Aleksandra Fedorova <afedorova at mirantis.com>
wrote:

> As we agreed, we have switched ISO builds to latest CentOS 7.2 snapshots.
>
> You can see now that each ISO build (see for ex. [1]) produces several
> *_id.txt artifacts.
> Note that centos_mirror_id.txt points to CentOS snapshot at
> http://mirror.fuel-infra.org/pkgs/
>
> BVT test is stable, see [2], and nightly system tests results are
> basically the same as they were before.
>
>
> [1] https://ci.fuel-infra.org/job/9.0-community.all/3868/
> [2]
> https://ci.fuel-infra.org/view/ISO/job/9.0.fuel_community.ubuntu.bvt_2/
>
> On Thu, Mar 3, 2016 at 3:46 AM, Dmitry Borodaenko
> <dborodaenko at mirantis.com> wrote:
> > Thanks for the detailed explanation, very helpful!
> >
> > Considering that this change is atomic and easily revertable, lets
> > proceed with the change, the sooner we do that the more time we'll have
> > to confirm that there is no impact and revert if necessary.
> >
> > --
> > Dmitry Borodaenko
> >
> > On Thu, Mar 03, 2016 at 03:40:22AM +0300, Aleksandra Fedorova wrote:
> >> Hi,
> >>
> >> let me add some details about the change:
> >>
> >> 1) There are two repositories used to build Fuel ISO: base OS
> >> repository [1], and mos repository [2], where we put Fuel dependencies
> >> and packages we rebuild due to certain version requirements.
> >>
> >> The CentOS 7.2 feature is related to the upstream repo only. Packages
> >> like RabbitMQ, MCollective, Puppet, MySQL and PostgreSQL live in mos
> >> repository, which has higher priority then upstream.
> >>
> >> I think we need to setup a separate discussion about our policy
> >> regarding these packages, but for now they are fixed and won't be
> >> updated by CentOS 7.2 switch.
> >>
> >> 2) This change doesn't affect Fuel codebase.
> >>
> >> The upstream mirror we use for ISO build is controlled by environment
> >> variable which is set via Jenkins [3] and can be changed anytime.
> >>
> >> As we have daily snapshots of CentOS repository available at [4], in
> >> case of regression in upstream we can pin our builds to stable
> >> snapshot and work on the issue without blocking the main development
> >> flow.
> >
> > Please make sure that the current snapshot of CentOS 7.1 is not rotated
> > away so that we don't loose the point we can revert to.
> >
> >> 3) The "improve snapshotting" work item which is at the moment in
> >> progress, will prevent any possibility to "accidentally" migrate to
> >> CentOS 7.3, when it becomes available.
> >> Thus the only changes which we can fetch from upstream are changes
> >> which are published to updates/ component of CentOS 7.2 repo.
> >>
> >>
> >> As latest BVT on master is green
> >>    https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/69/
> >> I think we should proceed with Jenkins reconfiguration [5] and switch
> >> to latest snapshots by default.
> >>
> >> [1] currently http://vault.centos.org/7.1.1503/
> >> [2]
> http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7-fuel/os/x86_64/
> >> [3]
> https://github.com/fuel-infra/jenkins-jobs/blob/76b5cdf1828b7db1957f7967180d20be099b0c63/common/scripts/all.sh#L84
> >> [4] http://mirror.fuel-infra.org/pkgs/
> >> [5] https://review.fuel-infra.org/#/c/17712/
> >>
> >> On Wed, Mar 2, 2016 at 9:22 PM, Mike Scherbakov
> >> <mscherbakov at mirantis.com> wrote:
> >> > It is not just about BVT. I'd suggest to monitor situation overall,
> >> > including failures of system tests [1]. If we see regressions there,
> or some
> >> > test cases will start flapping (what is even worse), then we'd have to
> >> > revert back to CentOS 7.1.
> >> >
> >> > [1] https://github.com/openstack/fuel-qa
> >> >
> >> > On Wed, Mar 2, 2016 at 10:16 AM Dmitry Borodaenko <
> dborodaenko at mirantis.com>
> >> > wrote:
> >> >>
> >> >> I agree with Mike's concerns, and propose to make these limitations
> (4
> >> >> weeks before FF for OS upgrades, 2 weeks for upgrades of key
> >> >> dependencies -- RabbitMQ, MCollective, Puppet, MySQL, PostgreSQL,
> >> >> anything else?) official for 10.0/Newton.
> >> >>
> >> >> For 9.0/Mitaka, it is too late to impose them, so we just have to be
> >> >> very careful and conservative with this upgrade. First of all, we
> need
> >> >> to have a green BVT before and after switching to the CentOS 7.2 repo
> >> >> snapshot, so while I approved the spec, we can't move forward with
> this
> >> >> until BVT is green again, and right now it's red:
> >> >>
> >> >> https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/
> >> >>
> >> >> If we get it back to green but it becomes red after the upgrade, you
> >> >> must switch back to CentOS 7.1 *immediately*. If you are able to
> stick
> >> >> to this plan, there is still time to complete the transition today
> >> >> without requiring an FFE.
> >> >>
> >> >> --
> >> >> Dmitry Borodaenko
> >> >>
> >> >>
> >> >> On Wed, Mar 02, 2016 at 05:53:53PM +0000, Mike Scherbakov wrote:
> >> >> > Formally, we can merge it today. Historically, every update of OS
> caused
> >> >> > us
> >> >> > instability for some time: from days to a couple of month.
> >> >> > Taking this into account and number of other exceptions requested,
> >> >> > overall
> >> >> > stability of code, my opinion would be to postpone this to 10.0.
> >> >> >
> >> >> > Also, I'd suggest to change the process, and have freeze date for
> all OS
> >> >> > updates no later than a month before official FF date. This will
> give us
> >> >> > time to stabilize, and ensure that base on which all new code is
> being
> >> >> > developed is stable when approaching FF.
> >> >> >
> >> >> > I'd also propose to have freeze for major upgrades of 3rd party
> packages
> >> >> > no
> >> >> > later than 2 weeks before FF, which Fuel depends heavily upon. For
> >> >> > instance, such will include RabbitMQ, MCollective, Puppet.
> >> >> >
> >> >> > On Wed, Mar 2, 2016 at 7:34 AM Igor Marnat <imarnat at mirantis.com>
> wrote:
> >> >> >
> >> >> > > Igor,
> >> >> > > couple of points from my side.
> >> >> > >
> >> >> > > CentOS 7.2 will be getting updates for several more months, and
> we
> >> >> > > have
> >> >> > > snapshots and all the mechanics in place to switch to the next
> version
> >> >> > > when
> >> >> > > needed.
> >> >> > >
> >> >> > > Speaking of getting this update into 9.0, we actually don't need
> FFE,
> >> >> > > we
> >> >> > > can merge remaining staff today. It has enough reviews, so if
> you add
> >> >> > > your
> >> >> > > +1 today, we don't need FFE.
> >> >> > >
> >> >> > > https://review.openstack.org/#/c/280338/
> >> >> > > https://review.fuel-infra.org/#/c/17400/
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > Regards,
> >> >> > > Igor Marnat
> >> >> > >
> >> >> > > On Wed, Mar 2, 2016 at 6:23 PM, Dmitry Teselkin
> >> >> > > <dteselkin at mirantis.com>
> >> >> > > wrote:
> >> >> > >
> >> >> > >> Igor,
> >> >> > >>
> >> >> > >> Your statement about updates for 7.2 isn't correct - it will
> receive
> >> >> > >> updates,  because it's the latest release ATM. There is *no*
> pinning
> >> >> > >> inside
> >> >> > >> ISO, and the only place where it was 8.0 were docker containers
> just
> >> >> > >> because we had to workaround some issues. But there are no
> docker
> >> >> > >> containers in 9.0, so that's not the case.
> >> >> > >> The proposed solution to switch to CentOS-7.2 in fact is based
> on
> >> >> > >> selecting the right snapshot with packages. There is no pinning
> in
> >> >> > >> ISO (it
> >> >> > >> was in earlier versions of the spec but was removed).
> >> >> > >>
> >> >> > >> On Wed, Mar 2, 2016 at 6:11 PM, Igor Kalnitsky
> >> >> > >> <ikalnitsky at mirantis.com>
> >> >> > >> wrote:
> >> >> > >>
> >> >> > >>> Dmitry, Igor,
> >> >> > >>>
> >> >> > >>> > Very important thing is that CentOS 7.1 which master node is
> based
> >> >> > >>> > now
> >> >> > >>> > don't get updates any longer.
> >> >> > >>>
> >> >> > >>> If you are using "fixed" release you must be ready that you
> won't
> >> >> > >>> get
> >> >> > >>> any updates. So with CentOS 7.2 the problem still the same.
> >> >> > >>>
> >> >> > >>> However, let's wait for Fuel PTL decision. I only shared my
> POV:
> >> >> > >>> that's not a critical feature, and taking into account the
> risks of
> >> >> > >>> regression - I'd prefer to do not accept it in 9.0.
> >> >> > >>>
> >> >> > >>> Regards,
> >> >> > >>> Igor
> >> >> > >>>
> >> >> > >>> On Wed, Mar 2, 2016 at 4:42 PM, Igor Marnat <
> imarnat at mirantis.com>
> >> >> > >>> wrote:
> >> >> > >>> > Igor,
> >> >> > >>> > please note that this is pretty much not like update of
> master
> >> >> > >>> > node
> >> >> > >>> which we
> >> >> > >>> > had in 8.0. This is minor _update_ of CentOS from 7.1 to 7.2
> which
> >> >> > >>> > team
> >> >> > >>> > tested for more than 2 months already.
> >> >> > >>> >
> >> >> > >>> > We don't expect it to require any additional efforts from
> core or
> >> >> > >>> > qa
> >> >> > >>> team.
> >> >> > >>> >
> >> >> > >>> > Very important thing is that CentOS 7.1 which master node is
> based
> >> >> > >>> > now
> >> >> > >>> don't
> >> >> > >>> > get updates any longer. Updates are only provided for CentOS
> 7.2.
> >> >> > >>> >
> >> >> > >>> > So we'll have to switch CentOS 7.1 to CentOS 7.2 anyways.
> >> >> > >>> >
> >> >> > >>> > We can do it now for more or less free, later in release
> cycle for
> >> >> > >>> higher
> >> >> > >>> > risk and QA efforts and after the release for 2x price
> because of
> >> >> > >>> additional
> >> >> > >>> > QA cycle we'll need to pass through.
> >> >> > >>> >
> >> >> > >>> >
> >> >> > >>> >
> >> >> > >>> > Regards,
> >> >> > >>> > Igor Marnat
> >> >> > >>> >
> >> >> > >>> > On Wed, Mar 2, 2016 at 2:57 PM, Dmitry Teselkin <
> >> >> > >>> dteselkin at mirantis.com>
> >> >> > >>> > wrote:
> >> >> > >>> >>
> >> >> > >>> >> Hi Igor,
> >> >> > >>> >>
> >> >> > >>> >> Postponing this till Fuel 10 means we have to elaborate a
> plan to
> >> >> > >>> >> do
> >> >> > >>> such
> >> >> > >>> >> upgrade for Fuel 9 after the release - the underlying
> system will
> >> >> > >>> >> not
> >> >> > >>> get
> >> >> > >>> >> updated on it's own, and the security issues will not close
> >> >> > >>> themselves. The
> >> >> > >>> >> problem here is that such upgrade of deployed master node
> >> >> > >>> >> requires a
> >> >> > >>> lot of
> >> >> > >>> >> QA work also.
> >> >> > >>> >>
> >> >> > >>> >> Since we are not going to update package we build on our own
> >> >> > >>> >> (they
> >> >> > >>> still
> >> >> > >>> >> targeted 7.1) switching master node base to CentOS-72 is
> not that
> >> >> > >>> dangerous
> >> >> > >>> >> as it seems.
> >> >> > >>> >>
> >> >> > >>> >> On Wed, Mar 2, 2016 at 1:54 PM, Igor Kalnitsky <
> >> >> > >>> ikalnitsky at mirantis.com>
> >> >> > >>> >> wrote:
> >> >> > >>> >>>
> >> >> > >>> >>> Hey Dmitry,
> >> >> > >>> >>>
> >> >> > >>> >>> No offence, but I rather against that exception. We have
> too
> >> >> > >>> >>> many
> >> >> > >>> >>> things to do in Mitaka, and moving to CentOS 7.2 means
> >> >> > >>> >>>
> >> >> > >>> >>> * extra effort from core team
> >> >> > >>> >>> * extra effort from qa team
> >> >> > >>> >>>
> >> >> > >>> >>> Moreover, it might block development by introducing
> >> >> > >>> >>> unpredictable
> >> >> > >>> >>> regressions. Remember 8.0? So I think it'd be better to
> reduce
> >> >> > >>> >>> risk
> >> >> > >>> of
> >> >> > >>> >>> regressions that affects so many developers by postponing
> CentOS
> >> >> > >>> >>> 7.2
> >> >> > >>> >>> till Fuel 10.
> >> >> > >>> >>>
> >> >> > >>> >>> Thanks,
> >> >> > >>> >>> Igor
> >> >> > >>> >>>
> >> >> > >>> >>>
> >> >> > >>> >>> On Mon, Feb 29, 2016 at 7:13 PM, Dmitry Teselkin <
> >> >> > >>> dteselkin at mirantis.com>
> >> >> > >>> >>> wrote:
> >> >> > >>> >>> > I'd like to ask for a feature freeze exception for
> switching
> >> >> > >>> >>> > to
> >> >> > >>> >>> > CentOS-7.2
> >> >> > >>> >>> > feature [0].
> >> >> > >>> >>> >
> >> >> > >>> >>> > CentOS-7.2 ISO's have been tested periodically since the
> >> >> > >>> >>> > beginning
> >> >> > >>> of
> >> >> > >>> >>> > the
> >> >> > >>> >>> > year, and all major issues were addressed / fixed at the
> >> >> > >>> >>> > moment.
> >> >> > >>> During
> >> >> > >>> >>> > the
> >> >> > >>> >>> > last weekend I've made 70 BVT runs to verify that the
> >> >> > >>> >>> > solution
> >> >> > >>> [2] for
> >> >> > >>> >>> > the
> >> >> > >>> >>> > last issue - e1000 transmit unit hangs works. And it
> works, 0
> >> >> > >>> tests of
> >> >> > >>> >>> > 35
> >> >> > >>> >>> > failed [3].
> >> >> > >>> >>> >
> >> >> > >>> >>> > Benefits of switching to CentOS-7.2 are quite obvious -
> we
> >> >> > >>> >>> > will
> >> >> > >>> return
> >> >> > >>> >>> > to
> >> >> > >>> >>> > latest supported CentOS release, will fix a lot of bugs /
> >> >> > >>> >>> > security
> >> >> > >>> >>> > issues
> >> >> > >>> >>> > [4] and will make further updates easier.
> >> >> > >>> >>> >
> >> >> > >>> >>> > Risk of regression still exists, but it's quite low, 35
> >> >> > >>> >>> > successful
> >> >> > >>> BVTs
> >> >> > >>> >>> > can't be wrong.
> >> >> > >>> >>> >
> >> >> > >>> >>> > To finish that feature the following should be done:
> >> >> > >>> >>> > * review and merge e1000 workaround [2]
> >> >> > >>> >>> > * review and merge spec [0]
> >> >> > >>> >>> > * review and merge request that switches build CI to
> >> >> > >>> >>> > CentOS-7.2 [5]
> >> >> > >>> >>> >
> >> >> > >>> >>> > I expect the last day it could be done is March, 4.
> >> >> > >>> >>> >
> >> >> > >>> >>> > [0] https://review.openstack.org/#/c/280338/
> >> >> > >>> >>> > [1] https://bugs.launchpad.net/fuel/+bug/1526544
> >> >> > >>> >>> > [2] https://review.openstack.org/#/c/285306/
> >> >> > >>> >>> > [3]
> >> >> > >>>
> https://etherpad.openstack.org/p/r.1c4cfee8185326d6922d6c9321404350
> >> >> > >>> >>> > [4]
> >> >> > >>>
> https://etherpad.openstack.org/p/r.a7fe0b575d891ed81206765fa5be6630
> >> >> > >>> >>> > [5] https://review.fuel-infra.org/#/c/17400/
> >> >> > >>> >>> >
> >> >> > >>> >>> >
> >> >> > >>> >>> > --
> >> >> > >>> >>> > Thanks,
> >> >> > >>> >>> > Dmitry Teselkin
> >> >> > >>> >>> > Mirantis
> >> >> > >>> >>> > http://www.mirantis.com
> >> >> > >>> >>> >
> >> >> > >>> >>> >
> >> >> > >>> >>> >
> >> >> > >>>
> >> >> > >>>
> __________________________________________________________________________
> >> >> > >>> >>> > OpenStack Development Mailing List (not for usage
> questions)
> >> >> > >>> >>> > Unsubscribe:
> >> >> > >>> >>> >
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> > >>> >>> >
> >> >> > >>> >>> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >> > >>> >>> >
> >> >> > >>> >>>
> >> >> > >>> >>>
> >> >> > >>> >>>
> >> >> > >>>
> >> >> > >>>
> __________________________________________________________________________
> >> >> > >>> >>> OpenStack Development Mailing List (not for usage
> questions)
> >> >> > >>> >>> Unsubscribe:
> >> >> > >>> >>>
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> > >>> >>>
> >> >> > >>> >>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >> > >>> >>
> >> >> > >>> >>
> >> >> > >>> >>
> >> >> > >>> >>
> >> >> > >>> >> --
> >> >> > >>> >> Thanks,
> >> >> > >>> >> Dmitry Teselkin
> >> >> > >>> >> Mirantis
> >> >> > >>> >> http://www.mirantis.com
> >> >> > >>> >>
> >> >> > >>> >>
> >> >> > >>>
> >> >> > >>>
> __________________________________________________________________________
> >> >> > >>> >> OpenStack Development Mailing List (not for usage questions)
> >> >> > >>> >> Unsubscribe:
> >> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> > >>> >>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >> > >>> >>
> >> >> > >>> >
> >> >> > >>> >
> >> >> > >>> >
> >> >> > >>>
> >> >> > >>>
> __________________________________________________________________________
> >> >> > >>> > OpenStack Development Mailing List (not for usage questions)
> >> >> > >>> > Unsubscribe:
> >> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> > >>> >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >> > >>> >
> >> >> > >>>
> >> >> > >>>
> >> >> > >>>
> >> >> > >>>
> __________________________________________________________________________
> >> >> > >>> OpenStack Development Mailing List (not for usage questions)
> >> >> > >>> Unsubscribe:
> >> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> > >>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >> > >>>
> >> >> > >>
> >> >> > >>
> >> >> > >>
> >> >> > >> --
> >> >> > >> Thanks,
> >> >> > >> Dmitry Teselkin
> >> >> > >> Mirantis
> >> >> > >> http://www.mirantis.com
> >> >> > >>
> >> >> > >>
> >> >> > >>
> __________________________________________________________________________
> >> >> > >> OpenStack Development Mailing List (not for usage questions)
> >> >> > >> Unsubscribe:
> >> >> > >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> > >>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >> > >>
> >> >> > >>
> >> >> > >
> >> >> > >
> __________________________________________________________________________
> >> >> > > OpenStack Development Mailing List (not for usage questions)
> >> >> > > Unsubscribe:
> >> >> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> > >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >> > >
> >> >> > --
> >> >> > Mike Scherbakov
> >> >> > #mihgen
> >> >>
> >> >> >
> >> >> >
> __________________________________________________________________________
> >> >> > OpenStack Development Mailing List (not for usage questions)
> >> >> > Unsubscribe:
> >> >> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>
> >> >>
> >> >>
> __________________________________________________________________________
> >> >> OpenStack Development Mailing List (not for usage questions)
> >> >> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >> > --
> >> > Mike Scherbakov
> >> > #mihgen
> >> >
> >> >
> __________________________________________________________________________
> >> > OpenStack Development Mailing List (not for usage questions)
> >> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >>
> >>
> >>
> >> --
> >> Aleksandra Fedorova
> >> CI Team Lead
> >> bookwar
> >>
> >>
> __________________________________________________________________________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Aleksandra Fedorova
> CI Team Lead
> bookwar
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160304/4172e919/attachment.html>