Open Stack

As we agreed, we have switched ISO builds to latest CentOS 7.2 snapshots.

You can see now that each ISO build (see for ex. [1]) produces several
*_id.txt artifacts.
Note that centos_mirror_id.txt points to CentOS snapshot at
http://mirror.fuel-infra.org/pkgs/

BVT test is stable, see [2], and nightly system tests results are
basically the same as they were before.


[1] https://ci.fuel-infra.org/job/9.0-community.all/3868/
[2]
https://ci.fuel-infra.org/view/ISO/job/9.0.fuel_community.ubuntu.bvt_2/

On Thu, Mar 3, 2016 at 3:46 AM, Dmitry Borodaenko
<dborodaenko at mirantis.com> wrote:
> Thanks for the detailed explanation, very helpful!
>
> Considering that this change is atomic and easily revertable, lets
> proceed with the change, the sooner we do that the more time we'll have
> to confirm that there is no impact and revert if necessary.
>
> --
> Dmitry Borodaenko
>
> On Thu, Mar 03, 2016 at 03:40:22AM +0300, Aleksandra Fedorova wrote:
>> Hi,
>>
>> let me add some details about the change:
>>
>> 1) There are two repositories used to build Fuel ISO: base OS
>> repository [1], and mos repository [2], where we put Fuel dependencies
>> and packages we rebuild due to certain version requirements.
>>
>> The CentOS 7.2 feature is related to the upstream repo only. Packages
>> like RabbitMQ, MCollective, Puppet, MySQL and PostgreSQL live in mos
>> repository, which has higher priority then upstream.
>>
>> I think we need to setup a separate discussion about our policy
>> regarding these packages, but for now they are fixed and won't be
>> updated by CentOS 7.2 switch.
>>
>> 2) This change doesn't affect Fuel codebase.
>>
>> The upstream mirror we use for ISO build is controlled by environment
>> variable which is set via Jenkins [3] and can be changed anytime.
>>
>> As we have daily snapshots of CentOS repository available at [4], in
>> case of regression in upstream we can pin our builds to stable
>> snapshot and work on the issue without blocking the main development
>> flow.
>
> Please make sure that the current snapshot of CentOS 7.1 is not rotated
> away so that we don't loose the point we can revert to.
>
>> 3) The "improve snapshotting" work item which is at the moment in
>> progress, will prevent any possibility to "accidentally" migrate to
>> CentOS 7.3, when it becomes available.
>> Thus the only changes which we can fetch from upstream are changes
>> which are published to updates/ component of CentOS 7.2 repo.
>>
>>
>> As latest BVT on master is green
>>    https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/69/
>> I think we should proceed with Jenkins reconfiguration [5] and switch
>> to latest snapshots by default.
>>
>> [1] currently http://vault.centos.org/7.1.1503/
>> [2] http://mirror.fuel-infra.org/mos-repos/centos/mos9.0-centos7-fuel/os/x86_64/
>> [3] https://github.com/fuel-infra/jenkins-jobs/blob/76b5cdf1828b7db1957f7967180d20be099b0c63/common/scripts/all.sh#L84
>> [4] http://mirror.fuel-infra.org/pkgs/
>> [5] https://review.fuel-infra.org/#/c/17712/
>>
>> On Wed, Mar 2, 2016 at 9:22 PM, Mike Scherbakov
>> <mscherbakov at mirantis.com> wrote:
>> > It is not just about BVT. I'd suggest to monitor situation overall,
>> > including failures of system tests [1]. If we see regressions there, or some
>> > test cases will start flapping (what is even worse), then we'd have to
>> > revert back to CentOS 7.1.
>> >
>> > [1] https://github.com/openstack/fuel-qa
>> >
>> > On Wed, Mar 2, 2016 at 10:16 AM Dmitry Borodaenko <dborodaenko at mirantis.com>
>> > wrote:
>> >>
>> >> I agree with Mike's concerns, and propose to make these limitations (4
>> >> weeks before FF for OS upgrades, 2 weeks for upgrades of key
>> >> dependencies -- RabbitMQ, MCollective, Puppet, MySQL, PostgreSQL,
>> >> anything else?) official for 10.0/Newton.
>> >>
>> >> For 9.0/Mitaka, it is too late to impose them, so we just have to be
>> >> very careful and conservative with this upgrade. First of all, we need
>> >> to have a green BVT before and after switching to the CentOS 7.2 repo
>> >> snapshot, so while I approved the spec, we can't move forward with this
>> >> until BVT is green again, and right now it's red:
>> >>
>> >> https://ci.fuel-infra.org/job/9.0.fuel_community.ubuntu.bvt_2/
>> >>
>> >> If we get it back to green but it becomes red after the upgrade, you
>> >> must switch back to CentOS 7.1 *immediately*. If you are able to stick
>> >> to this plan, there is still time to complete the transition today
>> >> without requiring an FFE.
>> >>
>> >> --
>> >> Dmitry Borodaenko
>> >>
>> >>
>> >> On Wed, Mar 02, 2016 at 05:53:53PM +0000, Mike Scherbakov wrote:
>> >> > Formally, we can merge it today. Historically, every update of OS caused
>> >> > us
>> >> > instability for some time: from days to a couple of month.
>> >> > Taking this into account and number of other exceptions requested,
>> >> > overall
>> >> > stability of code, my opinion would be to postpone this to 10.0.
>> >> >
>> >> > Also, I'd suggest to change the process, and have freeze date for all OS
>> >> > updates no later than a month before official FF date. This will give us
>> >> > time to stabilize, and ensure that base on which all new code is being
>> >> > developed is stable when approaching FF.
>> >> >
>> >> > I'd also propose to have freeze for major upgrades of 3rd party packages
>> >> > no
>> >> > later than 2 weeks before FF, which Fuel depends heavily upon. For
>> >> > instance, such will include RabbitMQ, MCollective, Puppet.
>> >> >
>> >> > On Wed, Mar 2, 2016 at 7:34 AM Igor Marnat <imarnat at mirantis.com> wrote:
>> >> >
>> >> > > Igor,
>> >> > > couple of points from my side.
>> >> > >
>> >> > > CentOS 7.2 will be getting updates for several more months, and we
>> >> > > have
>> >> > > snapshots and all the mechanics in place to switch to the next version
>> >> > > when
>> >> > > needed.
>> >> > >
>> >> > > Speaking of getting this update into 9.0, we actually don't need FFE,
>> >> > > we
>> >> > > can merge remaining staff today. It has enough reviews, so if you add
>> >> > > your
>> >> > > +1 today, we don't need FFE.
>> >> > >
>> >> > > https://review.openstack.org/#/c/280338/
>> >> > > https://review.fuel-infra.org/#/c/17400/
>> >> > >
>> >> > >
>> >> > >
>> >> > > Regards,
>> >> > > Igor Marnat
>> >> > >
>> >> > > On Wed, Mar 2, 2016 at 6:23 PM, Dmitry Teselkin
>> >> > > <dteselkin at mirantis.com>
>> >> > > wrote:
>> >> > >
>> >> > >> Igor,
>> >> > >>
>> >> > >> Your statement about updates for 7.2 isn't correct - it will receive
>> >> > >> updates,  because it's the latest release ATM. There is *no* pinning
>> >> > >> inside
>> >> > >> ISO, and the only place where it was 8.0 were docker containers just
>> >> > >> because we had to workaround some issues. But there are no docker
>> >> > >> containers in 9.0, so that's not the case.
>> >> > >> The proposed solution to switch to CentOS-7.2 in fact is based on
>> >> > >> selecting the right snapshot with packages. There is no pinning in
>> >> > >> ISO (it
>> >> > >> was in earlier versions of the spec but was removed).
>> >> > >>
>> >> > >> On Wed, Mar 2, 2016 at 6:11 PM, Igor Kalnitsky
>> >> > >> <ikalnitsky at mirantis.com>
>> >> > >> wrote:
>> >> > >>
>> >> > >>> Dmitry, Igor,
>> >> > >>>
>> >> > >>> > Very important thing is that CentOS 7.1 which master node is based
>> >> > >>> > now
>> >> > >>> > don't get updates any longer.
>> >> > >>>
>> >> > >>> If you are using "fixed" release you must be ready that you won't
>> >> > >>> get
>> >> > >>> any updates. So with CentOS 7.2 the problem still the same.
>> >> > >>>
>> >> > >>> However, let's wait for Fuel PTL decision. I only shared my POV:
>> >> > >>> that's not a critical feature, and taking into account the risks of
>> >> > >>> regression - I'd prefer to do not accept it in 9.0.
>> >> > >>>
>> >> > >>> Regards,
>> >> > >>> Igor
>> >> > >>>
>> >> > >>> On Wed, Mar 2, 2016 at 4:42 PM, Igor Marnat <imarnat at mirantis.com>
>> >> > >>> wrote:
>> >> > >>> > Igor,
>> >> > >>> > please note that this is pretty much not like update of master
>> >> > >>> > node
>> >> > >>> which we
>> >> > >>> > had in 8.0. This is minor _update_ of CentOS from 7.1 to 7.2 which
>> >> > >>> > team
>> >> > >>> > tested for more than 2 months already.
>> >> > >>> >
>> >> > >>> > We don't expect it to require any additional efforts from core or
>> >> > >>> > qa
>> >> > >>> team.
>> >> > >>> >
>> >> > >>> > Very important thing is that CentOS 7.1 which master node is based
>> >> > >>> > now
>> >> > >>> don't
>> >> > >>> > get updates any longer. Updates are only provided for CentOS 7.2.
>> >> > >>> >
>> >> > >>> > So we'll have to switch CentOS 7.1 to CentOS 7.2 anyways.
>> >> > >>> >
>> >> > >>> > We can do it now for more or less free, later in release cycle for
>> >> > >>> higher
>> >> > >>> > risk and QA efforts and after the release for 2x price because of
>> >> > >>> additional
>> >> > >>> > QA cycle we'll need to pass through.
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>> > Regards,
>> >> > >>> > Igor Marnat
>> >> > >>> >
>> >> > >>> > On Wed, Mar 2, 2016 at 2:57 PM, Dmitry Teselkin <
>> >> > >>> dteselkin at mirantis.com>
>> >> > >>> > wrote:
>> >> > >>> >>
>> >> > >>> >> Hi Igor,
>> >> > >>> >>
>> >> > >>> >> Postponing this till Fuel 10 means we have to elaborate a plan to
>> >> > >>> >> do
>> >> > >>> such
>> >> > >>> >> upgrade for Fuel 9 after the release - the underlying system will
>> >> > >>> >> not
>> >> > >>> get
>> >> > >>> >> updated on it's own, and the security issues will not close
>> >> > >>> themselves. The
>> >> > >>> >> problem here is that such upgrade of deployed master node
>> >> > >>> >> requires a
>> >> > >>> lot of
>> >> > >>> >> QA work also.
>> >> > >>> >>
>> >> > >>> >> Since we are not going to update package we build on our own
>> >> > >>> >> (they
>> >> > >>> still
>> >> > >>> >> targeted 7.1) switching master node base to CentOS-72 is not that
>> >> > >>> dangerous
>> >> > >>> >> as it seems.
>> >> > >>> >>
>> >> > >>> >> On Wed, Mar 2, 2016 at 1:54 PM, Igor Kalnitsky <
>> >> > >>> ikalnitsky at mirantis.com>
>> >> > >>> >> wrote:
>> >> > >>> >>>
>> >> > >>> >>> Hey Dmitry,
>> >> > >>> >>>
>> >> > >>> >>> No offence, but I rather against that exception. We have too
>> >> > >>> >>> many
>> >> > >>> >>> things to do in Mitaka, and moving to CentOS 7.2 means
>> >> > >>> >>>
>> >> > >>> >>> * extra effort from core team
>> >> > >>> >>> * extra effort from qa team
>> >> > >>> >>>
>> >> > >>> >>> Moreover, it might block development by introducing
>> >> > >>> >>> unpredictable
>> >> > >>> >>> regressions. Remember 8.0? So I think it'd be better to reduce
>> >> > >>> >>> risk
>> >> > >>> of
>> >> > >>> >>> regressions that affects so many developers by postponing CentOS
>> >> > >>> >>> 7.2
>> >> > >>> >>> till Fuel 10.
>> >> > >>> >>>
>> >> > >>> >>> Thanks,
>> >> > >>> >>> Igor
>> >> > >>> >>>
>> >> > >>> >>>
>> >> > >>> >>> On Mon, Feb 29, 2016 at 7:13 PM, Dmitry Teselkin <
>> >> > >>> dteselkin at mirantis.com>
>> >> > >>> >>> wrote:
>> >> > >>> >>> > I'd like to ask for a feature freeze exception for switching
>> >> > >>> >>> > to
>> >> > >>> >>> > CentOS-7.2
>> >> > >>> >>> > feature [0].
>> >> > >>> >>> >
>> >> > >>> >>> > CentOS-7.2 ISO's have been tested periodically since the
>> >> > >>> >>> > beginning
>> >> > >>> of
>> >> > >>> >>> > the
>> >> > >>> >>> > year, and all major issues were addressed / fixed at the
>> >> > >>> >>> > moment.
>> >> > >>> During
>> >> > >>> >>> > the
>> >> > >>> >>> > last weekend I've made 70 BVT runs to verify that the
>> >> > >>> >>> > solution
>> >> > >>> [2] for
>> >> > >>> >>> > the
>> >> > >>> >>> > last issue - e1000 transmit unit hangs works. And it works, 0
>> >> > >>> tests of
>> >> > >>> >>> > 35
>> >> > >>> >>> > failed [3].
>> >> > >>> >>> >
>> >> > >>> >>> > Benefits of switching to CentOS-7.2 are quite obvious - we
>> >> > >>> >>> > will
>> >> > >>> return
>> >> > >>> >>> > to
>> >> > >>> >>> > latest supported CentOS release, will fix a lot of bugs /
>> >> > >>> >>> > security
>> >> > >>> >>> > issues
>> >> > >>> >>> > [4] and will make further updates easier.
>> >> > >>> >>> >
>> >> > >>> >>> > Risk of regression still exists, but it's quite low, 35
>> >> > >>> >>> > successful
>> >> > >>> BVTs
>> >> > >>> >>> > can't be wrong.
>> >> > >>> >>> >
>> >> > >>> >>> > To finish that feature the following should be done:
>> >> > >>> >>> > * review and merge e1000 workaround [2]
>> >> > >>> >>> > * review and merge spec [0]
>> >> > >>> >>> > * review and merge request that switches build CI to
>> >> > >>> >>> > CentOS-7.2 [5]
>> >> > >>> >>> >
>> >> > >>> >>> > I expect the last day it could be done is March, 4.
>> >> > >>> >>> >
>> >> > >>> >>> > [0] https://review.openstack.org/#/c/280338/
>> >> > >>> >>> > [1] https://bugs.launchpad.net/fuel/+bug/1526544
>> >> > >>> >>> > [2] https://review.openstack.org/#/c/285306/
>> >> > >>> >>> > [3]
>> >> > >>> https://etherpad.openstack.org/p/r.1c4cfee8185326d6922d6c9321404350
>> >> > >>> >>> > [4]
>> >> > >>> https://etherpad.openstack.org/p/r.a7fe0b575d891ed81206765fa5be6630
>> >> > >>> >>> > [5] https://review.fuel-infra.org/#/c/17400/
>> >> > >>> >>> >
>> >> > >>> >>> >
>> >> > >>> >>> > --
>> >> > >>> >>> > Thanks,
>> >> > >>> >>> > Dmitry Teselkin
>> >> > >>> >>> > Mirantis
>> >> > >>> >>> > http://www.mirantis.com
>> >> > >>> >>> >
>> >> > >>> >>> >
>> >> > >>> >>> >
>> >> > >>>
>> >> > >>> __________________________________________________________________________
>> >> > >>> >>> > OpenStack Development Mailing List (not for usage questions)
>> >> > >>> >>> > Unsubscribe:
>> >> > >>> >>> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > >>> >>> >
>> >> > >>> >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> > >>> >>> >
>> >> > >>> >>>
>> >> > >>> >>>
>> >> > >>> >>>
>> >> > >>>
>> >> > >>> __________________________________________________________________________
>> >> > >>> >>> OpenStack Development Mailing List (not for usage questions)
>> >> > >>> >>> Unsubscribe:
>> >> > >>> >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > >>> >>>
>> >> > >>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> > >>> >>
>> >> > >>> >>
>> >> > >>> >>
>> >> > >>> >>
>> >> > >>> >> --
>> >> > >>> >> Thanks,
>> >> > >>> >> Dmitry Teselkin
>> >> > >>> >> Mirantis
>> >> > >>> >> http://www.mirantis.com
>> >> > >>> >>
>> >> > >>> >>
>> >> > >>>
>> >> > >>> __________________________________________________________________________
>> >> > >>> >> OpenStack Development Mailing List (not for usage questions)
>> >> > >>> >> Unsubscribe:
>> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > >>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> > >>> >>
>> >> > >>> >
>> >> > >>> >
>> >> > >>> >
>> >> > >>>
>> >> > >>> __________________________________________________________________________
>> >> > >>> > OpenStack Development Mailing List (not for usage questions)
>> >> > >>> > Unsubscribe:
>> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > >>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> > >>> >
>> >> > >>>
>> >> > >>>
>> >> > >>>
>> >> > >>> __________________________________________________________________________
>> >> > >>> OpenStack Development Mailing List (not for usage questions)
>> >> > >>> Unsubscribe:
>> >> > >>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> > >>>
>> >> > >>
>> >> > >>
>> >> > >>
>> >> > >> --
>> >> > >> Thanks,
>> >> > >> Dmitry Teselkin
>> >> > >> Mirantis
>> >> > >> http://www.mirantis.com
>> >> > >>
>> >> > >>
>> >> > >> __________________________________________________________________________
>> >> > >> OpenStack Development Mailing List (not for usage questions)
>> >> > >> Unsubscribe:
>> >> > >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> > >>
>> >> > >>
>> >> > >
>> >> > > __________________________________________________________________________
>> >> > > OpenStack Development Mailing List (not for usage questions)
>> >> > > Unsubscribe:
>> >> > > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> > >
>> >> > --
>> >> > Mike Scherbakov
>> >> > #mihgen
>> >>
>> >> >
>> >> > __________________________________________________________________________
>> >> > OpenStack Development Mailing List (not for usage questions)
>> >> > Unsubscribe:
>> >> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>
>> >> __________________________________________________________________________
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> > --
>> > Mike Scherbakov
>> > #mihgen
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>>
>>
>> --
>> Aleksandra Fedorova
>> CI Team Lead
>> bookwar
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Aleksandra Fedorova
CI Team Lead
bookwar