Open Stack

A clarification on 2: we are going to keep fuel_5.1_* jobs around for
the benefict 5.1.x maintenance releases, that should take care of
Icehouse testing for us, so I don't think we should keep Icehouse jobs
in 6.0/master after Juno is stabilized. What we should do instead is
drop Icehouse jobs and introduce Kilo jobs tracking OpenStack master,
and try to keep manifests in Fuel master forward-compatible with Kilo
through the 6.x release series.

On Tue, Sep 9, 2014 at 1:39 PM, Mike Scherbakov
<mscherbakov at mirantis.com> wrote:
> Aleksandra,
> you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but "4
> fuel-library tests" should happen right after we create stable/5.1. Also,
> for Fuel CI for OSTF - I don't think it's actually necessary to support <5.0
> envs.
>
> Your questions:
>
> Create jobs for both Icehouse and Juno, but it doesn't make sense to do
> staging for Juno till it starts to pass deployment in HA mode. Once it
> passes deployment in HA, staging should be enabled. Then, once it passes
> OSTF - we extend criteria, and pass only those mirrors which also pass OSTF
> phase
> Once Juno starts to pass BVT with OSTF check enabled, I think we can disable
> Icehouse checks. Not sure about fuel-library tests on Fuel CI with Icehouse
> - we might want to continue using them.
>
> Thanks,
>
> On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova
> <afedorova at mirantis.com> wrote:
>>
>> > Our Fuel CI can do 4 builds against puppet modules: 2 voting, with
>> > Icehouse packages; 2 non-voting, with Juno packages.
>> > Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno)
>> > actually before Juno becomes stable. We will be able to run 2 sets of BVTs
>> > (against Icehouse and Juno), and it means that we will be able to see almost
>> > immediately if something in nailgun/astute/puppet integration broke. For
>> > Juno builds it's going to be all red initially.
>>
>> Let me rephrase:
>>
>> We keep one Fuel master branch for two OpenStack releases. And we make
>> sure that Fuel master code is compatible with both of them. And we use
>> current release (Icehouse) as a reference for test results of upcoming
>> release, till we obtain stable enough reference point in Juno itself.
>> Moreover we'd like to have OSTF code running on all previous Fuel releases.
>>
>> Changes to CI workflow look as follows:
>>
>> Nightly builds:
>>   1) We build two mirrors: one for Icehouse and one for Juno.
>>   2) From each mirror we build Fuel ISO using exactly the same fuel master
>> branch code.
>>   3) Then we run BVT tests on both (using the same fuel-main code for
>> system tests).
>>   4) If Icehouse BVT tests pass, we deploy both ISO images (even with
>> failed Juno tests) onto Fuel CI.
>>
>> On Fuel CI we should run:
>>   - 4 fuel-library tests (revert master node, inject fuel-library code in
>> master node and run deployment):
>>             2 (ubuntu and centos) voting Icehouse tests and 2 non-voting
>> Juno tests
>>   - 5 OSTF tests (revert deployed environment, inject OSTF code into
>> master node, run OSTF):
>>             voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on
>> master/Juno
>>   - other tests, which don't use prebuilt environment, work as before
>>
>> The major action point here would be OSTF tests, as we don't have yet
>> working implementation of injecting OSTF code into deployed environment. And
>> we don't run any tests on old environments.
>>
>>
>> Questions:
>>
>> 1) How should we test mirrors?
>>
>> Current master mirrors go through the 4 hours test cycle involving Fuel
>> ISO build:
>>   1. we build temporary mirror
>>   2. build custom iso from it
>>   3. run two custom bvt jobs
>>   4. if they pass we move mirror to stable and sitch to it for our
>> "primary" fuel_master_iso
>>
>> Should we test only Icehouse mirrors, or both, but ignoring again failed
>> BVT for Juno? Maybe we should enable these tests only later in release
>> cycle, say, after SCF?
>>
>> 2) It is not clear for me when and how we will switch from supporting two
>> releases back to one.
>> Should we add one more milestone to our release process? The "Switching
>> point", when we disable and remove Icehouse tasks and move to Juno
>> completely? I guess it should happen before next SCF?
>>
>>
>>
>> On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov <mscherbakov at mirantis.com>
>> wrote:
>>>
>>> > What we need to achieve that is have 2 build series based on Fuel
>>> master: one with Icehouse packages, and one with Juno, and, as Mike
>>> proposed, keep our manifests backwards compatible with Icehouse.
>>> Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting,
>>> with Icehouse packages; 2 non-voting, with Juno packages.
>>>
>>> Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually
>>> before Juno becomes stable. We will be able to run 2 sets of BVTs (against
>>> Icehouse and Juno), and it means that we will be able to see almost
>>> immediately if something in nailgun/astute/puppet integration broke. For
>>> Juno builds it's going to be all red initially.
>>>
>>> Another suggestion would be to lower green switch in BVTs for Juno:
>>> first, when it passes deployment; and then, if it finally passes OSTF.
>>>
>>> I'd like to hear QA & DevOps opinion on all the above. Immediately we
>>> would need just standard stuff which is in checklists for OSCI & DevOps
>>> teams, and ideally soon after that - ability to have Fuel CI running 4
>>> builds, not 2, against our master, as mentioned above.
>>>
>>> On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov <rvyalov at mirantis.com>
>>> wrote:
>>>>
>>>> All OSCI action items for prepare HCF check list  has been done
>>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov
>>>> <mscherbakov at mirantis.com> wrote:
>>>>>
>>>>> Thanks Alexandra.
>>>>>
>>>>> We land a few patches a day currently, so I think we can open stable
>>>>> branch. If we see no serious objections in next 12 hours, let's do it. We
>>>>> would need to immediately notify everyone in mailing list - that for every
>>>>> patch for 5.1, it should go first to master, and then to stable/5.1.
>>>>>
>>>>> Is everything ready from DevOps, OSCI (packaging) side to do this? Fuel
>>>>> CI, OBS, etc.?
>>>>>
>>>>> On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova
>>>>> <afedorova at mirantis.com> wrote:
>>>>>>
>>>>>> As I understand your proposal, we need to split our HCF milestone into
>>>>>> two check points: Branching Point and HCF itself.
>>>>>>
>>>>>> Branching point should happen somewhere in between SCF and HCF. And
>>>>>> though It may coincide with HCF, it needs its own list of requirements. This
>>>>>> will give us the possibility to untie two events and make a separate
>>>>>> decision on branching without enforcing all HCF criteria.
>>>>>>
>>>>>> From the DevOps point of view it changes almost nothing, it just adds
>>>>>> a bit more discussion items on the management side and slight modifications
>>>>>> to our checklists.
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko
>>>>>> <dborodaenko at mirantis.com> wrote:
>>>>>>>
>>>>>>> TL;DR: Yes, our work on 6.0 features is currently blocked and it is
>>>>>>> becoming a major problem. No, I don't think we should create
>>>>>>> pre-release or feature branches. Instead, we should create stable/5.1
>>>>>>> branches and open master for 6.0 work.
>>>>>>>
>>>>>>> We have reached a point in 5.1 release cycle where the scope of
>>>>>>> issues
>>>>>>> we are willing to address in this release is narrow enough to not
>>>>>>> require full attention of the whole team. We have engineers working
>>>>>>> on
>>>>>>> 6.0 features, and their work is essentially blocked until they have
>>>>>>> somewhere to commit their changes.
>>>>>>>
>>>>>>> Simply creating new branches is not even close to solving this
>>>>>>> problem: we have a whole CI infrastructure around every active
>>>>>>> release
>>>>>>> series (currently 5.1, 5.0, 4.1), including test jobs for gerrit
>>>>>>> commits, package repository mirrors updates, ISO image builds, smoke,
>>>>>>> build verification, and swarm tests for ISO images, documentation
>>>>>>> builds, etc. A branch without all that infrastructure isn't any
>>>>>>> better
>>>>>>> than current status quo: every developer tracking their own 6.0 work
>>>>>>> locally.
>>>>>>>
>>>>>>> Unrelated to all that, we also had a lot of very negative experience
>>>>>>> with feature branches in the past [0] [1], which is why we have
>>>>>>> decided to follow the OpenStack branching strategy: commit all
>>>>>>> feature
>>>>>>> changes directly to master and track bugfixes for stable releases in
>>>>>>> stable/* branches.
>>>>>>>
>>>>>>> [0] https://lists.launchpad.net/fuel-dev/msg00127.html
>>>>>>> [1] https://lists.launchpad.net/fuel-dev/msg00028.html
>>>>>>>
>>>>>>> I'm also against declaring a "hard code freeze with exceptions", HCF
>>>>>>> should remain tied to our ability to declare a release candidate. If
>>>>>>> we can't release with the bugs we already know about, declaring HCF
>>>>>>> before fixing these bugs would be an empty gesture.
>>>>>>>
>>>>>>> Creating stable/5.1 now instead of waiting for hard code freeze for
>>>>>>> 5.1 will cost us two things:
>>>>>>>
>>>>>>> 1) DevOps team will have to update our CI infrastructure for one more
>>>>>>> release series. It's something we have to do for 6.0 sooner or later,
>>>>>>> so this may be a disruption, but not an additional effort.
>>>>>>>
>>>>>>> 2) All commits targeted for 5.1 will have to be proposed for two
>>>>>>> branches (master and stable/5.1) instead of just one (master). This
>>>>>>> will require additional effort, but I think that it is significantly
>>>>>>> smaller than the cost of spinning our wheels on 6.0 efforts.
>>>>>>>
>>>>>>> -DmitryB
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov
>>>>>>> <dmescheryakov at mirantis.com> wrote:
>>>>>>> > Hello Fuelers,
>>>>>>> >
>>>>>>> > Right now we have the following policy in place: the branches for a
>>>>>>> > release are opened only after its 'parent' release have reached
>>>>>>> > hard
>>>>>>> > code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1
>>>>>>> > and
>>>>>>> > 6.0.
>>>>>>> >
>>>>>>> > And that is the problem: if parent release is delayed, we can't
>>>>>>> > properly start development of a child release because we don't have
>>>>>>> > branches to commit. That is current issue with 6.0: we already
>>>>>>> > started
>>>>>>> > to work on pushing Juno in to 6.0, but if we are to make changes to
>>>>>>> > our deployment code we have nowhere to store them.
>>>>>>> >
>>>>>>> > IMHO the issue could easily be resolved by creation of pre-release
>>>>>>> > branches, which are merged together with parent branches once the
>>>>>>> > parent reaches HCF. Say, we use branch 'pre-6.0' for initial
>>>>>>> > development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into
>>>>>>> > master
>>>>>>> > and continue development here. After that pre-6.0 is abandoned.
>>>>>>> >
>>>>>>> > What do you think?
>>>>>>> >
>>>>>>> > Thanks,
>>>>>>> >
>>>>>>> > Dmitry
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > OpenStack-dev mailing list
>>>>>>> > OpenStack-dev at lists.openstack.org
>>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Dmitry Borodaenko
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Aleksandra Fedorova
>>>>>> bookwar
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mike Scherbakov
>>>>> #mihgen
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> --
>>> Mike Scherbakov
>>> #mihgen
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Aleksandra Fedorova
>> bookwar
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Mike Scherbakov
> #mihgen
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Dmitry Borodaenko