Open Stack

Aleksandra,
you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but "4
fuel-library tests" should happen right after we create stable/5.1. Also,
for Fuel CI for OSTF - I don't think it's actually necessary to support
<5.0 envs.

Your questions:

   1. Create jobs for both Icehouse and Juno, but it doesn't make sense to
   do staging for Juno till it starts to pass deployment in HA mode. Once it
   passes deployment in HA, staging should be enabled. Then, once it passes
   OSTF - we extend criteria, and pass only those mirrors which also pass OSTF
   phase
   2. Once Juno starts to pass BVT with OSTF check enabled, I think we can
   disable Icehouse checks. Not sure about fuel-library tests on Fuel CI with
   Icehouse - we might want to continue using them.

Thanks,

On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova <
afedorova at mirantis.com> wrote:

> > Our Fuel CI can do 4 builds against puppet modules: 2 voting, with
> Icehouse packages; 2 non-voting, with Juno packages.
> > Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno)
> actually before Juno becomes stable. We will be able to run 2 sets of BVTs
> (against Icehouse and Juno), and it means that we will be able to see
> almost immediately if something in nailgun/astute/puppet integration broke.
> For Juno builds it's going to be all red initially.
>
> Let me rephrase:
>
> We keep one Fuel master branch for two OpenStack releases. And we make
> sure that Fuel master code is compatible with both of them. And we use
> current release (Icehouse) as a reference for test results of upcoming
> release, till we obtain stable enough reference point in Juno itself.
> Moreover we'd like to have OSTF code running on all previous Fuel releases.
>
> Changes to CI workflow look as follows:
>
> Nightly builds:
>   1) We build two mirrors: one for Icehouse and one for Juno.
>   2) From each mirror we build Fuel ISO using exactly the same fuel master
> branch code.
>   3) Then we run BVT tests on both (using the same fuel-main code for
> system tests).
>   4) If Icehouse BVT tests pass, we deploy both ISO images (even with
> failed Juno tests) onto Fuel CI.
>
> On Fuel CI we should run:
>   - 4 fuel-library tests (revert master node, inject fuel-library code in
> master node and run deployment):
>             2 (ubuntu and centos) voting Icehouse tests and 2 non-voting
> Juno tests
>   - 5 OSTF tests (revert deployed environment, inject OSTF code into
> master node, run OSTF):
>             voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on
> master/Juno
>   - other tests, which don't use prebuilt environment, work as before
>
> The major action point here would be OSTF tests, as we don't have yet
> working implementation of injecting OSTF code into deployed environment.
> And we don't run any tests on old environments.
>
>
> Questions:
>
> 1) How should we test mirrors?
>
> Current master mirrors go through the 4 hours test cycle involving Fuel
> ISO build:
>   1. we build temporary mirror
>   2. build custom iso from it
>   3. run two custom bvt jobs
>   4. if they pass we move mirror to stable and sitch to it for our
> "primary" fuel_master_iso
>
> Should we test only Icehouse mirrors, or both, but ignoring again failed
> BVT for Juno? Maybe we should enable these tests only later in release
> cycle, say, after SCF?
>
> 2) It is not clear for me when and how we will switch from supporting two
> releases back to one.
> Should we add one more milestone to our release process? The "Switching
> point", when we disable and remove Icehouse tasks and move to Juno
> completely? I guess it should happen before next SCF?
>
>
>
> On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov <mscherbakov at mirantis.com>
> wrote:
>
>> > What we need to achieve that is have 2 build series based on Fuel
>> master: one with Icehouse packages, and one with Juno, and, as Mike
>> proposed, keep our manifests backwards compatible with Icehouse.
>> Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting,
>> with Icehouse packages; 2 non-voting, with Juno packages.
>>
>> Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually
>> before Juno becomes stable. We will be able to run 2 sets of BVTs (against
>> Icehouse and Juno), and it means that we will be able to see almost
>> immediately if something in nailgun/astute/puppet integration broke. For
>> Juno builds it's going to be all red initially.
>>
>> Another suggestion would be to lower green switch in BVTs for Juno:
>> first, when it passes deployment; and then, if it finally passes OSTF.
>>
>> I'd like to hear QA & DevOps opinion on all the above. Immediately we
>> would need just standard stuff which is in checklists for OSCI & DevOps
>> teams, and ideally soon after that - ability to have Fuel CI running 4
>> builds, not 2, against our master, as mentioned above.
>>
>> On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov <rvyalov at mirantis.com>
>> wrote:
>>
>>> All OSCI action items for prepare HCF check list  has been done
>>>
>>>
>>> On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov <
>>> mscherbakov at mirantis.com> wrote:
>>>
>>>> Thanks Alexandra.
>>>>
>>>> We land a few patches a day currently, so I think we can open stable
>>>> branch. If we see no serious objections in next 12 hours, let's do it. We
>>>> would need to immediately notify everyone in mailing list - that for every
>>>> patch for 5.1, it should go first to master, and then to stable/5.1.
>>>>
>>>> Is everything ready from DevOps, OSCI (packaging) side to do this? Fuel
>>>> CI, OBS, etc.?
>>>>
>>>> On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova <
>>>> afedorova at mirantis.com> wrote:
>>>>
>>>>> As I understand your proposal, we need to split our HCF milestone into
>>>>> two check points: Branching Point and HCF itself.
>>>>>
>>>>> Branching point should happen somewhere in between SCF and HCF. And
>>>>> though It may coincide with HCF, it needs its own list of requirements.
>>>>> This will give us the possibility to untie two events and make a separate
>>>>> decision on branching without enforcing all HCF criteria.
>>>>>
>>>>> From the DevOps point of view it changes almost nothing, it just adds
>>>>> a bit more discussion items on the management side and slight modifications
>>>>> to our checklists.
>>>>>
>>>>>
>>>>> On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko <
>>>>> dborodaenko at mirantis.com> wrote:
>>>>>
>>>>>> TL;DR: Yes, our work on 6.0 features is currently blocked and it is
>>>>>> becoming a major problem. No, I don't think we should create
>>>>>> pre-release or feature branches. Instead, we should create stable/5.1
>>>>>> branches and open master for 6.0 work.
>>>>>>
>>>>>> We have reached a point in 5.1 release cycle where the scope of issues
>>>>>> we are willing to address in this release is narrow enough to not
>>>>>> require full attention of the whole team. We have engineers working on
>>>>>> 6.0 features, and their work is essentially blocked until they have
>>>>>> somewhere to commit their changes.
>>>>>>
>>>>>> Simply creating new branches is not even close to solving this
>>>>>> problem: we have a whole CI infrastructure around every active release
>>>>>> series (currently 5.1, 5.0, 4.1), including test jobs for gerrit
>>>>>> commits, package repository mirrors updates, ISO image builds, smoke,
>>>>>> build verification, and swarm tests for ISO images, documentation
>>>>>> builds, etc. A branch without all that infrastructure isn't any better
>>>>>> than current status quo: every developer tracking their own 6.0 work
>>>>>> locally.
>>>>>>
>>>>>> Unrelated to all that, we also had a lot of very negative experience
>>>>>> with feature branches in the past [0] [1], which is why we have
>>>>>> decided to follow the OpenStack branching strategy: commit all feature
>>>>>> changes directly to master and track bugfixes for stable releases in
>>>>>> stable/* branches.
>>>>>>
>>>>>> [0] https://lists.launchpad.net/fuel-dev/msg00127.html
>>>>>> [1] https://lists.launchpad.net/fuel-dev/msg00028.html
>>>>>>
>>>>>> I'm also against declaring a "hard code freeze with exceptions", HCF
>>>>>> should remain tied to our ability to declare a release candidate. If
>>>>>> we can't release with the bugs we already know about, declaring HCF
>>>>>> before fixing these bugs would be an empty gesture.
>>>>>>
>>>>>> Creating stable/5.1 now instead of waiting for hard code freeze for
>>>>>> 5.1 will cost us two things:
>>>>>>
>>>>>> 1) DevOps team will have to update our CI infrastructure for one more
>>>>>> release series. It's something we have to do for 6.0 sooner or later,
>>>>>> so this may be a disruption, but not an additional effort.
>>>>>>
>>>>>> 2) All commits targeted for 5.1 will have to be proposed for two
>>>>>> branches (master and stable/5.1) instead of just one (master). This
>>>>>> will require additional effort, but I think that it is significantly
>>>>>> smaller than the cost of spinning our wheels on 6.0 efforts.
>>>>>>
>>>>>> -DmitryB
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov
>>>>>> <dmescheryakov at mirantis.com> wrote:
>>>>>> > Hello Fuelers,
>>>>>> >
>>>>>> > Right now we have the following policy in place: the branches for a
>>>>>> > release are opened only after its 'parent' release have reached hard
>>>>>> > code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 and
>>>>>> > 6.0.
>>>>>> >
>>>>>> > And that is the problem: if parent release is delayed, we can't
>>>>>> > properly start development of a child release because we don't have
>>>>>> > branches to commit. That is current issue with 6.0: we already
>>>>>> started
>>>>>> > to work on pushing Juno in to 6.0, but if we are to make changes to
>>>>>> > our deployment code we have nowhere to store them.
>>>>>> >
>>>>>> > IMHO the issue could easily be resolved by creation of pre-release
>>>>>> > branches, which are merged together with parent branches once the
>>>>>> > parent reaches HCF. Say, we use branch 'pre-6.0' for initial
>>>>>> > development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into
>>>>>> master
>>>>>> > and continue development here. After that pre-6.0 is abandoned.
>>>>>> >
>>>>>> > What do you think?
>>>>>> >
>>>>>> > Thanks,
>>>>>> >
>>>>>> > Dmitry
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > OpenStack-dev mailing list
>>>>>> > OpenStack-dev at lists.openstack.org
>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dmitry Borodaenko
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Aleksandra Fedorova
>>>>> bookwar
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Mike Scherbakov
>>>> #mihgen
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Mike Scherbakov
>> #mihgen
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Aleksandra Fedorova
> bookwar
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Mike Scherbakov
#mihgen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140910/10badd4a/attachment.html>