Open Stack

> Our Fuel CI can do 4 builds against puppet modules: 2 voting, with
Icehouse packages; 2 non-voting, with Juno packages.
> Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually
before Juno becomes stable. We will be able to run 2 sets of BVTs (against
Icehouse and Juno), and it means that we will be able to see almost
immediately if something in nailgun/astute/puppet integration broke. For
Juno builds it's going to be all red initially.

Let me rephrase:

We keep one Fuel master branch for two OpenStack releases. And we make sure
that Fuel master code is compatible with both of them. And we use current
release (Icehouse) as a reference for test results of upcoming release,
till we obtain stable enough reference point in Juno itself. Moreover we'd
like to have OSTF code running on all previous Fuel releases.

Changes to CI workflow look as follows:

Nightly builds:
  1) We build two mirrors: one for Icehouse and one for Juno.
  2) From each mirror we build Fuel ISO using exactly the same fuel master
branch code.
  3) Then we run BVT tests on both (using the same fuel-main code for
system tests).
  4) If Icehouse BVT tests pass, we deploy both ISO images (even with
failed Juno tests) onto Fuel CI.

On Fuel CI we should run:
  - 4 fuel-library tests (revert master node, inject fuel-library code in
master node and run deployment):
            2 (ubuntu and centos) voting Icehouse tests and 2 non-voting
Juno tests
  - 5 OSTF tests (revert deployed environment, inject OSTF code into master
node, run OSTF):
            voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on
master/Juno
  - other tests, which don't use prebuilt environment, work as before

The major action point here would be OSTF tests, as we don't have yet
working implementation of injecting OSTF code into deployed environment.
And we don't run any tests on old environments.


Questions:

1) How should we test mirrors?

Current master mirrors go through the 4 hours test cycle involving Fuel ISO
build:
  1. we build temporary mirror
  2. build custom iso from it
  3. run two custom bvt jobs
  4. if they pass we move mirror to stable and sitch to it for our
"primary" fuel_master_iso

Should we test only Icehouse mirrors, or both, but ignoring again failed
BVT for Juno? Maybe we should enable these tests only later in release
cycle, say, after SCF?

2) It is not clear for me when and how we will switch from supporting two
releases back to one.
Should we add one more milestone to our release process? The "Switching
point", when we disable and remove Icehouse tasks and move to Juno
completely? I guess it should happen before next SCF?



On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov <mscherbakov at mirantis.com>
wrote:

> > What we need to achieve that is have 2 build series based on Fuel
> master: one with Icehouse packages, and one with Juno, and, as Mike
> proposed, keep our manifests backwards compatible with Icehouse.
> Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting,
> with Icehouse packages; 2 non-voting, with Juno packages.
>
> Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno) actually
> before Juno becomes stable. We will be able to run 2 sets of BVTs (against
> Icehouse and Juno), and it means that we will be able to see almost
> immediately if something in nailgun/astute/puppet integration broke. For
> Juno builds it's going to be all red initially.
>
> Another suggestion would be to lower green switch in BVTs for Juno: first,
> when it passes deployment; and then, if it finally passes OSTF.
>
> I'd like to hear QA & DevOps opinion on all the above. Immediately we
> would need just standard stuff which is in checklists for OSCI & DevOps
> teams, and ideally soon after that - ability to have Fuel CI running 4
> builds, not 2, against our master, as mentioned above.
>
> On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov <rvyalov at mirantis.com> wrote:
>
>> All OSCI action items for prepare HCF check list  has been done
>>
>>
>> On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov <mscherbakov at mirantis.com
>> > wrote:
>>
>>> Thanks Alexandra.
>>>
>>> We land a few patches a day currently, so I think we can open stable
>>> branch. If we see no serious objections in next 12 hours, let's do it. We
>>> would need to immediately notify everyone in mailing list - that for every
>>> patch for 5.1, it should go first to master, and then to stable/5.1.
>>>
>>> Is everything ready from DevOps, OSCI (packaging) side to do this? Fuel
>>> CI, OBS, etc.?
>>>
>>> On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova <
>>> afedorova at mirantis.com> wrote:
>>>
>>>> As I understand your proposal, we need to split our HCF milestone into
>>>> two check points: Branching Point and HCF itself.
>>>>
>>>> Branching point should happen somewhere in between SCF and HCF. And
>>>> though It may coincide with HCF, it needs its own list of requirements.
>>>> This will give us the possibility to untie two events and make a separate
>>>> decision on branching without enforcing all HCF criteria.
>>>>
>>>> From the DevOps point of view it changes almost nothing, it just adds a
>>>> bit more discussion items on the management side and slight modifications
>>>> to our checklists.
>>>>
>>>>
>>>> On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko <
>>>> dborodaenko at mirantis.com> wrote:
>>>>
>>>>> TL;DR: Yes, our work on 6.0 features is currently blocked and it is
>>>>> becoming a major problem. No, I don't think we should create
>>>>> pre-release or feature branches. Instead, we should create stable/5.1
>>>>> branches and open master for 6.0 work.
>>>>>
>>>>> We have reached a point in 5.1 release cycle where the scope of issues
>>>>> we are willing to address in this release is narrow enough to not
>>>>> require full attention of the whole team. We have engineers working on
>>>>> 6.0 features, and their work is essentially blocked until they have
>>>>> somewhere to commit their changes.
>>>>>
>>>>> Simply creating new branches is not even close to solving this
>>>>> problem: we have a whole CI infrastructure around every active release
>>>>> series (currently 5.1, 5.0, 4.1), including test jobs for gerrit
>>>>> commits, package repository mirrors updates, ISO image builds, smoke,
>>>>> build verification, and swarm tests for ISO images, documentation
>>>>> builds, etc. A branch without all that infrastructure isn't any better
>>>>> than current status quo: every developer tracking their own 6.0 work
>>>>> locally.
>>>>>
>>>>> Unrelated to all that, we also had a lot of very negative experience
>>>>> with feature branches in the past [0] [1], which is why we have
>>>>> decided to follow the OpenStack branching strategy: commit all feature
>>>>> changes directly to master and track bugfixes for stable releases in
>>>>> stable/* branches.
>>>>>
>>>>> [0] https://lists.launchpad.net/fuel-dev/msg00127.html
>>>>> [1] https://lists.launchpad.net/fuel-dev/msg00028.html
>>>>>
>>>>> I'm also against declaring a "hard code freeze with exceptions", HCF
>>>>> should remain tied to our ability to declare a release candidate. If
>>>>> we can't release with the bugs we already know about, declaring HCF
>>>>> before fixing these bugs would be an empty gesture.
>>>>>
>>>>> Creating stable/5.1 now instead of waiting for hard code freeze for
>>>>> 5.1 will cost us two things:
>>>>>
>>>>> 1) DevOps team will have to update our CI infrastructure for one more
>>>>> release series. It's something we have to do for 6.0 sooner or later,
>>>>> so this may be a disruption, but not an additional effort.
>>>>>
>>>>> 2) All commits targeted for 5.1 will have to be proposed for two
>>>>> branches (master and stable/5.1) instead of just one (master). This
>>>>> will require additional effort, but I think that it is significantly
>>>>> smaller than the cost of spinning our wheels on 6.0 efforts.
>>>>>
>>>>> -DmitryB
>>>>>
>>>>>
>>>>> On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov
>>>>> <dmescheryakov at mirantis.com> wrote:
>>>>> > Hello Fuelers,
>>>>> >
>>>>> > Right now we have the following policy in place: the branches for a
>>>>> > release are opened only after its 'parent' release have reached hard
>>>>> > code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1 and
>>>>> > 6.0.
>>>>> >
>>>>> > And that is the problem: if parent release is delayed, we can't
>>>>> > properly start development of a child release because we don't have
>>>>> > branches to commit. That is current issue with 6.0: we already
>>>>> started
>>>>> > to work on pushing Juno in to 6.0, but if we are to make changes to
>>>>> > our deployment code we have nowhere to store them.
>>>>> >
>>>>> > IMHO the issue could easily be resolved by creation of pre-release
>>>>> > branches, which are merged together with parent branches once the
>>>>> > parent reaches HCF. Say, we use branch 'pre-6.0' for initial
>>>>> > development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into
>>>>> master
>>>>> > and continue development here. After that pre-6.0 is abandoned.
>>>>> >
>>>>> > What do you think?
>>>>> >
>>>>> > Thanks,
>>>>> >
>>>>> > Dmitry
>>>>> >
>>>>> > _______________________________________________
>>>>> > OpenStack-dev mailing list
>>>>> > OpenStack-dev at lists.openstack.org
>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dmitry Borodaenko
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Aleksandra Fedorova
>>>> bookwar
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Mike Scherbakov
>>> #mihgen
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Mike Scherbakov
> #mihgen
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Aleksandra Fedorova
bookwar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140910/d719fa69/attachment.html>