Open Stack

What would be your suggestions then on the issue? We want to avoid breaking
of our fuel-library and other code without knowing about it, so it's not an
option to simply forget about compatibility with Icehouse packages.

And I'm actually +1 for introducing Kilo CI jobs as soon as we can, so to
keep our CI cycle very close to the upstream.

On Thu, Sep 11, 2014 at 11:22 PM, Roman Vyalov <rvyalov at mirantis.com> wrote:

> Mike,
> 2 jobs for Icehouse and Juno equal 2 different repository with packages
> for Fuel 6.0. This can be problem for current osci workflow.
> For example:  We need building new packages. Which repository we must put
> packages? to icehouse or/and Juno ?
> if new packages will break icehouse repository, but required for Juno ...
>
> On Wed, Sep 10, 2014 at 12:39 AM, Mike Scherbakov <
> mscherbakov at mirantis.com> wrote:
>
>> Aleksandra,
>> you've got us exactly right. Fuel CI for OSTF can wait a bit longer, but "4
>> fuel-library tests" should happen right after we create stable/5.1. Also,
>> for Fuel CI for OSTF - I don't think it's actually necessary to support
>> <5.0 envs.
>>
>> Your questions:
>>
>>    1. Create jobs for both Icehouse and Juno, but it doesn't make sense
>>    to do staging for Juno till it starts to pass deployment in HA mode. Once
>>    it passes deployment in HA, staging should be enabled. Then, once it passes
>>    OSTF - we extend criteria, and pass only those mirrors which also pass OSTF
>>    phase
>>    2. Once Juno starts to pass BVT with OSTF check enabled, I think we
>>    can disable Icehouse checks. Not sure about fuel-library tests on Fuel CI
>>    with Icehouse - we might want to continue using them.
>>
>> Thanks,
>>
>> On Wed, Sep 10, 2014 at 12:22 AM, Aleksandra Fedorova <
>> afedorova at mirantis.com> wrote:
>>
>>> > Our Fuel CI can do 4 builds against puppet modules: 2 voting, with
>>> Icehouse packages; 2 non-voting, with Juno packages.
>>> > Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno)
>>> actually before Juno becomes stable. We will be able to run 2 sets of BVTs
>>> (against Icehouse and Juno), and it means that we will be able to see
>>> almost immediately if something in nailgun/astute/puppet integration broke.
>>> For Juno builds it's going to be all red initially.
>>>
>>> Let me rephrase:
>>>
>>> We keep one Fuel master branch for two OpenStack releases. And we make
>>> sure that Fuel master code is compatible with both of them. And we use
>>> current release (Icehouse) as a reference for test results of upcoming
>>> release, till we obtain stable enough reference point in Juno itself.
>>> Moreover we'd like to have OSTF code running on all previous Fuel releases.
>>>
>>> Changes to CI workflow look as follows:
>>>
>>> Nightly builds:
>>>   1) We build two mirrors: one for Icehouse and one for Juno.
>>>   2) From each mirror we build Fuel ISO using exactly the same fuel
>>> master branch code.
>>>   3) Then we run BVT tests on both (using the same fuel-main code for
>>> system tests).
>>>   4) If Icehouse BVT tests pass, we deploy both ISO images (even with
>>> failed Juno tests) onto Fuel CI.
>>>
>>> On Fuel CI we should run:
>>>   - 4 fuel-library tests (revert master node, inject fuel-library code
>>> in master node and run deployment):
>>>             2 (ubuntu and centos) voting Icehouse tests and 2 non-voting
>>> Juno tests
>>>   - 5 OSTF tests (revert deployed environment, inject OSTF code into
>>> master node, run OSTF):
>>>             voting on 4.1, 5.0, 5.1, master/icehouse and non-voting on
>>> master/Juno
>>>   - other tests, which don't use prebuilt environment, work as before
>>>
>>> The major action point here would be OSTF tests, as we don't have yet
>>> working implementation of injecting OSTF code into deployed environment.
>>> And we don't run any tests on old environments.
>>>
>>>
>>> Questions:
>>>
>>> 1) How should we test mirrors?
>>>
>>> Current master mirrors go through the 4 hours test cycle involving Fuel
>>> ISO build:
>>>   1. we build temporary mirror
>>>   2. build custom iso from it
>>>   3. run two custom bvt jobs
>>>   4. if they pass we move mirror to stable and sitch to it for our
>>> "primary" fuel_master_iso
>>>
>>> Should we test only Icehouse mirrors, or both, but ignoring again failed
>>> BVT for Juno? Maybe we should enable these tests only later in release
>>> cycle, say, after SCF?
>>>
>>> 2) It is not clear for me when and how we will switch from supporting
>>> two releases back to one.
>>> Should we add one more milestone to our release process? The "Switching
>>> point", when we disable and remove Icehouse tasks and move to Juno
>>> completely? I guess it should happen before next SCF?
>>>
>>>
>>>
>>> On Tue, Sep 9, 2014 at 9:52 PM, Mike Scherbakov <
>>> mscherbakov at mirantis.com> wrote:
>>>
>>>> > What we need to achieve that is have 2 build series based on Fuel
>>>> master: one with Icehouse packages, and one with Juno, and, as Mike
>>>> proposed, keep our manifests backwards compatible with Icehouse.
>>>> Exactly. Our Fuel CI can do 4 builds against puppet modules: 2 voting,
>>>> with Icehouse packages; 2 non-voting, with Juno packages.
>>>>
>>>> Then, I'd suggest to create ISO with 2 releases (Icehouse, Juno)
>>>> actually before Juno becomes stable. We will be able to run 2 sets of BVTs
>>>> (against Icehouse and Juno), and it means that we will be able to see
>>>> almost immediately if something in nailgun/astute/puppet integration broke.
>>>> For Juno builds it's going to be all red initially.
>>>>
>>>> Another suggestion would be to lower green switch in BVTs for Juno:
>>>> first, when it passes deployment; and then, if it finally passes OSTF.
>>>>
>>>> I'd like to hear QA & DevOps opinion on all the above. Immediately we
>>>> would need just standard stuff which is in checklists for OSCI & DevOps
>>>> teams, and ideally soon after that - ability to have Fuel CI running 4
>>>> builds, not 2, against our master, as mentioned above.
>>>>
>>>> On Tue, Sep 9, 2014 at 9:28 PM, Roman Vyalov <rvyalov at mirantis.com>
>>>> wrote:
>>>>
>>>>> All OSCI action items for prepare HCF check list  has been done
>>>>>
>>>>>
>>>>> On Tue, Sep 9, 2014 at 6:27 PM, Mike Scherbakov <
>>>>> mscherbakov at mirantis.com> wrote:
>>>>>
>>>>>> Thanks Alexandra.
>>>>>>
>>>>>> We land a few patches a day currently, so I think we can open stable
>>>>>> branch. If we see no serious objections in next 12 hours, let's do it. We
>>>>>> would need to immediately notify everyone in mailing list - that for every
>>>>>> patch for 5.1, it should go first to master, and then to stable/5.1.
>>>>>>
>>>>>> Is everything ready from DevOps, OSCI (packaging) side to do this?
>>>>>> Fuel CI, OBS, etc.?
>>>>>>
>>>>>> On Tue, Sep 9, 2014 at 2:28 PM, Aleksandra Fedorova <
>>>>>> afedorova at mirantis.com> wrote:
>>>>>>
>>>>>>> As I understand your proposal, we need to split our HCF milestone
>>>>>>> into two check points: Branching Point and HCF itself.
>>>>>>>
>>>>>>> Branching point should happen somewhere in between SCF and HCF. And
>>>>>>> though It may coincide with HCF, it needs its own list of requirements.
>>>>>>> This will give us the possibility to untie two events and make a separate
>>>>>>> decision on branching without enforcing all HCF criteria.
>>>>>>>
>>>>>>> From the DevOps point of view it changes almost nothing, it just
>>>>>>> adds a bit more discussion items on the management side and slight
>>>>>>> modifications to our checklists.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 9, 2014 at 5:55 AM, Dmitry Borodaenko <
>>>>>>> dborodaenko at mirantis.com> wrote:
>>>>>>>
>>>>>>>> TL;DR: Yes, our work on 6.0 features is currently blocked and it is
>>>>>>>> becoming a major problem. No, I don't think we should create
>>>>>>>> pre-release or feature branches. Instead, we should create
>>>>>>>> stable/5.1
>>>>>>>> branches and open master for 6.0 work.
>>>>>>>>
>>>>>>>> We have reached a point in 5.1 release cycle where the scope of
>>>>>>>> issues
>>>>>>>> we are willing to address in this release is narrow enough to not
>>>>>>>> require full attention of the whole team. We have engineers working
>>>>>>>> on
>>>>>>>> 6.0 features, and their work is essentially blocked until they have
>>>>>>>> somewhere to commit their changes.
>>>>>>>>
>>>>>>>> Simply creating new branches is not even close to solving this
>>>>>>>> problem: we have a whole CI infrastructure around every active
>>>>>>>> release
>>>>>>>> series (currently 5.1, 5.0, 4.1), including test jobs for gerrit
>>>>>>>> commits, package repository mirrors updates, ISO image builds,
>>>>>>>> smoke,
>>>>>>>> build verification, and swarm tests for ISO images, documentation
>>>>>>>> builds, etc. A branch without all that infrastructure isn't any
>>>>>>>> better
>>>>>>>> than current status quo: every developer tracking their own 6.0 work
>>>>>>>> locally.
>>>>>>>>
>>>>>>>> Unrelated to all that, we also had a lot of very negative experience
>>>>>>>> with feature branches in the past [0] [1], which is why we have
>>>>>>>> decided to follow the OpenStack branching strategy: commit all
>>>>>>>> feature
>>>>>>>> changes directly to master and track bugfixes for stable releases in
>>>>>>>> stable/* branches.
>>>>>>>>
>>>>>>>> [0] https://lists.launchpad.net/fuel-dev/msg00127.html
>>>>>>>> [1] https://lists.launchpad.net/fuel-dev/msg00028.html
>>>>>>>>
>>>>>>>> I'm also against declaring a "hard code freeze with exceptions", HCF
>>>>>>>> should remain tied to our ability to declare a release candidate. If
>>>>>>>> we can't release with the bugs we already know about, declaring HCF
>>>>>>>> before fixing these bugs would be an empty gesture.
>>>>>>>>
>>>>>>>> Creating stable/5.1 now instead of waiting for hard code freeze for
>>>>>>>> 5.1 will cost us two things:
>>>>>>>>
>>>>>>>> 1) DevOps team will have to update our CI infrastructure for one
>>>>>>>> more
>>>>>>>> release series. It's something we have to do for 6.0 sooner or
>>>>>>>> later,
>>>>>>>> so this may be a disruption, but not an additional effort.
>>>>>>>>
>>>>>>>> 2) All commits targeted for 5.1 will have to be proposed for two
>>>>>>>> branches (master and stable/5.1) instead of just one (master). This
>>>>>>>> will require additional effort, but I think that it is significantly
>>>>>>>> smaller than the cost of spinning our wheels on 6.0 efforts.
>>>>>>>>
>>>>>>>> -DmitryB
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Sep 8, 2014 at 10:10 AM, Dmitry Mescheryakov
>>>>>>>> <dmescheryakov at mirantis.com> wrote:
>>>>>>>> > Hello Fuelers,
>>>>>>>> >
>>>>>>>> > Right now we have the following policy in place: the branches for
>>>>>>>> a
>>>>>>>> > release are opened only after its 'parent' release have reached
>>>>>>>> hard
>>>>>>>> > code freeze (HCF). Say, 5.1 release is parent releases for 5.1.1
>>>>>>>> and
>>>>>>>> > 6.0.
>>>>>>>> >
>>>>>>>> > And that is the problem: if parent release is delayed, we can't
>>>>>>>> > properly start development of a child release because we don't
>>>>>>>> have
>>>>>>>> > branches to commit. That is current issue with 6.0: we already
>>>>>>>> started
>>>>>>>> > to work on pushing Juno in to 6.0, but if we are to make changes
>>>>>>>> to
>>>>>>>> > our deployment code we have nowhere to store them.
>>>>>>>> >
>>>>>>>> > IMHO the issue could easily be resolved by creation of pre-release
>>>>>>>> > branches, which are merged together with parent branches once the
>>>>>>>> > parent reaches HCF. Say, we use branch 'pre-6.0' for initial
>>>>>>>> > development of 6.0. Once 5.1 reaches HCF, we merge pre-6.0 into
>>>>>>>> master
>>>>>>>> > and continue development here. After that pre-6.0 is abandoned.
>>>>>>>> >
>>>>>>>> > What do you think?
>>>>>>>> >
>>>>>>>> > Thanks,
>>>>>>>> >
>>>>>>>> > Dmitry
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > OpenStack-dev mailing list
>>>>>>>> > OpenStack-dev at lists.openstack.org
>>>>>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Dmitry Borodaenko
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Aleksandra Fedorova
>>>>>>> bookwar
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Mike Scherbakov
>>>>>> #mihgen
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Mike Scherbakov
>>>> #mihgen
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Aleksandra Fedorova
>>> bookwar
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>>
>> --
>> Mike Scherbakov
>> #mihgen
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Mike Scherbakov
#mihgen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140912/479b2fd5/attachment.html>