[openstack-dev] [stable] juno is fubar in the gate

Joe Gordon joe.gordon0 at gmail.com
Mon Feb 9 22:26:20 UTC 2015


On Mon, Feb 9, 2015 at 2:20 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:

>
>
> On Mon, Feb 9, 2015 at 1:02 PM, John Griffith <john.griffith8 at gmail.com>
> wrote:
>
>> On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish <mtreinish at kortar.org>
>> wrote:
>> > On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
>> >>
>> >>
>> >> On 2/9/2015 12:23 PM, Joe Gordon wrote:
>> >> >
>> >> >On Feb 9, 2015 10:04 AM, "Matt Riedemann" <mriedem at linux.vnet.ibm.com
>> >> ><mailto:mriedem at linux.vnet.ibm.com>> wrote:
>> >> > >
>> >> > > There are at least two blocking bugs:
>> >> > >
>> >> > > 1. https://bugs.launchpad.net/grenade/+bug/1419913
>> >> > >
>> >> > > Sounds like jogo is working a javelin fix for this. I'm not aware
>> of
>> >> >a patch to review though.
>> >> >
>> >> >We need to stop trying to install tempest in the same env as stable/*
>> code.
>> >> >
>> >> >I should be able to revise/respond to comments shortly.
>> >> >
>> >> >https://review.openstack.org/#/c/153080/
>> >> >
>> >> >https://review.openstack.org/#/c/153702/
>> >> >
>> >> >This is also blocking my effort to pin stable dependencies (Dean's
>> >> >devstack changes are needed before we can pin stable dependencies as
>> well).
>> >> >
>> >> > >
>> >> > > 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
>> >> > >
>> >> > > I'm not sure yet what's going on with this one.
>> >> > >
>> >>
>> >> Tracking etherpad:
>> >>
>> >> https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
>> >
>> >
>> > So I think it's time we called the icehouse branch and marked it EOL. We
>> > originally conditioned the longer support window on extra people
>> stepping
>> > forward to keep things working. I believe this latest issue is just the
>> latest
>> > indication that this hasn't happened. Issue 1 listed above is being
>> caused by
>> > the icehouse branch during upgrades. The fact that a stable release was
>> pushed
>> > at the same time things were wedged on the juno branch is just the
>> latest
>> > evidence to me that things aren't being maintained as they should be.
>> Looking at
>> > the #openstack-qa irc log from today or the etherpad about trying to
>> sort this
>> > issue should be an indication that no one has stepped up to help with
>> the
>> > maintenance and it shows given the poor state of the branch.
>> >
>> > If I'm not mistaken with our original support window lengths Icehouse
>> would be
>> > EOL'd around now. So it's time we stopped pretending we'll be
>> maintaining this
>> > branch for several more months and just go through the normal EOL
>> procedure.
>> >
>>
>
>> Was this serious?  I mean, we just say; 'sorry, yes we said support
>> until X; but now it's hard so we're going to drop it'.
>>
>> Tell me I'm missing something here?
>>
>
> You are missing the fact that a bunch of us (Matt Treinish, myself and
> others) are frustrated by the fact that we end up fixing stable branches
> whenever they break because we touch tempest, grenade and other projects
> that require working stable branches. But we do not want to be working on
> stable branches ourselves.  I begrudgingly stepped up to work on pinning
> all requirements on stable branches, to reduce the number of times stable
> branches break and ruin my day. But my plan to cap dependencies has been
> delayed several times by stable branches breaking again and again, along
> with unwinding undesired behaviors in our testing harnesses.
>

Note: At least 3 of us just spent most of the day working on this instead
of working on developing on other things.


>
> Most recently, stable/juno grenade broke on February 4th (due to the
> release of tempest-lib 0.2.0). This caused bug
> https://bugs.launchpad.net/grenade/+bug/1419913
> <https://bugs.launchpad.net/grenade/+bug/1419913>
>  (" pkg_resources.ContextualVersionConflict: (oslo.config 1.4.0
> (/usr/local/lib/python2.7/dist-packages),
> Requirement.parse('oslo.config>=1.6.0'), set(['tempest-lib']))". This
> specific bug is caused because we install master tempest (due to branchless
> tempest) on stable/icehouse and sync in stable/icehouse global requirements
> which not surprisingly has a conflict with tempest's requirements.  So the
> solution here is stop installing tempest and requiring it  to work with
> stable/icehouse, stable/juno and master's version of global-requirements.
> But that doesn't work because master tempest has an uncapped version of
> boto but nova stable/icehouse only works with the capped version of
> Icehouse. So we get this https://review.openstack.org/#/c/154217/1/. So
> now we are exploring dropping the EC2 tests on stable/icehouse. If that
> works, we still need to land roughly 4 more patches to unwedge this
> stable/juno grenade and prevent this type of issue from happening in the
> future.
>
> Lets say we EOL Icehouse, we stop running grenade on stable/juno patches.
> Meaning this bug goes away all together and stable/juno is unwedged and I
> can move forward with pinning all stable/juno requirements.
>
> What I expect to happen when issues like this arise is interested parties
> work together to fix things and be proactive and make stable testing more
> robust. Instead we currently have people who have no desire to work on
> stable branches maintaining them. Pinning all direct stable/* requirements
> isn't enough to make sure stable/* doesn't break. There are transitive
> dependencies that can change (I have a plan on how to pin those too, but it
> will take time and I can use some help), and changing packages etc. can
> break things as well.  Having a reactive stable maintenance team is
> insufficient.
>
> Until we have a proactive team of people actually maintaining stable
> branches and related testing infrastructure, versus the skeleton crew we
> have now, I don't think we should support stable branches for 15 months.
> Because supporting Icehouse for 15 months means we will have to support 3
> stable branches 50% of the time. Currently the plan is to EOL Icehouse 4
> months into the M release, so we will need to support Icehouse, Juno, Kilo
> and Master. And considering the issues we are having with supporting two
> stable branches, supporting 3 scares me.
>
>
>
>
>> > -Matt Treinish
>> >
>> >
>> __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150209/9e68d480/attachment.html>


More information about the OpenStack-dev mailing list