[openstack-dev] [stable] juno is fubar in the gate

Joe Gordon joe.gordon0 at gmail.com
Mon Feb 9 22:20:29 UTC 2015


On Mon, Feb 9, 2015 at 1:02 PM, John Griffith <john.griffith8 at gmail.com>
wrote:

> On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish <mtreinish at kortar.org>
> wrote:
> > On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:
> >>
> >>
> >> On 2/9/2015 12:23 PM, Joe Gordon wrote:
> >> >
> >> >On Feb 9, 2015 10:04 AM, "Matt Riedemann" <mriedem at linux.vnet.ibm.com
> >> ><mailto:mriedem at linux.vnet.ibm.com>> wrote:
> >> > >
> >> > > There are at least two blocking bugs:
> >> > >
> >> > > 1. https://bugs.launchpad.net/grenade/+bug/1419913
> >> > >
> >> > > Sounds like jogo is working a javelin fix for this. I'm not aware of
> >> >a patch to review though.
> >> >
> >> >We need to stop trying to install tempest in the same env as stable/*
> code.
> >> >
> >> >I should be able to revise/respond to comments shortly.
> >> >
> >> >https://review.openstack.org/#/c/153080/
> >> >
> >> >https://review.openstack.org/#/c/153702/
> >> >
> >> >This is also blocking my effort to pin stable dependencies (Dean's
> >> >devstack changes are needed before we can pin stable dependencies as
> well).
> >> >
> >> > >
> >> > > 2. https://bugs.launchpad.net/ceilometer/+bug/1419919
> >> > >
> >> > > I'm not sure yet what's going on with this one.
> >> > >
> >>
> >> Tracking etherpad:
> >>
> >> https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015
> >
> >
> > So I think it's time we called the icehouse branch and marked it EOL. We
> > originally conditioned the longer support window on extra people stepping
> > forward to keep things working. I believe this latest issue is just the
> latest
> > indication that this hasn't happened. Issue 1 listed above is being
> caused by
> > the icehouse branch during upgrades. The fact that a stable release was
> pushed
> > at the same time things were wedged on the juno branch is just the latest
> > evidence to me that things aren't being maintained as they should be.
> Looking at
> > the #openstack-qa irc log from today or the etherpad about trying to
> sort this
> > issue should be an indication that no one has stepped up to help with the
> > maintenance and it shows given the poor state of the branch.
> >
> > If I'm not mistaken with our original support window lengths Icehouse
> would be
> > EOL'd around now. So it's time we stopped pretending we'll be
> maintaining this
> > branch for several more months and just go through the normal EOL
> procedure.
> >
>

> Was this serious?  I mean, we just say; 'sorry, yes we said support
> until X; but now it's hard so we're going to drop it'.
>
> Tell me I'm missing something here?
>

You are missing the fact that a bunch of us (Matt Treinish, myself and
others) are frustrated by the fact that we end up fixing stable branches
whenever they break because we touch tempest, grenade and other projects
that require working stable branches. But we do not want to be working on
stable branches ourselves.  I begrudgingly stepped up to work on pinning
all requirements on stable branches, to reduce the number of times stable
branches break and ruin my day. But my plan to cap dependencies has been
delayed several times by stable branches breaking again and again, along
with unwinding undesired behaviors in our testing harnesses.

Most recently, stable/juno grenade broke on February 4th (due to the
release of tempest-lib 0.2.0). This caused bug
https://bugs.launchpad.net/grenade/+bug/1419913
<https://bugs.launchpad.net/grenade/+bug/1419913>
 (" pkg_resources.ContextualVersionConflict: (oslo.config 1.4.0
(/usr/local/lib/python2.7/dist-packages),
Requirement.parse('oslo.config>=1.6.0'), set(['tempest-lib']))". This
specific bug is caused because we install master tempest (due to branchless
tempest) on stable/icehouse and sync in stable/icehouse global requirements
which not surprisingly has a conflict with tempest's requirements.  So the
solution here is stop installing tempest and requiring it  to work with
stable/icehouse, stable/juno and master's version of global-requirements.
But that doesn't work because master tempest has an uncapped version of
boto but nova stable/icehouse only works with the capped version of
Icehouse. So we get this https://review.openstack.org/#/c/154217/1/. So now
we are exploring dropping the EC2 tests on stable/icehouse. If that works,
we still need to land roughly 4 more patches to unwedge this stable/juno
grenade and prevent this type of issue from happening in the future.

Lets say we EOL Icehouse, we stop running grenade on stable/juno patches.
Meaning this bug goes away all together and stable/juno is unwedged and I
can move forward with pinning all stable/juno requirements.

What I expect to happen when issues like this arise is interested parties
work together to fix things and be proactive and make stable testing more
robust. Instead we currently have people who have no desire to work on
stable branches maintaining them. Pinning all direct stable/* requirements
isn't enough to make sure stable/* doesn't break. There are transitive
dependencies that can change (I have a plan on how to pin those too, but it
will take time and I can use some help), and changing packages etc. can
break things as well.  Having a reactive stable maintenance team is
insufficient.

Until we have a proactive team of people actually maintaining stable
branches and related testing infrastructure, versus the skeleton crew we
have now, I don't think we should support stable branches for 15 months.
Because supporting Icehouse for 15 months means we will have to support 3
stable branches 50% of the time. Currently the plan is to EOL Icehouse 4
months into the M release, so we will need to support Icehouse, Juno, Kilo
and Master. And considering the issues we are having with supporting two
stable branches, supporting 3 scares me.




> > -Matt Treinish
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150209/bebeb831/attachment.html>


More information about the OpenStack-dev mailing list