<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 9, 2015 at 2:20 PM, Joe Gordon <span dir="ltr"><<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Mon, Feb 9, 2015 at 1:02 PM, John Griffith <span dir="ltr"><<a href="mailto:john.griffith8@gmail.com" target="_blank">john.griffith8@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div>On Mon, Feb 9, 2015 at 1:56 PM, Matthew Treinish <<a href="mailto:mtreinish@kortar.org" target="_blank">mtreinish@kortar.org</a>> wrote:<br>
> On Mon, Feb 09, 2015 at 01:24:34PM -0600, Matt Riedemann wrote:<br>
>><br>
>><br>
>> On 2/9/2015 12:23 PM, Joe Gordon wrote:<br>
>> ><br>
>> >On Feb 9, 2015 10:04 AM, "Matt Riedemann" <<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a><br>
>> ><mailto:<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>>> wrote:<br>
>> > ><br>
>> > > There are at least two blocking bugs:<br>
>> > ><br>
>> > > 1. <a href="https://bugs.launchpad.net/grenade/+bug/1419913" target="_blank">https://bugs.launchpad.net/grenade/+bug/1419913</a><br>
>> > ><br>
>> > > Sounds like jogo is working a javelin fix for this. I'm not aware of<br>
>> >a patch to review though.<br>
>> ><br>
>> >We need to stop trying to install tempest in the same env as stable/* code.<br>
>> ><br>
>> >I should be able to revise/respond to comments shortly.<br>
>> ><br>
>> ><a href="https://review.openstack.org/#/c/153080/" target="_blank">https://review.openstack.org/#/c/153080/</a><br>
>> ><br>
>> ><a href="https://review.openstack.org/#/c/153702/" target="_blank">https://review.openstack.org/#/c/153702/</a><br>
>> ><br>
>> >This is also blocking my effort to pin stable dependencies (Dean's<br>
>> >devstack changes are needed before we can pin stable dependencies as well).<br>
>> ><br>
>> > ><br>
>> > > 2. <a href="https://bugs.launchpad.net/ceilometer/+bug/1419919" target="_blank">https://bugs.launchpad.net/ceilometer/+bug/1419919</a><br>
>> > ><br>
>> > > I'm not sure yet what's going on with this one.<br>
>> > ><br>
>><br>
>> Tracking etherpad:<br>
>><br>
>> <a href="https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015" target="_blank">https://etherpad.openstack.org/p/wedged-stable-gate-feb-2015</a><br>
><br>
><br>
> So I think it's time we called the icehouse branch and marked it EOL. We<br>
> originally conditioned the longer support window on extra people stepping<br>
> forward to keep things working. I believe this latest issue is just the latest<br>
> indication that this hasn't happened. Issue 1 listed above is being caused by<br>
> the icehouse branch during upgrades. The fact that a stable release was pushed<br>
> at the same time things were wedged on the juno branch is just the latest<br>
> evidence to me that things aren't being maintained as they should be. Looking at<br>
> the #openstack-qa irc log from today or the etherpad about trying to sort this<br>
> issue should be an indication that no one has stepped up to help with the<br>
> maintenance and it shows given the poor state of the branch.<br>
><br>
> If I'm not mistaken with our original support window lengths Icehouse would be<br>
> EOL'd around now. So it's time we stopped pretending we'll be maintaining this<br>
> branch for several more months and just go through the normal EOL procedure.<br>
><span style="color:rgb(34,34,34)"> </span></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div>
<br>
</div></div>Was this serious? I mean, we just say; 'sorry, yes we said support<br>
until X; but now it's hard so we're going to drop it'.<br>
<br>
Tell me I'm missing something here?<br></blockquote><div><br></div></div></div><div>You are missing the fact that a bunch of us (Matt Treinish, myself and others) are frustrated by the fact that we end up fixing stable branches whenever they break because we touch tempest, grenade and other projects that require working stable branches. But we do not want to be working on stable branches ourselves. I begrudgingly stepped up to work on pinning all requirements on stable branches, to reduce the number of times stable branches break and ruin my day. But my plan to cap dependencies has been delayed several times by stable branches breaking again and again, along with unwinding undesired behaviors in our testing harnesses.</div></div></div></div></blockquote><div><br></div><div>Note: At least 3 of us just spent most of the day working on this instead of working on developing on other things.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Most recently, stable/juno grenade broke on February 4th (due to the release of tempest-lib 0.2.0). This caused bug <a href="https://bugs.launchpad.net/grenade/+bug/1419913" target="_blank">https://bugs.launchpad.net/grenade/+bug/1419913 </a></div><div> (" pkg_resources.ContextualVersionConflict: (oslo.config 1.4.0 (/usr/local/lib/python2.7/dist-packages), Requirement.parse('oslo.config>=1.6.0'), set(['tempest-lib']))". This specific bug is caused because we install master tempest (due to branchless tempest) on stable/icehouse and sync in stable/icehouse global requirements which not surprisingly has a conflict with tempest's requirements. So the solution here is stop installing tempest and requiring it to work with stable/icehouse, stable/juno and master's version of global-requirements. But that doesn't work because master tempest has an uncapped version of boto but nova stable/icehouse only works with the capped version of Icehouse. So we get this <a href="https://review.openstack.org/#/c/154217/1/" target="_blank">https://review.openstack.org/#/c/154217/1/</a>. So now we are exploring dropping the EC2 tests on stable/icehouse. If that works, we still need to land roughly 4 more patches to unwedge this stable/juno grenade and prevent this type of issue from happening in the future.</div><div><br></div><div>Lets say we EOL Icehouse, we stop running grenade on stable/juno patches. Meaning this bug goes away all together and stable/juno is unwedged and I can move forward with pinning all stable/juno requirements.<br></div><div><br></div><div>What I expect to happen when issues like this arise is interested parties work together to fix things and be proactive and make stable testing more robust. Instead we currently have people who have no desire to work on stable branches maintaining them. Pinning all direct stable/* requirements isn't enough to make sure stable/* doesn't break. There are transitive dependencies that can change (I have a plan on how to pin those too, but it will take time and I can use some help), and changing packages etc. can break things as well. Having a reactive stable maintenance team is insufficient.</div><div><br></div><div>Until we have a proactive team of people actually maintaining stable branches and related testing infrastructure, versus the skeleton crew we have now, I don't think we should support stable branches for 15 months. Because supporting Icehouse for 15 months means we will have to support 3 stable branches 50% of the time. Currently the plan is to EOL Icehouse 4 months into the M release, so we will need to support Icehouse, Juno, Kilo and Master. And considering the issues we are having with supporting two stable branches, supporting 3 scares me.</div><span class=""><div><br></div><div> <br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
> -Matt Treinish<br>
<div><div>><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>