Open Stack

On 02/27/2013 04:29 PM, Sean Dague wrote:
> On 02/27/2013 03:51 PM, Russell Bryant wrote:
>> So I'm not sure that the problems Sean has described here should be
>> considered a disaster.  They seem to be just one of the things our
>> testing should be catching for us.
> 
> I'm not sure I'd call them disaster, but they at least are a calamity.
> 
>> It's not just about getting updates for free.  It's about finding out
>> when our software is no longer compatible with an upstream dependency.
>> It means we found out as soon as we could that there is work to do to
>> update project X to be compatible with new version of dependency Y.  We
>> have to find this out somehow sometime.
> 
> Here's the thing though, because all our software requires all the rest
> of our software to go through the gate, the current process isn't
> getting that answer. The pain threshold of a broken merge queue is too
> high, and the ability to explore removing dependency limitations is too
> hard.
> 
> I'll use the example from yesterday:
> 
> django 1.5 comes out. Horizon as setup by devstack doesn't do something
> right by it. The gate stops merging any code. Nova, glance, tempest
> development is now blocked.
> 
> So the answer is pin it as fast as you can because right now horizon is
> preventing all other projects from doing anything. There really isn't
> time to figure out the right fix, because you've got 400 contributors
> sitting on their hands, unable to get anything through CI.
> 
> so we pin, move on. it didn't solve the issue, and the issue got thrown
> on the back burner because there are bigger fish to fry. Most of the
> pins in pip-requires showed up because of
> similar issues.
> 
>> Pinning seems like a fine short term fix for this, but only along with a
>> bug for fixing compatibility with this new version of the dependency.
>>
>> I think I could buy pinning everything in stable branches since we
>> wouldn't be doing work in stable branches to chase new versions of
>> dependencies, but not in master.
> 
> I think that for sure we need to pin in stable branches, otherwise it's
> pretty bad bit rot.
> 
> I think the tension is fundamentally between "OpenStack should always
> work from git" and "we should use git to figure out what OpenStack works
> with when we get to release". Our CI system enforces point #1.
> 
> I feel like the sane medium term approach is *when we enter freeze* we
> cap everything, then the first havana commits are to uncap, and let us
> run with the calamity during open development.
> 
> 
> 
> In an ideal world I'd say a solution that pins everything to maximum
> known working versions in the tree. Then a background set of jobs that
> tries canary builds of new dependency versions (unpin them one at a
> time), and reports success or fail (registering a bugs to either
> increase our cap, or that component X doesn't work with the new cap)
> would be what we want. It's pretty complicated, but given the level of
> infrastructure we currently have, is something someone actually could
> build in the havana time frame. The centralized dependency list would
> make life easier, but you could even pull it off without that.

I'd really like this. I think step one is getting the list going and at
least doing something (dhellman and I were just discussing that on IRC)
... but I think we should set aside a session and some time at the
summit to design the thing you're talking about above. I don't think
it's impossible - it's a combinations problem, but we have clouds. The
biggest issue with things like this before has been how-to-communicate -
and somehow I've never considered having the testing system file a bug. Duh.

> Honestly, a big part of the fact that we even have this issue is our
> testing is good enough that master can't really bitrot. On most projects
> these sort of breaks would be sitting in the tree for weeks because it
> worked fine on a developer's workstation and ci was stale in what it's
> base image looked like. So kuddos for us having a system that's giving
> us new and interesting problems to solve. :)
> 
>     -Sean
>