Open Stack

On 02/27/2013 03:51 PM, Russell Bryant wrote:
> So I'm not sure that the problems Sean has described here should be
> considered a disaster.  They seem to be just one of the things our
> testing should be catching for us.

I'm not sure I'd call them disaster, but they at least are a calamity.

> It's not just about getting updates for free.  It's about finding out
> when our software is no longer compatible with an upstream dependency.
> It means we found out as soon as we could that there is work to do to
> update project X to be compatible with new version of dependency Y.  We
> have to find this out somehow sometime.

Here's the thing though, because all our software requires all the rest 
of our software to go through the gate, the current process isn't 
getting that answer. The pain threshold of a broken merge queue is too 
high, and the ability to explore removing dependency limitations is too 
hard.

I'll use the example from yesterday:

django 1.5 comes out. Horizon as setup by devstack doesn't do something 
right by it. The gate stops merging any code. Nova, glance, tempest 
development is now blocked.

So the answer is pin it as fast as you can because right now horizon is 
preventing all other projects from doing anything. There really isn't 
time to figure out the right fix, because you've got 400 contributors 
sitting on their hands, unable to get anything through CI.

so we pin, move on. it didn't solve the issue, and the issue got thrown 
on the back burner because there are bigger fish to fry. Most of the 
pins in pip-requires showed up because of
similar issues.

> Pinning seems like a fine short term fix for this, but only along with a
> bug for fixing compatibility with this new version of the dependency.
>
> I think I could buy pinning everything in stable branches since we
> wouldn't be doing work in stable branches to chase new versions of
> dependencies, but not in master.

I think that for sure we need to pin in stable branches, otherwise it's 
pretty bad bit rot.

I think the tension is fundamentally between "OpenStack should always 
work from git" and "we should use git to figure out what OpenStack works 
with when we get to release". Our CI system enforces point #1.

I feel like the sane medium term approach is *when we enter freeze* we 
cap everything, then the first havana commits are to uncap, and let us 
run with the calamity during open development.



In an ideal world I'd say a solution that pins everything to maximum 
known working versions in the tree. Then a background set of jobs that 
tries canary builds of new dependency versions (unpin them one at a 
time), and reports success or fail (registering a bugs to either 
increase our cap, or that component X doesn't work with the new cap) 
would be what we want. It's pretty complicated, but given the level of 
infrastructure we currently have, is something someone actually could 
build in the havana time frame. The centralized dependency list would 
make life easier, but you could even pull it off without that.

Honestly, a big part of the fact that we even have this issue is our 
testing is good enough that master can't really bitrot. On most projects 
these sort of breaks would be sitting in the tree for weeks because it 
worked fine on a developer's workstation and ci was stale in what it's 
base image looked like. So kuddos for us having a system that's giving 
us new and interesting problems to solve. :)

	-Sean

-- 
Sean Dague
IBM Linux Technology Center
email: sdague at linux.vnet.ibm.com
alt-email: sldague at us.ibm.com