Open Stack

On Wed, Feb 27, 2013 at 2:04 PM, Monty Taylor <mordred at inaugust.com> wrote:
>
>
> On 02/27/2013 04:29 PM, Sean Dague wrote:
>> On 02/27/2013 03:51 PM, Russell Bryant wrote:
>>> So I'm not sure that the problems Sean has described here should be
>>> considered a disaster.  They seem to be just one of the things our
>>> testing should be catching for us.
>>
>> I'm not sure I'd call them disaster, but they at least are a calamity.
>>
>>> It's not just about getting updates for free.  It's about finding out
>>> when our software is no longer compatible with an upstream dependency.
>>> It means we found out as soon as we could that there is work to do to
>>> update project X to be compatible with new version of dependency Y.  We
>>> have to find this out somehow sometime.
>>
>> Here's the thing though, because all our software requires all the rest
>> of our software to go through the gate, the current process isn't
>> getting that answer. The pain threshold of a broken merge queue is too
>> high, and the ability to explore removing dependency limitations is too
>> hard.
>>
>> I'll use the example from yesterday:
>>
>> django 1.5 comes out. Horizon as setup by devstack doesn't do something
>> right by it. The gate stops merging any code. Nova, glance, tempest
>> development is now blocked.
>>
>> So the answer is pin it as fast as you can because right now horizon is
>> preventing all other projects from doing anything. There really isn't
>> time to figure out the right fix, because you've got 400 contributors
>> sitting on their hands, unable to get anything through CI.
>>
>> so we pin, move on. it didn't solve the issue, and the issue got thrown
>> on the back burner because there are bigger fish to fry. Most of the
>> pins in pip-requires showed up because of
>> similar issues.
>>
>>> Pinning seems like a fine short term fix for this, but only along with a
>>> bug for fixing compatibility with this new version of the dependency.
>>>
>>> I think I could buy pinning everything in stable branches since we
>>> wouldn't be doing work in stable branches to chase new versions of
>>> dependencies, but not in master.
>>
>> I think that for sure we need to pin in stable branches, otherwise it's
>> pretty bad bit rot.
>>
>> I think the tension is fundamentally between "OpenStack should always
>> work from git" and "we should use git to figure out what OpenStack works
>> with when we get to release". Our CI system enforces point #1.
>>
>> I feel like the sane medium term approach is *when we enter freeze* we
>> cap everything, then the first havana commits are to uncap, and let us
>> run with the calamity during open development.
>>
>>
>>
>> In an ideal world I'd say a solution that pins everything to maximum
>> known working versions in the tree. Then a background set of jobs that
>> tries canary builds of new dependency versions (unpin them one at a
>> time), and reports success or fail (registering a bugs to either
>> increase our cap, or that component X doesn't work with the new cap)
>> would be what we want. It's pretty complicated, but given the level of
>> infrastructure we currently have, is something someone actually could
>> build in the havana time frame. The centralized dependency list would
>> make life easier, but you could even pull it off without that.
>
> I'd really like this. I think step one is getting the list going and at
> least doing something (dhellman and I were just discussing that on IRC)
> ... but I think we should set aside a session and some time at the
> summit to design the thing you're talking about above. I don't think
> it's impossible - it's a combinations problem, but we have clouds. The
> biggest issue with things like this before has been how-to-communicate -
> and somehow I've never considered having the testing system file a bug. Duh.

+1

"Pinning is only temporary" seems like a nonstarter to me.

>
>> Honestly, a big part of the fact that we even have this issue is our
>> testing is good enough that master can't really bitrot. On most projects
>> these sort of breaks would be sitting in the tree for weeks because it
>> worked fine on a developer's workstation and ci was stale in what it's
>> base image looked like. So kuddos for us having a system that's giving
>> us new and interesting problems to solve. :)
>>
>>     -Sean
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev