Open Stack

On 02/27/2013 02:59 PM, Monty Taylor wrote:
> 
> 
> On 02/27/2013 02:48 PM, Doug Hellmann wrote:
>>
>>
>> On Wed, Feb 27, 2013 at 1:18 PM, Sean Dague <sdague at linux.vnet.ibm.com
>> <mailto:sdague at linux.vnet.ibm.com>> wrote:
>>
>>     After lots of running around this morning, we've got a new
>>     python-quantumclient v.2.1.2 that fixes OpenStack CI. The gate merge
>>     was only broken for 4 hours, before California woke up, so overall
>>     project impact was pretty bad.
>>
>>     I think there are a couple lessons learned out of this, at least in
>>     my opinion. As someone that's chased 2 big breaks like this in 2
>>     days, I'd like to figure out how we stop doing that. :)
>>
>>     1) python-FOOclients should release more often
>>
>>     All it takes is setting a tag on a python-FOOclient to generate a
>>     release, CI does the push into pypi. We should be releasing early
>>     and often here. This issue was known to be an issue back in
>>     December, which is when the fix landed in the python-quantumclient
>>     tree, but because there wasn't a python-quantumclient release it
>>     managed to be a gate blocker.
>>
>>     Release early, release often. clients don't need to be on the 6
>>     month cadence that the server projects are on, as clients should
>>     work with multiple versions of the servers.
>>
>>     It's also probably worth making sure that all cores on a project
>>     known and understand the python-client release process -
>>     https://wiki.openstack.org/__wiki/GerritJenkinsGithub#__Tagging_a_Release
>>     <https://wiki.openstack.org/wiki/GerritJenkinsGithub#Tagging_a_Release>
>>
>>     It's super easy, and should be doable at any time.
>>
>>     2) open ended requires are going to hurt us once we go to freeze.
>>
>>     We've had 3 breaks of our gate this week due to pip updates. Horizon
>>     due to django, Glance due to jsonscheme, and Quantum Client due to
>>     pyparsing. These typically break stable as well (I known the django
>>     one did).
>>
>>     I've chased the gate fixes on 2 of these, it's definitely getting
>>     old. I understand the desire to not artificially limit ourselves,
>>     but it feels like once we get into freeze we should start capping
>>     these things. We can keep them open in open development, but I think
>>     we need to start clamping these down once we hit freeze.
>>
>>
>> I'm having the same issue with prettytable and some of the client
>> libraries under devstack right now, too. I wonder if that's the other
>> issue you chased down?
> 
> I totally understand the argument against stricter pinning, but I'd like
> to repeat my argument for it.
> 
> That is - while in theory it might seem nice to be able to "get upgrades
> for free" from upstreams, experience in this project has shown over and
> over that it's almost always a disaster. It _shouldn't_ be - but it is.

So I'm not sure that the problems Sean has described here should be
considered a disaster.  They seem to be just one of the things our
testing should be catching for us.

It's not just about getting updates for free.  It's about finding out
when our software is no longer compatible with an upstream dependency.
It means we found out as soon as we could that there is work to do to
update project X to be compatible with new version of dependency Y.  We
have to find this out somehow sometime.

Pinning seems like a fine short term fix for this, but only along with a
bug for fixing compatibility with this new version of the dependency.

I think I could buy pinning everything in stable branches since we
wouldn't be doing work in stable branches to chase new versions of
dependencies, but not in master.

-- 
Russell Bryant