Open Stack

After lots of running around this morning, we've got a new 
python-quantumclient v.2.1.2 that fixes OpenStack CI. The gate merge was 
only broken for 4 hours, before California woke up, so overall project 
impact was pretty bad.

I think there are a couple lessons learned out of this, at least in my 
opinion. As someone that's chased 2 big breaks like this in 2 days, I'd 
like to figure out how we stop doing that. :)

1) python-FOOclients should release more often

All it takes is setting a tag on a python-FOOclient to generate a 
release, CI does the push into pypi. We should be releasing early and 
often here. This issue was known to be an issue back in December, which 
is when the fix landed in the python-quantumclient tree, but because 
there wasn't a python-quantumclient release it managed to be a gate blocker.

Release early, release often. clients don't need to be on the 6 month 
cadence that the server projects are on, as clients should work with 
multiple versions of the servers.

It's also probably worth making sure that all cores on a project known 
and understand the python-client release process - 
https://wiki.openstack.org/wiki/GerritJenkinsGithub#Tagging_a_Release

It's super easy, and should be doable at any time.

2) open ended requires are going to hurt us once we go to freeze.

We've had 3 breaks of our gate this week due to pip updates. Horizon due 
to django, Glance due to jsonscheme, and Quantum Client due to 
pyparsing. These typically break stable as well (I known the django one 
did).

I've chased the gate fixes on 2 of these, it's definitely getting old. I 
understand the desire to not artificially limit ourselves, but it feels 
like once we get into freeze we should start capping these things. We 
can keep them open in open development, but I think we need to start 
clamping these down once we hit freeze.

3) order in pip-requires matters :)

The eventual fix for python-quantumclient was to bump cliff. There was 
nothing wrong with cliff, but cliff depends on pyparsing as well, and 
didn't specify strict version until 1.3.1. In reality this probably 
wasn't even a needed change for the pypi package, but it did let it get 
through our pip based pep8 scans.

The CI fix should really have been to move pyparsing up in the file to 
install before cliff, and 1.2.1 would then still work.


Anyway, alternative thoughts are really appreciated. I realize point #2 
is a little contentious, and there is also another thread on it. But 
it's also no good to have git code stop working for people, and us to 
not realize it until someone tries to merge code.

	-Sean

-- 
Sean Dague
IBM Linux Technology Center
email: sdague at linux.vnet.ibm.com
alt-email: sldague at us.ibm.com