<br><br><div class="gmail_quote">On Wed, Feb 27, 2013 at 1:18 PM, Sean Dague <span dir="ltr"><<a href="mailto:sdague@linux.vnet.ibm.com" target="_blank">sdague@linux.vnet.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
After lots of running around this morning, we've got a new python-quantumclient v.2.1.2 that fixes OpenStack CI. The gate merge was only broken for 4 hours, before California woke up, so overall project impact was pretty bad.<br>
<br>
I think there are a couple lessons learned out of this, at least in my opinion. As someone that's chased 2 big breaks like this in 2 days, I'd like to figure out how we stop doing that. :)<br>
<br>
1) python-FOOclients should release more often<br>
<br>
All it takes is setting a tag on a python-FOOclient to generate a release, CI does the push into pypi. We should be releasing early and often here. This issue was known to be an issue back in December, which is when the fix landed in the python-quantumclient tree, but because there wasn't a python-quantumclient release it managed to be a gate blocker.<br>
<br>
Release early, release often. clients don't need to be on the 6 month cadence that the server projects are on, as clients should work with multiple versions of the servers.<br>
<br>
It's also probably worth making sure that all cores on a project known and understand the python-client release process - <a href="https://wiki.openstack.org/wiki/GerritJenkinsGithub#Tagging_a_Release" target="_blank">https://wiki.openstack.org/<u></u>wiki/GerritJenkinsGithub#<u></u>Tagging_a_Release</a><br>
<br>
It's super easy, and should be doable at any time.<br>
<br>
2) open ended requires are going to hurt us once we go to freeze.<br>
<br>
We've had 3 breaks of our gate this week due to pip updates. Horizon due to django, Glance due to jsonscheme, and Quantum Client due to pyparsing. These typically break stable as well (I known the django one did).<br>
<br>
I've chased the gate fixes on 2 of these, it's definitely getting old. I understand the desire to not artificially limit ourselves, but it feels like once we get into freeze we should start capping these things. We can keep them open in open development, but I think we need to start clamping these down once we hit freeze.<br>
</blockquote><div><br></div><div>I'm having the same issue with prettytable and some of the client libraries under devstack right now, too. I wonder if that's the other issue you chased down?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
3) order in pip-requires matters :)<br>
<br>
The eventual fix for python-quantumclient was to bump cliff. There was nothing wrong with cliff, but cliff depends on pyparsing as well, and didn't specify strict version until 1.3.1. In reality this probably wasn't even a needed change for the pypi package, but it did let it get through our pip based pep8 scans. </blockquote>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The CI fix should really have been to move pyparsing up in the file to install before cliff, and 1.2.1 would then still work.<br></blockquote><div><br></div><div><div>Yes, cliff didn't require any code changes. I borrowed some setup code from an unreleased version cmd2 (the library that actually specifies the pyparsing dependency) and copied it into cliff's requirements list to pin the version earlier. As you say, it probably would have been possible to do that right in the quantumclient dependency list, too. OTOH, cliff 1.3.1 also includes some UI fixes requested by the Quantum team, and I had promised a release soon anyway, so two birds, one stone, etc.</div>
</div><div><br></div><div>Doug</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Anyway, alternative thoughts are really appreciated. I realize point #2 is a little contentious, and there is also another thread on it. But it's also no good to have git code stop working for people, and us to not realize it until someone tries to merge code.<span class="HOEnZb"><font color="#888888"><br>
<br>
-Sean<br>
<br>
-- <br>
Sean Dague<br>
IBM Linux Technology Center<br>
email: <a href="mailto:sdague@linux.vnet.ibm.com" target="_blank">sdague@linux.vnet.ibm.com</a><br>
alt-email: <a href="mailto:sldague@us.ibm.com" target="_blank">sldague@us.ibm.com</a><br>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</font></span></blockquote></div><br>