Open Stack

We have around 60 repos using the 'mccabe' package to do cyclomatic 
complexity checks in flake8 in their gate jobs.[1] Based on advice 
received at the time[2], the version of mccabe that we use in the gate 
is effectively pinned to 0.2.1 by hacking (the project, not... never 
mind).[3]

This is starting to cause problems, because version 0.2.1 does not work 
with Python 3. By 'does not work' I don't mean that it refuses to run, I 
mean that it returns completely different numbers. (As far as I can tell 
they're always the same or lower.) tox creates the pep8 environment 
using the version of python that tox itself is running in, so as distros 
increasingly prefer to install packages using python3, people will 
increasingly be running flake8 under python3, and they'll run into 
situations like the one I had today where their pep8 tests pass locally 
but fail in the gate (which still runs under python2). When the gate 
switches to python3, as I assume it eventually must, we'll get bogus 
results.

Unfortunately, moving to a later version is complicated by the fact that 
0.2.1 was... not good software. There is exactly one regression test.[4] 
And if you're about to ask whether it was to check the trivial case that 
the function:

   def foo():
       return None

has a cyclomatic complexity of 1, then the joke is on you because it 
actually returns 2 for that function.

Later versions have more tests and appear to be consistent with each 
other and across python versions, but they return completely different 
results to 0.2.1. And the results are mostly higher, so if we bump the 
requirements many of those 60 projects will break.

(I could also go into details on how the numbers that later versions 
report are also wrong in different ways, but I'll save it for another day.)

Here's how the allegedly most complex function in Heat[5] shakes out, 
for example:

   mccabe  py27  py36
   0.2.1    14     6
   0.3.1    23    23
   0.6.1    23    23

I don't have a solution to suggest, but I burned most of my day digging 
into this so I just wanted to let y'all know how screwed we are.

cheers,
Zane.

[1] 
https://github.com/search?q=org%3Aopenstack+max-complexity+extension%3A.ini+path%3A%2F&type=Code
[2] 
http://lists.openstack.org/pipermail/openstack-dev/2014-October/048701.html
[3] 
http://git.openstack.org/cgit/openstack-dev/hacking/tree/requirements.txt#n9
[4] https://github.com/PyCQA/mccabe/blob/0.2.1/test_mccabe.py
[5] 
http://git.openstack.org/cgit/openstack/heat/tree/heat/engine/scheduler.py#n299