[openstack-dev] [all][code quality] Voting coverage job (-1 if coverage get worse after patch)

Boris Pavlovic boris at pavlovic.me
Sun Apr 19 01:30:02 UTC 2015

Hi stackers,

Code coverage is one of the very important metric of overall code quality
especially in case of Python. It's quite important to ensure that code is
covered fully with well written unit tests.

One of the nice thing is coverage job.

In Rally we are running it against every check which allows us to get
detailed information about
coverage before merging patch:

This helped Rally core team to automate checking that new/changed code is
covered by unit tests and we raised unit test coverage from ~78% to almost

But it produces few issues:
1) >9k nitpicking - core reviewers have to put -1 if something is not
covered by unit tests
2) core team scaling issues - core team members spend a lot of time just
checking that whole code is covered by unit test and leaving messages like
this should be covered by unit test
3) not friendly community - it's not nice to get on your code -1 from
somebody that is asking just to write unit tests
4) coverage regressions - sometimes we accidentally accept patches that
reduce coverage

To resolve this issue I improved a bit coverage job in Rally project, and
now it compares master vs master + patch coverage. If new coverage is less
than master job is marked as -1.

Here is the patch for job enhancement:

Here is coverage job in action:
patch https://review.openstack.org/#/c/174677/
job message

Best regards,
Boris Pavlovic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150419/17153d9c/attachment.html>

More information about the OpenStack-dev mailing list