One concern I have is this: suppose we find that a code block is unnecessary, or can be refactored more compactly, but it has test coverage. Then removing it would make the % coverage fall.<div><br></div><div>We want to remove the code, but we'd have to add unrelated tests to the same merge because otherwise the test coverage % would fall?<div>
<br></div><div>I think we can certainly enhance the metrics, but I do have concerns over strict gating (particularly per file, where the problem is more likely to occur than per-project)</div><div><br></div><div>Maybe the gate could be that line count of uncovered lines must not increase, unless the new % coverage > 80%.</div>
<div><br></div><div>Or we could simply have a gate bypass.</div><div><br></div><div>Justin</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 25, 2012 at 2:45 PM, Monty Taylor <span dir="ltr"><<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey - funny story - in responding to Justin I re-read the original email<br>
and realized it was asking for a static low number, which we _can_ do -<br>
at least project-wide. We can't do per-file yet, nor can we fail on a<br>
downward inflection... and I've emailed Justin about that.<br>
<br>
If we have consensus on gating on project-wide threshold, I can<br>
certainly add adding that to the gate to the todo list. (If we decide to<br>
do that, I'd really like to make that be openstack-wide rather than just<br>
nova... although I imagine it might take a few weeks to come to<br>
consensus on what the project-wide low number should be.<br>
<br>
Current numbers on project-wide lines numbers:<br>
<br>
nova: 79%<br>
glance: 75%<br>
keystone: 81%<br>
swift: 80%<br>
horizon: 91%<br>
<br>
Perhaps we get nova and glance up to 80 and then set the threshold for 80?<br>
<br>
Also, turns out we're not running this on the client libs...<br>
<br>
Monty<br>
<div class="im"><br>
On 04/25/2012 03:53 PM, Justin Santa Barbara wrote:<br>
</div><div class="im">> If you let me know in a bit more detail what you're looking for, I can<br>
> probably whip something up. Email me direct?<br>
><br>
> Justin<br>
><br>
><br>
> On Wed, Apr 25, 2012 at 6:59 AM, Monty Taylor <<a href="mailto:mordred@inaugust.com">mordred@inaugust.com</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:mordred@inaugust.com">mordred@inaugust.com</a>>> wrote:<br>
><br>
><br>
><br>
> On 04/24/2012 10:08 PM, Lorin Hochstein wrote:<br>
> ><br>
> > On Apr 24, 2012, at 4:11 PM, Joe Gordon wrote:<br>
> ><br>
> >> Hi All,<br>
> >><br>
> >> I would like to propose a minimum required code coverage level per<br>
> >> file in Nova. Say 80%. This would mean that any new feature/file<br>
> >> should only be accepted if it has over 80% code coverage. Exceptions<br>
> >> to this rule would be allowed for code that is covered by skipped<br>
> >> tests (as long as 80% is reached when the tests are not skipped).<br>
> >><br>
> ><br>
> > I like the idea of looking at code coverage numbers. For any<br>
> particular<br>
> > merge proposal, I'd also like to know whether it increases or<br>
> decreases<br>
> > the overall code coverage of the project. I don't think we should gate<br>
> > on this, but it would be helpful for a reviewer to see that,<br>
> especially<br>
> > for larger proposals.<br>
><br>
> Yup... Nati requested this a couple of summits ago - main issue is that<br>
> while we run code coverage and use the jenkins code coverage plugin to<br>
> track the coverage numbers, the plugin doesn't fully support this<br>
> particular kind of report.<br>
><br>
> HOWEVER - if any of our fine java friends out there want to chat with me<br>
> about adding support to the jenkins code coverage plugin to track and<br>
> report this, I will be thrilled to put it in as a piece of reported<br>
> information.<br>
><br>
> >> With 193 python files in nova/tests, Nova unit tests produce 85%<br>
> >> overall code coverage (calculated with ./run_test.sh -c [1]).<br>
> But 23%<br>
> >> of files (125 files) have lower then 80% code coverage (30 tests<br>
> >> skipped on my machine). Getting all files to hit the 80% code<br>
> >> coverage mark should be one of the goals for Folsom.<br>
> >><br>
> ><br>
> > I would really like to see a visualization of the code coverage<br>
> > distribution, in order to help spot the outliers.<br>
> ><br>
> ><br>
> > Along these lines, there's been a lot of work in the software<br>
> > engineering research community about predicting which parts of the<br>
> code<br>
> > are most likely to contain bugs ("fault prone" is a good keyword<br>
> to find<br>
> > this stuff, e.g.: <a href="http://scholar.google.com/scholar?q=fault+prone" target="_blank">http://scholar.google.com/scholar?q=fault+prone</a>, big<br>
> > names include Nachi Nagappan at MS Research and Elaine Weyuker,<br>
> formerly<br>
> > of AT&T Research). I would *love* to see some academic researchers try<br>
> > to apply those techniques to OpenStack to help guide QA activities by<br>
> > identifying which parts of the code should get more rigorous testing<br>
> > and review.<br>
><br>
> ++<br>
><br>
> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
</div></div>> <mailto:<a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a>><br>
<div class="HOEnZb"><div class="h5">> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
><br>
><br>
</div></div></blockquote></div><br></div></div>