<div class="gmail_extra">It would nice to initially see the code coverage delta per merge proposal as a comment in gerrit (similar to SmokeStack), and not as a gating factor. </div><div class="gmail_extra"><p style="margin:0px 0px 0px 0px;font:13.0px Arial;color:#232323;min-height:15.0px">
<br></p>
<p style="margin:0px 0px 0px 0px;font:13.0px Arial;color:#232323">Kevin, should we start copying<span class="Apple-style-span" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif"> openstack-common</span> tests to client projects? Or just make sure to not count openstack-common code in the code coverage numbers for client projects?</p>
<div class="gmail_extra"><br></div><div class="gmail_extra">best,</div><div class="gmail_extra">Joe</div><br><div class="gmail_quote">On Wed, Apr 25, 2012 at 7:30 PM, Tim Simpson <span dir="ltr"><<a href="mailto:tim.simpson@rackspace.com" target="_blank">tim.simpson@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-size:10pt;font-family:Tahoma">Great point Justin. I've worked on projects where this has happened repeatedly and it's a drag.
<br>
<br>
<div style="font-size:16px;font-family:Times New Roman">
<hr>
<div style="direction:ltr"><font face="Tahoma" color="#000000"><b>From:</b> openstack-bounces+tim.simpson=<a href="mailto:rackspace.com@lists.launchpad.net" target="_blank">rackspace.com@lists.launchpad.net</a> [openstack-bounces+tim.simpson=<a href="mailto:rackspace.com@lists.launchpad.net" target="_blank">rackspace.com@lists.launchpad.net</a>] on behalf of Justin Santa
Barbara [<a href="mailto:justin@fathomdb.com" target="_blank">justin@fathomdb.com</a>]<br>
<b>Sent:</b> Wednesday, April 25, 2012 5:20 PM<br>
<b>To:</b> Monty Taylor<div class="im"><br>
<b>Cc:</b> <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<b>Subject:</b> Re: [Openstack] [OpenStack][Nova] Minimum required code coverage per file<br>
</div></font><br>
</div><div><div class="h5">
<div></div>
<div>One concern I have is this: suppose we find that a code block is unnecessary, or can be refactored more compactly, but it has test coverage. Then removing it would make the % coverage fall.
<div><br>
</div>
<div>We want to remove the code, but we'd have to add unrelated tests to the same merge because otherwise the test coverage % would fall?
<div><br>
</div>
<div>I think we can certainly enhance the metrics, but I do have concerns over strict gating (particularly per file, where the problem is more likely to occur than per-project)</div>
<div><br>
</div>
<div>Maybe the gate could be that line count of uncovered lines must not increase, unless the new % coverage > 80%.</div>
<div><br>
</div>
<div>Or we could simply have a gate bypass.</div>
<div><br>
</div>
<div>Justin</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Apr 25, 2012 at 2:45 PM, Monty Taylor <span dir="ltr">
<<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hey - funny story - in responding to Justin I re-read the original email<br>
and realized it was asking for a static low number, which we _can_ do -<br>
at least project-wide. We can't do per-file yet, nor can we fail on a<br>
downward inflection... and I've emailed Justin about that.<br>
<br>
If we have consensus on gating on project-wide threshold, I can<br>
certainly add adding that to the gate to the todo list. (If we decide to<br>
do that, I'd really like to make that be openstack-wide rather than just<br>
nova... although I imagine it might take a few weeks to come to<br>
consensus on what the project-wide low number should be.<br>
<br>
Current numbers on project-wide lines numbers:<br>
<br>
nova: 79%<br>
glance: 75%<br>
keystone: 81%<br>
swift: 80%<br>
horizon: 91%<br>
<br>
Perhaps we get nova and glance up to 80 and then set the threshold for 80?<br>
<br>
Also, turns out we're not running this on the client libs...<br>
<br>
Monty<br>
<div><br>
On 04/25/2012 03:53 PM, Justin Santa Barbara wrote:<br>
</div>
<div>> If you let me know in a bit more detail what you're looking for, I can<br>
> probably whip something up. Email me direct?<br>
><br>
> Justin<br>
><br>
><br>
> On Wed, Apr 25, 2012 at 6:59 AM, Monty Taylor <<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a><br>
</div>
<div>
<div>> <mailto:<a href="mailto:mordred@inaugust.com" target="_blank">mordred@inaugust.com</a>>> wrote:<br>
><br>
><br>
><br>
> On 04/24/2012 10:08 PM, Lorin Hochstein wrote:<br>
> ><br>
> > On Apr 24, 2012, at 4:11 PM, Joe Gordon wrote:<br>
> ><br>
> >> Hi All,<br>
> >><br>
> >> I would like to propose a minimum required code coverage level per<br>
> >> file in Nova. Say 80%. This would mean that any new feature/file<br>
> >> should only be accepted if it has over 80% code coverage. Exceptions<br>
> >> to this rule would be allowed for code that is covered by skipped<br>
> >> tests (as long as 80% is reached when the tests are not skipped).<br>
> >><br>
> ><br>
> > I like the idea of looking at code coverage numbers. For any<br>
> particular<br>
> > merge proposal, I'd also like to know whether it increases or<br>
> decreases<br>
> > the overall code coverage of the project. I don't think we should gate<br>
> > on this, but it would be helpful for a reviewer to see that,<br>
> especially<br>
> > for larger proposals.<br>
><br>
> Yup... Nati requested this a couple of summits ago - main issue is that<br>
> while we run code coverage and use the jenkins code coverage plugin to<br>
> track the coverage numbers, the plugin doesn't fully support this<br>
> particular kind of report.<br>
><br>
> HOWEVER - if any of our fine java friends out there want to chat with me<br>
> about adding support to the jenkins code coverage plugin to track and<br>
> report this, I will be thrilled to put it in as a piece of reported<br>
> information.<br>
><br>
> >> With 193 python files in nova/tests, Nova unit tests produce 85%<br>
> >> overall code coverage (calculated with ./run_test.sh -c [1]).<br>
> But 23%<br>
> >> of files (125 files) have lower then 80% code coverage (30 tests<br>
> >> skipped on my machine). Getting all files to hit the 80% code<br>
> >> coverage mark should be one of the goals for Folsom.<br>
> >><br>
> ><br>
> > I would really like to see a visualization of the code coverage<br>
> > distribution, in order to help spot the outliers.<br>
> ><br>
> ><br>
> > Along these lines, there's been a lot of work in the software<br>
> > engineering research community about predicting which parts of the<br>
> code<br>
> > are most likely to contain bugs ("fault prone" is a good keyword<br>
> to find<br>
> > this stuff, e.g.: <a href="http://scholar.google.com/scholar?q=fault+prone" target="_blank">
http://scholar.google.com/scholar?q=fault+prone</a>, big<br>
> > names include Nachi Nagappan at MS Research and Elaine Weyuker,<br>
> formerly<br>
> > of AT&T Research). I would *love* to see some academic researchers try<br>
> > to apply those techniques to OpenStack to help guide QA activities by<br>
> > identifying which parts of the code should get more rigorous testing<br>
> > and review.<br>
><br>
> ++<br>
><br>
> _______________________________________________<br>
> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">
openstack@lists.launchpad.net</a><br>
</div>
</div>
> <mailto:<a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a>><br>
<div>
<div>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">
https://launchpad.net/~openstack</a><br>
> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">
https://help.launchpad.net/ListHelp</a><br>
><br>
><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
<br>_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br></blockquote></div><br></div>