<div dir="ltr">[REPOSTING THIS REPLY]<div><br></div><div>Please let me know if others want to jump in the discussion on Quantum unit tests<br><div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">Hi Maru,</span><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px">thanks for contributing to this topic.</div><div style="font-family:arial,sans-serif;font-size:13px">More comments inline.</div><div style="font-family:arial,sans-serif;font-size:13px">
<br></div><div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">I was thinking of providing code samples during the summit <span class="">session</span> explaining how existing <span class="">tests</span> could be refactored. If you can provide some samples it would be great; also if you're ok with walking the audience through these samples, that would be awesome!</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Salvatore<br><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="im">On 4 April 2013 03:34, Maru Newby <span dir="ltr"><<a href="mailto:marun@redhat.com" target="_blank">marun@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Salvatore,<br><br>I'm working on a patch to swap pep8 for flake8, which will increase the automation of mechanical review. In the process of doing the cleanup required for the patch, I have seen some ways that quantum's test suite could be improved:<br>
<br> - An api method and its serialization format (xml, json) are orthogonal (or should be) and could be tested as such. For plugins that test both serialization formats for each api method, test execution time is effectively doubled with no benefit to coverage.<br>
</blockquote><div><br></div></div><div>I totally agree. Once we get past the WSGI layer, there is not difference between the JSON or the XML use case. The Serialization <span class="">tests</span> should just validate data serialization and deserialization.</div>
<div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br> - Many plugins contain duplicate code and <span class="">tests</span> (e.g. get_port_from_device and its associated <span class="">tests</span> appear in each of the nec, openvswitch, ryu, and linuxbridge plugins). Common code could be factored out and tested only once.<br>
</blockquote><div><br></div></div><div>That's another good point that I wanted to address. It seems these <span class="">tests</span> have the same coverage, so, as you say, they should be factored out in a test module aimed at covering that specific functionality.</div>
<div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>- Many quantum <span class="">tests</span> evaluate low-level functionality through a web api rather than isolating the functionality under test. Writing more focused <span class="">unit</span> <span class="">tests</span> (and fewer functional and integration <span class="">tests</span> ala <a href="http://www.martinfowler.com/bliki/TestPyramid.html" target="_blank">http://www.martinfowler.com/bliki/TestPyramid.html</a>) could allow for better coverage at a lower cost in both maintenance and execution.<br>
</blockquote><div><br></div></div><div>This is the first goal of reworking our <span class="">unit</span> <span class="">tests</span>. We have always been aware that those were not <span class="">unit</span> <span class="">tests</span>. However, those were the closest think we could provide to a gate<span class="">tests</span>. Now that we have proper gate <span class="">tests</span> - we should make sure our <span class="">unit</span> <span class="">tests</span> are proper <span class="">unit</span> <span class="">tests</span>! Also, it would be good to discuss what to make of the current <span class="">unit</span> <span class="">tests</span>, which are probably suitable for some sort of automated functional testing.</div>
<div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>The technical aspects of these potential improvements are straightforward, but will require a degree of developer and reviewer education to ensure that they are properly implemented and maintained going forward. Do you think the summit <span class="">session</span> you've proposed would be an appropriate venue, or should it be done separately?<br>
</blockquote><div><br></div></div><div>I think the summit <span class="">session</span> is just a starter. 'Education' should be enforced via code reviews. The main goal of the summit <span class="">session</span> is achieve consensus on the general direction, identify priorities, and possibly resources.</div>
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>Cheers,<br><br><br>Maru</blockquote>
</div></div></div></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 3 April 2013 18:34, Maru Newby <span dir="ltr"><<a href="mailto:marun@redhat.com" target="_blank">marun@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Salvatore,<br>
<br>
I'm working on a patch to swap pep8 for flake8, which will increase the automation of mechanical review. In the process of doing the cleanup required for the patch, I have seen some ways that quantum's test suite could be improved:<br>
<br>
- An api method and its serialization format (xml, json) are orthogonal (or should be) and could be tested as such. For plugins that test both serialization formats for each api method, test execution time is effectively doubled with no benefit to coverage.<br>
<br>
- Many plugins contain duplicate code and tests (e.g. get_port_from_device and its associated tests appear in each of the nec, openvswitch, ryu, and linuxbridge plugins). Common code could be factored out and tested only once.<br>
<br>
- Many quantum tests evaluate low-level functionality through a web api rather than isolating the functionality under test. Writing more focused unit tests (and fewer functional and integration tests ala <a href="http://www.martinfowler.com/bliki/TestPyramid.html" target="_blank">http://www.martinfowler.com/bliki/TestPyramid.html</a>) could allow for better coverage at a lower cost in both maintenance and execution.<br>
<br>
The technical aspects of these potential improvements are straightforward, but will require a degree of developer and reviewer education to ensure that they are properly implemented and maintained going forward. Do you think the summit session you've proposed would be an appropriate venue, or should it be done separately?<br>
<br>
Cheers,<br>
<br>
<br>
Maru</blockquote></div><br></div>