<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">čt 6. 10. 2016 v 16:41 odesílatel Andrea Frittoli <<a href="mailto:andrea.frittoli@gmail.com">andrea.frittoli@gmail.com</a>> napsal:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">The difficulty with integration testing is that the services under test run in processes separated from the test one(s). <div class="gmail_msg"><br class="gmail_msg"><div class="gmail_msg">There is not obvious / existing mechanism to collect coverage data in this case. Several cycles back used to be a backdoor built into nova to enable coverage data collection during integration testing, but it has been removed long ago.</div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div></div></div></blockquote><div>andrea, AFAIK it is possible to create a hook like this[1]: <br><br></div><div> path = os.path.join(sysconfig.get_python_lib(), '<span class="inbox-inbox-pl-c1">hack.pth'</span>)</div><div> with open(path, 'w') as fd:<br></div><div> fd.write("""<br></div><div> import coverage;<span class="inbox-inbox-n"> coverage</span><span class="inbox-inbox-o">.</span><span class="inbox-inbox-n">process_startup</span><span class="inbox-inbox-p">()</span><br> """<br><br></div><div>to always enable instrumenting any python process.<br></div><div>This can get more fancy with white/black listing paths to consider.<br></div><div>The project I've referenced in the beginning of this thread used to enable tracing this way (with some filtering applied).<br><br></div><div>Cheers,<br></div><div>milan<br><br>[1] <a href="https://coverage.readthedocs.io/en/coverage-4.2/subprocess.html#configuring-python-for-sub-process-coverage">https://coverage.readthedocs.io/en/coverage-4.2/subprocess.html#configuring-python-for-sub-process-coverage</a><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"></div><div class="gmail_msg">andrea</div></div></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Thu, Sep 29, 2016 at 12:12 PM Assaf Muller <<a href="mailto:assaf@redhat.com" class="gmail_msg" target="_blank">assaf@redhat.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">On Thu, Sep 29, 2016 at 5:27 AM, milanisko k <span dir="ltr" class="gmail_msg"><<a href="mailto:vetrisko@gmail.com" class="gmail_msg" target="_blank">vetrisko@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">út 27. 9. 2016 v 20:12 odesílatel Assaf Muller <<a href="mailto:assaf@redhat.com" class="gmail_msg" target="_blank">assaf@redhat.com</a>> napsal:<br class="gmail_msg"></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><span class="m_-7334834038753116405m_-3004969265291985821gmail- gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">On Tue, Sep 27, 2016 at 2:05 PM, Assaf Muller <span dir="ltr" class="gmail_msg"><<a href="mailto:assaf@redhat.com" class="gmail_msg" target="_blank">assaf@redhat.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><br class="gmail_msg"><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg"><span class="gmail_msg">On Tue, Sep 27, 2016 at 12:18 PM, Timur Nurlygayanov <span dir="ltr" class="gmail_msg"><<a href="mailto:tnurlygayanov@mirantis.com" class="gmail_msg" target="_blank">tnurlygayanov@mirantis.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg">Hi milan,<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">we have measured the test coverage for OpenStack components with coverage.py tool [1]. It is very easy tool and it allows measure the coverage by lines of code and etc. (several metrics are available).</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">[1] <a href="https://coverage.readthedocs.io/en/coverage-4.2/" class="gmail_msg" target="_blank">https://coverage.readthedocs.io/en/coverage-4.2/</a></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></span><div class="gmail_msg">coverage also supports aggregating results from multiple runs, so you can measure results from combinations such as:<br class="gmail_msg"> </div></div></div></div></blockquote></div></div></div></blockquote><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">1) Unit tests</div><div class="gmail_msg">2) Functional tests</div><div class="gmail_msg">3) Integration tests</div><div class="gmail_msg">4) 1 + 2</div><div class="gmail_msg">5) 1 + 2 + 3</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">To my eyes 3 and 4 make the most sense. Unit and functional tests are supposed to give you low level coverage, keeping in mind that 'functional tests' is an overloaded term and actually means something else in every community. Integration tests aren't about code coverage, they're about user facing flows, so it'd be interesting to measure coverage from integration tests,</div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Sorry, replace integration with unit + functional.</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">then comparing coverage coming from integration tests, and getting the set difference between the two: That's the area that needs more unit and functional tests.</div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">To reiterate:</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Run coverage from integration tests, let this be c</div><div class="gmail_msg">Run coverage from unit and functional tests, let this be c'</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Let diff = c \ c'</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">'diff' is where you're missing unit and functional tests coverage.</div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></span><div class="gmail_msg">Assaf, the tool I linked is a monkey-patched coverage.py but the collector stores the stats in Redis --- gives the same accumulative collecting.<br class="gmail_msg"></div><div class="gmail_msg">Is there any interest/effort to collect coverage stats from selected jobs in CI, no matter the tool used?<br class="gmail_msg"></div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Some projects already collect coverage stats on their post-merge queue:</div><div class="gmail_msg"><a href="http://logs.openstack.org/61/61af70a734b99e61e751cfb494ddc93a85eec394/post/nova-coverage-db-ubuntu-xenial/55210aa/" class="gmail_msg" target="_blank">http://logs.openstack.org/61/61af70a734b99e61e751cfb494ddc93a85eec394/post/nova-coverage-db-ubuntu-xenial/55210aa/</a><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">It's invoked with 'tox -e cover' which you define in your project's tox.ini file, I imagine most projects if not all have it set up to gather coverage from a unit tests run.</div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"></div><div class="gmail_msg"><div class="m_-7334834038753116405m_-3004969265291985821gmail-h5 gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><div class="gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Tue, Sep 27, 2016 at 1:06 PM, Jordan Pittier <span dir="ltr" class="gmail_msg"><<a href="mailto:jordan.pittier@scality.com" class="gmail_msg" target="_blank">jordan.pittier@scality.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg">Hi,<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><span class="gmail_msg"><span class="gmail_msg">On Tue, Sep 27, 2016 at 11:43 AM, milanisko k <span dir="ltr" class="gmail_msg"><<a href="mailto:vetrisko@gmail.com" class="gmail_msg" target="_blank">vetrisko@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_msg">Dear Stackers,<br class="gmail_msg"></div><div class="gmail_msg">I'd like to gather some overview on the $Sub: is there some infrastructure in place to gather such stats? Are there any groups interested in it? Any plans to establish such infrastructure?<br class="gmail_msg"></div></div></blockquote></span></span><div class="gmail_msg">I am working on such a tool with mixed results so far. Here's my approach taking let's say Nova as an example:</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">1) Print all the routes known to nova (available as a python-routes object: nova.api.openstack.compute.APIRouterV21())</div><div class="gmail_msg">2) "Normalize" the Nova routes </div><div class="gmail_msg">3) Take the logs produced by Tempest during a tempest run (in logs/tempest.txt.gz). Grep for what looks like a Nova URL (based on port 8774)</div><div class="gmail_msg">4) "Normalize" the tested-by-tempest Nova routes.</div><div class="gmail_msg">5) Compare the two sets of routes</div><div class="gmail_msg">6) ????</div><div class="gmail_msg">7) Profit !!</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">So the hard part is obviously the normalizing of the URLs. I am currently using a tons of regex.... :) That's not fun.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I'll let you guys know if I have something to show.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I think there's real interest on the topic (it comes up every year or so), but no definitive answer/tool.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Cheers,</div><div class="gmail_msg">Jordan</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div>
<br class="gmail_msg">
<a href="https://www.scality.com/backup/?utm_source=signatures&utm_medium=email&utm_campaign=backup2016" class="gmail_msg" target="_blank"><img src="https://support.scality.com/Email_Footers/Current/Scality_Email_Footer.png" class="gmail_msg"></a><br class="gmail_msg">__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
<br class="gmail_msg"></blockquote></div><span class="gmail_msg"><font class="gmail_msg" color="#888888"><br class="gmail_msg"><br class="gmail_msg" clear="all"><div class="gmail_msg"><br class="gmail_msg"></div>-- <br class="gmail_msg"><div class="gmail_msg"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><div dir="ltr" class="gmail_msg"><font class="gmail_msg" color="#888888"><font class="gmail_msg" color="#888888"><br class="gmail_msg"></font></font><div style="font-family:arial;font-size:small" class="gmail_msg">Timur,</div><div style="font-family:arial;font-size:small" class="gmail_msg">Senior QA Manager</div><div style="font-family:arial;font-size:small" class="gmail_msg">OpenStack Projects</div><div style="font-family:arial;font-size:small" class="gmail_msg">Mirantis Inc</div></div></div></div></div></div></div></div></div>
</font></span></div>
<br class="gmail_msg">__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
<br class="gmail_msg"></blockquote></div></div></div><br class="gmail_msg"></div></div>
</blockquote></div></div></div>
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</blockquote></div></div></div></div></div>
<br class="gmail_msg">__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
<br class="gmail_msg"></blockquote></div></div></div>
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</blockquote></div>
__________________________________________________________________________<br class="gmail_msg">
OpenStack Development Mailing List (not for usage questions)<br class="gmail_msg">
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" class="gmail_msg" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br class="gmail_msg">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="gmail_msg">
</blockquote></div></div>