Open Stack

On Mon, Feb 04, 2013 at 01:53:09PM +0100, Jaroslav Henner wrote:
> On Po 4. únor 2013, 13:09:54 CET, Sean Dague wrote:
> >Coverage for tempest is different than coverage for unit tests. It
> >doesn't start at process start, it is turned on once nova is running.
> I am aware of that.
> 
> >
> >Which means the eval lines will have already executed prior to
> >coverage counters being enabled. And as far as I can tell, in most
> >cases, python coverage only counts the def line when the function is
> >first loaded (not executed). As such it's not surprising that the
> >decorators look weird.
> >
> >I'm honestly surprised any of the imports are marked as executed,
> >wonder if Matt's got any other thoughts.
> 
> Sure. The def lines in module are executed when module is being
> loaded. The content of the def is executed when the function is
> called.
> 
> I think the problem here is that gevent, greenlet and similar are
> doing some magic with the stack. See [1] and [2].
> 
I had a feeling this is what was causing the weird behavior. When I
was working on the coverage extension I saw those posts about coverage,
but, I wasn't aware of that 'fixed' version. I'll give that a try on 
some local runs to see if it fixes the weirdness in the coverage reports.
> 
> 
> In any case, the coverage reports like this are quite inaccurate, if
> not invalid. Maybe we should rather use the "fixed" coverage tool
> [3]
When I made the list of tests missing, I tried to manually confirm the
coverage report by looking through our tests. That's part of the reason
the test list isn't as long as the analysis yet. (Because it takes so long)
I also may have missed some, so if that patched coverage will work that
would be great.
> 
> [1] https://bitbucket.org/ned/coveragepy/issue/149/coverage-gevent-looks-broken
> [2] http://nedbatchelder.com/code/coverage/trouble.html
> [3] https://github.com/newbrough/coverage

-Matt Treinish

> >On 02/04/2013 06:21 AM, Jaroslav Henner wrote:
> >>Hi
> >>
> >>I fail to understand the coverage report. For example in
> >>http://logs.openstack.org/periodic/periodic-tempest-devstack-coverage-vm-full/15/logs/coverage-report/nova_compute_api.html
> >>
> >>
> >>  1) the import section. How is it possible some lines are reported to
> >>be not-executed, while the others are?
> >>  2) 91-95 The same question as 1)
> >>  3) How can it be that the condition (109) was not exectued, but the
> >>110 -- the statement was?
> >>
> >>
> >>
> >>On 30.1.2013 08:21, Yaniv Kaul wrote:
> >>>On 30/01/13 00:54, Matthew Treinish wrote:
> >>>>Hi Everyone,
> >>>>
> >>>>I've started working on an analysis of the coverage reports generated
> >>>>by the
> >>>>periodic Jenkins jobs:
> >>>>
> >>>>https://jenkins.openstack.org/job/periodic-tempest-devstack-coverage-vm-full/
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>I wanted to see how good a job tempest is doing for covering the Nova
> >>>>API code.
> >>>>Right now we've got some gaps in what is being tested in a tempest
> >>>>full gate
> >>>>run. I've started putting up a list of what functionality is missing
> >>>>from the
> >>>>tempest runs here:
> >>>>
> >>>>https://etherpad.openstack.org/coverage-analysis
> >>>>
> >>>>The periodic jobs uses the same configuration as our current gate
> >>>>jobs, so
> >>>>there are couple cases where we having testing but they depend on
> >>>>things not
> >>>>set in tempest.conf (Resize action for example)
> >>>>
> >>>>Then based on the results of the coverage report I put together a
> >>>>list of
> >>>>proposed additional tests here:
> >>>>
> >>>>https://etherpad.openstack.org/MissingTempestTests
> >>>
> >>>Will you be converting some of the bigger and/or contained tasks to
> >>>blueprints for Tempest?
> >>>Y.
> >>>
> >>>>
> >>>>Keep in mind that both of these etherpads are still WIP (and quite
> >>>>incomplete)
> >>>>and I'll be continually updating them over the next few days until I
> >>>>go through
> >>>>the rest of the coverage reports. I just wanted to share what I'm
> >>>>working on
> >>>>and some of the initial findings. Feel free to ping me on IRC for
> >>>>specifics or
> >>>>questions about any of this.
> >>>>
> >>>>Thanks,
> >>>>
> >>>>Matt Treinish
> >>>>
> >>>>
> >>>>_______________________________________________
> >>>>openstack-qa mailing list
> >>>>openstack-qa at lists.openstack.org
> >>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >>>
> >>>
> >>>_______________________________________________
> >>>openstack-qa mailing list
> >>>openstack-qa at lists.openstack.org
> >>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >>
> >>
> >>_______________________________________________
> >>openstack-qa mailing list
> >>openstack-qa at lists.openstack.org
> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> >>
> >
> >
> 
> 
> 
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa