Open Stack

On Thu, Sep 29, 2016 at 5:27 AM, milanisko k <vetrisko at gmail.com> wrote:

>
>
> út 27. 9. 2016 v 20:12 odesílatel Assaf Muller <assaf at redhat.com> napsal:
>
>> On Tue, Sep 27, 2016 at 2:05 PM, Assaf Muller <assaf at redhat.com> wrote:
>>
>>>
>>>
>>> On Tue, Sep 27, 2016 at 12:18 PM, Timur Nurlygayanov <
>>> tnurlygayanov at mirantis.com> wrote:
>>>
>>>> Hi milan,
>>>>
>>>> we have measured the test coverage for OpenStack components with
>>>> coverage.py tool [1]. It is very easy tool and it allows measure the
>>>> coverage by lines of code and etc. (several metrics are available).
>>>>
>>>> [1] https://coverage.readthedocs.io/en/coverage-4.2/
>>>>
>>>
>>> coverage also supports aggregating results from multiple runs, so you
>>> can measure results from combinations such as:
>>>
>>>
>>
>>> 1) Unit tests
>>> 2) Functional tests
>>> 3) Integration tests
>>> 4) 1 + 2
>>> 5) 1 + 2 + 3
>>>
>>> To my eyes 3 and 4 make the most sense. Unit and functional tests are
>>> supposed to give you low level coverage, keeping in mind that 'functional
>>> tests' is an overloaded term and actually means something else in every
>>> community. Integration tests aren't about code coverage, they're about user
>>> facing flows, so it'd be interesting to measure coverage
>>> from integration tests,
>>>
>>
>> Sorry, replace integration with unit + functional.
>>
>>
>>> then comparing coverage coming from integration tests, and getting the
>>> set difference between the two: That's the area that needs more unit and
>>> functional tests.
>>>
>>
>> To reiterate:
>>
>> Run coverage from integration tests, let this be c
>> Run coverage from unit and functional tests, let this be c'
>>
>> Let diff = c \ c'
>>
>> 'diff' is where you're missing unit and functional tests coverage.
>>
>
> Assaf, the tool I linked is a monkey-patched coverage.py but the collector
> stores the stats in Redis --- gives the same accumulative collecting.
> Is there any interest/effort to collect coverage stats from selected jobs
> in CI, no matter the tool used?
>

Some projects already collect coverage stats on their post-merge queue:
http://logs.openstack.org/61/61af70a734b99e61e751cfb494ddc93a85eec394/post/nova-coverage-db-ubuntu-xenial/55210aa/

It's invoked with 'tox -e cover' which you define in your project's tox.ini
file, I imagine most projects if not all have it set up to gather coverage
from a unit tests run.


>
>
>>
>>
>>>
>>>
>>>>
>>>> On Tue, Sep 27, 2016 at 1:06 PM, Jordan Pittier <
>>>> jordan.pittier at scality.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On Tue, Sep 27, 2016 at 11:43 AM, milanisko k <vetrisko at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Dear Stackers,
>>>>>> I'd like to gather some overview on the $Sub: is there some
>>>>>> infrastructure in place to gather such stats? Are there any groups
>>>>>> interested in it? Any plans to establish such infrastructure?
>>>>>>
>>>>> I am working on such a tool with mixed results so far. Here's my
>>>>> approach taking let's say Nova as an example:
>>>>>
>>>>> 1) Print all the routes known to nova (available as a python-routes
>>>>> object:  nova.api.openstack.compute.APIRouterV21())
>>>>> 2) "Normalize" the Nova routes
>>>>> 3) Take the logs produced by Tempest during a tempest run (in
>>>>> logs/tempest.txt.gz). Grep for what looks like a Nova URL (based on port
>>>>> 8774)
>>>>> 4) "Normalize" the tested-by-tempest Nova routes.
>>>>> 5) Compare the two sets of routes
>>>>> 6) ????
>>>>> 7) Profit !!
>>>>>
>>>>> So the hard part is obviously the normalizing of the URLs. I am
>>>>> currently using a tons of regex.... :) That's not fun.
>>>>>
>>>>> I'll let you guys know if I have something to show.
>>>>>
>>>>> I think there's real interest on the topic (it comes up every year or
>>>>> so), but no definitive answer/tool.
>>>>>
>>>>> Cheers,
>>>>> Jordan
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> <https://www.scality.com/backup/?utm_source=signatures&utm_medium=email&utm_campaign=backup2016>
>>>>> ____________________________________________________________
>>>>> ______________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
>>>>> unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Timur,
>>>> Senior QA Manager
>>>> OpenStack Projects
>>>> Mirantis Inc
>>>>
>>>> ____________________________________________________________
>>>> ______________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
>>>> unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:
>> unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160929/e3a1c043/attachment.html>