[openstack-dev] [cinder] Proposal: changes to our current testing process

Eric Harney eharney at redhat.com
Wed Mar 2 15:16:14 UTC 2016


On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote:
> Eric,
> 
> For now, we test Cinder API with some concurrency only with Rally, so, IMO,
> it's reasonable get more scenarios for API races fixes.
> 
> It's not a hard task to implement new scenarios, they are pretty simple:
> [11] and [12]
> 

Sure, these are simple, but I think it's nowhere near that simple to
write a scenario which will prove that "remove API races" works correctly.

> [11]
> https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535
> [12]
> https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 
> On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney <eharney at redhat.com> wrote:
> 
>> On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
>>> Eric,
>>>
>>> There are Gorka's patches [10] to remove API Races
>>>
>>>
>>> [10]
>>>
>> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
>>>
>>> Regards,
>>> Ivan Kolodyazhny,
>>> http://blog.e0ne.info/
>>>
>>
>> So the second part of my question is, is writing a Rally job to prove
>> out that code a reasonable task?
>>
>> How hard is that to do and what does it look like?
>>
>>> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney <eharney at redhat.com> wrote:
>>>
>>>> On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
>>>>> Hi Team,
>>>>>
>>>>> Here are my thoughts and proposals how to make Cinder testing process
>>>>> better. I won't cover "3rd party CI's" topic here. I will share my
>>>> opinion
>>>>> about current and feature jobs.
>>>>>
>>>>>
>>>>> Unit-tests
>>>>>
>>>>>    - Long-running tests. I hope, everybody will agree that unit-tests
>>>> must
>>>>>    be quite simple and very fast. Unit tests which takes more than 3-5
>>>> seconds
>>>>>    should be refactored and/or moved to 'integration' tests.
>>>>>    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>>>>>    good to have some hacking checks to prevent such issues in a future.
>>>>>
>>>>>    - Tests coverage. We don't check it in an automatic way on gates.
>>>>>    Usually, we require to add some unit-tests during code review
>>>> process. Why
>>>>>    can't we add coverage job to our CI and do not merge new patches,
>> with
>>>>>    will decrease tests coverage rate? Maybe, such job could be voting
>> in
>>>> a
>>>>>    future to not ignore it. For now, there is not simple way to check
>>>> coverage
>>>>>    because 'tox -e cover' output is not useful [2].
>>>>>
>>>>>
>>>>> Functional tests for Cinder
>>>>>
>>>>> We introduced some functional tests last month [3]. Here is a patch to
>>>>> infra to add new job [4]. Because these tests were moved from
>>>> unit-tests, I
>>>>> think we're OK to make this job voting. Such tests should not be a
>>>>> replacement for Tempest. They even could tests Cinder with Fake Driver
>> to
>>>>> make it faster and not related on storage backends issues.
>>>>>
>>>>>
>>>>> Tempest in-tree tests
>>>>>
>>>>> Sean started work on it [5] and I think it's a good idea to get them in
>>>>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a
>> real
>>>>> backend.
>>>>>
>>>>>
>>>>> Functional tests for python-brick-cinderclient-ext
>>>>>
>>>>> There are patches that introduces functional tests [6] and new job [7].
>>>>>
>>>>>
>>>>> Functional tests for python-cinderclient
>>>>>
>>>>> We've got a very limited set of such tests and non-voting job. IMO, we
>>>> can
>>>>> run them even with Cinder Fake Driver to make them not depended on a
>>>>> storage backend and make it faster. I believe, we can make this job
>>>> voting
>>>>> soon. Also, we need more contributors to this kind of tests.
>>>>>
>>>>>
>>>>> Integrated tests for python-cinderclient
>>>>>
>>>>> We need such tests to make sure that we won't break Nova, Heat or other
>>>>> python-cinderclient consumers with a next merged patch. There is a
>> thread
>>>>> in openstack-dev ML about such tests [8] and proposal [9] to introduce
>>>> them
>>>>> to python-cinderclient.
>>>>>
>>>>>
>>>>> Rally tests
>>>>>
>>>>> IMO, it would be good to have new Rally scenarios for every patches
>> like
>>>>> 'improves performance', 'fixes concurrency issues', etc. Even if we as
>> a
>>>>> Cinder community don't have enough time to implement them, we have to
>> ask
>>>>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints
>> if
>>>>> needed.
>>>>>
>>>>
>>>> Are there any recent examples of a fix like this recently where it would
>>>> seem like a reasonable task to write a Rally scenario along with the
>> patch?
>>>>
>>>> Not being very familiar with Rally (as I think most of us aren't), I'm
>>>> having a hard time picturing this.
>>>>
>>>>>
>>>>> [1] https://review.openstack.org/#/c/282861/
>>>>> [2] http://paste.openstack.org/show/488925/
>>>>> [3] https://review.openstack.org/#/c/267801/
>>>>> [4] https://review.openstack.org/#/c/287115/
>>>>> [5] https://review.openstack.org/#/c/274471/
>>>>> [6] https://review.openstack.org/#/c/265811/
>>>>> [7] https://review.openstack.org/#/c/265925/
>>>>> [8]
>>>>>
>>>>
>> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
>>>>> [9] https://review.openstack.org/#/c/279432/
>>>>>
>>>>>
>>>>> Regards,
>>>>> Ivan Kolodyazhny,
>>>>> http://blog.e0ne.info/
>>>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 




More information about the OpenStack-dev mailing list