Open Stack

On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:
> Eric,
> 
> There are Gorka's patches [10] to remove API Races
> 
> 
> [10]
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> 
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
> 

So the second part of my question is, is writing a Rally job to prove
out that code a reasonable task?

How hard is that to do and what does it look like?

> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney <eharney at redhat.com> wrote:
> 
>> On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:
>>> Hi Team,
>>>
>>> Here are my thoughts and proposals how to make Cinder testing process
>>> better. I won't cover "3rd party CI's" topic here. I will share my
>> opinion
>>> about current and feature jobs.
>>>
>>>
>>> Unit-tests
>>>
>>>    - Long-running tests. I hope, everybody will agree that unit-tests
>> must
>>>    be quite simple and very fast. Unit tests which takes more than 3-5
>> seconds
>>>    should be refactored and/or moved to 'integration' tests.
>>>    Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
>>>    good to have some hacking checks to prevent such issues in a future.
>>>
>>>    - Tests coverage. We don't check it in an automatic way on gates.
>>>    Usually, we require to add some unit-tests during code review
>> process. Why
>>>    can't we add coverage job to our CI and do not merge new patches, with
>>>    will decrease tests coverage rate? Maybe, such job could be voting in
>> a
>>>    future to not ignore it. For now, there is not simple way to check
>> coverage
>>>    because 'tox -e cover' output is not useful [2].
>>>
>>>
>>> Functional tests for Cinder
>>>
>>> We introduced some functional tests last month [3]. Here is a patch to
>>> infra to add new job [4]. Because these tests were moved from
>> unit-tests, I
>>> think we're OK to make this job voting. Such tests should not be a
>>> replacement for Tempest. They even could tests Cinder with Fake Driver to
>>> make it faster and not related on storage backends issues.
>>>
>>>
>>> Tempest in-tree tests
>>>
>>> Sean started work on it [5] and I think it's a good idea to get them in
>>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
>>> backend.
>>>
>>>
>>> Functional tests for python-brick-cinderclient-ext
>>>
>>> There are patches that introduces functional tests [6] and new job [7].
>>>
>>>
>>> Functional tests for python-cinderclient
>>>
>>> We've got a very limited set of such tests and non-voting job. IMO, we
>> can
>>> run them even with Cinder Fake Driver to make them not depended on a
>>> storage backend and make it faster. I believe, we can make this job
>> voting
>>> soon. Also, we need more contributors to this kind of tests.
>>>
>>>
>>> Integrated tests for python-cinderclient
>>>
>>> We need such tests to make sure that we won't break Nova, Heat or other
>>> python-cinderclient consumers with a next merged patch. There is a thread
>>> in openstack-dev ML about such tests [8] and proposal [9] to introduce
>> them
>>> to python-cinderclient.
>>>
>>>
>>> Rally tests
>>>
>>> IMO, it would be good to have new Rally scenarios for every patches like
>>> 'improves performance', 'fixes concurrency issues', etc. Even if we as a
>>> Cinder community don't have enough time to implement them, we have to ask
>>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
>>> needed.
>>>
>>
>> Are there any recent examples of a fix like this recently where it would
>> seem like a reasonable task to write a Rally scenario along with the patch?
>>
>> Not being very familiar with Rally (as I think most of us aren't), I'm
>> having a hard time picturing this.
>>
>>>
>>> [1] https://review.openstack.org/#/c/282861/
>>> [2] http://paste.openstack.org/show/488925/
>>> [3] https://review.openstack.org/#/c/267801/
>>> [4] https://review.openstack.org/#/c/287115/
>>> [5] https://review.openstack.org/#/c/274471/
>>> [6] https://review.openstack.org/#/c/265811/
>>> [7] https://review.openstack.org/#/c/265925/
>>> [8]
>>>
>> http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
>>> [9] https://review.openstack.org/#/c/279432/
>>>
>>>
>>> Regards,
>>> Ivan Kolodyazhny,
>>> http://blog.e0ne.info/
>>>