<div dir="ltr">I'll try to implement such scenario and step-by-step guideline soon.</div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div>Regards,<br>Ivan Kolodyazhny,<br><a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></div></div></div></div>
<br><div class="gmail_quote">On Wed, Mar 2, 2016 at 5:16 PM, Eric Harney <span dir="ltr"><<a href="mailto:eharney@redhat.com" target="_blank">eharney@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 03/02/2016 10:07 AM, Ivan Kolodyazhny wrote:<br>
> Eric,<br>
><br>
</span><span class="">> For now, we test Cinder API with some concurrency only with Rally, so, IMO,<br>
> it's reasonable get more scenarios for API races fixes.<br>
><br>
> It's not a hard task to implement new scenarios, they are pretty simple:<br>
> [11] and [12]<br>
><br>
<br>
</span>Sure, these are simple, but I think it's nowhere near that simple to<br>
write a scenario which will prove that "remove API races" works correctly.<br>
<div class="HOEnZb"><div class="h5"><br>
> [11]<br>
> <a href="https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535" rel="noreferrer" target="_blank">https://github.com/openstack/rally/blob/master/rally/plugins/openstack/scenarios/cinder/volumes.py#L535</a><br>
> [12]<br>
> <a href="https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516" rel="noreferrer" target="_blank">https://github.com/openstack/rally/blob/master/rally-jobs/cinder.yaml#L516</a><br>
><br>
> Regards,<br>
> Ivan Kolodyazhny,<br>
> <a href="http://blog.e0ne.info/" rel="noreferrer" target="_blank">http://blog.e0ne.info/</a><br>
><br>
> On Wed, Mar 2, 2016 at 4:50 PM, Eric Harney <<a href="mailto:eharney@redhat.com">eharney@redhat.com</a>> wrote:<br>
><br>
>> On 03/02/2016 09:36 AM, Ivan Kolodyazhny wrote:<br>
>>> Eric,<br>
>>><br>
>>> There are Gorka's patches [10] to remove API Races<br>
>>><br>
>>><br>
>>> [10]<br>
>>><br>
>> <a href="https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified" rel="noreferrer" target="_blank">https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified</a><br>
>>><br>
>>> Regards,<br>
>>> Ivan Kolodyazhny,<br>
>>> <a href="http://blog.e0ne.info/" rel="noreferrer" target="_blank">http://blog.e0ne.info/</a><br>
>>><br>
>><br>
>> So the second part of my question is, is writing a Rally job to prove<br>
>> out that code a reasonable task?<br>
>><br>
>> How hard is that to do and what does it look like?<br>
>><br>
>>> On Wed, Mar 2, 2016 at 4:27 PM, Eric Harney <<a href="mailto:eharney@redhat.com">eharney@redhat.com</a>> wrote:<br>
>>><br>
>>>> On 03/02/2016 06:25 AM, Ivan Kolodyazhny wrote:<br>
>>>>> Hi Team,<br>
>>>>><br>
>>>>> Here are my thoughts and proposals how to make Cinder testing process<br>
>>>>> better. I won't cover "3rd party CI's" topic here. I will share my<br>
>>>> opinion<br>
>>>>> about current and feature jobs.<br>
>>>>><br>
>>>>><br>
>>>>> Unit-tests<br>
>>>>><br>
>>>>> - Long-running tests. I hope, everybody will agree that unit-tests<br>
>>>> must<br>
>>>>> be quite simple and very fast. Unit tests which takes more than 3-5<br>
>>>> seconds<br>
>>>>> should be refactored and/or moved to 'integration' tests.<br>
>>>>> Thanks to Tom Barron for several fixes like [1]. IMO, we it would be<br>
>>>>> good to have some hacking checks to prevent such issues in a future.<br>
>>>>><br>
>>>>> - Tests coverage. We don't check it in an automatic way on gates.<br>
>>>>> Usually, we require to add some unit-tests during code review<br>
>>>> process. Why<br>
>>>>> can't we add coverage job to our CI and do not merge new patches,<br>
>> with<br>
>>>>> will decrease tests coverage rate? Maybe, such job could be voting<br>
>> in<br>
>>>> a<br>
>>>>> future to not ignore it. For now, there is not simple way to check<br>
>>>> coverage<br>
>>>>> because 'tox -e cover' output is not useful [2].<br>
>>>>><br>
>>>>><br>
>>>>> Functional tests for Cinder<br>
>>>>><br>
>>>>> We introduced some functional tests last month [3]. Here is a patch to<br>
>>>>> infra to add new job [4]. Because these tests were moved from<br>
>>>> unit-tests, I<br>
>>>>> think we're OK to make this job voting. Such tests should not be a<br>
>>>>> replacement for Tempest. They even could tests Cinder with Fake Driver<br>
>> to<br>
>>>>> make it faster and not related on storage backends issues.<br>
>>>>><br>
>>>>><br>
>>>>> Tempest in-tree tests<br>
>>>>><br>
>>>>> Sean started work on it [5] and I think it's a good idea to get them in<br>
>>>>> Cinder repo to run them on Tempest jobs and 3-rd party CIs against a<br>
>> real<br>
>>>>> backend.<br>
>>>>><br>
>>>>><br>
>>>>> Functional tests for python-brick-cinderclient-ext<br>
>>>>><br>
>>>>> There are patches that introduces functional tests [6] and new job [7].<br>
>>>>><br>
>>>>><br>
>>>>> Functional tests for python-cinderclient<br>
>>>>><br>
>>>>> We've got a very limited set of such tests and non-voting job. IMO, we<br>
>>>> can<br>
>>>>> run them even with Cinder Fake Driver to make them not depended on a<br>
>>>>> storage backend and make it faster. I believe, we can make this job<br>
>>>> voting<br>
>>>>> soon. Also, we need more contributors to this kind of tests.<br>
>>>>><br>
>>>>><br>
>>>>> Integrated tests for python-cinderclient<br>
>>>>><br>
>>>>> We need such tests to make sure that we won't break Nova, Heat or other<br>
>>>>> python-cinderclient consumers with a next merged patch. There is a<br>
>> thread<br>
>>>>> in openstack-dev ML about such tests [8] and proposal [9] to introduce<br>
>>>> them<br>
>>>>> to python-cinderclient.<br>
>>>>><br>
>>>>><br>
>>>>> Rally tests<br>
>>>>><br>
>>>>> IMO, it would be good to have new Rally scenarios for every patches<br>
>> like<br>
>>>>> 'improves performance', 'fixes concurrency issues', etc. Even if we as<br>
>> a<br>
>>>>> Cinder community don't have enough time to implement them, we have to<br>
>> ask<br>
>>>>> for them in reviews, openstack-dev ML, file Rally bugs and blueprints<br>
>> if<br>
>>>>> needed.<br>
>>>>><br>
>>>><br>
>>>> Are there any recent examples of a fix like this recently where it would<br>
>>>> seem like a reasonable task to write a Rally scenario along with the<br>
>> patch?<br>
>>>><br>
>>>> Not being very familiar with Rally (as I think most of us aren't), I'm<br>
>>>> having a hard time picturing this.<br>
>>>><br>
>>>>><br>
>>>>> [1] <a href="https://review.openstack.org/#/c/282861/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/282861/</a><br>
>>>>> [2] <a href="http://paste.openstack.org/show/488925/" rel="noreferrer" target="_blank">http://paste.openstack.org/show/488925/</a><br>
>>>>> [3] <a href="https://review.openstack.org/#/c/267801/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/267801/</a><br>
>>>>> [4] <a href="https://review.openstack.org/#/c/287115/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/287115/</a><br>
>>>>> [5] <a href="https://review.openstack.org/#/c/274471/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/274471/</a><br>
>>>>> [6] <a href="https://review.openstack.org/#/c/265811/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/265811/</a><br>
>>>>> [7] <a href="https://review.openstack.org/#/c/265925/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/265925/</a><br>
>>>>> [8]<br>
>>>>><br>
>>>><br>
>> <a href="http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html" rel="noreferrer" target="_blank">http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html</a><br>
>>>>> [9] <a href="https://review.openstack.org/#/c/279432/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/279432/</a><br>
>>>>><br>
>>>>><br>
>>>>> Regards,<br>
>>>>> Ivan Kolodyazhny,<br>
>>>>> <a href="http://blog.e0ne.info/" rel="noreferrer" target="_blank">http://blog.e0ne.info/</a><br>
>>>>><br>
>><br>
>><br>
>> __________________________________________________________________________<br>
>> OpenStack Development Mailing List (not for usage questions)<br>
>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>