Open Stack

Hi Team,

Here are my thoughts and proposals how to make Cinder testing process
better. I won't cover "3rd party CI's" topic here. I will share my opinion
about current and feature jobs.


Unit-tests

   - Long-running tests. I hope, everybody will agree that unit-tests must
   be quite simple and very fast. Unit tests which takes more than 3-5 seconds
   should be refactored and/or moved to 'integration' tests.
   Thanks to Tom Barron for several fixes like [1]. IMO, we it would be
   good to have some hacking checks to prevent such issues in a future.

   - Tests coverage. We don't check it in an automatic way on gates.
   Usually, we require to add some unit-tests during code review process. Why
   can't we add coverage job to our CI and do not merge new patches, with
   will decrease tests coverage rate? Maybe, such job could be voting in a
   future to not ignore it. For now, there is not simple way to check coverage
   because 'tox -e cover' output is not useful [2].


Functional tests for Cinder

We introduced some functional tests last month [3]. Here is a patch to
infra to add new job [4]. Because these tests were moved from unit-tests, I
think we're OK to make this job voting. Such tests should not be a
replacement for Tempest. They even could tests Cinder with Fake Driver to
make it faster and not related on storage backends issues.


Tempest in-tree tests

Sean started work on it [5] and I think it's a good idea to get them in
Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
backend.


Functional tests for python-brick-cinderclient-ext

There are patches that introduces functional tests [6] and new job [7].


Functional tests for python-cinderclient

We've got a very limited set of such tests and non-voting job. IMO, we can
run them even with Cinder Fake Driver to make them not depended on a
storage backend and make it faster. I believe, we can make this job voting
soon. Also, we need more contributors to this kind of tests.


Integrated tests for python-cinderclient

We need such tests to make sure that we won't break Nova, Heat or other
python-cinderclient consumers with a next merged patch. There is a thread
in openstack-dev ML about such tests [8] and proposal [9] to introduce them
to python-cinderclient.


Rally tests

IMO, it would be good to have new Rally scenarios for every patches like
'improves performance', 'fixes concurrency issues', etc. Even if we as a
Cinder community don't have enough time to implement them, we have to ask
for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
needed.


[1] https://review.openstack.org/#/c/282861/
[2] http://paste.openstack.org/show/488925/
[3] https://review.openstack.org/#/c/267801/
[4] https://review.openstack.org/#/c/287115/
[5] https://review.openstack.org/#/c/274471/
[6] https://review.openstack.org/#/c/265811/
[7] https://review.openstack.org/#/c/265925/
[8]
http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
[9] https://review.openstack.org/#/c/279432/


Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160302/fbb3cfa3/attachment.html>