Open Stack

On 02/03, Ivan Kolodyazhny wrote:
> Eric,
> 
> There are Gorka's patches [10] to remove API Races
> 
> 
> [10]
> https://review.openstack.org/#/q/project:openstack/cinder+branch:master+topic:fix/api-races-simplified
> 

I looked at Rally a long time ago so apologies if I'm totally off base
here, but it looked like it was a performance evaluation tool, which
means that it probably won't help to check for API Races (at least I
didn't see how when I looked).

Many of the API races only happen if you simultaneously try the same
operation multiple times against the same resource or if there are
different operations that are trying to operate on the same resource.

On the first case if Rally allowed it we could test it because we know
only 1 of the operations should succeed, but on the second case when we
are talking about preventing races from different operations there is no
way to know what the result should be, since the order in which those
operations are executed on each test run will determine which one will
fail and which one will succeed.

I'm not trying to go against the general idea of adding rally tests, I
just think that they won't help in the case of the API races.

> > > [ ... ]
> > >
> > >    - Tests coverage. We don't check it in an automatic way on gates.
> > >    Usually, we require to add some unit-tests during code review
> > process. Why
> > >    can't we add coverage job to our CI and do not merge new patches, with
> > >    will decrease tests coverage rate? Maybe, such job could be voting in
> > a
> > >    future to not ignore it. For now, there is not simple way to check
> > coverage
> > >    because 'tox -e cover' output is not useful [2].

In my experience preventing patches that reduce test coverage can be
sometimes problematic because it's a percentage, and many times code
refactoring gives false positives.  :-(


Cheers,
Gorka.


> > >
> > >
> > > Functional tests for Cinder
> > >
> > > We introduced some functional tests last month [3]. Here is a patch to
> > > infra to add new job [4]. Because these tests were moved from
> > unit-tests, I
> > > think we're OK to make this job voting. Such tests should not be a
> > > replacement for Tempest. They even could tests Cinder with Fake Driver to
> > > make it faster and not related on storage backends issues.
> > >
> > >
> > > Tempest in-tree tests
> > >
> > > Sean started work on it [5] and I think it's a good idea to get them in
> > > Cinder repo to run them on Tempest jobs and 3-rd party CIs against a real
> > > backend.
> > >
> > >
> > > Functional tests for python-brick-cinderclient-ext
> > >
> > > There are patches that introduces functional tests [6] and new job [7].
> > >
> > >
> > > Functional tests for python-cinderclient
> > >
> > > We've got a very limited set of such tests and non-voting job. IMO, we
> > can
> > > run them even with Cinder Fake Driver to make them not depended on a
> > > storage backend and make it faster. I believe, we can make this job
> > voting
> > > soon. Also, we need more contributors to this kind of tests.
> > >
> > >
> > > Integrated tests for python-cinderclient
> > >
> > > We need such tests to make sure that we won't break Nova, Heat or other
> > > python-cinderclient consumers with a next merged patch. There is a thread
> > > in openstack-dev ML about such tests [8] and proposal [9] to introduce
> > them
> > > to python-cinderclient.
> > >
> > >
> > > Rally tests
> > >
> > > IMO, it would be good to have new Rally scenarios for every patches like
> > > 'improves performance', 'fixes concurrency issues', etc. Even if we as a
> > > Cinder community don't have enough time to implement them, we have to ask
> > > for them in reviews, openstack-dev ML, file Rally bugs and blueprints if
> > > needed.
> > >
> >
> > Are there any recent examples of a fix like this recently where it would
> > seem like a reasonable task to write a Rally scenario along with the patch?
> >
> > Not being very familiar with Rally (as I think most of us aren't), I'm
> > having a hard time picturing this.
> >
> > >
> > > [1] https://review.openstack.org/#/c/282861/
> > > [2] http://paste.openstack.org/show/488925/
> > > [3] https://review.openstack.org/#/c/267801/
> > > [4] https://review.openstack.org/#/c/287115/
> > > [5] https://review.openstack.org/#/c/274471/
> > > [6] https://review.openstack.org/#/c/265811/
> > > [7] https://review.openstack.org/#/c/265925/
> > > [8]
> > >
> > http://lists.openstack.org/pipermail/openstack-dev/2016-March/088027.html
> > > [9] https://review.openstack.org/#/c/279432/
> > >
> > >
> > > Regards,
> > > Ivan Kolodyazhny,
> > > http://blog.e0ne.info/
> > >
> > >
> > >
> > >
> > __________________________________________________________________________
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev