Open Stack

On Mon, Feb 15, 2016 at 1:02 PM, Clark Boylan <cboylan at sapwetik.org> wrote:

> On Mon, Feb 15, 2016, at 11:48 AM, Ivan Kolodyazhny wrote:
> > Hi all,
> >
> > I'll talk mostly about python-cinderclient but the same question could be
> > related for other clients.
> >
> > Now, for python-cinderclient we've got to kinds for functional/integrated
> > jobs:
> >
> > 1) gate-cinderclient-dsvm-functional - a very limited (for now) set of
> > functional tests, most of them were part of tempest CLI tests in the
> > past.
> >
> > 2) gate-tempest-dsvm-neutron-src-python-cinderclient - if I understand
> > right, the idea os this job was to have integrated tests to test
> > cinderclient with other projects to verify that new patch to
> > python-cinderclietn won't break any other project.
> > But it does *not* test cinderclient at all, except few attach-related
> > tests
> > because Tempest doesn't use python-*client.
>
> Tempest doesn't use python-*client to talk to the APIs but the various
> OpenStack services do use python-*client to talk to the other services.
> Using cinderclient as an example, nova consumes cinderclient to perform
> volume operations in nova/volume/cinder.py. There is value in this
> existing test if those code paths are exercised. Basically ensuring the
> next release of cinderclient does not break nova. It may be the case
> that cinderclient is a bad example because tempest doesn't do volume
> operations through nova, but I am sure for many of the other clients
> these tests do provide value.
>
> >
> > The same job was added for python-heatclient but was removed because
> > devstack didn't install Heat for that job [1].
> >
> > We agreed [2] to remove this job from cinderclient gates too, once
> > functional or integration tests will be implemented.
>
> Just make sure that you don't lose exercising of the above code paths
> when this transition happens. If we don't currently test that code it
> would be a good goal for any new integration testing to do so.
>
> >
> >
> > There is a proposal to python-cinderclient tests to implement some
> > cross-project testing to make sure, that new python-cinderclient won't
> > break any of existing project who use it.
> >
> > After discussing in IRC with John Griffith (jgriffith) I'm realized that
> > it
> > could be an cross-project initiative in such kind of integration tests.
> > OpenStack Client (OSC) could cover some part of such tests, but does it
> > mean that we'll run OSC tests on every patch to python-*client? We can
> > run
> > only cinder-realated OSC tests on our gates to verify that it doesn't
> > breack OSC and, may be other project.
> >
> > The other option, is to implement tests like [3] per project basis and
> > call
> > it "integration".  Such tests could cover more cases than OSC functional
> > tests and have more project-related test cases, e.g.: test some
> > python-cinderclient specific corner cases, which is not related to OSC.
> >
> > IMO, It would be good to have some cross-project decision on how will be
> > implement clients' integration tests per project.
> >
> >
> > [1] https://review.openstack.org/#/c/272411/
> > [2]
> >
> http://eavesdrop.openstack.org/meetings/cinder/2015/cinder.2015-12-16-16.00.log.html
> > [3] https://review.openstack.org/#/c/279432/8
> >
> > Regards,
> > Ivan Kolodyazhny,
> > http://blog.e0ne.info/
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Hey Everyone,

So this started after after I made some comments on a patch in CinderClient
that added attach/detach integration tests to cinder's local test repo.  My
first thought was that we should focus on just Cinder functional tests
first, and that maybe the integration tests (including with the clients)
should be centralized, or have a more standardized approach to them.

What I was getting at is that while the Tempest tests don't use the clients
directly, there are a number of places where tempest does end up calling
them indirectly. Volume attach in Nova is a good example of this, while we
don't call NovaClient to do this, the Nova API drills down into
volume/cinder.py which just loads and calls CinderClient in order to issue
the volume related calls that it does.  My thought was that maybe it would
be useful to have a more cross-project effort for things like this. There
are other places we do this in a few projects with Glance, Keystone and
Swift as I recall.

I was actually thinking of a common test-run (similar to dsvm-full) but
something that focuses exclusively on the cross-project calls that are made
via clients.  The idea being that any xxx-client change would have to load
this scenario up with current master and run successfully.  Maybe this is
overkill?  Maybe we don't do the "import xxxClient" in as many places as I
thought we did?

Or... maybe just having good tests in each of the client projects just
solves this problem and by the nature of what we're trying to test there
isn't a bunch of overlap and the problem just sort of takes care of itself?

As far as the CinderClient changes, I guess I was hoping to focus on
functional tests within Cinder/CinderClient itself first before branching
out to various integration points.  That's what raised the discussion, and
prompted me to wonder if this was maybe a topic that other projects have
been thinking about or have ideas about what might be good ways to deal
with test coverage.

I'll admit, it's probably not as big a deal as I may have initially
thought; but it doesn't hurt to kick the idea around on the ML a bit.

Thanks,
John
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160215/e5086dc5/attachment.html>