Open Stack

On Wed, 28 Oct 2015, Emilien Macchi wrote:

> As a user[1], I would like to functionally test OpenStack services.

Bless you. Let's all be more like that.

> Until now I was happy, until this bug [4] (TL;DR: Aodh can't be testing
> with Tempest which is a bug I'm working on, and not really related to
> this thread).
> I realized Aodh [5] (and apparently some other projects like Ceilometer)
> were using something else (gabbi [6]) for testing.

There's a fair amount of history here for which I don't know all the
details so I'll just try to address the gabbi related questions below
after this initial comment:

aodh is not yet in tempest, but that's simply because it is not yet
done, not because there are not plans to do it. Tomorrow morning at
9am in the Kiri room we're having a design session about functional
and integration testing in all three of ceilometer, aodh and gnocchi
and one of the primary topics is: getting all three to have tempest
plugins. One of the other primary topics is making new and existing
in-tree functional test as good and useful as possible. A lot of those
tests are not tempest because they've been extracted from pre-existing
so-called unit tests but were determined to not be because they use a
database. A small segment are API tests driven by gabbi as a result of
this spec[1]

If you or anyone else have time to show up to that session, that would
be great.

> How come some big tent projects do not use Tempest anymore for
> functional testing? I thought there was/is a move with tempest plugins
> that will help projects to host their tempest tests in their repos.

Ceilometer projects started a migration to more in-tree functional
tests in advance of tempest-lib existing. These manifest as tests that
require a storage backend and have their own non-tempest job
description in project-config.

> Am I missing something? Any official decision taken?
> Is gabbi supported by OpenStack?

When I first released gabbi there was talk about it becoming either
part of QA or oslo but after showing it around the world a bit I had
feedback from several non-openstack people that they would be _far_
more likely to contribute to it if it was not subject to the openstack
development model[0]. So it lives now as a project on github to which
people submit pull requests and travis takes care of CI. To avoid bus
termination errors, there are two other admins in addition to me who
have all the same rights with regard to merging and releasing and
maintaining on python. One of them is another OpenStack contributor
(jasonamyers).

I'm obviously biased, but I think gabbi is teh ossum and makes writing
API tests easy and perhaps more importantly makes reading them later
fantastic.

Within gnocchi and aodh, gabbi has been so successful at making it easy
to write API tests that it is now being used for writing integration
tests of aodh+gnocchi+ceilometer+heat.

> I feel like there is currently 2 paths that try to do the same thing and
> as a user, I'm not happy.

Yeah, that's perhaps a problem. One thing I'm hoping to explore (or
hoping someone else will explore) is making it easy to do gabbi
tests within a tempest plugin. This ought to be possible but there
are some humps to deal with regard to how gabbi orders and groups
tests.

Again, I'm biased, but I think that gabbi would be a _huge_ asset for
API tests in tempest because by its very nature it is very close to
HTTP without a notion of a (python) client being involved. I think
this is an excellent guard for ensuring that OpenStack APIs are
sufficiently agnostic about their context.

> Please help me to understand,

I hope that adds a bit more info. I know it doesn't actually answer
the real question though. I hope some other voices will come along.

[0] I'm happy to discuss this elsewhere (in person, on another thread,
whatever) but it would be bad to let it distract this current thread.
[1] http://specs.openstack.org/openstack/ceilometer-specs/specs/kilo/declarative-http-tests.html

-- 
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent