Open Stack

On 10/27/2015 9:55 PM, Emilien Macchi wrote:
> As a user[1], I would like to functionally test OpenStack services.
>
> I'm using Tempest (which is AFIK [2] the official OpenStack project for
> functional testing) and am able to validate that Puppet OpenStack module
> actually deploy services & make them work together which is the goal of
> Puppet OpenStack Integration testing [3].
>
> Until now I was happy, until this bug [4] (TL;DR: Aodh can't be testing
> with Tempest which is a bug I'm working on, and not really related to
> this thread).
> I realized Aodh [5] (and apparently some other projects like Ceilometer)
> were using something else (gabbi [6]) for testing.
>
> How come some big tent projects do not use Tempest anymore for
> functional testing? I thought there was/is a move with tempest plugins
> that will help projects to host their tempest tests in their repos.
>
> Am I missing something? Any official decision taken?
> Is gabbi supported by OpenStack?
>
> I feel like there is currently 2 paths that try to do the same thing and
> as a user, I'm not happy.
>
> Please help me to understand,
> Thank you.
>
> [1] a user who deploy Puppet OpenStack modules in OpenStack Infra for CI
> purposes
> [2] http://goo.gl/sgI2D8
>      http://goo.gl/DTR1cL
> [3] https://github.com/openstack/puppet-openstack-integration#overview
> [4] https://bugs.launchpad.net/tempest/+bug/1509885
> [5] https://github.com/openstack/aodh
> [6] https://pypi.python.org/pypi/gabbi/
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

This might be helpful. I noticed recently that nova didn't have any API 
functional testing for a wrinkle in one of it's APIs. I opened a nova 
bug to add the functional testing (in nova's tree) since it only relies 
on having a wsgi server for the API call, and I think a sqlite database 
for stubbing out some resources.

A patch was proposed to Tempest [1] which I -1'ed since I thought we 
could just contain that code in nova's tree since there wasn't any 
cross-project interaction needed (nova functional tests don't run 
against a live devstack so we don't have glance/cinder/neutron).

mtreinish elaborated on the reason for not having it in Tempest in the 
review:

"Yeah, for something like this I feel it's more appropriate inside nova 
itself. It'll be far more efficient to test it there. This only really 
needs to be a tempest test if you want it to be externally covered.

Since it can be more effectively tested inside of nova, and is self 
contained in nova, the rule of thumb I use for a case like this is 
basically: if this is something you want to enforce every cloud provider 
to do correctly for all time. (in both directions, since deployments 
aren't always running the latest) You also have to weigh that against 
the cost of enforcing this on a change to every project's commits.

This is because the primary advantage tempest gives you over an in tree 
functional test is external visibility. Tempest is used a ton of 
different places to test real clouds, not just for gating. (things like 
defcore, CD setups, etc) So if we can test it more effectively inside of 
nova's functional tests the only major reason to add it to tempest would 
to take advantage of that."

Hopefully that helps. I think that statement is something that we should 
get into the Tempest docs, i.e. a section on 'when is a test appropriate 
for tempest'.

[1] https://review.openstack.org/#/c/233808/

-- 

Thanks,

Matt Riedemann