[openstack-dev] What's up with functional testing?
mriedem at linux.vnet.ibm.com
Thu Oct 29 14:22:22 UTC 2015
On 10/27/2015 9:55 PM, Emilien Macchi wrote:
> As a user, I would like to functionally test OpenStack services.
> I'm using Tempest (which is AFIK  the official OpenStack project for
> functional testing) and am able to validate that Puppet OpenStack module
> actually deploy services & make them work together which is the goal of
> Puppet OpenStack Integration testing .
> Until now I was happy, until this bug  (TL;DR: Aodh can't be testing
> with Tempest which is a bug I'm working on, and not really related to
> this thread).
> I realized Aodh  (and apparently some other projects like Ceilometer)
> were using something else (gabbi ) for testing.
> How come some big tent projects do not use Tempest anymore for
> functional testing? I thought there was/is a move with tempest plugins
> that will help projects to host their tempest tests in their repos.
> Am I missing something? Any official decision taken?
> Is gabbi supported by OpenStack?
> I feel like there is currently 2 paths that try to do the same thing and
> as a user, I'm not happy.
> Please help me to understand,
> Thank you.
>  a user who deploy Puppet OpenStack modules in OpenStack Infra for CI
>  http://goo.gl/sgI2D8
>  https://github.com/openstack/puppet-openstack-integration#overview
>  https://bugs.launchpad.net/tempest/+bug/1509885
>  https://github.com/openstack/aodh
>  https://pypi.python.org/pypi/gabbi/
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
This might be helpful. I noticed recently that nova didn't have any API
functional testing for a wrinkle in one of it's APIs. I opened a nova
bug to add the functional testing (in nova's tree) since it only relies
on having a wsgi server for the API call, and I think a sqlite database
for stubbing out some resources.
A patch was proposed to Tempest  which I -1'ed since I thought we
could just contain that code in nova's tree since there wasn't any
cross-project interaction needed (nova functional tests don't run
against a live devstack so we don't have glance/cinder/neutron).
mtreinish elaborated on the reason for not having it in Tempest in the
"Yeah, for something like this I feel it's more appropriate inside nova
itself. It'll be far more efficient to test it there. This only really
needs to be a tempest test if you want it to be externally covered.
Since it can be more effectively tested inside of nova, and is self
contained in nova, the rule of thumb I use for a case like this is
basically: if this is something you want to enforce every cloud provider
to do correctly for all time. (in both directions, since deployments
aren't always running the latest) You also have to weigh that against
the cost of enforcing this on a change to every project's commits.
This is because the primary advantage tempest gives you over an in tree
functional test is external visibility. Tempest is used a ton of
different places to test real clouds, not just for gating. (things like
defcore, CD setups, etc) So if we can test it more effectively inside of
nova's functional tests the only major reason to add it to tempest would
to take advantage of that."
Hopefully that helps. I think that statement is something that we should
get into the Tempest docs, i.e. a section on 'when is a test appropriate
More information about the OpenStack-dev