Open Stack

So, I pushed out https://review.openstack.org/#/c/37320/ yesterday afternoon
which switches from using cinder endpoint discovery to skip the volume api tests
to using a new config option, 'cinder_available' to indicate whether cinder is
enabled or not. This spawned a discussion in the review regarding the right 
approach to determine which service tests are ok to run. So I thought I'd bring
the discussion out to the mailing list to reach a wider audience.

So from what I could see in the review is that there were 3 competing approaches
to doing this in Tempest brought up in the review: automatic endpoint discovery, 
config options, and adding service tags to the tests. My opinion on this is that
we should do both service tagging and also have service config options. 

The service tags are useful to have in general since it gives us more flexibility
on which subset of tests to run. Which is useful for running the tests manually
but isn't a replacement for configuration. It'll also enable us to specify
specific projects to be tested for gating purposes if we ever need to split up
things that way. I'll submit a blueprint to document this work at some point
today. But, I feel that this is independent of the discussion of automatic
skipping vs a configuration option.

As for config vs endpoint generation the issue I see here is gating. If we use
endpoint detection in the gate we can silently cover up issues where we intended
tests to be run. For example, because of a misconfiguration in devstack or
devstack-gate we could end up skipping tests that we intended to gate on because
a service wasn't started. This has happened in the past with neutron (well, it
was quantum then) a misconfiguration switched all of the quantum runs to use
nova-network which skipped the network tests because the endpoint wasn't found.
So all the quantum runs were passing because they weren't actually running
tests. This doesn't happen with using a config option because whether or not
the service is expected to be running is explicitly defined and the tests will
fail if something is enabled but isn't actually running.


Thanks,

-Matt Treinish