Open Stack

On 07/17/2013 11:37 AM, Matthew Treinish wrote:
> So, I pushed out https://review.openstack.org/#/c/37320/ yesterday afternoon
> which switches from using cinder endpoint discovery to skip the volume api tests
> to using a new config option, 'cinder_available' to indicate whether cinder is
> enabled or not. This spawned a discussion in the review regarding the right
> approach to determine which service tests are ok to run. So I thought I'd bring
> the discussion out to the mailing list to reach a wider audience.
>
> So from what I could see in the review is that there were 3 competing approaches
> to doing this in Tempest brought up in the review: automatic endpoint discovery,
> config options, and adding service tags to the tests. My opinion on this is that
> we should do both service tagging and also have service config options.
>
> The service tags are useful to have in general since it gives us more flexibility
> on which subset of tests to run. Which is useful for running the tests manually
> but isn't a replacement for configuration. It'll also enable us to specify
> specific projects to be tested for gating purposes if we ever need to split up
> things that way. I'll submit a blueprint to document this work at some point
> today. But, I feel that this is independent of the discussion of automatic
> skipping vs a configuration option.
>
> As for config vs endpoint generation the issue I see here is gating. If we use
> endpoint detection in the gate we can silently cover up issues where we intended
> tests to be run. For example, because of a misconfiguration in devstack or
> devstack-gate we could end up skipping tests that we intended to gate on because
> a service wasn't started. This has happened in the past with neutron (well, it
> was quantum then) a misconfiguration switched all of the quantum runs to use
> nova-network which skipped the network tests because the endpoint wasn't found.
> So all the quantum runs were passing because they weren't actually running
> tests. This doesn't happen with using a config option because whether or not
> the service is expected to be running is explicitly defined and the tests will
> fail if something is enabled but isn't actually running.
>
>
> Thanks,
>
> -Matt Treinish
>
> _
I agree with all of this. Do we actually have any tests for endpoint 
discovery, other than the implicit tests for keystone.

  -David