Open Stack

On 07/17/2013 12:50 PM, David Kranz wrote:
> On 07/17/2013 11:37 AM, Matthew Treinish wrote:
>> So, I pushed out https://review.openstack.org/#/c/37320/ yesterday
>> afternoon
>> which switches from using cinder endpoint discovery to skip the volume
>> api tests
>> to using a new config option, 'cinder_available' to indicate whether
>> cinder is
>> enabled or not. This spawned a discussion in the review regarding the
>> right
>> approach to determine which service tests are ok to run. So I thought
>> I'd bring
>> the discussion out to the mailing list to reach a wider audience.
>>
>> So from what I could see in the review is that there were 3 competing
>> approaches
>> to doing this in Tempest brought up in the review: automatic endpoint
>> discovery,
>> config options, and adding service tags to the tests. My opinion on
>> this is that
>> we should do both service tagging and also have service config options.
>>
>> The service tags are useful to have in general since it gives us more
>> flexibility
>> on which subset of tests to run. Which is useful for running the tests
>> manually
>> but isn't a replacement for configuration. It'll also enable us to
>> specify
>> specific projects to be tested for gating purposes if we ever need to
>> split up
>> things that way. I'll submit a blueprint to document this work at some
>> point
>> today. But, I feel that this is independent of the discussion of
>> automatic
>> skipping vs a configuration option.
>>
>> As for config vs endpoint generation the issue I see here is gating.
>> If we use
>> endpoint detection in the gate we can silently cover up issues where
>> we intended
>> tests to be run. For example, because of a misconfiguration in
>> devstack or
>> devstack-gate we could end up skipping tests that we intended to gate
>> on because
>> a service wasn't started. This has happened in the past with neutron
>> (well, it
>> was quantum then) a misconfiguration switched all of the quantum runs
>> to use
>> nova-network which skipped the network tests because the endpoint
>> wasn't found.
>> So all the quantum runs were passing because they weren't actually
>> running
>> tests. This doesn't happen with using a config option because whether
>> or not
>> the service is expected to be running is explicitly defined and the
>> tests will
>> fail if something is enabled but isn't actually running.
>>
>>
>> Thanks,
>>
>> -Matt Treinish
>>
>> _
> I agree with all of this. Do we actually have any tests for endpoint
> discovery, other than the implicit tests for keystone.

Actually, swift is currently doing endpoint detection instead of config 
at the moment. I only realized this this morning in dealing with a 
devstack proposed commit to not create the swift accounts to skip the 
tempest tests.

https://github.com/openstack/tempest/blob/master/tempest/api/object_storage/base.py#L47

Mostly being explicit is about protecting ourselves from fat fingering 
devstack-gate and thinking we are good. That happened with Postgresql 
not that long ago.

And I'm +1 on Matt's approach here.

	-Sean

-- 
Sean Dague
http://dague.net