Open Stack

On 10/17/2013 10:46 AM, Matthew Treinish wrote:
> On Thu, Oct 17, 2013 at 10:10:15AM -0400, Jay Pipes wrote:
>> On 10/17/2013 08:46 AM, Salvatore Orlando wrote:
>>> Hi,
>>>
>>> in the discussion for patch https://review.openstack.org/#/c/50880/ Sean
>>> asked a very reasonable question:
>>>
>>> "so are all these [Neutron] extensions always loaded on all
>>> environments? If not, how are we detecting which are?"
>>>
>>> So far we've been relying on devstack-gate setup, I think. If the
>>> extension is enabled by devstack-gate, then you can add the test. This
>>> would make sense if tempest is used exclusively by the upstream gate
>>> jobs, which however appears not to be the case. It should therefore be
>>> possible to select which tests should be run according to currently
>>> enabled Neutron extensions, which depend on core and service plugins
>>>
>>> One way to solve this issue would be to use tempest configuration
>>> variables; while this is the simplest approach, there are already 25
>>> extensions in Neutron, not counting plugin-specific extensions. This
>>> number can only grow in future releases, so I have a few concerns about
>>> maintainability of this approach because of the high number of
>>> configuration variables.
>>>
>>> Another way is to use Neutron API to query available extensions, by
>>> querying /v2.0/extensions
>>
>> This is actually how we used to control for Nova extensions. I just
>> went to look at the latest code, though, and while I see the tempest.services.compute.json.extensions_client.ExtensionsClientJSON.is_enabled()
>> method exists, I don't see it called any more :(
>>
>> https://github.com/openstack/tempest/blob/master/tempest/services/compute/json/extensions_client.py#L36
>>
>> Instead, it looks like a bunch of variables have been added to the
>> Tempest configuration file that are then checked to see if an
>> extension should be tested:
>>
>> https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_flavors_extra_specs.py#L39
>>
>>> This will return the list of currently loaded extensions, from which
>>> extension aliases can be easily extracted; these values can then be used
>>> to conditionally skip tests. This Neutron query can either be done each
>>> time an API test class is initialised  or just once - by storing in
>>> memory the list of enabled extensions. If you deem the last approach
>>> reasonable, I'd be happy to provide some code for it. I see many patches
>>> pushed into tempest without an accompanying blueprint/bug; should I
>>> register anyway a blueprint for this work?
>>
>> I would fully support getting rid of tempest configuration file
>> variables that "enable" these tests and instead just use the
>> is_enabled() method that checks a memoized list of extensions pulled
>> from the /extensions resource to determine whether a test should be
>> run.
>>
>
> We've actually been going the opposite direction in tempest on purpose. The
> concern with auto discovery is that if there is a bug we might end up masking
> it if something were to go wrong. With auto discovery we can't tell exactly
> what is supposed to be run, so if a test is skipped in the >1000 tests because
> something went wrong with auto-discovery it'd be very difficult to tell. By
> explicitly stating up front in the config what is enabled we know what is
> expected to run. Granted this does shift the burden to devstack (or whatever
> else you're using to configure tempest) to set this up properly.
>
> Jay, I originally agreed with you that we can auto-discover things nicely and
> trim down the config file. But, we had a couple cases where we found things not
> running as expected in the gate and when we fixed it the tests didn't work.

Hi Matt, thanks for sharing the above history. I can see your point and 
understand the rationale behind the change in direction.

One thing that might be nice is to make the tempest configuration 
variables that switch on/off these extension tests consistently named. 
It's tough right now to scan the tempest config file and try to 
determine which config options refer to extensions... some use 
"ENABLED", others use "AVAILABLE", none of them mention extension or 
have "extension" or "ext" prefixes, etc.

Best,
-jay