Open Stack

On Thu, Oct 17, 2013 at 10:10:15AM -0400, Jay Pipes wrote:
> On 10/17/2013 08:46 AM, Salvatore Orlando wrote:
> >Hi,
> >
> >in the discussion for patch https://review.openstack.org/#/c/50880/ Sean
> >asked a very reasonable question:
> >
> >"so are all these [Neutron] extensions always loaded on all
> >environments? If not, how are we detecting which are?"
> >
> >So far we've been relying on devstack-gate setup, I think. If the
> >extension is enabled by devstack-gate, then you can add the test. This
> >would make sense if tempest is used exclusively by the upstream gate
> >jobs, which however appears not to be the case. It should therefore be
> >possible to select which tests should be run according to currently
> >enabled Neutron extensions, which depend on core and service plugins
> >
> >One way to solve this issue would be to use tempest configuration
> >variables; while this is the simplest approach, there are already 25
> >extensions in Neutron, not counting plugin-specific extensions. This
> >number can only grow in future releases, so I have a few concerns about
> >maintainability of this approach because of the high number of
> >configuration variables.
> >
> >Another way is to use Neutron API to query available extensions, by
> >querying /v2.0/extensions
> 
> This is actually how we used to control for Nova extensions. I just
> went to look at the latest code, though, and while I see the tempest.services.compute.json.extensions_client.ExtensionsClientJSON.is_enabled()
> method exists, I don't see it called any more :(
> 
> https://github.com/openstack/tempest/blob/master/tempest/services/compute/json/extensions_client.py#L36
> 
> Instead, it looks like a bunch of variables have been added to the
> Tempest configuration file that are then checked to see if an
> extension should be tested:
> 
> https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_flavors_extra_specs.py#L39
> 
> >This will return the list of currently loaded extensions, from which
> >extension aliases can be easily extracted; these values can then be used
> >to conditionally skip tests. This Neutron query can either be done each
> >time an API test class is initialised  or just once - by storing in
> >memory the list of enabled extensions. If you deem the last approach
> >reasonable, I'd be happy to provide some code for it. I see many patches
> >pushed into tempest without an accompanying blueprint/bug; should I
> >register anyway a blueprint for this work?
> 
> I would fully support getting rid of tempest configuration file
> variables that "enable" these tests and instead just use the
> is_enabled() method that checks a memoized list of extensions pulled
> from the /extensions resource to determine whether a test should be
> run.
> 

We've actually been going the opposite direction in tempest on purpose. The
concern with auto discovery is that if there is a bug we might end up masking
it if something were to go wrong. With auto discovery we can't tell exactly
what is supposed to be run, so if a test is skipped in the >1000 tests because
something went wrong with auto-discovery it'd be very difficult to tell. By
explicitly stating up front in the config what is enabled we know what is
expected to run. Granted this does shift the burden to devstack (or whatever
else you're using to configure tempest) to set this up properly.

Jay, I originally agreed with you that we can auto-discover things nicely and
trim down the config file. But, we had a couple cases where we found things not
running as expected in the gate and when we fixed it the tests didn't work.


-Matt Treinish