Open Stack

On Thu, Oct 17, 2013 at 10:34:13PM +0200, Salvatore Orlando wrote:
> Matt,
> 
> Thanks for the input; after reading Jay's story I was kind of sure that
> there was a precise reason for not using extension auto-discovery.
> I will then start adding configuration variarables to tests exercising
> Neutron extensions; and ensure in reviews that these variables are added.
> I think we should follow Jay's advice on naming. Something like the follow
> should work
> 
> neutron_<ext_alias> = {on|off}

You should probably make a separate group for all of these extenstions.
Something like what I'm doing in the commit I pointed out below.


> On 17 October 2013 17:34, Jay Pipes <jaypipes at gmail.com> wrote:
> 
> > On 10/17/2013 10:46 AM, Matthew Treinish wrote:
> >
> >> On Thu, Oct 17, 2013 at 10:10:15AM -0400, Jay Pipes wrote:
> >>
> >>> On 10/17/2013 08:46 AM, Salvatore Orlando wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> in the discussion for patch https://review.openstack.org/#**/c/50880/<https://review.openstack.org/#/c/50880/>Sean
> >>>> asked a very reasonable question:
> >>>>
> >>>> "so are all these [Neutron] extensions always loaded on all
> >>>> environments? If not, how are we detecting which are?"
> >>>>
> >>>> So far we've been relying on devstack-gate setup, I think. If the
> >>>> extension is enabled by devstack-gate, then you can add the test. This
> >>>> would make sense if tempest is used exclusively by the upstream gate
> >>>> jobs, which however appears not to be the case. It should therefore be
> >>>> possible to select which tests should be run according to currently
> >>>> enabled Neutron extensions, which depend on core and service plugins
> >>>>
> >>>> One way to solve this issue would be to use tempest configuration
> >>>> variables; while this is the simplest approach, there are already 25
> >>>> extensions in Neutron, not counting plugin-specific extensions. This
> >>>> number can only grow in future releases, so I have a few concerns about
> >>>> maintainability of this approach because of the high number of
> >>>> configuration variables.
> >>>>
> >>>> Another way is to use Neutron API to query available extensions, by
> >>>> querying /v2.0/extensions
> >>>>
> >>>
> >>> This is actually how we used to control for Nova extensions. I just
> >>> went to look at the latest code, though, and while I see the
> >>> tempest.services.compute.json.**extensions_client.**
> >>> ExtensionsClientJSON.is_**enabled()
> >>> method exists, I don't see it called any more :(
> >>>
> >>> https://github.com/openstack/**tempest/blob/master/tempest/**
> >>> services/compute/json/**extensions_client.py#L36<https://github.com/openstack/tempest/blob/master/tempest/services/compute/json/extensions_client.py#L36>
> >>>
> >>> Instead, it looks like a bunch of variables have been added to the
> >>> Tempest configuration file that are then checked to see if an
> >>> extension should be tested:
> >>>
> >>> https://github.com/openstack/**tempest/blob/master/tempest/**
> >>> api/compute/admin/test_**flavors_extra_specs.py#L39<https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_flavors_extra_specs.py#L39>
> >>>
> >>>  This will return the list of currently loaded extensions, from which
> >>>> extension aliases can be easily extracted; these values can then be used
> >>>> to conditionally skip tests. This Neutron query can either be done each
> >>>> time an API test class is initialised  or just once - by storing in
> >>>> memory the list of enabled extensions. If you deem the last approach
> >>>> reasonable, I'd be happy to provide some code for it. I see many patches
> >>>> pushed into tempest without an accompanying blueprint/bug; should I
> >>>> register anyway a blueprint for this work?
> >>>>
> >>>
> >>> I would fully support getting rid of tempest configuration file
> >>> variables that "enable" these tests and instead just use the
> >>> is_enabled() method that checks a memoized list of extensions pulled
> >>> from the /extensions resource to determine whether a test should be
> >>> run.
> >>>
> >>>
> >> We've actually been going the opposite direction in tempest on purpose.
> >> The
> >> concern with auto discovery is that if there is a bug we might end up
> >> masking
> >> it if something were to go wrong. With auto discovery we can't tell
> >> exactly
> >> what is supposed to be run, so if a test is skipped in the >1000 tests
> >> because
> >> something went wrong with auto-discovery it'd be very difficult to tell.
> >> By
> >> explicitly stating up front in the config what is enabled we know what is
> >> expected to run. Granted this does shift the burden to devstack (or
> >> whatever
> >> else you're using to configure tempest) to set this up properly.
> >>
> >> Jay, I originally agreed with you that we can auto-discover things nicely
> >> and
> >> trim down the config file. But, we had a couple cases where we found
> >> things not
> >> running as expected in the gate and when we fixed it the tests didn't
> >> work.
> >>
> >
> > Hi Matt, thanks for sharing the above history. I can see your point and
> > understand the rationale behind the change in direction.
> >
> > One thing that might be nice is to make the tempest configuration
> > variables that switch on/off these extension tests consistently named. It's
> > tough right now to scan the tempest config file and try to determine which
> > config options refer to extensions... some use "ENABLED", others use
> > "AVAILABLE", none of them mention extension or have "extension" or "ext"
> > prefixes, etc.
> >

Jay, I really liked this idea take a look at:
https://review.openstack.org/52524

It's my first rough(I haven't tested it locally) attempt to make this a bit more
obvious about which features/extensions are enabled in each project and making
the naming more consistent.


-Matt Treinish