Open Stack

Matt,

Thanks for the input; after reading Jay's story I was kind of sure that
there was a precise reason for not using extension auto-discovery.
I will then start adding configuration variarables to tests exercising
Neutron extensions; and ensure in reviews that these variables are added.
I think we should follow Jay's advice on naming. Something like the follow
should work

neutron_<ext_alias> = {on|off}

Thanks,
Salvatore


On 17 October 2013 17:34, Jay Pipes <jaypipes at gmail.com> wrote:

> On 10/17/2013 10:46 AM, Matthew Treinish wrote:
>
>> On Thu, Oct 17, 2013 at 10:10:15AM -0400, Jay Pipes wrote:
>>
>>> On 10/17/2013 08:46 AM, Salvatore Orlando wrote:
>>>
>>>> Hi,
>>>>
>>>> in the discussion for patch https://review.openstack.org/#**/c/50880/<https://review.openstack.org/#/c/50880/>Sean
>>>> asked a very reasonable question:
>>>>
>>>> "so are all these [Neutron] extensions always loaded on all
>>>> environments? If not, how are we detecting which are?"
>>>>
>>>> So far we've been relying on devstack-gate setup, I think. If the
>>>> extension is enabled by devstack-gate, then you can add the test. This
>>>> would make sense if tempest is used exclusively by the upstream gate
>>>> jobs, which however appears not to be the case. It should therefore be
>>>> possible to select which tests should be run according to currently
>>>> enabled Neutron extensions, which depend on core and service plugins
>>>>
>>>> One way to solve this issue would be to use tempest configuration
>>>> variables; while this is the simplest approach, there are already 25
>>>> extensions in Neutron, not counting plugin-specific extensions. This
>>>> number can only grow in future releases, so I have a few concerns about
>>>> maintainability of this approach because of the high number of
>>>> configuration variables.
>>>>
>>>> Another way is to use Neutron API to query available extensions, by
>>>> querying /v2.0/extensions
>>>>
>>>
>>> This is actually how we used to control for Nova extensions. I just
>>> went to look at the latest code, though, and while I see the
>>> tempest.services.compute.json.**extensions_client.**
>>> ExtensionsClientJSON.is_**enabled()
>>> method exists, I don't see it called any more :(
>>>
>>> https://github.com/openstack/**tempest/blob/master/tempest/**
>>> services/compute/json/**extensions_client.py#L36<https://github.com/openstack/tempest/blob/master/tempest/services/compute/json/extensions_client.py#L36>
>>>
>>> Instead, it looks like a bunch of variables have been added to the
>>> Tempest configuration file that are then checked to see if an
>>> extension should be tested:
>>>
>>> https://github.com/openstack/**tempest/blob/master/tempest/**
>>> api/compute/admin/test_**flavors_extra_specs.py#L39<https://github.com/openstack/tempest/blob/master/tempest/api/compute/admin/test_flavors_extra_specs.py#L39>
>>>
>>>  This will return the list of currently loaded extensions, from which
>>>> extension aliases can be easily extracted; these values can then be used
>>>> to conditionally skip tests. This Neutron query can either be done each
>>>> time an API test class is initialised  or just once - by storing in
>>>> memory the list of enabled extensions. If you deem the last approach
>>>> reasonable, I'd be happy to provide some code for it. I see many patches
>>>> pushed into tempest without an accompanying blueprint/bug; should I
>>>> register anyway a blueprint for this work?
>>>>
>>>
>>> I would fully support getting rid of tempest configuration file
>>> variables that "enable" these tests and instead just use the
>>> is_enabled() method that checks a memoized list of extensions pulled
>>> from the /extensions resource to determine whether a test should be
>>> run.
>>>
>>>
>> We've actually been going the opposite direction in tempest on purpose.
>> The
>> concern with auto discovery is that if there is a bug we might end up
>> masking
>> it if something were to go wrong. With auto discovery we can't tell
>> exactly
>> what is supposed to be run, so if a test is skipped in the >1000 tests
>> because
>> something went wrong with auto-discovery it'd be very difficult to tell.
>> By
>> explicitly stating up front in the config what is enabled we know what is
>> expected to run. Granted this does shift the burden to devstack (or
>> whatever
>> else you're using to configure tempest) to set this up properly.
>>
>> Jay, I originally agreed with you that we can auto-discover things nicely
>> and
>> trim down the config file. But, we had a couple cases where we found
>> things not
>> running as expected in the gate and when we fixed it the tests didn't
>> work.
>>
>
> Hi Matt, thanks for sharing the above history. I can see your point and
> understand the rationale behind the change in direction.
>
> One thing that might be nice is to make the tempest configuration
> variables that switch on/off these extension tests consistently named. It's
> tough right now to scan the tempest config file and try to determine which
> config options refer to extensions... some use "ENABLED", others use
> "AVAILABLE", none of them mention extension or have "extension" or "ext"
> prefixes, etc.
>
> Best,
> -jay
>
>
>
> ______________________________**_________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.**org <OpenStack-dev at lists.openstack.org>
> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131017/7926d91f/attachment-0001.html>