Open Stack

On 04/22/2014 07:33 PM, Adam Gandelman wrote:
> On Tue, Apr 22, 2014 at 4:23 AM, Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
> 
> 
>     Agreed. Though I think we probably want the Nova API to be explicit
>     about what parts of the API it's ok to throw a Not Supported. Because I
>     don't think it's a blanket ok. On API endpoints where this is ok, we can
>     convert not supported to a skip.
> 
>             -Sean
> 
> 
> I'd favor going even further and let any such exception convert to a
> skip, at least in the main test suite.  Keep Tempest a point-and-shoot
> suite that can be pointed at any cloud and "do the right thing."  We can
> add another test or utility (perhaps in tools/?) to interpret results
> and attach meaning to them WRT skips/fails validated against individual
> projects' current notion of what is mandatory.   A list of "these tests
> must pass against any driver" instead of a driver feature matrix.   This
> would allow such policies to easily change over time outside of the
> actual test code.

I think the policy about what's allowed to be not implemented or not
shouldn't be changing so quickly that it needs to be left to the
projects to decide after the fact.

It also really needs to be documented in our API. If a not implemented
is allowable for a particular API call, the client needs to know that's
valid response, in advance.

There are a lot of challenges with interpreting things after the fact,
especially if your path was cut short because things just started
skipping. And past experiences with auto skipping mostly meant that
project breaks got in (which is why we don't do that any more).

	-Sean

-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140422/5ed8a26e/attachment.pgp>