Open Stack

On 01/12/2014 10:14 PM, Matthew Treinish wrote:
> <snip>
>>> Last, is a question, is it possible to currently run the full API an build a
>>> json schema for it all? Or to query these validating schemas? We *really*
>>> want that over in tempest, so we can completely drop the manual creation of
>>> negative testing, and just fuzz known bad against the schema definitions.
>> Sorry, I'm not sure I understand this question correctly.
>> We need to define schemas for each API with the separated schema patches.
>> It is impossible to define one schema for all APIs.
>>
>>
>> Hi  David, Marc,
>>
>> I guess the negative test generator of Tempest would need each API definition.
>> Glance can provide API definitions through API with jsonschema format, but
>> Nova does not have such feature.
>> We need to port these API schema from Nova to Tempest, I guess. right?
>>
> As I understand things after reviewing the first versions of the negative test
> generator patch is that we have to hard code the schema into a file (right now
> it's in the test file, but eventually it'll be an external input file). One of
> my issues with doing that is it's highly manual process, essentially a copy and
> paste from the nova tree. I think what we're looking for from this jsonschema
> validation work is an API which we can query the API and get the jsonschema
> definitions; similar to what the glance API offers.
I looked at the glance schema api and it seemed to just return the 
schema for the  json returned by the call, not for the arguments to the 
request. Am I wrong about that?
Additionally, the schema for the request json in these patches is not 
enough for the negative test generator. The schema for negative 
generation also
needs the http method type and a description of  the resources that are 
part of the url of the request. The negative test patch in the shared 
fork https://github.com/mkoderer/tempest/tree/feature/negative_tests 
defines such a schema. The tempest patch will be updated from this fork 
on Monday.
>
> I wouldn't advocate making the negative test generator be fully dynamic for the
> same reason we don't autodetect which API versions and extensions/features are
> enabled in tempest, but rather rely on the config file. But, instead have an
> additional tool which could query the schema for all the endpoints in nova and
> generate an input file for the negative test generator. That way we'll still
> catch breaking API changes in the gate, but it's not a manual process to update
> the input file with the schema definitions in tempest when there is a breaking
> API change. (Which hopefully should almost never happen)
I agree, it would be ideal if there were a way for tempest to grab the 
appropriate schemas from somewhere else rather than hard-coding them in 
tempest itself. But as far as I can see,  most of the services don't 
even have json schemas defined.

  -Daivd