Open Stack

Hi David,

2014/1/13 David Kranz <dkranz at redhat.com>:
> On 01/12/2014 10:14 PM, Matthew Treinish wrote:
>>
>> <snip>
>>>>
>>>> Last, is a question, is it possible to currently run the full API an
>>>> build a
>>>> json schema for it all? Or to query these validating schemas? We
>>>> *really*
>>>> want that over in tempest, so we can completely drop the manual creation
>>>> of
>>>> negative testing, and just fuzz known bad against the schema
>>>> definitions.
>>>
>>> Sorry, I'm not sure I understand this question correctly.
>>> We need to define schemas for each API with the separated schema patches.
>>> It is impossible to define one schema for all APIs.
>>>
>>>
>>> Hi  David, Marc,
>>>
>>> I guess the negative test generator of Tempest would need each API
>>> definition.
>>> Glance can provide API definitions through API with jsonschema format,
>>> but
>>> Nova does not have such feature.
>>> We need to port these API schema from Nova to Tempest, I guess. right?
>>>
>> As I understand things after reviewing the first versions of the negative
>> test
>> generator patch is that we have to hard code the schema into a file (right
>> now
>> it's in the test file, but eventually it'll be an external input file).
>> One of
>> my issues with doing that is it's highly manual process, essentially a
>> copy and
>> paste from the nova tree. I think what we're looking for from this
>> jsonschema
>> validation work is an API which we can query the API and get the
>> jsonschema
>> definitions; similar to what the glance API offers.
>
> I looked at the glance schema api and it seemed to just return the schema
> for the  json returned by the call, not for the arguments to the request. Am
> I wrong about that?

I think you are right. Glance feature provides the schemas of request body only.
That does not contain the http method type and the endpoint.

> Additionally, the schema for the request json in these patches is not enough
> for the negative test generator. The schema for negative generation also
> needs the http method type and a description of  the resources that are part
> of the url of the request.

Thanks for your explanation, I see. Current Nova's validation feature does not
seem enough for the negative test generator. It would be necessary to hard-code
API schemas in Tempest.


Thanks
Ken'ichi Ohmichi

---

> The negative test patch in the shared fork
> https://github.com/mkoderer/tempest/tree/feature/negative_tests defines such
> a schema. The tempest patch will be updated from this fork on Monday.
>
>>
>> I wouldn't advocate making the negative test generator be fully dynamic
>> for the
>> same reason we don't autodetect which API versions and extensions/features
>> are
>> enabled in tempest, but rather rely on the config file. But, instead have
>> an
>> additional tool which could query the schema for all the endpoints in nova
>> and
>> generate an input file for the negative test generator. That way we'll
>> still
>> catch breaking API changes in the gate, but it's not a manual process to
>> update
>> the input file with the schema definitions in tempest when there is a
>> breaking
>> API change. (Which hopefully should almost never happen)
>
> I agree, it would be ideal if there were a way for tempest to grab the
> appropriate schemas from somewhere else rather than hard-coding them in
> tempest itself. But as far as I can see,  most of the services don't even
> have json schemas defined.