[openstack-dev] [nova] [qa] api schema validation pattern changes
ken1ohmichi at gmail.com
Mon Jan 13 08:25:19 UTC 2014
2014/1/13 David Kranz <dkranz at redhat.com>:
> On 01/12/2014 10:14 PM, Matthew Treinish wrote:
>>>> Last, is a question, is it possible to currently run the full API an
>>>> build a
>>>> json schema for it all? Or to query these validating schemas? We
>>>> want that over in tempest, so we can completely drop the manual creation
>>>> negative testing, and just fuzz known bad against the schema
>>> Sorry, I'm not sure I understand this question correctly.
>>> We need to define schemas for each API with the separated schema patches.
>>> It is impossible to define one schema for all APIs.
>>> Hi David, Marc,
>>> I guess the negative test generator of Tempest would need each API
>>> Glance can provide API definitions through API with jsonschema format,
>>> Nova does not have such feature.
>>> We need to port these API schema from Nova to Tempest, I guess. right?
>> As I understand things after reviewing the first versions of the negative
>> generator patch is that we have to hard code the schema into a file (right
>> it's in the test file, but eventually it'll be an external input file).
>> One of
>> my issues with doing that is it's highly manual process, essentially a
>> copy and
>> paste from the nova tree. I think what we're looking for from this
>> validation work is an API which we can query the API and get the
>> definitions; similar to what the glance API offers.
> I looked at the glance schema api and it seemed to just return the schema
> for the json returned by the call, not for the arguments to the request. Am
> I wrong about that?
I think you are right. Glance feature provides the schemas of request body only.
That does not contain the http method type and the endpoint.
> Additionally, the schema for the request json in these patches is not enough
> for the negative test generator. The schema for negative generation also
> needs the http method type and a description of the resources that are part
> of the url of the request.
Thanks for your explanation, I see. Current Nova's validation feature does not
seem enough for the negative test generator. It would be necessary to hard-code
API schemas in Tempest.
> The negative test patch in the shared fork
> https://github.com/mkoderer/tempest/tree/feature/negative_tests defines such
> a schema. The tempest patch will be updated from this fork on Monday.
>> I wouldn't advocate making the negative test generator be fully dynamic
>> for the
>> same reason we don't autodetect which API versions and extensions/features
>> enabled in tempest, but rather rely on the config file. But, instead have
>> additional tool which could query the schema for all the endpoints in nova
>> generate an input file for the negative test generator. That way we'll
>> catch breaking API changes in the gate, but it's not a manual process to
>> the input file with the schema definitions in tempest when there is a
>> API change. (Which hopefully should almost never happen)
> I agree, it would be ideal if there were a way for tempest to grab the
> appropriate schemas from somewhere else rather than hard-coding them in
> tempest itself. But as far as I can see, most of the services don't even
> have json schemas defined.
More information about the OpenStack-dev