Open Stack

> Von: David Kranz [mailto:dkranz at redhat.com]
> Gesendet: Montag, 13. Januar 2014 05:04
> An: openstack-dev at lists.openstack.org
> Betreff: Re: [openstack-dev] [nova] [qa] api schema validation pattern
> changes
> 
> On 01/12/2014 10:14 PM, Matthew Treinish wrote:
> > <snip>
> >>> Last, is a question, is it possible to currently run the full API
> an
> >>> build a json schema for it all? Or to query these validating
> >>> schemas? We *really* want that over in tempest, so we can
> completely
> >>> drop the manual creation of negative testing, and just fuzz known
> bad against the schema definitions.
> >> Sorry, I'm not sure I understand this question correctly.
> >> We need to define schemas for each API with the separated schema
> patches.
> >> It is impossible to define one schema for all APIs.
> >>
> >>
> >> Hi  David, Marc,
> >>
> >> I guess the negative test generator of Tempest would need each API
> definition.
> >> Glance can provide API definitions through API with jsonschema
> >> format, but Nova does not have such feature.
> >> We need to port these API schema from Nova to Tempest, I guess.
> right?
> >>
> > As I understand things after reviewing the first versions of the
> > negative test generator patch is that we have to hard code the schema
> > into a file (right now it's in the test file, but eventually it'll be
> > an external input file). One of my issues with doing that is it's
> > highly manual process, essentially a copy and paste from the nova
> > tree. I think what we're looking for from this jsonschema validation
> > work is an API which we can query the API and get the jsonschema
> definitions; similar to what the glance API offers.
> I looked at the glance schema api and it seemed to just return the
> schema for the  json returned by the call, not for the arguments to the
> request. Am I wrong about that?
> Additionally, the schema for the request json in these patches is not
> enough for the negative test generator. The schema for negative
> generation also needs the http method type and a description of  the
> resources that are part of the url of the request. The negative test
> patch in the shared fork
> https://github.com/mkoderer/tempest/tree/feature/negative_tests
> defines such a schema. The tempest patch will be updated from this fork
> on Monday.
> >
> > I wouldn't advocate making the negative test generator be fully
> > dynamic for the same reason we don't autodetect which API versions
> and
> > extensions/features are enabled in tempest, but rather rely on the
> > config file. But, instead have an additional tool which could query
> > the schema for all the endpoints in nova and generate an input file
> > for the negative test generator. That way we'll still catch breaking
> > API changes in the gate, but it's not a manual process to update the
> > input file with the schema definitions in tempest when there is a
> > breaking API change. (Which hopefully should almost never happen)
> I agree, it would be ideal if there were a way for tempest to grab the
> appropriate schemas from somewhere else rather than hard-coding them in
> tempest itself. But as far as I can see,  most of the services don't
> even have json schemas defined.

I think at the end the hard coded json schema should be optional.
If the endpoint supports an export of the schema it simply takes it and
put the result automatically into the description of the negative test.
For me this is a second step since currently we already "duplicating"
somehow this in our manual negative tests. What we would need is
at least one interface that exposes the schema.

 Marc