Open Stack

<snip>
> > Last, is a question, is it possible to currently run the full API an build a
> > json schema for it all? Or to query these validating schemas? We *really*
> > want that over in tempest, so we can completely drop the manual creation of
> > negative testing, and just fuzz known bad against the schema definitions.
> 
> Sorry, I'm not sure I understand this question correctly.
> We need to define schemas for each API with the separated schema patches.
> It is impossible to define one schema for all APIs.
> 
> 
> Hi  David, Marc,
> 
> I guess the negative test generator of Tempest would need each API definition.
> Glance can provide API definitions through API with jsonschema format, but
> Nova does not have such feature.
> We need to port these API schema from Nova to Tempest, I guess. right?
> 

As I understand things after reviewing the first versions of the negative test
generator patch is that we have to hard code the schema into a file (right now
it's in the test file, but eventually it'll be an external input file). One of
my issues with doing that is it's highly manual process, essentially a copy and
paste from the nova tree. I think what we're looking for from this jsonschema
validation work is an API which we can query the API and get the jsonschema
definitions; similar to what the glance API offers.

I wouldn't advocate making the negative test generator be fully dynamic for the
same reason we don't autodetect which API versions and extensions/features are
enabled in tempest, but rather rely on the config file. But, instead have an
additional tool which could query the schema for all the endpoints in nova and
generate an input file for the negative test generator. That way we'll still
catch breaking API changes in the gate, but it's not a manual process to update
the input file with the schema definitions in tempest when there is a breaking
API change. (Which hopefully should almost never happen)


-Matt Treinish