Open Stack

On Tue, Mar 18, 2014 at 12:34:57PM +0100, Koderer, Marc wrote:
> On Tue, 18 Marc 2014 12:00:00 +0100
> Christopher Yeoh [mailto:cbkyeoh at gmail.com] wrote:
> > On Tue, 18 Mar 2014 10:39:19 +0100
> > "Koderer, Marc" <m.koderer at telekom.de> wrote:
> > >
> > > I just recognized that we have very similar interface definitions in
> > > tempest/api_schema and etc/schema:
> > >
> > > https://github.com/openstack/tempest/tree/master/etc/schemas
> > > https://github.com/openstack/tempest/tree/master/tempest/api_schema
> > >
> > > Any objections if I move them to a single location? I'd prefer to use
> > > json as file format instead of .py. As final goal we should find a way
> > > how to merge them competently but I feel like this is something for
> > > the design summit ;)
> > >
> > 
> > Heh we just moved them but I'm open to other suggestions - they are are
> > specific to API testing though aren't they? Long term the idea is that
> > they should be generated by Nova rather than tempest.  I think to prevent
> > unintentional changes we'd probably cache a copy in tempest though rather
> > than dynamically query them.

The idea was never to dynamically query them; there should always be a copy in
the tempest tree. Like you said to prevent unintentional changes which is the
same reason we don't auto-discover api versions. The idea for querying nova to
get the schemas was to enable a tool which could populate the schemas
automatically so that we didn't have to manually generate them individually. I'd
say, to a certain extent, that this new round of validation patches could use
the same kind of tool.

> 
> Sry that I didn't recognized this review.
> Both definitions are coupled to API testing, yes.
> 
> > 
> > My feeling at the moment is that they should  .py files.
> > Because I think there's cases where we will want to have some schema
> > definitions based on others or share common parts and use bits of python
> > code to achieve this. For example availability zone list and detailed
> > listing  have a lot in common (detailed listing just has a more
> > parameters). I think there'll be similar cases for v2 and v3 versions as
> > well.  While we're still manually generating them and keeping them up to
> > date I think it's worth sharing as much as we can.
> 
> Ok understood. We just converted the negative testing
> definitions to json files due to review findings..

Well, when I left the review comment about it being a json file, I didn't think
about inheritance. Chris has a good point about reusing common bits and just
extending it. That wasn't how you proposed the negative test schemas would be
used which is why I suggested using a raw json file.

> It's just very confusing for new people if they see
> two separate folders with schema definitions.
> 
> But unfortunately there isn't an easy way.

Am I missing something or are these schemas being added now just a subset of
what is being used for negative testing? Why can't we either add the extra
negative test info around the new test validation patches and get the double
benefit. Or have the negative test schemas just extend these new schemas being
added?

> 
> > 
> > I agree there's a lot of commonality and we should long term just have one
> > schema definition. There's quite a bit to discuss (eg level of strictness
> > is currently different) in this area and a summit session about it would
> > be very useful.
> > 
> 
> +1
> 

I agree there is probably enough here to discuss during a summit session on
where schema validation fits into tempest. As a part of that discussing how to
store and manage schema definitions for both the negative test framework and
validation tests.


-Matt Treinish