Open Stack

On 10/23/2014 06:27 AM, Chris Dent wrote:
>
> I've proposed a spec to Ceilometer
>
>    https://review.openstack.org/#/c/129669/
>
> for a suite of declarative HTTP tests that would be runnable both in
> gate check jobs and in local dev environments.
>
> There's been some discussion that this may be generally applicable
> and could be best served by a generic tool. My original assertion
> was "let's make something work and then see if people like it" but I
> thought I also better check with the larger world:
>
> * Is this a good idea?
I think so
>
> * Do other projects have similar ideas in progress?
Tempest faced a similar problem around negative tests in particular. We 
have code in tempest that automatically generates a series of negative
test cases based on illegal variations of a schema. If you want to look 
at it the NegativeAutoTest class is probably a good place to start. We have
discussed using a similar methodology for positive test cases but never 
did anything with that.

Currently only a few of the previous negative tests have been replaced 
with auto-gen tests. In addition to the issue of how to represent the 
schema, the other major issue we encountered was the need to create 
resources used by the auto-generated tests and a way to integrate a 
resource description into the schema. We use json for the schema and 
hoped one day to be able to receive base schemas from the projects 
themselves.
>
> * Is this concept something for which a generic tool should be
>   created _prior_ to implementation in an individual project?
>
> * Is there prior art? What's a good format?
Marc Koderer and I did a lot of searching and asking folks if there was 
some python code that we could use as a starting point but in the end 
did not find anything. I do not have a list of what we considered and 
rejected.

  -David
>
> Thanks.
>