Open Stack

On Jul 2, 2013, at 11:35 AM, Daryl Walleck <daryl.walleck at RACKSPACE.COM> wrote:

> I don't see a link between testing and being able to generate a client from documents. The time to create/maintain the clients for me at least has been negligible in comparison to the time spent writing tests and interfaces with other critical parts of the OpenStack infrastructure I want to test. I do think an architectural discussion of the clients in worthwhile (for what it's worth, the json/xml client structure was based on the code I originally submitted for Tempest and even I hate the concept now), but at some point we need to move forward and talk about the lower level details of OpenStack that we're not testing and developing interfaces for that.

The point is to remove mechanical effort, not just be able to generate clients.  The majority of api tests could be generated too - including serialization-specific validation and fuzz testing - if sufficiently detailed api specs were available.  The api clients themselves could be generated from those same specs.  And then, we'd have more cycles to test the truly interesting stuff.


m.
  

> Daryl
> ________________________________________
> From: Maru Newby [marun at redhat.com]
> Sent: Monday, July 01, 2013 10:55 AM
> To: All Things QA.
> Subject: Re: [openstack-qa] Refactoring Tempest service clients
> 
> On Jul 1, 2013, at 11:45 AM, Daryl Walleck <daryl.walleck at RACKSPACE.COM> wrote:
> 
>> Based on my previous experiences, I have to advocate the opposite. My quick bullet point list about language bindings:
>>      • Are too helpful (re-tries, re-auth)
>>      • Hide specifics about the response from the tester
>>      • Stop the tester from manipulating a request in meaningful ways
>>      • Block the tester from injecting useful logging/timing of requests etc
>> For simple, positive testing I can see the argument that language bindings provide a quick and dirty way to get started, but the downsides are too great for anything beyond basic smoke API testing.
> 
> If we are talking about doing this _properly_, it should be possible to generate bindings (both official and for testing) automatically from a machine-readable spec.  For all the negatives of something like WSDL, the alternative is surely worse.
> 
> 
> m.
> 
> 
>> Daryl
>> ________________________________________
>> From: Monty Taylor [mordred at inaugust.com]
>> Sent: Monday, July 01, 2013 9:15 AM
>> To: openstack-qa at lists.openstack.org
>> Subject: Re: [openstack-qa] Refactoring Tempest service clients
>> 
>> 
>> I think that if you do this, you will wind up implementing a parallel
>> set of client libraries.
>> 
>> I'm going to take this moment to advocate for just using the client
>> libraries in tempest instead of direct HTTP, btw. 2 reasons - one - what
>> Martina brings up - it's hard to write things with the direct HTTP. Two
>> - we have integration testing for real at this point, whereas we did not
>> have it when tempest was first written/designed. So the fact that the
>> client libs are part of the gate should mitigate that.
>> 
>> But - as always- only my $0.02.
>> 
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> 
> 
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
> 
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa