Open Stack

On 05/25/2016 02:54 PM, Andrew Laski wrote:
> 
> 
> On Wed, May 25, 2016, at 11:13 AM, Chris Dent wrote:
>>
>> Earlier this year I worked with jaypipes to compose a spec[1] for using
>> gabbi[2] with nova. Summit rolled around and there were some legitimate
>> concerns about the focus of the spec being geared towards replacing the
>> api sample tests. I wasn't at summit ☹ but my understanding of the
>> outcome of the discussion was (please correct me if I'm wrong):
>>
>> * gabbi is not a straight replacement for the api-samples (notably
>>    it doesn't address the documentation functionality provided by
>>    api-samples)
>>
>> * there are concerns, because of the style of response validation
>>    that gabbi does, that there could be a coverage gap[3] when a
>>    representation changes (in, for example, a microversion bump)
>>
>> * we'll see how things go with the placement API work[4], which uses
>>    gabbi for TDD, and allow people to learn more about gabbi from
>>    that
>>
>> Since that all seems to make sense, I've gone ahead and abandoned
>> the review associated with the spec as overreaching for the time
>> being.
>>
>> I'd like, however, to replace it with a spec that is somewhat less
>> reaching in its plans. Rather than replace api-samples with gabbi,
>> augment existing tests of the API with gabbi-based tests. I think
>> this is a useful endeavor that will find and fix inconsistencies but
>> I'd like to get some feedback from people so I can formulate a spec
>> that will actually be useful.
>>
>> For reference, I started working on some integration of tempest and
>> gabbi[5] (based on some work that Mehdi did), and in the first few
>> minutes of writing tests found and reported bugs against nova and
>> glance, some of which have even been fixed since then. Win! We like
>> win.
>>
>> The difficulty here, and the reason I'm writing this message, is
>> simply this: The biggest benefit of gabbi is the actual writing and
>> initial (not the repeated) running of the tests. You write tests, you
>> find bugs and inconsistencies. The second biggest benefit is going
>> back and being a human and reading the tests and being able to see
>> what the API is doing, request and response in the same place. That's
>> harder to write a spec about than "I want to add or change feature X".
>> There's no feature here.
> 
> After reading this my first thought is that gabbi would handle what I'm
> testing in
> https://review.openstack.org/#/c/263927/33/nova/tests/functional/wsgi/test_servers.py,
> or any of the other tests in that directory. Does that seem accurate?
> And what would the advantage of gabbi be versus what I have currently
> written?

It would.

I still would rather not put gabbi into the compute API testing this
cycle. Instead learn from the placement side, let people see good
patterns there, and not confuse contributors with multiple ways to test
things in the compute API. Because that requires a lot of digging out
from later (example: mox & mock).

And we still have this whole api-ref site which is only 50% verified
(and we still need to address a number of microversion issues) -
http://burndown.dague.org/. We said at the beginning of the cycle
api-ref and policy in code were our 2 API priorities. Until those are
well in the bag I don't want to take the energy and care to make sure we
do a pivot on test strategy to something completely new in a way that is
easy for everyone to contribute to and review.

I feel like we have a good sandbox for this in the placement API, and we
can evaluate at end of cycle for next steps.

	-Sean

-- 
Sean Dague
http://dague.net