Open Stack

On 02/12/2014 04:25 PM, Maru Newby wrote:
> 
> On Feb 12, 2014, at 12:36 PM, Sean Dague <sean at dague.net> wrote:
> 
>> On 02/12/2014 01:48 PM, Maru Newby wrote:
>>> At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest.  I've finally submitted some patches that demonstrate this concept:
>>>
>>> https://review.openstack.org/#/c/72585/  (implements a unit test for the lifecycle of the network resource)
>>> https://review.openstack.org/#/c/72588/  (runs the test with tempest rest clients)
>>>
>>> My hope is to make API test maintenance a responsibility of the Neutron team.  The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them.
>>>
>>> I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion.
>>>
>>>
>>> m.
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> Realistically, having API tests duplicated in the Tempest tree is a
>> feature, not a bug.
>>
>> tempest/api is there for double book keep accounting, and it has been
>> really effective at preventing accidental breakage of our APIs (which
>> used to happen all the time), so I don't think putting API testing in
>> neutron obviates that.
> 
> Given how limited our testing resources are, might it be worth considering whether 'double-entry accounting' is actually the best way to preventing accidental breakage going forward?  Might reasonable alternatives exist, such as clearly separating api tests from other tests in the neutron tree and giving review oversight only to qualified individuals?

Our direct experience is that if we don't do this, within 2 weeks some
project will have landed API breaking changes. This approach actually
takes a lot of review load off the core reviewers, so reverting to a
model which puts more work back on the review team (given the current
review load), isn't something I think we want.

I get that there is a cost with this. But there is a cost of all of
this. And because API tests should be write once for the API (and
specifically not evolving), the upfront cost vs. the continually paid
cost of review time in tree has been the better trade off.

	-Sean

-- 
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 547 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140212/4777410f/attachment.pgp>