Open Stack

On Feb 12, 2014, at 12:36 PM, Sean Dague <sean at dague.net> wrote:

> On 02/12/2014 01:48 PM, Maru Newby wrote:
>> At the last 2 summits, I've suggested that API tests could be maintained in the Neutron tree and reused by Tempest.  I've finally submitted some patches that demonstrate this concept:
>> 
>> https://review.openstack.org/#/c/72585/  (implements a unit test for the lifecycle of the network resource)
>> https://review.openstack.org/#/c/72588/  (runs the test with tempest rest clients)
>> 
>> My hope is to make API test maintenance a responsibility of the Neutron team.  The API compatibility of each Neutron plugin has to be validated by Neutron tests anyway, and if the tests are structured as I am proposing, Tempest can reuse those efforts rather than duplicating them.
>> 
>> I've added this topic to this week's agenda, and I would really appreciate it interested parties would take a look at the patches in question to prepare themselves to participate in the discussion.
>> 
>> 
>> m.
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> Realistically, having API tests duplicated in the Tempest tree is a
> feature, not a bug.
> 
> tempest/api is there for double book keep accounting, and it has been
> really effective at preventing accidental breakage of our APIs (which
> used to happen all the time), so I don't think putting API testing in
> neutron obviates that.

Given how limited our testing resources are, might it be worth considering whether 'double-entry accounting' is actually the best way to preventing accidental breakage going forward?  Might reasonable alternatives exist, such as clearly separating api tests from other tests in the neutron tree and giving review oversight only to qualified individuals?


> 
> Today most projects (excepting swift... which I'll get to in a second)
> think about testing in 2 ways. Unit tests driven by tox in a venv, and
> tempest tests in a devstack environment.
> 
> Because of this dualism, people have put increasingly more awkward live
> environments in the tox unit tests jobs. For instance, IIRC, neutron
> actually starts a full wsgi stack to tests every single in tree plugin,
> instead of just testing the driver call down path.

Yeah, and this full-stack approach means that neutron tests are incredibly hard to maintain and debug, to say nothing of the time penalty (approaching the duration of a tempest smoke run).  Part of the benefit of the proposal in question is to allow targeting the plugins directly with the same tests that will run against the full stack.


m.

> 
> Swift did something a little different. They have 3 classes of things.
> Unit tests, Tempest Tests, and Swift Functional tests. The Swift
> functional tests run in a devstack, but not with Tempest. Instead they
> run their own suite.
> 
> This test suite only runs on the swift project, not on other projects,
> and it's something they can use to test functional scenarios that
> wouldn't fit in tempest, and extend to their heart's content, not having
> to worry about the interaction with other components, because it's only
> pushing on swift.
> 
> Going down this third path with neutron testing is I think very
> valuable. Honestly, I'd like to encourage more projects to do this as
> well. It would give them a greater component assuredness before entering
> the integrated gate.
> 
> 	-Sean
> 
> -- 
> Sean Dague
> Samsung Research America
> sean at dague.net / sean.dague at samsung.com
> http://dague.net
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev