[openstack-dev] [qa] In-tree functional test vision

Joe Gordon joe.gordon0 at gmail.com
Mon Aug 25 20:44:27 UTC 2014

On Mon, Aug 4, 2014 at 3:29 AM, Chris Dent <chdent at redhat.com> wrote:

> In the "Thoughts on the patch test failure rate and moving forward"
> thread[1] there's discussion of moving some of the burden for
> functional testing to the individual projects. This seems like a good
> idea to me, but also seems like it could be a source of confusion so I
> thought I'd start another thread to focus on the details of just this
> topic, separate from the gate-oriented discussion in the other.
> In a couple of messages[2] Sean mentions "the vision". Is there a wiki
> page or spec or other kind of document where this nascent vision is
> starting to form? Even if we can't quite get started just yet, it
> would be good to have an opporunity to think about the constraints and
> goals that we'll be working with.

There is no single document on this that I know of. But two good places to
start are the functional testing systems we have for swift and neutron:



> Not just the goal of moving tests around, but what, for example, makes
> a good functional test?

Here is a non-exhaustive list of some differences between what belongs in
tempest and functional tests

* functional tests should ideally not need external services (external
services makes it closer to an integration test)
* functional tests can be whitebox tests
* a functional test environment should be easier to set up then devstack

> For constraints: Will tempest be available as a stable library? Is using
> tempest (or other same library across all projects) a good or bad thing?
> Seems there's some disagreement on both of these.

Yes, there is a separate thread on spinning out a tempest-lib (not sure on
what final name will be yet) that functional tests can use. Although I
think there is a lot  to be done before  needing the tempest-lib.

> Personally I'm quite eager to to vastly increase the amount of testing
> I can do on my own machine(s) before letting the gate touch my code.

Why can't you run devstack locally? Maybe there are some changes we can
make so its easier to run devstack locally first.

> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2014-
> July/thread.html#41057
> [2]
> http://lists.openstack.org/pipermail/openstack-dev/2014-July/041188.html
> http://lists.openstack.org/pipermail/openstack-dev/2014-July/041252.html
> --
> Chris Dent tw:@anticdent freenode:cdent
> https://tank.peermore.com/tanks/cdent
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140825/66a444ca/attachment.html>

More information about the OpenStack-dev mailing list