Open Stack

> Hi everyone,
> 
> Before we start the larger discussion at summit next week about the future of
> testing in OpenStack - specifically about spinning up functional testing and
> how
> it relates to tempest - I would like to share some of my thoughts on how we
> can
> get things started and how I think they'll eventually come together.
> 
> Currently in tempest we have a large number of tests (mostly api-focused)
> which are probably a better fit for a project's functional test suite. The
> best
> example I can think of is the nova flavors tests. Validation of flavor
> manipulation doesn't need to run in the integrated test suite on every commit
> to
> every project because it only requires Nova. A simple win for initiating
> in-tree
> functional testing would be to move these kinds of tests into the projects
> and
> run the tests from the project repos instead of from tempest.
> 
> This would have the advantage of making tempest slimmer for every project
> and begin the process of getting projects to take responsibility for their
> functional testing rather than relying on tempest. As tests are moved tempest
> can start to become the integration test suite it was meant to be. It would
> retain only tests that involve multiple projects and stop being the OpenStack
> black box testing suite. I think that this is the right direction for tempest
> moving forward, especially as we move to having project-specific functional
> testing.
> 
> Doing this migration is dependent on some refactors in tempest and moving
> the required bits to tempest-lib so they can be easily consumed by the
> other projects. This will be discussed at summit, is being planned
> for implementation this cycle, and is similar to what is currently in
> progress
> for the cli tests.
> 
> The only reason this testing existed in tempest in the first place was as
> mechanism to block and then add friction against breaking api changes.
> Tempest's
> api testing has been been pretty successful at achieving these goals. We'll
> want
> to ensure that migrated tests retain these characteristics. If we are using
> clients from tempest-lib we should get this automatically since to break
> the api you'd have to change the api client. Another option proposed was to
> introduce a hacking rule that would block changes to api tests at the same
> time
> other code was being changed.
> 
> There is also a concern for external consumers of tempest if we move the
> tests
> out of the tempest tree (I'm thinking refstack). I think the solution is
> to maintain a load_tests discovery method inside of tempest or elsewhere that
> will run the appropriate tests from the other repos for something like
> refstack.
> Assuming that things are built in a compatible way using the same framework
> then
> running the tests from separate repos should be a simple matter of pointing
> the
> test runner in the right direction.
> 
> I also want to comment on the role of functional testing. What I've proposed
> here is only one piece of what project specific functional testing should be
> and just what I feel is a good/easy start. I don't feel that this should be
> the only testing done in the projects.  I'm suggesting this as a first
> step because the tests already exist and it should be a relatively simple
> task.
> I also feel that using tempest-lib like this shouldn't be a hard requirement.
> Ideally the client definitions shouldn't have to live externally, or if they
> did
> they would be the official clients, but I am suggesting this as a first step
> to
> start a migration out of tempest.
> 
> I don't want anyone to feel that they need block their functional testing
> efforts until tempest-lib becomes more useable. The larger value from
> functional
> testing is actually in enabling testing more tightly coupled to the projects
> (e.g. whitebox testing). I feel that any work necessary to enable functional
> testing should be prioritized.

Thanks Matt for getting the ball rolling on this conversation in advance
of summit.

Probably stating the obvious here, but I feel we should make a concious
effort to keep the approaches to in-tree functional testing as consistent
as possible across projects.

Towards that end, it would be good for folks with an interest in this area
to attend each other's sessions where possible:
  
 Cross-project: Tue, 12:05 [1]
 Heat:          Wed, 13:50 [2]
 Nova:          Wed, 16:30 [3]
 Ceilometer:    Wed, 17:20 [4]
 QA:            Wed, 17:20 [5]

Unfortunately there's a clash there between the QA tempest-lib session
and the ceilo session. I'm not sure how reasonable it would be to make
a last-minute schedule change to avoid that clash.

Cheers,
Eoghan

[1] http://kilodesignsummit.sched.org/event/575938e4837e8293615845582d7e3e7f
[2] http://kilodesignsummit.sched.org/event/eb261fb08b18ec1eaa2c67492e7fc385
[3] http://kilodesignsummit.sched.org/event/271a9075c1ced6c1269100ff4b8efc37
[4] http://kilodesignsummit.sched.org/event/daf63526a1883e84cec107c70cc6cad3
[5] http://kilodesignsummit.sched.org/event/a45e9af23c2c3fc17c12cc22e8bb3704