Open Stack

TL;DR: branchless Tempest shouldn't impact on backporting policy, yet
       makes it difficult to test new features not discoverable via APIs

Folks,

At the project/release status meeting yesterday[1], I raised the issue
that featureful backports to stable are beginning to show up[2], purely
to facilitate branchless Tempest. We had a useful exchange of views on
IRC but ran out of time, so this thread is intended to capture and
complete the discussion.

The issues, as I see it, are:

 * Tempest is expected to do double-duty as both the integration testing
   harness for upstream CI and as a tool for externally probing capabilities
   in public clouds

 * Tempest has an implicit bent towards pure API tests, yet not all
   interactions between OpenStack services that we want to test are
   mediated by APIs

 * We don't have another integration test harness other than Tempest
   that we could use to host tests that don't just make assertions
   about the correctness/presence of versioned APIs

 * We want to be able to add new features to Juno, or fix bugs of
   omission, in ways that aren't necessarily discoverable in the API;
   without backporting these patches to stable if we wouldn't have
   done so under the normal stable-maint policy[3]

 * Integrated projects are required[4] to provide Tempest coverage,
   so the rate of addition of tests to Tempest is unlikely to slow
   down anytime soon

So the specific type of test that I have in mind would be common
for Ceilometer, but also possibly for Ironic and others:

 1. an end-user initiates some action via an API
    (e.g. calls the cinder snapshot API)

 2. this initiates some actions behind the scenes
    (e.g. a volume is snapshot'd and a notification emitted)

 3. the test reasons over some expected side-effect
    (e.g. some metering data shows up in ceilometer)

The branchless Tempest spec envisages new features will be added and
need to be skipped when testing stable/previous, but IIUC requires
that the presence of new behaviors is externally discoverable[5].

One approach mooted for allowing these kind of scenarios to be tested
was to split off the pure-API aspects of Tempest so that it can be used
for probing public-cloud-capabilities as well as upstream CI, and then
build project-specific mini-Tempests to test integration with other
projects.

Personally, I'm not a fan of that approach as it would require a lot
of QA expertise in each project, lead to inefficient use of CI
nodepool resources to run all the mini-Tempests, and probably lead to
a divergent hotchpotch of per-project approaches.

Another idea would be to keep all tests in Tempest, while also
micro-versioning the services such that tests can be skipped on the
basis of whether a particular feature-adding commit is present.

When this micro-versioning can't be discovered by the test (as in the
public cloud capabilities probing case), those tests would be skipped
anyway.

The final, less palatable, approach that occurs to me would be to
revert to branchful Tempest.

Any other ideas, or preferences among the options laid out above? 

Cheers,
Eoghan

[1] http://eavesdrop.openstack.org/meetings/project/2014/project.2014-07-08-21.03.html
[2] https://review.openstack.org/104863
[3] https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes
[4] https://github.com/openstack/governance/blob/master/reference/incubation-integration-requirements.rst#qa-1
[5] https://github.com/openstack/qa-specs/blob/master/specs/implemented/branchless-tempest.rst#scenario-1-new-tests-for-new-features