Open Stack

On Wed, Jul 09, 2014 at 05:41:10AM -0400, Eoghan Glynn wrote:
> 
> TL;DR: branchless Tempest shouldn't impact on backporting policy, yet
>        makes it difficult to test new features not discoverable via APIs
> 
> Folks,
> 
> At the project/release status meeting yesterday[1], I raised the issue
> that featureful backports to stable are beginning to show up[2], purely
> to facilitate branchless Tempest. We had a useful exchange of views on
> IRC but ran out of time, so this thread is intended to capture and
> complete the discussion.

So, [2] is definitely not something that should be backported. But, doesn't it
mean that cinder snapshot notifications don't work at all in icehouse? Is this
reflected in the release notes or docs somewhere, because it seems like
something that would be expected to work, which, I think, is actually a bigger
bug being exposed by branchless tempest. As a user how do I know that with
the cloud I'm using whether cinder snapshot notifications are supported?

> 
> The issues, as I see it, are:
> 
>  * Tempest is expected to do double-duty as both the integration testing
>    harness for upstream CI and as a tool for externally probing capabilities
>    in public clouds
> 
>  * Tempest has an implicit bent towards pure API tests, yet not all
>    interactions between OpenStack services that we want to test are
>    mediated by APIs

I think this is the bigger issue. If there is cross-service communication it
should have an API contract. (and probably be directly tested too) It doesn't
necessarily have to be a REST API, although in most cases that's easier. This
is probably something for the TC to discuss/mandate, though.

> 
>  * We don't have another integration test harness other than Tempest
>    that we could use to host tests that don't just make assertions
>    about the correctness/presence of versioned APIs
> 
>  * We want to be able to add new features to Juno, or fix bugs of
>    omission, in ways that aren't necessarily discoverable in the API;
>    without backporting these patches to stable if we wouldn't have
>    done so under the normal stable-maint policy[3]
> 
>  * Integrated projects are required[4] to provide Tempest coverage,
>    so the rate of addition of tests to Tempest is unlikely to slow
>    down anytime soon
> 
> So the specific type of test that I have in mind would be common
> for Ceilometer, but also possibly for Ironic and others:
> 
>  1. an end-user initiates some action via an API
>     (e.g. calls the cinder snapshot API)
> 
>  2. this initiates some actions behind the scenes
>     (e.g. a volume is snapshot'd and a notification emitted)
> 
>  3. the test reasons over some expected side-effect
>     (e.g. some metering data shows up in ceilometer)
> 
> The branchless Tempest spec envisages new features will be added and
> need to be skipped when testing stable/previous, but IIUC requires
> that the presence of new behaviors is externally discoverable[5].

I think the test case you proposed is fine. I know some people will argue that
it is expanding the scope of tempest to include more whitebox like testing,
because the notification are an internal side-effect of the api call, but I
don't see it that way. It feels more like exactly what tempest is there to
enable testing, a cross-project interaction using the api.

I'm pretty sure that most of the concerns around tests like this were from the
gate maintenance and debug side of things. In other words when things go wrong
how impossible will it be to debug that a notification wasn't generated or not
counted? Right now I think it would be pretty difficult to debug a notification
test failure, which is where the problem is. While I think testing like this is
definitely valid, that doesn't mean we should rush in a bunch of sloppy tests
that are impossible to debug, because that'll just make everyone sad panda.

But, they're is also a slight misunderstanding here. Having a feature be
externally discoverable isn't a hard requirement for a config option in tempest,
it's just *strongly* recommended. Mostly, because if there isn't a way to
discover it how are end users expected to know what will work.

For this specific case I think it's definitely fair to have an option for which
notifications services are expected to be generated. That's something that is
definitely a configurable option when setting up a deployment, and is something
that feels like a valid tempest config option, so we know which tests will work.
We already have similar feature flags for config time options in the services,
and having options like that would also get you out of that backport mess you
have right now. However, it does raise the question of being an end user how am
I expected to know which notifications get counted? Which is why having the
feature discoverability is generally a really good idea.

> 
> One approach mooted for allowing these kind of scenarios to be tested
> was to split off the pure-API aspects of Tempest so that it can be used
> for probing public-cloud-capabilities as well as upstream CI, and then
> build project-specific mini-Tempests to test integration with other
> projects.
> 
> Personally, I'm not a fan of that approach as it would require a lot
> of QA expertise in each project, lead to inefficient use of CI
> nodepool resources to run all the mini-Tempests, and probably lead to
> a divergent hotchpotch of per-project approaches.

I think the proposal here was for people interested in doing whitebox testing,
where there is a desire to test an internal project mechanism. I could see the
argument for testing notifications this way, but that would have to be for every
project individually. There are already several projects that have functional
testing like this in tree and run them as a gating job. There are definitely
certain classes of testing where doing this makes sense.

> 
> Another idea would be to keep all tests in Tempest, while also
> micro-versioning the services such that tests can be skipped on the
> basis of whether a particular feature-adding commit is present.
> 
> When this micro-versioning can't be discovered by the test (as in the
> public cloud capabilities probing case), those tests would be skipped
> anyway.

Yeah, I'm not a fan of this approach at all. It is just a bad way of
reimplementing a temporally-aware tempest. But, instead of using branches we
have arbitrary service versions. It sacrifices all the real advantages of having
branchless tempest, but adds more complexity around the version discovery to all
the projects and tempest. If we want to revert back to a temporally aware
tempest, which I don't think we should, then going back the branched model is
what we should do.

> 
> The final, less palatable, approach that occurs to me would be to
> revert to branchful Tempest.

So I don't think we're anywhere near this. I think what we're hitting here is
more a matter of projects trying to map out exactly how to test things for real
in the gate with tempest. While at the same time coming to understand that
things don't quite work as well as we expected. I think that we have to remember
that this is the first cycle with branchless tempest it's still new for everyone
and what we're hitting here are just some of the growing pains around it. Having
discussions like this and mapping out the requirements more completely is th

I recognize that for projects that didn't have any real testing before we
started branchless tempest it's harder to get things going with it. Especially
because in my experience the adage "if it isn't tested it's broken" tends to
hold true. So I expect there will be a lot of non-backportable fixes just to
enable testing. What this friction with branchless tempest is showing us is that
these fixes, besides fixing the bug, will also have implications for people
using OpenStack clouds. Which I feel is invaluable information to collect, and
definitely something we should gate on. The open question is how do we make it
easier the to enable testing for new things.

> 
> Any other ideas, or preferences among the options laid out above? 
> 
> Cheers,
> Eoghan
> 
> [1] http://eavesdrop.openstack.org/meetings/project/2014/project.2014-07-08-21.03.html
> [2] https://review.openstack.org/104863
> [3] https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes
> [4] https://github.com/openstack/governance/blob/master/reference/incubation-integration-requirements.rst#qa-1
> [5] https://github.com/openstack/qa-specs/blob/master/specs/implemented/branchless-tempest.rst#scenario-1-new-tests-for-new-features
> 

-Matt Treinish
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140709/e6a5ae1c/attachment.pgp>