Open Stack

On Wed, Jul 09, 2014 at 09:16:01AM -0400, Sean Dague wrote:
> I think we need to actually step back a little and figure out where we
> are, how we got here, and what the future of validation might need to
> look like in OpenStack. Because I think there has been some
> communication gaps. (Also, for people I've had vigorous conversations
> about this before, realize my positions have changed somewhat,
> especially on separation of concerns.)
> 
> (Also note, this is all mental stream right now, so I will not pretend
> that it's an entirely coherent view of the world, my hope in getting
> things down is we can come up with that coherent view of the wold together.)
> 
> == Basic History ==
> 
> In the essex time frame Tempest was 70 tests. It was basically a barely
> adequate sniff test for integration for OpenStack. So much so that our
> first 3rd Party CI system, SmokeStack, used it's own test suite, which
> legitimately found completely different bugs than Tempest. Not
> surprising, Tempest was a really small number of integration tests.
> 
> As we got to Grizzly Tempest had grown to 1300 tests, somewhat
> organically. People were throwing a mix of tests into the fold, some
> using Tempest's client, some using official clients, some trying to hit
> the database doing white box testing. It had become kind of a mess and a
> rorshack test. We had some really weird design summit sessions because
> many people had only looked at a piece of Tempest, and assumed the rest
> was like it.
> 
> So we spent some time defining scope. Tempest couldn't really be
> everything to everyone. It would be a few things:
>  * API testing for public APIs with a contract
>  * Some throughput integration scenarios to test some common flows
> (these were expected to be small in number)
>  * 3rd Party API testing (because it had existed previously)
> 
> But importantly, Tempest isn't a generic function test suite. Focus is
> important, because Tempests mission always was highly aligned with what
> eventually became called Defcore. Some way to validate some
> compatibility between clouds. Be that clouds built from upstream (is the
> cloud of 5 patches ago compatible with the cloud right now), clouds from
> different vendors, public clouds vs. private clouds, etc.
> 
> == The Current Validation Environment ==
> 
> Today most OpenStack projects have 2 levels of validation. Unit tests &
> Tempest. That's sort of like saying your house has a basement and a
> roof. For sufficiently small values of house, this is fine. I don't
> think our house is sufficiently small any more.
> 
> This has caused things like Neutron's unit tests, which actually bring
> up a full wsgi functional stack and test plugins through http calls
> through the entire wsgi stack, replicated 17 times. It's the reason that
> Neutron unit tests takes many GB of memory to run, and often run longer
> than Tempest runs. (Maru has been doing hero's work to fix much of this.)
> 
> In the last year we made it *really* easy to get a devstack node of your
> own, configured any way you want, to do any project level validation you
> like. Swift uses it to drive their own functional testing. Neutron is
> working on heading down this path.
> 
> == New Challenges with New Projects ==
> 
> When we started down this path all projects had user APIs. So all
> projects were something we could think about from a tenant usage
> environment. Looking at both Ironic and Ceilometer, we really have
> projects that are Admin API only.
> 
> == Contracts or lack thereof ==
> 
> I think this is where we start to overlap with Eoghan's thread most.
> Because branchless Tempest assumes that the test in Tempest are governed
> by a stable contract. The behavior should only change based on API
> version, not on day of the week. In the case that triggered this what
> was really being tested was not an API, but the existence of a meter
> that only showed up in Juno.
> 
> Ceilometer is also another great instance of something that's often in a
> state of huge amounts of stack tracing because it depends on some
> internals interface in a project which isn't a contract. Or notification
> formats, which aren't (largely) versioned.
> 
> Ironic has a Nova driver in their tree, which implements the Nova driver
> internals interface. Which means they depend on something that's not a
> contract. It gets broken a lot.
> 
> == Depth of reach of a test suite ==
> 
> Tempest can only reach so far into a stack given that it's levers are
> basically public API calls. That's ok. But it means that things like
> testing a bunch of different dbs in the gate (i.e. the postgresql job)
> are pretty ineffectual. Trying to exercise code 4 levels deep through
> API calls is like driving a rover on Mars. You can do it, but only very
> carefully.
> 
> == Replication ==
> 
> Because there is such a huge gap between unit tests, and Tempest tests,
> replication of issues is often challenging. We have the ability to see
> races in the gate due to volume of results, that don't show up for
> developers very easily. When you do 30k runs a week, a ton of data falls
> out of it.
> 
> A good instance is the live snapshot bug. It was failing on about 3% of
> Tempest runs, which means that it had about a 10% chance of killing a
> patch on it's own. So it's definitely real. It's real enough that if we
> enable that path, there are a ton of extra rechecks required by people.
> However it's at a frequency that reproducing on demand is hard. And
> reproducing with enough signal to make it debuggable is also hard.
> 
> == The Fail Pit ==
> 
> All of which has somewhat led us to the fail pit. Where keeping
> OpenStack in a state that it can actually pass Tempest consistently is a
> full time job. It's actually more than a full time job, it's a full time
> program. If it was it's own program it would probably be larger than 1/2
> the official programs in OpenStack.
> 
> Also, when the Gate "program" is understaffed, it means that all the
> rest of the OpenStack programs (possibly excepting infra and tripleo
> because they aren't in the integrated gate) are slowed down
> dramatically. That velocity loss has real community and people power
> implications.
> 
> This is especially true of people trying to get time, review, mentoring,
> otherwise, out of the QA team. As there is kind of a natural overlap
> with folks that actually want us to be able to merge code, so while the
> Gate is under water, getting help on Tempest issues isn't going to
> happen in any really responsive rate.
> 
> Also, all the folks that have been the work horses here, myself, joe
> gordon, matt treinish, matt riedemann, are pretty burnt out on this.
> Every time we seem to nail one issue, 3 more crop up. Having no ending
> in sight and spending all your time shoveling out other project bugs is
> not a happy place to be.
> 
> == New Thinking about our validation layers ==
> 
> I feel like an ideal world would be the following:
> 
> 1. all projects have unit tests for their own internal testing, and
> these pass 100% of the time (note, most projects have races in their
> unit tests, and they don't pass 100% of the time. And they are low
> priority to fix).
> 2. all projects have a functional devstack job with tests *in their own
> tree* that pokes their project in interesting ways. This is akin to what
> neutron is trying and what swift is doing. These are *not* cogating.

So I'm not sure that this should be a mandatory thing, but an opt-in. My real
concern is the manpower, who is going to take the time to write all the test
suites for all of the projects. I think it would be better to add that on-demand
as the extra testing is required. That being said, I definitely view doing this
as a good thing and something to be encouraged, because tempest won't be able to
test everything. 

The other thing to also consider is duplicated effort between projects. For an
example, look at the CLI tests in Tempest, the functional testing framework for
testing CLI formatting was essentially the same between all the clients which is
why they're in tempest. Under your proposal here, CLI tests should be moved back
to the clients. But, would that mean we have a bunch of copy and pasted versions
of the CLI test framework between all the project. 

I really want to avoid a situation where every project does the same basic
testing differently just in a rush to spin up functional testing. I think coming
up with a solution for a place with common test patterns and frameworks that can
be maintained independently of all the projects and consumed for project
specific testing is something we should figure out first. (I'm not sure oslo
would be the right place for this necessarily)


> 3. all non public API contracts are shored up by landing contract tests
> in projects. We did this recently with Ironic in Nova -
> https://github.com/openstack/nova/blob/master/nova/tests/virt/test_ironic_api_contracts.py.

So I think that the contract unit tests work well specifically for the ironic
use case, but isn't a general solution. Mostly because the Nova driver api is an
unstable interface and there is no reason for that to change. It's also a
temporary thing because eventually the driver will be moved into Nova and then
the only cross-project interaction between Ironic and Nova will be over the
stable REST APIs.

I think in general we should try to avoid doing non REST API cross-project
communication. So hopefully there won't be more of these class of things, and
if there are we can tackle them on a per case basis. But, even if it's a non
REST API I don't think we should ever encourage or really allow any
cross-project interactions over unstable interfaces.

As a solution for notifications I'd rather see a separate notification
white/grey (or any other monochrome shade) box test suite. If as a project we
say that notifications have to be versioned for any change we can then enforce
that easily with an external test suite that contains the definitions for all
the notification. It then just makes a bunch of api calls and sits on RPC
verifying the notification format. (or something of that ilk)

I agree that normally whitebox testing needs to be tightly coupled with the data
models in the projects, but I feel like notifications are slightly different.
Mostly, because the basic format is the same between all the projects to make
consumption simpler. So instead of duplicating the work to validate the
notifications in all the projects it would be better to just implement it once.
I also think tempest being an external audit on the API has been invaluable so
enforcing that for notifications would have similar benefits.

As an aside I think it would probably be fair if this was maintained as part of
ceilometer or the telemetry program, since that's really all notifications are
used for. (or least as AIUI) But, it would still be a co-gating test suite for
anything that emits notifications. 

> 
> 4. all public API contracts are tested in Tempest (these are co-gating,
> and ensure a contract breakage in keystone doesn't break swift).
> 
> Out of these 4 levels, we currently have 2 (1 and 4). In some projects
> we're making #1 cover 1 & 2. And we're making #4 cover 4, 3, and
> sometimes 2. And the problem with this is it's actually pretty wasteful,
> and when things fail, they fail so far away from the test, that the
> reproduce is hard.

I think the only real issue in your proposal is that the boundaries between all
the test classifications aren't as well defined as they seem. I agree that
having more intermediate classes of testing is definitely a good thing to do.
Especially, since there is a great deal of hand waving on the details between
what is being run in between tempest and unit tests. But, the issue as I see it
is without guidelines on what type of tests belong where we'll end up with a
bunch duplicated work. 

It's the same problem we have all the time in tempest, where we get a lot of
patches that exceed the scope of tempest, despite it being arguably clearly
outlined in the developer docs. But, the complexity is higher in this situation,
because of having a bunch of different types of test suites that are available
to add a new test to. I just think before we adopt #2 as mandatory it's
important to have a better definition on the scope of the project specific
functional testing.

> 
> I actually think that if we went down this path we could actually make
> Tempest smaller. For instance, negative API testing is something I'd say
> is really #2. While these tests don't take a ton of time, they do add a
> certain amount of complexity. It might also mean that admin tests, whose
> side effects are hard to understand sometimes without white/greybox
> interactions might migrated into #2.

I think that negative testing is still part of tempest in your proposal. I still
feel that the negative space of an API still is part of the contract, and should
be externally validated. As part of tempest I think we need to revisit the
negative space solution again, because I haven't seen much growth on the
automatic test generation. We also can probably be way more targeted about what
we're running, but I don't think punting on negative testing in tempest is
something we should do.

I actually think that testing on the admin api is doubly important because of
the inadvertent side effects that they can cause. I think attempting to map that
out is useful. (I don't think we can assume that the admin api is being used in
a vacuum) I agree that in your proposal that tests for those weird interactions
might be more fitting for #2. (to avoid more heisenbugs in tempest, etc) But,
I'm on the fence about that. Mostly because I still think an admin api should
conform to the api guidelines and thus needs tempest tests for that. I know
you've expressed the opposite opinion about stability on the admin apis. But, I
fail to see the distinction between an admin api and any other api when it comes
to the stable api guarantees.

For a real world example look at the default-quotas api which was probably the
most recent example of this. (and I suspect why you mentioned it here :) ) The
reason the test was added was because it was previously removed from nova, while
horizon depended on it, which is exactly the kind of thing we should be using
tempest for. (even under your proposal since it's a co-gating rest api issue)
What's better about this example is that the test added had all the harms you
outlined about a weird cross interactions from this extension and the other
tests. I think when we weigh the complexity against the benefits for testing
admin-apis in tempest there isn't a compelling reason to pull them out of
tempest. But, as an alternative should start attempting to get clever about
scheduling tests to avoid some strange cross interactions.

> 
> I also think that #3 would help expose much more surgically what the
> cross project pain points are instead of proxy efforts through Tempest
> for these subtle issues. Because Tempest is probably a terrible tool to
> discover that notifications in nova changed. The results is some weird
> failure in a ceilometer test which says some instance didn't run when it
> was expected, then you have to dig through 5 different openstack logs to
> figure out that it was really a deep exception somewhere. If it was
> logged, which it often isn't. (I actually challenge anyone to figure out
> the reason for a ceilometer failure from a Tempest test based on it's
> current logging. :) )

I agree that we should be directly testing the cross-project integration points
which aren't REST APIs. I just don't think that we should decrease the level of
api testing in tempest for something that consumes that integration point. I
just feel if we ignore the top level too much we're going to expose more api
bugs. What I think the real path forward is validating both separately.
Hopefully, that'll let us catch bugs at each level independently.

> 
> And by ensuring specific functionality earlier in the stack, and letting
> Nova beat up Nova the way they think they should in a functional test
> (or land a Neutron functional test to ensure that it's doing the right
> thing), would make the Tempests runs which were cogating, a ton more
> predictable.
> 
> == Back to Branchless Tempest ==
> 
> I think the real issues that projects are running into with Branchless
> Tempest is they are coming forward with tests not in class #4, which
> fail, because while the same API existed 4 months ago as today, the
> semantics of the project have changed in a non discoverable way. Which
> I'd say was bad, however until we tried the radical idea of running the
> API test suite against all releases that declared they had the same API,
> we didn't see it. :)
> 
> 
> Ok, that was a lot. Hopefully it was vaguely coherent. I want to preface
> that I don't consider this all fully formed, but it's a lot of what's
> been rattling around in my brain.
> 

So here are some of my initial thoughts, I still need to stew some more on some
of the details, so certain things may be more of a knee-jerk reaction and I
might still be missing certain intricacies. Also a part of my response here is
just me playing devil's advocate. I definitely think more testing is always
better. I just want to make sure we're targeting the right things, because this
proposal is pushing for a lot extra work for everyone. I want to make sure that
before we commit to something this large that it's the right direction.

-Matt Treinish


> 
> On 07/09/2014 05:41 AM, Eoghan Glynn wrote:
> > 
> > TL;DR: branchless Tempest shouldn't impact on backporting policy, yet
> >        makes it difficult to test new features not discoverable via APIs
> > 
> > Folks,
> > 
> > At the project/release status meeting yesterday[1], I raised the issue
> > that featureful backports to stable are beginning to show up[2], purely
> > to facilitate branchless Tempest. We had a useful exchange of views on
> > IRC but ran out of time, so this thread is intended to capture and
> > complete the discussion.
> > 
> > The issues, as I see it, are:
> > 
> >  * Tempest is expected to do double-duty as both the integration testing
> >    harness for upstream CI and as a tool for externally probing capabilities
> >    in public clouds
> > 
> >  * Tempest has an implicit bent towards pure API tests, yet not all
> >    interactions between OpenStack services that we want to test are
> >    mediated by APIs
> > 
> >  * We don't have another integration test harness other than Tempest
> >    that we could use to host tests that don't just make assertions
> >    about the correctness/presence of versioned APIs
> > 
> >  * We want to be able to add new features to Juno, or fix bugs of
> >    omission, in ways that aren't necessarily discoverable in the API;
> >    without backporting these patches to stable if we wouldn't have
> >    done so under the normal stable-maint policy[3]
> > 
> >  * Integrated projects are required[4] to provide Tempest coverage,
> >    so the rate of addition of tests to Tempest is unlikely to slow
> >    down anytime soon
> > 
> > So the specific type of test that I have in mind would be common
> > for Ceilometer, but also possibly for Ironic and others:
> > 
> >  1. an end-user initiates some action via an API
> >     (e.g. calls the cinder snapshot API)
> > 
> >  2. this initiates some actions behind the scenes
> >     (e.g. a volume is snapshot'd and a notification emitted)
> > 
> >  3. the test reasons over some expected side-effect
> >     (e.g. some metering data shows up in ceilometer)
> > 
> > The branchless Tempest spec envisages new features will be added and
> > need to be skipped when testing stable/previous, but IIUC requires
> > that the presence of new behaviors is externally discoverable[5].
> > 
> > One approach mooted for allowing these kind of scenarios to be tested
> > was to split off the pure-API aspects of Tempest so that it can be used
> > for probing public-cloud-capabilities as well as upstream CI, and then
> > build project-specific mini-Tempests to test integration with other
> > projects.
> > 
> > Personally, I'm not a fan of that approach as it would require a lot
> > of QA expertise in each project, lead to inefficient use of CI
> > nodepool resources to run all the mini-Tempests, and probably lead to
> > a divergent hotchpotch of per-project approaches.
> > 
> > Another idea would be to keep all tests in Tempest, while also
> > micro-versioning the services such that tests can be skipped on the
> > basis of whether a particular feature-adding commit is present.
> > 
> > When this micro-versioning can't be discovered by the test (as in the
> > public cloud capabilities probing case), those tests would be skipped
> > anyway.
> > 
> > The final, less palatable, approach that occurs to me would be to
> > revert to branchful Tempest.
> > 
> > Any other ideas, or preferences among the options laid out above? 
> > 
> > Cheers,
> > Eoghan
> > 
> > [1] http://eavesdrop.openstack.org/meetings/project/2014/project.2014-07-08-21.03.html
> > [2] https://review.openstack.org/104863
> > [3] https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes
> > [4] https://github.com/openstack/governance/blob/master/reference/incubation-integration-requirements.rst#qa-1
> > [5] https://github.com/openstack/qa-specs/blob/master/specs/implemented/branchless-tempest.rst#scenario-1-new-tests-for-new-features
> > 
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > 
> 
> 
> -- 
> Sean Dague
> http://dague.net
> 



> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140710/8c7e1541/attachment.pgp>