Open Stack

On 06/09/14 04:10, Matthew Treinish wrote:
> On Fri, Sep 05, 2014 at 09:42:17AM +1200, Steve Baker wrote:
>> On 05/09/14 04:51, Matthew Treinish wrote:
>>> On Thu, Sep 04, 2014 at 04:32:53PM +0100, Steven Hardy wrote:
>>>> On Thu, Sep 04, 2014 at 10:45:59AM -0400, Jay Pipes wrote:
>>>>> On 08/29/2014 05:15 PM, Zane Bitter wrote:
>>>>>> On 29/08/14 14:27, Jay Pipes wrote:
>>>>>>> On 08/26/2014 10:14 AM, Zane Bitter wrote:
>>>>>>>> Steve Baker has started the process of moving Heat tests out of the
>>>>>>>> Tempest repository and into the Heat repository, and we're looking for
>>>>>>>> some guidance on how they should be packaged in a consistent way.
>>>>>>>> Apparently there are a few projects already packaging functional tests
>>>>>>>> in the package <projectname>.tests.functional (alongside
>>>>>>>> <projectname>.tests.unit for the unit tests).
>>>>>>>>
>>>>>>>> That strikes me as odd in our context, because while the unit tests run
>>>>>>>> against the code in the package in which they are embedded, the
>>>>>>>> functional tests run against some entirely different code - whatever
>>>>>>>> OpenStack cloud you give it the auth URL and credentials for. So these
>>>>>>>> tests run from the outside, just like their ancestors in Tempest do.
>>>>>>>>
>>>>>>>> There's all kinds of potential confusion here for users and packagers.
>>>>>>>> None of it is fatal and all of it can be worked around, but if we
>>>>>>>> refrain from doing the thing that makes zero conceptual sense then there
>>>>>>>> will be no problem to work around :)
>>>>>>>>
>>>>>>>> I suspect from reading the previous thread about "In-tree functional
>>>>>>>> test vision" that we may actually be dealing with three categories of
>>>>>>>> test here rather than two:
>>>>>>>>
>>>>>>>> * Unit tests that run against the package they are embedded in
>>>>>>>> * Functional tests that run against the package they are embedded in
>>>>>>>> * Integration tests that run against a specified cloud
>>>>>>>>
>>>>>>>> i.e. the tests we are now trying to add to Heat might be qualitatively
>>>>>>>> different from the <projectname>.tests.functional suites that already
>>>>>>>> exist in a few projects. Perhaps someone from Neutron and/or Swift can
>>>>>>>> confirm?
>>>>>>>>
>>>>>>>> I'd like to propose that tests of the third type get their own top-level
>>>>>>>> package with a name of the form <projectname>-integrationtests (second
>>>>>>>> choice: <projectname>-tempest on the principle that they're essentially
>>>>>>>> plugins for Tempest). How would people feel about standardising that
>>>>>>>> across OpenStack?
>>>>>>> By its nature, Heat is one of the only projects that would have
>>>>>>> integration tests of this nature. For Nova, there are some "functional"
>>>>>>> tests in nova/tests/integrated/ (yeah, badly named, I know) that are
>>>>>>> tests of the REST API endpoints and running service daemons (the things
>>>>>>> that are RPC endpoints), with a bunch of stuff faked out (like RPC
>>>>>>> comms, image services, authentication and the hypervisor layer itself).
>>>>>>> So, the "integrated" tests in Nova are really not testing integration
>>>>>>> with other projects, but rather integration of the subsystems and
>>>>>>> processes inside Nova.
>>>>>>>
>>>>>>> I'd support a policy that true integration tests -- tests that test the
>>>>>>> interaction between multiple real OpenStack service endpoints -- be left
>>>>>>> entirely to Tempest. Functional tests that test interaction between
>>>>>>> internal daemons and processes to a project should go into
>>>>>>> /$project/tests/functional/.
>>>>>>>
>>>>>>> For Heat, I believe tests that rely on faked-out other OpenStack
>>>>>>> services but stress the interaction between internal Heat
>>>>>>> daemons/processes should be in /heat/tests/functional/ and any tests the
>>>>>>> rely on working, real OpenStack service endpoints should be in Tempest.
>>>>>> Well, the problem with that is that last time I checked there was
>>>>>> exactly one Heat scenario test in Tempest because tempest-core doesn't
>>>>>> have the bandwidth to merge all (any?) of the other ones folks submitted.
>>>>>>
>>>>>> So we're moving them to openstack/heat for the pure practical reason
>>>>>> that it's the only way to get test coverage at all, rather than concerns
>>>>>> about overloading the gate or theories about the best venue for
>>>>>> cross-project integration testing.
>>>>> Hmm, speaking of passive aggressivity...
>>>>>
>>>>> Where can I see a discussion of the Heat integration tests with Tempest QA
>>>>> folks? If you give me some background on what efforts have been made already
>>>>> and what is remaining to be reviewed/merged/worked on, then I can try to get
>>>>> some resources dedicated to helping here.
>>>> We recieved some fairly strong criticism from sdague[1] earlier this year,
>>>> at which point we were  already actively working on improving test coverage
>>>> by writing new tests for tempest.
>>>>
>>>> Since then, several folks, myself included, commited very significant
>>>> amounts of additional effort to writing more tests for tempest, with some
>>>> success.
>>>>
>>>> Ultimately the review latency and overhead involved in constantly rebasing
>>>> changes between infrequent reviews has resulted in slow progress and
>>>> significant frustration for those attempting to contribute new test cases.
>>>>
>>>> It's been clear for a while that tempest-core have significant bandwidth
>>>> issues, as well as not necessarily always having the specific domain
>>>> expertise to thoroughly review some tests related to project-specific
>>>> behavior or functionality.
>>> So I view this as actually a breakdown in cross-team communication, with both
>>> sides at fault. For example, for a couple of months we had an outstanding
>>> meeting topic on heat testing which almost always no one brought up anything to
>>> discuss, eventually I just dropped it because it was never used. Instead I
>>> should have found someone to drive it forward. Or that the heat testing blueprint
>>> hasn't really seen much activity and only has 6 patches linked against it.
>> If I had been aware of the meeting topic I definitely would have taken
>> advantage of it.
>>> The QA team is also well aware of review latency issues, we have a few relief
>>> valves to try and help with it, like a meeting topic every week dedicated to
>>> reviews that need attention, and using review dashboards that prioritize reviews
>>> which need extra eyes. We also use the blueprints to track and prioritize
>>> reviews for efforts like bringing ramping up testing for a project. But if these
>>> aren't used it's hard to know that things aren't getting attention. Honestly, I
>>> think it's a major issue when the first I'd heard of this frustration about
>>> reviews on heat patches was when I happened to notice an abandoned patch that
>>> mentioned it.
>>>
>>> This case is actually why I'm planning on starting a QA liaison program soon so
>>> there is point of contact to push forward these things. Looking at neutron which
>>> had very little testing in havana and had ramped up the number of tests very
>>> quickly was having someone driving that effort and attending both meetings.
>>> Miguel Lavalle drove things forward by keeping on top of the patches in flight
>>> and letting people in both QA and Neutron know when something needed extra
>>> attention. I think the unspoken expectation from the QA team was that something
>>> like this was going to happen here. Hopefully, having a person formally take on
>>> this role in fostering communication between teams will be helpful in avoiding
>>> these issues in the future.
>>>
>> A QA liaison program sounds like a great idea.
>>
>>>> So it was with some relief that we saw the proposal[2] to move the burden
>>>> for reviewing project test-cases to the project teams, who will presumably
>>>> be more motivated to do the reviews, and have the knowledge of what needs
>>>> testing.
>>>>
>>>> [1] http://lists.openstack.org/pipermail/openstack-dev/2014-March/029661.html
>>>> [2] http://lists.openstack.org/pipermail/openstack-dev/2014-July/041057.html
>>>>
>>>>> I would greatly prefer just having a single source of integration testing in
>>>>> OpenStack, versus going back to the bad ol' days of everybody under the sun
>>>>> rewriting their own.
>>>>>
>>>>> Note that I'm not talking about functional testing here, just the
>>>>> integration testing...
>>>> You may have to define the terms functional and integration here, as IMO
>>>> there's already significant confusion about what the target of e.g API and
>>>> scenario tests in tempest are.
>>>>
>>>> This is also further complicated by the fact that all heat functional tests
>>>> also test integration of the various underlying services to some extent.
>>>>
>>>> My opinion is that any tests remaining in tempest should focus on API
>>>> correctness, e.g to keep us honest in terms of backwards incomaptible
>>>> changes to the API surface.
>>>>
>>>> Then for all tests which aim to prove the functionality of the project, e.g
>>>> my understanding of tempest scenario tests atm, we should allow project
>>>> teams to own them, and add to them as functionality develops over time.
>>> This is actually the opposite direction that things are pushing right now. The
>>> API tests are viewed as being mostly project specific, and besides for causing
>>> friction when attempting to make a breaking api change there isn't a reason to
>>> put them in an integrated test suite. While the scenario tests mostly involve
>>> cross-project interactions and would be outside the scope of project specific
>>> testing. Moving forward the expectation is that tempest's api tests will mostly
>>> move to the projects (once we have a solution to block breaking api changes) and
>>> the scenario tests will grow.
>>>
>> This sounds fine in the long term, but Heat needs a comprehensive
>> integration suite urgently, and developing them as tempest scenarios has
>> not delivered that yet. Tempest reviewer bandwidth has only been part of
>> the issue, not enough heat developers have been writing scenario tests
>> either. This has been a bit of a chicken-and-egg problem since we never
>> got to the point where there was enough existing scenario tests to -1
>> any new Heat feature that lacked one. Another issue is that it has taken
>> this long to get the devstack changes in which build a custom image
>> containing the required agents, which many of our tests will require.
>>
>> The existing scenario tests have been forklifted into
>> heat_integrationtests, and they can always be forklifted back again in
>> the future. I would like to propose that we go ahead with the in-tree
>> integration tests with a view to moving them back to tempest in the
>> future. We could agree on a set of preconditions for moving them back.
>>
>> On the heat side the preconditions could be:
>> - Good coverage of testing heat resources
>> - An established process for insisting on new integration tests for new
>> features
>>
>> On the tempest side:
>> - An established QA liaison program
>> - Completion of transition to tempest-lib and in-tree functional tests
> So this is actually something that is very similar to something we discussed at
> summit. [1] I don't have an issue with the model of developing tests in the heat
> tree to have testing be more tightly coupled with development. It has several
> advantages. Then we can do a graduation of tests into tempest when and where it
> would make sense to run against everyone, and move heat tests from nova into
> heat. However, I don't view any of this as a good reason to remove existing
> tests from tempest now. Maybe as part of the tempest cleanup that'll happen
> eventually some of the existing tests we'll find don't need to be in tempest.
> But for right now there isn't really any evidence supporting that, especially
> considering how limited heat test coverage is that would just seem like a
> premature action.
>
> I think what you've outlined for preconditions to migration makes sense for the
> most part. But, I think it should be for migration either way, not just for
> heat -> tempest. Because really when we're talking about test migrations we're
> talking about trying to optimize our test load so that we're only running things
> where and when they need to be.
>
> I'm also not entirely sure what you mean by: 'Completion of transition to
> tempest-lib and in-tree functional tests' I think you might have some
> unrealistic expectations around what is happening here. We're not going to be
> shrinking tempest scope in the short term, and I don't expect it will happen for
> some time. When we have discussions about moving a large chunk of api test
> coverage back into projects that's a *long* term goal. Several things will need
> to happen before we can even consider working on an en masse migration of tests
> out of tempest. (including having the discussion on what the procedure and
> prereqs are for doing that, which is really a summit topic) Just one of which is
> having the projects actually starting to spin up their own functional test
> suites capable of doing the same class of api validation. Honestly I feel that
> removing tests from tempest will probably start to happen at the end of Kilo at
> the very earliest, but more realistically it'll be a L task. I can understand
> this precondition being just the tempest-lib migration especially if your
> in-tree heat tests are going to essentially be a mini-tempest. That will make
> the migrations in either direction much simpler.
I like the idea of a 2-way flow of tests between tempest and
heat_integrationtests.

I can see that heat_integrationtests would be a good place to develop
and incubate integration tests for new and existing features.

But lets bring in-tree functional tests into this picture too. Lets
assume that as well as heat_integrationtests we have
heat_functionaltests which is run in a devstack provisioned heat but all
other API interactions are mocked by something like mimic. For example
in this scenario, our current autoscaling test can evolve in
heat_integrationtests until eventually I see the following happen:
* a minimal test moves to the integrated gate which just confirms that
metrics and alarms reaching heat from ceilometer/servers
* the full autoscaling test moves to heat_functionaltests with mocking
from mimic (and fake webhook triggers)

New tests can start in heat_integrationtest, but then mature into some
combination of heat_integrationtest, heat_functionaltests and tempest.

>>>> Ultimately I don't think it really matters which repo those tests live in,
>>>> provided we can write them and get them running in the gate (catching
>>>> regressions, which otherwise keep slipping through) in a timely manner.
>>> So for the most part this may be true, unless you are considering cross project
>>> testing and gating, which is what I think Jay's argument is here. Heat is in a
>>> different position that almost all of it's functionality is dependent on the
>>> other services. So if the expectation is to be running these tests in a full
>>> OpenStack deployment essentially you'll be duplicating the role of Tempest. But,
>>> by being a heat specific test suite you'll have symmetrical gating issues. 
>>>
>> This is touching on the limits of the gating infrastructure. We're
>> already at the limit of available cloud resources to run an integrated
>> gate, and the tests we'd like to write will by their nature consume
>> quite some resources. There is a human limit too, some of our best folk
>> are burning out on keeping on top of integrated gate issues.
>>
> Yes, I agree we're reaching resource limitations here, but that in itself isn't
> a reason to abandon the system we have now. We've identified a set of problems
> with our current methodology and have a plan to try and fix it but rushing to
> implement it by removing existing things is not a good way to handle it. We need
> to be targeted in how we change things so that we don't lose the advantages of
> our current system in the process.
Yes, fair enough.
>> There is a potential symmetrical gating issue, but in theory Heat is
>> just consuming stable tested APIs. sdague has suggested we only run
>> check-heat-dsvm-functional against heat for now, and any asymmetric
>> breakages be reverted/fixed as they occur, and prevented from recurring
>> in the test suites of the offending projects.
>>
> So I can safely say from experience that saying this and doing it are 2 very
> different things. Every time we've tried doing something like this with having
> asymmetry it's caused way more pain than anyone considers at first. Having seen
> neutron break their gate for weeks at a time because of asymmetry. Or trying to
> spin up a new gating test job that only runs on a subset of projects constantly
> delayed because something else broke it. Although this might not necessarily be
> a bad thing, I was just outlining it as a potential concern before. I actually
> think this will probably be one way to stimulate better cross project
> communication.
We can see how this goes, one option would be to consider moving a test
into the integrated gate if it is getting a lot of asymmetry caused fails.
> The counter example that's been brought up to get around this is having
> "contract tests" in the projects to enforce stability on interfaces consumed by
> other projects. However, I don't think for Heat this will work or scale too well
> because it consumes pretty much all of the projects' REST APIs, which really
> comes back to all of Jay's posts on this thread.
>
> -Matt Treinish
>
> [1] https://etherpad.openstack.org/p/juno-qa-functional-api
>