Open Stack

On Mar 26, 2014, at 1:52 PM, Sean Dague <sean at dague.net> wrote:

> On 03/25/2014 04:49 AM, Maru Newby wrote:
>> 
>> On Mar 21, 2014, at 9:01 AM, David Kranz <dkranz at redhat.com> wrote:
>> 
>>> On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote:
>>>> 
>>>>> -----Original Message-----
>>>>> From: Malini Kamalambal [mailto:malini.kamalambal at RACKSPACE.COM]
>>>>> Sent: Thursday, March 20, 2014 12:13 PM
>>>>> 
>>>>> 'project specific functional testing' in the Marconi context is
>>>>> treating
>>>>> Marconi as a complete system, making Marconi API calls & verifying the
>>>>> response - just like an end user would, but without keystone. If one of
>>>>> these tests fail, it is because there is a bug in the Marconi code ,
>>>>> and
>>>>> not because its interaction with Keystone caused it to fail.
>>>>> 
>>>>> "That being said there are certain cases where having a project
>>>>> specific
>>>>> functional test makes sense. For example swift has a functional test
>>>>> job
>>>>> that
>>>>> starts swift in devstack. But, those things are normally handled on a
>>>>> per
>>>>> case
>>>>> basis. In general if the project is meant to be part of the larger
>>>>> OpenStack
>>>>> ecosystem then Tempest is the place to put functional testing. That way
>>>>> you know
>>>>> it works with all of the other components. The thing is in openstack
>>>>> what
>>>>> seems
>>>>> like a project isolated functional test almost always involves another
>>>>> project
>>>>> in real use cases. (for example keystone auth with api requests)
>>>>> 
>>>>> "
>> 
>> 
>> 
>>>>> 
>>>>> One of the concerns we heard in the review was 'having the functional
>>>>> tests elsewhere (I.e within the project itself) does not count and they
>>>>> have to be in Tempest'.
>>>>> This has made us as a team wonder if we should migrate all our
>>>>> functional
>>>>> tests to Tempest.
>>>>> But from Matt's response, I think it is reasonable to continue in our
>>>>> current path & have the functional tests in Marconi coexist  along with
>>>>> the tests in Tempest.
>>>>> 
>>>> I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either "no Keystone" or "Keystone" or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated.  The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development.  If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job.
>>>> 
>>>> So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them.  Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository.
>>>> 
>>>> Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones.
>>>> 
>>>> --Rocky
>>> This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's "real" tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer.
>> 
>> I think the best place for functional api tests to be maintained is in the projects themselves.  The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge.  The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible.  Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default.
> 
> I disagree. I think all that ends up doing is creating greater variance
> in quality between components in OpenStack. And it means now no one
> feels responsible when a bad test in a project some where causes a gate
> block.

In my experience, projects have greater incentive to maintain tests that are in the same tree as the code they are testing.  The tests can run pre-merge, and the same eyes that review the code can review the tests for that code.  Any lag between when code is written and tests are written, at least below the integration level, can be detrimental due to the fact that the mental models for the code naturally decay over time.  Does your experience on this differ?

As you say, though, maybe its worth taking the hit of not putting as much testing in-tree as possible.  Maybe the OpenStack community is doing such a poor job of providing cross-project leadership on quality initiatives and developer education that we simply can't trust the projects to write and maintain good tests .  And maybe contributors for a given project don't feel sufficient responsibility to other projects to ensure they don't break them, and to ensure rapid resolution when they do.

If you agree with any of that, might the lack of trust in delegating greater responsibility for testing to the projects be merely a symptom of organizational malaise? And might it be worth addressing the root causes rather than responding only to the symptoms?


> 
> If a core project team can't be bothered to work in the docs, infra, and
> qa, and other project repositories repos as part normal flow, that's
> core project is very clearly not integrated with the rest of OpenStack.
> 
> Being integrated needs to not just be a badge you get from the TC on a
> vote, it actually means "being integrated", beyond just your own git
> tree that you have +2 on.

I couldn't agree more - we need more coordination at every level.  It's essential to our success.  However, I'm not sure it follows that tighter integration with the wider OpenStack ecosystem precludes bearing greater responsibility for project-specific quality.


> 
>> The patch in question [1] proposes allowing pre-merge functional api test maintenance and test reuse in an integration environment.
>> 
>> 
>> m.
>> 
>> 1: https://review.openstack.org/#/c/72585/
> 
> This effort is interesting, but I feel like it's so far down on maslo's
> hierarchy of needs, it's not really clear why it's a focus right now.
> 
> Neutron's biggest issues right now related to testing is:
> * massive over duplication in unit tests which means it is now
> regularly passing 60 minutes on 4 core machines.
> * inability to run the full test suites (getting close)
> * no upgrade job
> 
> All of those would be way more beneficial to get energy on.

The initiative in question is targeted to address exactly the 'massive over duplication' you mention.  As a fortuitous side-effect, the refactored tests will be able to be run against deployed neutron with minimal additional effort.  I am frustrated to hear that, despite all the discussions you and I have engaged in on this topic, I still haven't managed to accurately convey my intent, and that you seem to believe that my time could be spent more productively.


>>> IMO there are valid arguments on both sides, but I hope every one could agree that functional tests should not be arbitrarily split between projects and tempest as they are now. The Tempest README states a desire for "complete coverage of the OpenStack API" but Tempest is not close to that. We have been discussing and then ignoring this issue for some time but I think the recent action to say that Tempest will be used to determine if something can use the OpenStack trademark will force more completeness on tempest (more tests, that is). I think we need to resolve this issue but it won't be easy and modifying existing api tests to be more flexible will be a lot of work. But at least new projects could get on the right path sooner.
>> 
>> 
>> 
>> - to say nothing of the increased effort required to develop functional tests in an integration envirequired to maintain tests in Tempest splinters their efforts unnecessarily.  I'm not suggesting that    

> The beginning of this thread largely came from the fact that Marconi
> clearly doing most of their QA not in an upstream way. To be integrated,
> that needs to change.
> 
> I've seen this go down with many projects now to the point where it's a
> normal 5 stages of grief thing.
> 
> * Denial - we can totally do all this in our own tree, no need to work
> in Tempest
> * Anger - what, python*client shipped a new version and we're broken!
> how dare they? And why do I need to bother working outside my own git tree?
> * Bargaining - let me propose a whole other approach to testing that
> doesn't need Tempest
> * Depression - that's not going to work? why won't you let me gate all
> the projects on my own thing? ok, fine I'll work with Tempest.
> * Acceptance - oh, look at that, I just managed to block a Keystone
> change that would have broken me, but now I have a Tempest test that
> *they* have to pass as well.
> 
> Is Tempest a paragon of virtue? Far from it. But right now has very
> clearly shown it's value, and I think remains the right collaboration
> point to create the contract about what we all believe OpenStack is.

I don't think anybody is questioning the value of Tempest, nor that it is an effective collaboration point.  I do think that, given how critical it is to the success of OpenStack, Tempest - and its mandate - will need to evolve over time to stay maximally effective.  What worked in the past isn't necessarily what we will need in the future.


m.