Open Stack

On Mar 26, 2014, at 12:59 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:

> 
> 
> 
> On Tue, Mar 25, 2014 at 1:49 AM, Maru Newby <marun at redhat.com> wrote:
> 
> On Mar 21, 2014, at 9:01 AM, David Kranz <dkranz at redhat.com> wrote:
> 
> > On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote:
> >>
> >>> -----Original Message-----
> >>> From: Malini Kamalambal [mailto:malini.kamalambal at RACKSPACE.COM]
> >>> Sent: Thursday, March 20, 2014 12:13 PM
> >>>
> >>> 'project specific functional testing' in the Marconi context is
> >>> treating
> >>> Marconi as a complete system, making Marconi API calls & verifying the
> >>> response - just like an end user would, but without keystone. If one of
> >>> these tests fail, it is because there is a bug in the Marconi code ,
> >>> and
> >>> not because its interaction with Keystone caused it to fail.
> >>>
> >>> "That being said there are certain cases where having a project
> >>> specific
> >>> functional test makes sense. For example swift has a functional test
> >>> job
> >>> that
> >>> starts swift in devstack. But, those things are normally handled on a
> >>> per
> >>> case
> >>> basis. In general if the project is meant to be part of the larger
> >>> OpenStack
> >>> ecosystem then Tempest is the place to put functional testing. That way
> >>> you know
> >>> it works with all of the other components. The thing is in openstack
> >>> what
> >>> seems
> >>> like a project isolated functional test almost always involves another
> >>> project
> >>> in real use cases. (for example keystone auth with api requests)
> >>>
> >>> "
> 
> 
> 
> >>>
> >>> One of the concerns we heard in the review was 'having the functional
> >>> tests elsewhere (I.e within the project itself) does not count and they
> >>> have to be in Tempest'.
> >>> This has made us as a team wonder if we should migrate all our
> >>> functional
> >>> tests to Tempest.
> >>> But from Matt's response, I think it is reasonable to continue in our
> >>> current path & have the functional tests in Marconi coexist  along with
> >>> the tests in Tempest.
> >>>
> >> I think that what is being asked, really is that the functional tests could be a single set of tests that would become a part of the tempest repository and that these tests would have an ENV variable as part of the configuration that would allow either "no Keystone" or "Keystone" or some such, if that is the only configuration issue that separates running the tests isolated vs. integrated.  The functional tests need to be as much as possible a single set of tests to reduce duplication and remove the likelihood of two sets getting out of sync with each other/development.  If they only run in the integrated environment, that's ok, but if you want to run them isolated to make debugging easier, then it should be a configuration option and a separate test job.
> >>
> >> So, if my assumptions are correct, QA only requires functional tests for integrated runs, but if the project QAs/Devs want to run isolated for dev and devtest purposes, more power to them.  Just keep it a single set of functional tests and put them in the Tempest repository so that if a failure happens, anyone can find the test and do the debug work without digging into a separate project repository.
> >>
> >> Hopefully, the tests as designed could easily take a new configuration directive and a short bit of work with OS QA will get the integrated FTs working as well as the isolated ones.
> >>
> >> --Rocky
> > This issue has been much debated. There are some active members of our community who believe that all the functional tests should live outside of tempest in the projects, albeit with the same idea that such tests could be run either as part of today's "real" tempest runs or mocked in various ways to allow component isolation or better performance. Maru Newby posted a patch with an example of one way to do this but I think it expired and I don't have a pointer.
> 
> I think the best place for functional api tests to be maintained is in the projects themselves.  The domain expertise required to write api tests is likely to be greater among project resources, and they should be tasked with writing api tests pre-merge.  The current 'merge-first, test-later' procedure of maintaining api tests in the Tempest repo makes that impossible.  Worse, the cost of developing functional api tests is higher in the integration environment that is the Tempest default.
> 
> 
> If an API is made and documented properly what domain expertise would be needed to use it? The opposite is true for tempest and the tests themselves. The tempest team focuses on just tests so they know how to write good tests and are able to leverage common underlying framework code.

Given that documentation is typically finalized only late in the cycle, are you suggesting that we forego api testing until possibly well after the code has been written?  Plus, it is a rare api that doesn't have to evolve in response to real-world experience with early implementations.  The sooner we can write functional api tests, the sooner we can identify shortcomings that need to be addressed - and the less costly they will be to fix.

As to expertise, would you agree that we have a much greater need for QA expertise than the existing team can hope to meet?  I would argue that this is the case, and believe that the more we can diffuse a testing mentality from the QA team to the projects, the better off OpenStack will be.  Tempest has proven a very effective guardian against regression, but that role can only grow more difficult with time and a lack of focus will complicate the effort unnecessarily.

As to framework, the talk of 'libifying' Tempest promises to allow the projects the same benefit of common framework that Tempest currently enjoys.

> 
> Yes 'merge-first, test-later' is bad. But There are other ways of fixing this then moving the tests out of tempest. Hopefully cross project dependencies in zuul will help make this workflow easier.
> 
>  
> 
> The patch in question [1] proposes allowing pre-merge functional api test maintenance and test reuse in an integration environment.
> 
> 
> m.
> 
> 1: https://review.openstack.org/#/c/72585/
> 
> > IMO there are valid arguments on both sides, but I hope every one could agree that functional tests should not be arbitrarily split between projects and tempest as they are now. The Tempest README states a desire for "complete coverage of the OpenStack API" but Tempest is not close to that. We have been discussing and then ignoring this issue for some time but I think the recent action to say that Tempest will be used to determine if something can use the OpenStack trademark will force more completeness on tempest (more tests, that is). I think we need to resolve this issue but it won't be easy and modifying existing api tests to be more flexible will be a lot of work. But at least new projects could get on the right path sooner.
> 
> 
> This touches on another point that I forgot to bring up before.  As a cloud deployer I want a test suite that allows me to exercise all of my APIs to make sure everything is working. Although OpenStack runs extensive tests upstream, upstream cannot test every possible configuration, so as a deployer I would want rerun the same test suite. This is exactly why TripleO is getting ready to run tempest. Does this test suite have to just tempest? Perhaps not, but if we go down the road of moving more API tests into individual projects I think we still need easy way for a cloud deployer to run all the API tests against a live cloud from a single interface.

I don't think project-maintained functional api tests would in any way preclude Tempest from continuing to be the go-to test suite for deployers/operators.  There is nothing stopping project-maintained functional api tests from being consumed by Tempest.  Tempest could explicitly depend on the projects, either the whole package or a special package that contained only the api tests.  Or Tempest could copy the api tests, as is done with oslo incubator.  In either case, the process could be purely mechanical.


m.