[openstack-dev] [openstack-qa] Graduation Requirements + Scope of Tempest
malini.kamalambal at RACKSPACE.COM
Fri Mar 21 16:38:19 UTC 2014
On 3/21/14 12:01 PM, "David Kranz" <dkranz at redhat.com> wrote:
>On 03/20/2014 04:19 PM, Rochelle.RochelleGrober wrote:
>>> -----Original Message-----
>>> From: Malini Kamalambal [mailto:malini.kamalambal at RACKSPACE.COM]
>>> Sent: Thursday, March 20, 2014 12:13 PM
>>> 'project specific functional testing' in the Marconi context is
>>> Marconi as a complete system, making Marconi API calls & verifying the
>>> response - just like an end user would, but without keystone. If one of
>>> these tests fail, it is because there is a bug in the Marconi code ,
>>> not because its interaction with Keystone caused it to fail.
>>> "That being said there are certain cases where having a project
>>> functional test makes sense. For example swift has a functional test
>>> starts swift in devstack. But, those things are normally handled on a
>>> basis. In general if the project is meant to be part of the larger
>>> ecosystem then Tempest is the place to put functional testing. That way
>>> you know
>>> it works with all of the other components. The thing is in openstack
>>> like a project isolated functional test almost always involves another
>>> in real use cases. (for example keystone auth with api requests)
>>> One of the concerns we heard in the review was 'having the functional
>>> tests elsewhere (I.e within the project itself) does not count and they
>>> have to be in Tempest'.
>>> This has made us as a team wonder if we should migrate all our
>>> tests to Tempest.
>>> But from Matt's response, I think it is reasonable to continue in our
>>> current path & have the functional tests in Marconi coexist along with
>>> the tests in Tempest.
>> I think that what is being asked, really is that the functional tests
>>could be a single set of tests that would become a part of the tempest
>>repository and that these tests would have an ENV variable as part of
>>the configuration that would allow either "no Keystone" or "Keystone" or
>>some such, if that is the only configuration issue that separates
>>running the tests isolated vs. integrated. The functional tests need to
>>be as much as possible a single set of tests to reduce duplication and
>>remove the likelihood of two sets getting out of sync with each
>>other/development. If they only run in the integrated environment,
>>that's ok, but if you want to run them isolated to make debugging
>>easier, then it should be a configuration option and a separate test job.
>> So, if my assumptions are correct, QA only requires functional tests
>>for integrated runs, but if the project QAs/Devs want to run isolated
>>for dev and devtest purposes, more power to them. Just keep it a single
>>set of functional tests and put them in the Tempest repository so that
>>if a failure happens, anyone can find the test and do the debug work
>>without digging into a separate project repository.
>> Hopefully, the tests as designed could easily take a new configuration
>>directive and a short bit of work with OS QA will get the integrated FTs
>>working as well as the isolated ones.
>This issue has been much debated. There are some active members of our
>community who believe that all the functional tests should live outside
>of tempest in the projects, albeit with the same idea that such tests
>could be run either as part of today's "real" tempest runs or mocked in
>various ways to allow component isolation or better performance. Maru
>Newby posted a patch with an example of one way to do this but I think
>it expired and I don't have a pointer.
>IMO there are valid arguments on both sides, but I hope every one could
>agree that functional tests should not be arbitrarily split between
>projects and tempest as they are now. The Tempest README states a desire
>for "complete coverage of the OpenStack API" but Tempest is not close to
>that. We have been discussing and then ignoring this issue for some time
>but I think the recent action to say that Tempest will be used to
>determine if something can use the OpenStack trademark will force more
>completeness on tempest (more tests, that is). I think we need to
>resolve this issue but it won't be easy and modifying existing api tests
>to be more flexible will be a lot of work. But at least new projects
>could get on the right path sooner.
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
We are talking about different levels of testing,
1. Unit tests - which everybody agrees should be in the individual project
2. System Tests - 'System' referring to (& limited to), all the components
that make up the project. These are also the functional tests for the
3. Integration Tests - This is to verify that the OS components interact
well and don't break other components -Keystone being the most obvious
example. This is where I see getting the maximum mileage out of Tempest.
I see value in projects taking ownership of the System Tests - because if
the project is not 'functionally ready', it is not ready to integrate with
other components of Openstack.
But for this approach to be successful, projects should have diversity in
the team composition - we need more testers who focus on creating these
This will keep the teams honest in their quality standards.
As long as individual projects cannot guarantee functional test coverage,
we will need more tests in Tempest.
But that will shift focus away from Integration Testing, which can be done
ONLY in Tempest.
Regardless of whatever we end up deciding, it will be good to have these
discussions sooner than later.
This will help at least the new projects to move in the right direction.
More information about the OpenStack-dev