Open Stack

Hi everyone,

I have been reading the etherpad from the Paris summit wrt to moving the
functional tests into their respective projects 
(https://etherpad.openstack.org/p/kilo-crossproject-move-func-tests-to-projects).  I am mostly interested this from the nova project 
perspective. However, I still have a lot of questions.

For example, is it permissible (or a good idea) to use the python-
*clients as a library for the tasks?  I know these were not allowed in 
Tempest, but I don't see why they couldn't be used here (especially 
since, AFAIK, there is no testing done on the SDK clients themselves).

Another question is also about a difference between Tempest and these 
new functional tests.  In nova's case, it would be very useful to 
actually utilize the libvirt library in order to touch the hypervisor 
itself.  In Tempest, it's not allowed to do that.  Would it make sense 
to be able to make calls to libvirt within a nova functional test?

Basically, since Tempest was a "public" only library, there needs to be 
a different set of rules as to what can and can't be done.  Even the 
definition of what exactly a functional test is should be more clearly 
stated.  

For example, I have been working on a project for some nova tests that 
also use the glance and keystone clients (since I am using the python 
SDK clients).  I saw this quote from the etherpad:

"    Many "api" tests in Tempest require more than one service (eg, nova 
api tests require glance)

    Is this an API test or an integration test or a functional test? 
sounds to me like cross project integration tests +1+1"

I would disagree that a functional test should belong to only one 
project.  IMHO, a functional test is essentially a black box test that 
might span one or more projects, though the projects should be related.  
For example, I have worked on one of the new features where the config 
drive image property is set in the glance image itself, rather than 
specified during the nova boot call.  

I believe that's how a functional test can be defined.  A black box test 
which may require looking "under the hood" that Tempest does not allow.

Has there been any other work or thoughts on how functional testing 
should be done?

Thanks,
Sean Toner