Open Stack

On 12/12/2014 01:04 PM, Sean Toner wrote:
> Hi everyone,
> 
> I have been reading the etherpad from the Paris summit wrt to moving the
> functional tests into their respective projects 
> (https://etherpad.openstack.org/p/kilo-crossproject-move-func-tests-to-projects).  I am mostly interested this from the nova project 
> perspective. However, I still have a lot of questions.
> 
> For example, is it permissible (or a good idea) to use the python-
> *clients as a library for the tasks?  I know these were not allowed in 
> Tempest, but I don't see why they couldn't be used here (especially 
> since, AFAIK, there is no testing done on the SDK clients themselves).

Sure, though realistically I'd actually expect the clients to have their
own tests.

> Another question is also about a difference between Tempest and these 
> new functional tests.  In nova's case, it would be very useful to 
> actually utilize the libvirt library in order to touch the hypervisor 
> itself.  In Tempest, it's not allowed to do that.  Would it make sense 
> to be able to make calls to libvirt within a nova functional test?

Examples would be handy here.

> Basically, since Tempest was a "public" only library, there needs to be 
> a different set of rules as to what can and can't be done.  Even the 
> definition of what exactly a functional test is should be more clearly 
> stated.  
> 
> For example, I have been working on a project for some nova tests that 
> also use the glance and keystone clients (since I am using the python 
> SDK clients).  I saw this quote from the etherpad:
> 
> "    Many "api" tests in Tempest require more than one service (eg, nova 
> api tests require glance)
> 
>     Is this an API test or an integration test or a functional test? 
> sounds to me like cross project integration tests +1+1"
> 
> I would disagree that a functional test should belong to only one 
> project.  IMHO, a functional test is essentially a black box test that 
> might span one or more projects, though the projects should be related.  
> For example, I have worked on one of the new features where the config 
> drive image property is set in the glance image itself, rather than 
> specified during the nova boot call.  
> 
> I believe that's how a functional test can be defined.  A black box test 
> which may require looking "under the hood" that Tempest does not allow.

A black box test by definition doesn't look under the hood, and I think
that's where there has been a lot of disconnect.

> Has there been any other work or thoughts on how functional testing 
> should be done?

I've been working through early stages of this on the Nova side.

There are very pragmatic reasons for the tests to be owned by a single
project. When they are not, we get wedges, where due to factors beyond
our control you have 2 source trees in a bind over tests, and the
project doesn't have the ability to make an executive decision that
those tests aren't actually correct and should be changed.

I think the functional correctness of a project needs to be owned by
that project. As a community that's been largely pushed at the QA team
up until this point, and that both not scalable, as well as not very
debuggable (note how many folks just randomly type "recheck" because
they have no idea how to debug a failure).

	-Sean

-- 
Sean Dague
http://dague.net