Open Stack

On 06/24/2014 01:29 AM, Fei Long Wang wrote:
> Greetings,
> 
> We're leveraging the scenario test of Tempest to do the end-to-end
> functional test to make sure everything work great after upgrade,
> patching, etc. And We're happy to fill the gaps we found. However, I'm a
> little bit confused about the test policy from the scenario test
> perspective, especially comparing with the API test. IMHO, scenario test
> will cover some typical work flows of one specific service or mixed
> services, and it would be nice to make sure the function is really
> working instead of just checking the object status from OpenStack
> perspective. Is that correct?
> 
> For example, live migration of Nova, it has been covered in API test of
> Tempest (see
> https://github.com/openstack/tempest/blob/master/tempest/api/compute/test_live_block_migration.py).
> But as you see, it just checks if the instance is Active or not instead
> of checking if the instance can be login/ssh successfully. Obviously,
> from an real world view, we'd like to check if it's working indeed. So
> the question is, should this be improved? If so, the enhanced code
> should be in API test, scenario test or any other places? Thanks you.

The fact that computes aren't verified fully during the API testing is
mostly historical. I think they should be. The run_ssh flag used to be
used for this, however because of some long standing race conditions in
the networking stack, that wasn't able to be turned on in upstream
testing. My guess is that it's rotted now.

We've had some conversations in the QA team about a compute verifier
that would be run after any of the compute jobs to make sure they booted
correctly, and more importantly, did a very consistent set of debug
capture when they didn't. Would be great if that's something you'd like
to help out with.

	-Sean

-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140624/3f1f77f7/attachment.pgp>