Open Stack

+1 to this plan

>  From the above, I would surmise that smoke tests should have all three of
> the following characteristics:
> 
> * Test basic operations of an API, usually in a specific order that makes sense
> as a bare-bones use case of the API
> * Test only the correct action paths -- in other words, smoke tests do not
> throw random or invalid input at an API
> * Use the default client tools to complete the actions
> * Take a short amount of time to complete

[JG] That sounds good. I wonder if we should also restrict them to be user actions only? Maybe we should also try to concentrate on actions that are likely to be available on all OpenStack installations? I am thinking things like resize are less important than VM start. However, I like the idea of testing every user facing compute CLI call using Tempest, but that scope would seem too large for a smoke test.
 
> It would be super-awesome if Tempest could be used to remove some of
> this duplication of efforts.

[JG] +1

> * Create a base test class that uses the default CLI tools instead of raw HTTP
> calls

[JG] we need to test the default tools, so this sounds like a good plan

> * Create a set of Manager classes that would provide a config object and one
> or more client objects to test cases -- this is similar to the
> tempest.openstack.Manager class currently in the source code, but would
> allow using the default project client libraries instead of the raw HTTP client
> currently in Tempest
> * Allow test cases that derive from the base SmokeTest class to use an
> ordered test method system -- e.g.  test_001_foo, test_002_bar

[JG] I don't like order dependent tests, but I think it makes sense in this case

> * Have the base smoke test case class automatically annotate all of its test
> methods with type=smoke so that we can run all smoke tests with:
>      nosetests --attr=type=smoke tempest
> 
> I have put together some code that shows the general idea as a draft
> patchset [5]
 
[JG] I would like to take this and get it running on several XenServer setups.