Open Stack

Hello Georg,

We had tried to make sure that tests test only one capability and not several. We had a spec that describes what we are looking for in test:

https://github.com/openstack/interop/blob/master/working_materials/interop_test_spec.rst

Test guidelines:
Guidelines

  1.  Interop API functional tests should target and validate one capability. Some capabilities may require discovery calls and these should be included also in the test. Each capability should have at least one interop API functional test.
  2.  Interop API functional tests should not use non-required capabilities.
  3.  Interop scenario tests may use several capabilities to test end to end functionality and none of the capabilities used may be non-required capabilities.
  4.  Tests must be reliable; race conditions must be avoided; tests should not be time dependent. If any of these problems occur the test will be flagged and either fixed or removed from the guidelines.
  5.  Tests must be able to run consistently across all the releases covered by the Guidelines which are considered valid by the OpenStack Foundation for the OpenStack Powered program.
  6.  Tests should not require more credentials than required to test the capability itself.
  7.  Tests must not be harmful to the end user's environment, that is no changing credentials nor destroying user's data nor altering any other user resource other than those specifically created as part of the test.
  8.  Test cases must setup any data they need to run and clean up after themselves. It is not acceptable to corrupt the user's environment or leave any test data or test configuration behind.

Let us know if you have any questions!

-Egle


From: Georg Kunz <georg.kunz at ericsson.com<mailto:georg.kunz at ericsson.com>>
Date: Monday, June 26, 2017 at 2:54 AM
To: "interop-wg at lists.openstack.org<mailto:interop-wg at lists.openstack.org>" <interop-wg at lists.openstack.org<mailto:interop-wg at lists.openstack.org>>
Subject: [Interop-wg] Tempest scenario tests

Hi Egle, Rocky,

Thank you again for joining last week's OPNFV Dovetail call. Your input was very valuable and I believe we agreed across the board that a closer collaboration between OPNFV Dovatail and the OpenStack Interop WG is fundamental for both communities.

I have one follow-up question: the test suites of the three OpenStack certification programs currently include only tempest API tests (located in tempest.api.X) and none of the tempest scenarios (located in tempest.scenario.X). Is there a specific reason why those more complex end-to-end tests are currently not included?

Thanks
Georg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/interop-wg/attachments/20170626/03251d80/attachment.html>