Open Stack

Chris,

I don't think we should be testing against tag3 for Havana.  Tag 1 is the tag that was created just after Havana was released.  These are the tests that should define Havana capabilities.

Testing against Tag3 will ensure that the Havana capabilities are compatible with the Juno tests and capabilities that exist in Juno, but it doesn't address the fixed nature of the compatibility of a specific release.

I think as a group, both DefCore and the Refstack team, we need to specifically define how we identify the tests for a release certification.  Right now, Refstack test sets are changing as tag sets are chosen for testing against the Havana capabilities list.  Mayb that's what DefCore wants, but I don't think so.

>From a vendor perspective, I would want a *fixed* set of tests against which my Release would be tested.  The only time I would want to change that list is if I rolled out a maintenance release and that release used new or modified tests to validate a core capability.

This is a great email to start the discussion on exactly how DefCore will identify the tests associated with a capability/Release set.

--Rocky

-----Original Message-----
From: Chris Hoge [mailto:chris at openstack.org] 
Sent: Friday, January 30, 2015 10:14 AM
To: defcore-committee at lists.openstack.org
Subject: [OpenStack-DefCore] Removed and Versioned Havana Capability Tests

A note on the Havana capabilities. In comparing the capabilities list to the pinned version of Tempest we are testing against (tag 3), the following tests are no longer available (or possibly moved):

tempest.api.compute.admin.test_fixed_ips.FixedIPsTestXml.test_list_fixed_ip_details                                                 
tempest.api.compute.admin.test_fixed_ips.FixedIPsTestXml.test_set_reserve                                                           
tempest.api.compute.admin.test_fixed_ips.FixedIPsTestXml.test_set_unreserve
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_pause_unpause_server
tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_suspend_resume_server

The following volume tests have been replaced with a pair of versioned tests:

tempest.api.volume.admin.test_volume_types.VolumeTypesTest.test_create_get_delete_volume_with_volume_type_and_extra_specs
tempest.api.volume.admin.test_volume_types.VolumeTypesTest.test_volume_type_list
tempest.api.volume.admin.test_volume_types_extra_specs.VolumeTypesExtraSpecsTest.test_volume_type_extra_spec_create_get_delete
tempest.api.volume.admin.test_volume_types_extra_specs.VolumeTypesExtraSpecsTest.test_volume_type_extra_specs_list
tempest.api.volume.admin.test_volume_types_extra_specs.VolumeTypesExtraSpecsTest.test_volume_type_extra_specs_update
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_create_invalid_body
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_create_none_body
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_create_nonexistent_type_id
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_delete_nonexistent_volume_type_id
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_get_nonexistent_extra_spec_id
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_get_nonexistent_volume_type_id
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_list_nonexistent_volume_type_id
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_update_multiple_extra_spec
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_update_no_body
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_update_none_extra_spec_id
tempest.api.volume.admin.test_volume_types_extra_specs_negative.ExtraSpecsNegativeTest.test_update_nonexistent_extra_spec_id
tempest.api.volume.admin.test_volume_types_negative.VolumeTypesNegativeTest.test_create_with_empty_name
tempest.api.volume.admin.test_volume_types_negative.VolumeTypesNegativeTest.test_create_with_nonexistent_volume_type
tempest.api.volume.admin.test_volume_types_negative.VolumeTypesNegativeTest.test_delete_nonexistent_type_id
tempest.api.volume.admin.test_volume_types_negative.VolumeTypesNegativeTest.test_get_nonexistent_type_id

All of the core capability test names remained stable.

Chris Hoge
Interop Engineer
OpenStack Foundation


_______________________________________________
Defcore-committee mailing list
Defcore-committee at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee