Open Stack

On 20:49 Thu 26 Mar     , Ryan Hsu wrote:
> Hi Mike,
> 
> We (VMware CI) run "testr run tempest.api.volume" for our Cinder CI and this
> runs about ~240 tests for us. I'm guessing that the rest of the ~60 tests are
> not being run due to skips and disabled features. For example, here is
> a sampling of tests that are skipped in a recent run (note that this is using
> tempest.conf with no explicit disabling of Cinder services):
> 
> tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_iscsi_volume ... SKIPPED: Block Live migration not available
> setUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest) ... SKIPPED: Heat support is required
> setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV2Test) ... SKIPPED: Cinder multi-backend feature disabled
> tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume ... SKIPPED: SSH required for this test
> setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test) ... SKIPPED: Cinder backup feature disabled
> setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test) ... SKIPPED: Cinder backup feature disabled
> setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV1Test) ... SKIPPED: Cinder multi-backend feature disabled
> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved.
> tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern ... SKIPPED: Skipped until Bug: 1205344 is resolved.
> setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest) ... SKIPPED: The EC2 API is not available
> setUpClass (tempest.thirdparty.boto.test_ec2_volumes.EC2VolumesTest) ... SKIPPED: The EC2 API is not available
> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved.
> 
> As we are actually running the volume suite according to the FAQ and the
> above skipped tests are documented by our CI, would it be possible to add an
> exception to the rule?  I'm sure these numbers will be different for all CIs
> and as long as people are not abusing and hiding skipped tests, I don't see
> this as a problem.

There will be no exceptions. Everyone must pass the same tests or you're not an
approved volume driver for OpenStack Cinder.

You should also take this bug [1] Vmware hit as a lesson of doing any
excluding in your CI. The driver would've been seriously broken in Kilo if this
wasn't caught.

[1] - https://bugs.launchpad.net/cinder/+bug/1436603

-- 
Mike Perez