Open Stack

Rightfully so, but it doesn't hurt to offer suggestions that might improve the community. It would just be nice to have exclusions reconsidered if there are legitimate bugs behind them. You see them all the time in the tempest tests ala "SKIPPED: Skipped until Bug: 1373513 is resolved" so  it's hard to understand why we can't just apply the same principles to third-party CI.

Thanks,
Ryan

On Mar 26, 2015, at 4:42 PM, Anita Kuno <anteaya at anteaya.info> wrote:

> On 03/26/2015 06:48 PM, Ryan Hsu wrote:
>> Exclusions are legitimate and will always be necessary at some point. In the case of the linked bug, this was once a known issue for the VMware driver and we had excluded affected tests so that CI could continue to run. This is the same way we do it in Nova CI and oslo.vmware CI. 
> 
> This is Cinder, Ryan, and Mike is the PTL. It is his decision.
> 
> Thank you,
> Anita.
> 
>> Thanks,
>> Ryan
>> 
>> On Mar 26, 2015, at 3:30 PM, Mike Perez <thingee at gmail.com> wrote:
>> 
>>> On 20:49 Thu 26 Mar     , Ryan Hsu wrote:
>>>> Hi Mike,
>>>> 
>>>> We (VMware CI) run "testr run tempest.api.volume" for our Cinder CI and this
>>>> runs about ~240 tests for us. I'm guessing that the rest of the ~60 tests are
>>>> not being run due to skips and disabled features. For example, here is
>>>> a sampling of tests that are skipped in a recent run (note that this is using
>>>> tempest.conf with no explicit disabling of Cinder services):
>>>> 
>>>> tempest.api.compute.test_live_block_migration.LiveBlockMigrationTestJSON.test_iscsi_volume ... SKIPPED: Block Live migration not available
>>>> setUpClass (tempest.api.orchestration.stacks.test_volumes.CinderResourcesTest) ... SKIPPED: Heat support is required
>>>> setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV2Test) ... SKIPPED: Cinder multi-backend feature disabled
>>>> tempest.api.compute.volumes.test_attach_volume.AttachVolumeTestJSON.test_attach_detach_volume ... SKIPPED: SSH required for this test
>>>> setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV1Test) ... SKIPPED: Cinder backup feature disabled
>>>> setUpClass (tempest.api.volume.admin.test_volumes_backup.VolumesBackupsV2Test) ... SKIPPED: Cinder backup feature disabled
>>>> setUpClass (tempest.api.volume.admin.test_multi_backend.VolumeMultiBackendV1Test) ... SKIPPED: Cinder multi-backend feature disabled
>>>> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved.
>>>> tempest.scenario.test_stamp_pattern.TestStampPattern.test_stamp_pattern ... SKIPPED: Skipped until Bug: 1205344 is resolved.
>>>> setUpClass (tempest.thirdparty.boto.test_ec2_instance_run.InstanceRunTest) ... SKIPPED: The EC2 API is not available
>>>> setUpClass (tempest.thirdparty.boto.test_ec2_volumes.EC2VolumesTest) ... SKIPPED: The EC2 API is not available
>>>> tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern ... SKIPPED: Skipped until Bug: 1373513 is resolved.
>>>> 
>>>> As we are actually running the volume suite according to the FAQ and the
>>>> above skipped tests are documented by our CI, would it be possible to add an
>>>> exception to the rule?  I'm sure these numbers will be different for all CIs
>>>> and as long as people are not abusing and hiding skipped tests, I don't see
>>>> this as a problem.
>>> 
>>> There will be no exceptions. Everyone must pass the same tests or you're not an
>>> approved volume driver for OpenStack Cinder.
>>> 
>>> You should also take this bug [1] Vmware hit as a lesson of doing any
>>> excluding in your CI. The driver would've been seriously broken in Kilo if this
>>> wasn't caught.
>>> 
>>> [1] - https://bugs.launchpad.net/cinder/+bug/1436603
>>> 
>>> -- 
>>> Mike Perez
>>> 
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev