Open Stack

Hi Sam,

You’re correct that negative tests don’t generally prove that a capability works.  If that were the only test required for the “volumes-v2-create-delete” capability, I’d be a bit more worried—but it’s actually just one of the tests proposed.  Another one isn’t a negative test and does directly test the capability:

tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_get_update_delete

Negative tests are somewhat useful from a couple of perspectives:

1.)  They provide some assurance that the code providing the Capability is indeed OpenStack code.  E.g. if a vendor had replaced Cinder’s code with their own, they might well have done so with code that inappropriately fails negative tests.  That presents two problems: first, replacing designated sections of code (generally speaking, the code providing a Capability is designated) violates the Guideline.  Secondly, if there are behavioral differences this may actually cause interoperability problems because applications are often written against behavior that is observed.

2.)  They verify that in failure cases, the vendor’s product is behaving in a consistent way.  This is actually important from an interop perspective as well, because application code sometimes does silly things (like, say, not checking user input for required parameters like a source volume) and it’s useful to be able to code your error handlers against known server behaviors and responses…which you can’t do if they aren’t consistent from product to product.  

Make sense?

At Your Service,

Mark T. Voelker



> On Sep 24, 2015, at 1:55 PM, Sam Danes <sam.danes at rackspace.com> wrote:
> 
> Folks,
> 
> Not sure if this is the right forum or if I should bring it up in the next meeting, however, I had a question about the number of Negative tests that are getting included in the DefCore specs. I could soap box on the topic for a while but the short summary is that by their very definition, a negative test is not proving a capability other than a correct response or behavior from a system in a failure condition. They could easily pass and still provide no real visibility into whether or not a capability exists.
> 
> For example just to pick a random test, this is one of the tests in the current 2016.next patch for volumes: 
> "tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid" 
> 
> # Should not be able to create volume with non-existent source volume
> v_name = data_utils.rand_name('Volume')
> metadata = {'Type': 'work'}
> self.assertRaises(lib_exc.NotFound, self.client.create_volume,
>              size='1', source_volid=str(uuid.uuid4()),
>      display_name=v_name, metadata=metadata)
> 
> 
> Which means that this test will fail if it couldn't create a volume without passing an ID. This test could just as easily pass or fail for any number of other reasons. If the point of a DefCore test is to prove that Capability XYZ exists as described in an interoperable fashion, this test doesn’t really move that point forward. I would think DefCore would want to focus on only positive tests that confirm a specific capability exists, not any given negative test that might only be looking to prove some failure condition.
> 
> Thoughts?
> 
> Thanks,
> 
> 
> Sam
> 
> Sam Danes
> Architect, Software Development - Test
> 
> sam.danes at rackspace.com
> mobile: 412.689.1532
>    _______________________________________________
> Defcore-committee mailing list
> Defcore-committee at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee