Open Stack

Excerpts from Sam Danes's message of 2015-09-24 17:55:37 +0000:
> Folks,
> 
> Not sure if this is the right forum or if I should bring it up in the next meeting, however, I had a question about the number of Negative tests that are getting included in the DefCore specs. I could soap box on the topic for a while but the short summary is that by their very definition, a negative test is not proving a capability other than a correct response or behavior from a system in a failure condition. They could easily pass and still provide no real visibility into whether or not a capability exists.
> 
> For example just to pick a random test, this is one of the tests in the current 2016.next patch for volumes:
> "tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid"
> 
> # Should not be able to create volume with non-existent source volume
> v_name = data_utils.rand_name('Volume')
> metadata = {'Type': 'work'}
> self.assertRaises(lib_exc.NotFound, self.client.create_volume,
>              size='1', source_volid=str(uuid.uuid4()),
>      display_name=v_name, metadata=metadata)
> 
> 
> Which means that this test will fail if it couldn't create a volume without passing an ID. This test could just as easily pass or fail for any number of other reasons. If the point of a DefCore test is to prove that Capability XYZ exists as described in an interoperable fashion, this test doesn't really move that point forward. I would think DefCore would want to focus on only positive tests that confirm a specific capability exists, not any given negative test that might only be looking to prove some failure condition.

That test is expecting a NotFound error for the made up volume id
passed to the create_volume call.  If any other type of exception
is returned, the test will fail because the error type doesn't
match.

It's just as important to return consistent errors to callers as
it is to have consistent successful behaviors, so we need to continue
to include these sorts of tests.

Doug

> 
> Thoughts?
> 
> Thanks,
> 
> 
> Sam
> 
> Sam Danes
> Architect, Software Development - Test
> [http://600a2794aa4ab5bae6bd-8d3014ab8e4d12d3346853d589a26319.r53.cf1.rackcdn.com/signatures/images/rackspace_logo.png]
> 
> sam.danes at rackspace.com
> mobile: 412.689.1532
> 
> <https://www.rackspacemarketing.com/signatyourEmail/>
> [http://600a2794aa4ab5bae6bd-8d3014ab8e4d12d3346853d589a26319.r53.cf1.rackcdn.com/signatures/images/linkedin.png]<https://www.linkedin.com/in/samueldanes>