Open Stack

Folks,

Not sure if this is the right forum or if I should bring it up in the next meeting, however, I had a question about the number of Negative tests that are getting included in the DefCore specs. I could soap box on the topic for a while but the short summary is that by their very definition, a negative test is not proving a capability other than a correct response or behavior from a system in a failure condition. They could easily pass and still provide no real visibility into whether or not a capability exists.

For example just to pick a random test, this is one of the tests in the current 2016.next patch for volumes:
"tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_create_volume_with_nonexistent_source_volid"

# Should not be able to create volume with non-existent source volume
v_name = data_utils.rand_name('Volume')
metadata = {'Type': 'work'}
self.assertRaises(lib_exc.NotFound, self.client.create_volume,
             size='1', source_volid=str(uuid.uuid4()),
     display_name=v_name, metadata=metadata)


Which means that this test will fail if it couldn't create a volume without passing an ID. This test could just as easily pass or fail for any number of other reasons. If the point of a DefCore test is to prove that Capability XYZ exists as described in an interoperable fashion, this test doesn't really move that point forward. I would think DefCore would want to focus on only positive tests that confirm a specific capability exists, not any given negative test that might only be looking to prove some failure condition.

Thoughts?

Thanks,


Sam

Sam Danes
Architect, Software Development - Test
[http://600a2794aa4ab5bae6bd-8d3014ab8e4d12d3346853d589a26319.r53.cf1.rackcdn.com/signatures/images/rackspace_logo.png]

sam.danes at rackspace.com
mobile: 412.689.1532

<https://www.rackspacemarketing.com/signatyourEmail/>
[http://600a2794aa4ab5bae6bd-8d3014ab8e4d12d3346853d589a26319.r53.cf1.rackcdn.com/signatures/images/linkedin.png]<https://www.linkedin.com/in/samueldanes>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/defcore-committee/attachments/20150924/ed5f21fb/attachment.html>