Open Stack

Hey Mark,


Your two points below absolutely make sense and I would concur. The test I
picked was simply to have some kind of an example for the overall
question. If we agree on negative tests included specifically for the
purposes of consistent error behavior on a capability that already
has/will have the corresponding functional test(s) it would alleviate my
worry. As I start to get more involved with DefCore and OpenStack in
general I want to make sure that I am working towards the right common
goal. :-)

Thanks,


Sam

On 9/24/15, 12:20 PM, "Mark Voelker" <mvoelker at vmware.com> wrote:

>Hi Sam,
>
>You¹re correct that negative tests don¹t generally prove that a
>capability works.  If that were the only test required for the
>³volumes-v2-create-delete² capability, I¹d be a bit more worried‹but it¹s
>actually just one of the tests proposed.  Another one isn¹t a negative
>test and does directly test the capability:
>
>tempest.api.volume.test_volumes_get.VolumesV2GetTest.test_volume_create_ge
>t_update_delete
>
>Negative tests are somewhat useful from a couple of perspectives:
>
>1.)  They provide some assurance that the code providing the Capability
>is indeed OpenStack code.  E.g. if a vendor had replaced Cinder¹s code
>with their own, they might well have done so with code that
>inappropriately fails negative tests.  That presents two problems: first,
>replacing designated sections of code (generally speaking, the code
>providing a Capability is designated) violates the Guideline.  Secondly,
>if there are behavioral differences this may actually cause
>interoperability problems because applications are often written against
>behavior that is observed.
>
>2.)  They verify that in failure cases, the vendor¹s product is behaving
>in a consistent way.  This is actually important from an interop
>perspective as well, because application code sometimes does silly things
>(like, say, not checking user input for required parameters like a source
>volume) and it¹s useful to be able to code your error handlers against
>known server behaviors and responsesŠwhich you can¹t do if they aren¹t
>consistent from product to product.
>
>Make sense?
>
>At Your Service,
>
>Mark T. Voelker
>
>
>
>> On Sep 24, 2015, at 1:55 PM, Sam Danes <sam.danes at rackspace.com> wrote:
>> 
>> Folks,
>> 
>> Not sure if this is the right forum or if I should bring it up in the
>>next meeting, however, I had a question about the number of Negative
>>tests that are getting included in the DefCore specs. I could soap box
>>on the topic for a while but the short summary is that by their very
>>definition, a negative test is not proving a capability other than a
>>correct response or behavior from a system in a failure condition. They
>>could easily pass and still provide no real visibility into whether or
>>not a capability exists.
>> 
>> For example just to pick a random test, this is one of the tests in the
>>current 2016.next patch for volumes:
>> 
>>"tempest.api.volume.test_volumes_negative.VolumesV2NegativeTest.test_crea
>>te_volume_with_nonexistent_source_volid"
>> 
>> # Should not be able to create volume with non-existent source volume
>> v_name = data_utils.rand_name('Volume')
>> metadata = {'Type': 'work'}
>> self.assertRaises(lib_exc.NotFound, self.client.create_volume,
>>              size='1', source_volid=str(uuid.uuid4()),
>>      display_name=v_name, metadata=metadata)
>> 
>> 
>> Which means that this test will fail if it couldn't create a volume
>>without passing an ID. This test could just as easily pass or fail for
>>any number of other reasons. If the point of a DefCore test is to prove
>>that Capability XYZ exists as described in an interoperable fashion,
>>this test doesn¹t really move that point forward. I would think DefCore
>>would want to focus on only positive tests that confirm a specific
>>capability exists, not any given negative test that might only be
>>looking to prove some failure condition.
>> 
>> Thoughts?
>> 
>> Thanks,
>> 
>> 
>> Sam
>> 
>> Sam Danes
>> Architect, Software Development - Test
>> 
>> sam.danes at rackspace.com
>> mobile: 412.689.1532
>>    _______________________________________________
>> Defcore-committee mailing list
>> Defcore-committee at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee
>