Open Stack

This sounds like a good plan David. If the docs are ambiguous or outright missing info, we should file the doc bug too. I think we all agree that the checking for Nxx codes is a bad test. :-)

-----Original Message-----
From: David Kranz [mailto:david.kranz at qrclab.com] 
Sent: Wednesday, January 30, 2013 7:45 PM
To: openstack-qa at lists.openstack.org
Subject: Re: [openstack-qa] Return codes for OpenStack APIs

I agree with this as well, and this was the consensus of the Tempest team in the past. The problem is with APIs that are undocumented about the return value, or say that there are more than one 2xx possibility but don't say under what circumstances each code would be returned. I think that in addition to requiring exact checks, we file doc bugs for the undocumented cases as we come across them.

  -David

On 1/30/2013 7:45 PM, Sean Dague wrote:
> Realistically this started to come in during the 200 / 202 issues over 
> the summer.
>
> I think it's bad form for us to have generic 2xx testing, tempest is 
> really about testing exact behavior. My vote would be to -1 these 
> starts with tests and make the tests be explicit about what they are 
> supposed to return.
>
> This also lets tempest prevent accidental break of the API contract.
>
>     -Sean
>
> On 01/30/2013 06:45 PM, Sam Danes wrote:
>> I tend to agree. The specific response code is a contract from the 
>> API that in condition X, it will return exactly Y. I would give a nod 
>> to what Daryl is saying that if the response codes were externalized 
>> and vendor configurable you could make somewhat of an argument to 
>> generalize the check, however, at that point I would think that the 
>> specific configuration that the vendor has in place should still be 
>> validated with the exact codes.
>>
>> We have had issues as well where a call may return say a 401 instead 
>> of a 404 which would have caused a downstream issue. A generic check 
>> for a
>> startswith(‘4’) would have let that through.
>>
>> --Sam
>>
>> *From:*Daryl Walleck [mailto:daryl.walleck at RACKSPACE.COM]
>> *Sent:* Wednesday, January 30, 2013 4:55 PM
>> *To:* openstack-qa at lists.openstack.org; All Things QA.
>> *Subject:* Re: [openstack-qa] Return codes for OpenStack APIs
>>
>> I'm a proponent of checking the exact code. To me, its part of the 
>> API contract. From a practical point, it has also helped our 
>> automated tests catch bugs that nearly slipped by. One case I an 
>> remember is when some various POSTs started returning a 200 code as 
>> opposed to a 202, with the end result being that the requests 
>> responded to with a 200 actually were not processed. The only reason 
>> I could think of to generalize response code checking would be if it 
>> the response codes were ever vendor configurable.
>>
>> Daryl
>>
>> *From:* David Kranz
>> *Sent:* ‎January‎ ‎30‎, ‎2013 ‎4‎:‎08‎ ‎PM
>> *To:* openstack-qa at lists.openstack.org 
>> <mailto:openstack-qa at lists.openstack.org>
>> *Subject:* [openstack-qa] Return codes for OpenStack APIs
>>
>> There have been recent submissions to Tempest with code like:
>>
>> self.assertTrue(resp['status'].startswith('2'))
>>
>> It had been the consensus that we should be checking for actual 
>> status codes. Some projects document the return codes and others 
>> don't. This example was from a call to keystone to list services and 
>> the result is not documented but the list tenants calls says it could 
>> return 200 or 203. There should be a statement from each project 
>> about whether it considers the return code to be part of the spec.
>> Ideally this would be uniform across projects but it is not. Users of 
>> the API really need to be able to know this. Any comments?
>>
>>     -David
>>
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> <mailto:openstack-qa at lists.openstack.org>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>
>>
>>
>> _______________________________________________
>> openstack-qa mailing list
>> openstack-qa at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>>
>
>


_______________________________________________
openstack-qa mailing list
openstack-qa at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa