Open Stack

Realistically this started to come in during the 200 / 202 issues over 
the summer.

I think it's bad form for us to have generic 2xx testing, tempest is 
really about testing exact behavior. My vote would be to -1 these starts 
with tests and make the tests be explicit about what they are supposed 
to return.

This also lets tempest prevent accidental break of the API contract.

	-Sean

On 01/30/2013 06:45 PM, Sam Danes wrote:
> I tend to agree. The specific response code is a contract from the API
> that in condition X, it will return exactly Y. I would give a nod to
> what Daryl is saying that if the response codes were externalized and
> vendor configurable you could make somewhat of an argument to generalize
> the check, however, at that point I would think that the specific
> configuration that the vendor has in place should still be validated
> with the exact codes.
>
> We have had issues as well where a call may return say a 401 instead of
> a 404 which would have caused a downstream issue. A generic check for a
> startswith(‘4’) would have let that through.
>
> --Sam
>
> *From:*Daryl Walleck [mailto:daryl.walleck at RACKSPACE.COM]
> *Sent:* Wednesday, January 30, 2013 4:55 PM
> *To:* openstack-qa at lists.openstack.org; All Things QA.
> *Subject:* Re: [openstack-qa] Return codes for OpenStack APIs
>
> I'm a proponent of checking the exact code. To me, its part of the API
> contract. From a practical point, it has also helped our automated tests
> catch bugs that nearly slipped by. One case I an remember is when some
> various POSTs started returning a 200 code as opposed to a 202, with the
> end result being that the requests responded to with a 200 actually were
> not processed. The only reason I could think of to generalize response
> code checking would be if it the response codes were ever vendor
> configurable.
>
> Daryl
>
> *From:* David Kranz
> *Sent:* ‎January‎ ‎30‎, ‎2013 ‎4‎:‎08‎ ‎PM
> *To:* openstack-qa at lists.openstack.org
> <mailto:openstack-qa at lists.openstack.org>
> *Subject:* [openstack-qa] Return codes for OpenStack APIs
>
> There have been recent submissions to Tempest with code like:
>
> self.assertTrue(resp['status'].startswith('2'))
>
> It had been the consensus that we should be checking for actual status
> codes. Some projects document the return
> codes and others don't. This example was from a call to keystone to list
> services and the result is not documented but the list tenants
> calls says it could return 200 or 203. There should be a statement from
> each project about whether it considers the return code to be part of
> the spec.
> Ideally this would be uniform across projects but it is not. Users of
> the API really need to be able to know this. Any comments?
>
>     -David
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org <mailto:openstack-qa at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>
>
>
> _______________________________________________
> openstack-qa mailing list
> openstack-qa at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-qa
>


-- 
Sean Dague
IBM Linux Technology Center
email: sdague at linux.vnet.ibm.com
alt-email: sldague at us.ibm.com