Open Stack

On 11/16/2012 7:49 AM, Sean Dague wrote:
> After talking with Dave yesterday I was looking through 
> https://bugs.launchpad.net/nova/+bug/1079210 which has a link to a log 
> file with a lot of stack traces in nova-api
>
> I picked one at mostly random and worked through it, now fixed here - 
> https://bugs.launchpad.net/nova/+bug/1079387, tempest change to 
> confirm the behavior ready for review here - 
> https://review.openstack.org/#/c/16300/
>
> The root cause of Tempest passing, but nova exploding, is the fact 
> that our negative tests in Tempest are really broad. Any fail is 
> acceptable. So in this case a nova-api 500 and stack trace was deemed 
> a-ok.
>
> But most of these failures should happen in a controlled way, with a 
> known error state. The value of tempest in the gate goes way up in 
> these cases.
>
> I think we need a blueprint for making error codes specific in tempest 
> during the grizzly cycle. Thoughts?
>
> We should also be a bit more careful on reviews to make sure people 
> are being specific about their fail conditions in the future.
>
>     -Sean
>
The negative tests are supposed to expect a particular failure code but 
evidently a bunch of them do not. Presumably when
we move to fuzz testing the return code will be a required part of the 
spec for a test case, so I think your suggestion would
be part of the blueprint for fuzz testing. There wasn't one so I just 
added it https://blueprints.launchpad.net/tempest/+spec/fuzz-testing.

Once we can get a consistent passing run with no ERRORs in the log, we 
can change devstack-gate to fail if any ERRORs show up. Once such code 
is released,
operators will be able to set an alert on errors in the logs which will 
help them discover operator issues as well as get
better bug reporting.

  -David