[openstack-dev] [all] [api] gabbi: A tool for declarative testing of APIs

Chris Dent chdent at redhat.com
Mon Mar 2 11:56:36 UTC 2015

I (and a few others) have been using gabbi[1] for a couple of months now
and it has proven very useful and evolved a bit so I thought it would be
worthwhile to followup my original message and give an update.

Some recent reviews[1] give a sample of how it can be used to validate
an existing API as well as search for less than perfect HTTP behavior
(e.g sending a 404 when a 405 would be correct).

Regular use has lead to some important changes:

* It can now be integrated with other tox targets so it can run
   alongside other functional tests.
* Individual tests can be xfailed and skipped. An entire YAML test
   file can be skipped.
* For those APIs which provide insufficient hypermedia support, the
   ability to inspect and reference the prior test and use template
   variables in the current request has been expanded (with support for
   environment variables pending a merge).

My original motivation for creating the tool was to make it easier to
learn APIs by causing a body of readable YAML files to exist. This
remains important but what I've found is that writing the tests is
itself an incredible tool. Not only is it very easy to write tests
(throw some stuff at a URL and see what happen) and find (many) bugs
as a result, the exploratory nature of test writing drives a
learning process.

You'll note that the reviews below are just the YAML files. That's
because the test loading and fixture python code is already merged.
Adding tests is just a matter of adding more YAML. An interesting
trick is to run a small segment of the gabbi tests in a project (e.g.
just one file that represents one type of resource) while producing
coverage data. Reviewing the coverage of just the controller for that
resource can help drive test creation and separation.

[1] http://gabbi.readthedocs.org/en/latest/
[2] https://review.openstack.org/#/c/159945/

Chris Dent tw:@anticdent freenode:cdent

More information about the OpenStack-dev mailing list