Open Stack

On 01/12/2015 02:20 PM, Chris Dent wrote:
>
> After some discussion with Sean Dague and a few others it became
> clear that it would be a good idea to introduce a new tool I've been
> working on to the list to get a sense of its usefulness generally,
> work towards getting it into global requirements, and get the
> documentation fleshed out so that people can actually figure out how
> to use it well.
>
> tl;dr: Help me make this interesting tool useful to you and your
> HTTP testing by reading this message and following some of the links
> and asking any questions that come up.
>
> The tool is called gabbi
>
>      https://github.com/cdent/gabbi
>      http://gabbi.readthedocs.org/
>      https://pypi.python.org/pypi/gabbi
>
> It describes itself as a tool for running HTTP tests where requests
> and responses are represented in a declarative form. Its main
> purpose is to allow testing of APIs where the focus of test writing
> (and reading!) is on the HTTP requests and responses, not on a bunch of
> Python (that obscures the HTTP).
>
> The tests are written in YAML and the simplest test file has this form:
>
> ```
> tests:
> - name: a test
>    url: /
> ```
>
> This test will pass if the response status code is '200'.
>
> The test file is loaded by a small amount of python code which transforms
> the file into an ordered sequence of TestCases in a TestSuite[1].
>
> ```
> def load_tests(loader, tests, pattern):
>      """Provide a TestSuite to the discovery process."""
>          test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
>          return driver.build_tests(test_dir, loader, host=None,
>                                    intercept=SimpleWsgi,
>                                    fixture_module=sys.modules[__name__])
> ```
>
> The loader provides either:
>
> * a host to which real over-the-network requests are made
> * a WSGI app which is wsgi-intercept-ed[2]
>
> If an individual TestCase is asked to be run by the testrunner, those tests
> that are prior to it in the same file are run first, as prerequisites.
>
> Each test file can declare a sequence of nested fixtures to be loaded
> from a configured (in the loader) module. Fixtures are context managers
> (they establish the fixture upon __enter__ and destroy it upon
> __exit__).
>
> With a proper group_regex setting in .testr.conf each YAML file can
> run in its own process in a concurrent test runner.
>
> The docs contain information on the format of the test files:
>
>      http://gabbi.readthedocs.org/en/latest/format.html
>
> Each test can state request headers and bodies and evaluate both response
> headers and response bodies. Request bodies can be strings in the
> YAML, files read from disk, or JSON created from YAML structures.
> Response verifcation can use JSONPath[3] to inspect the details of
> response bodies. Response header validation may use regular
> expressions.
>
> There is limited support for refering to the previous request
> to construct URIs, potentially allowing traversal of a full HATEOAS
> compliant API.
>
> At the moment the most complete examples of how things work are:
>
> * Ceilometer's pending use of gabbi:
>    https://review.openstack.org/#/c/146187/
> * Gabbi's testing of gabbi:
>    https://github.com/cdent/gabbi/tree/master/gabbi/gabbits_intercept
>    (the loader and faked WSGI app for those yaml files is in:
>    https://github.com/cdent/gabbi/blob/master/gabbi/test_intercept.py)
>
> One obvious thing that will need to happen is a suite of concrete
> examples on how to use the various features. I'm hoping that
> feedback will help drive that.
>
> In my own experimentation with gabbi I've found it very useful. It's
> helped me explore and learn the ceilometer API in a way that existing
> test code has completely failed to do. It's also helped reveal
> several warts that will be very useful to fix. And it is fast. To
> run and to write. I hope that with some work it can be useful to you
> too.

Very impressive, Chris, thanks very much for bringing Gabbi into the 
OpenStack ecosystem. I very much look forward to replacing the API 
samples code in Nova with Gabbi, which looks very clean and 
easily-understandable for anyone.

Best,
-jay