Open Stack

Chris,

The Idea is brilliant. I may steal it! =)

But there are some issues that will be faced:

1) Using as a base unittest:

> python -m subunit.run discover -f gabbi | subunit2pyunit


So rally team won't be able to reuse it for load testing (if we directly
integrate it) because we will have huge overhead (of discover stuff)

2) Load testing.

Using unittest for functional testing adds a lot of troubles:
2.1) It makes things complicated:
Like reusing fixtures via input YAML will be painfull
2.2) It adds a lot of functionality that is not required
2.3) It makes it hardly integratabtle with other tools. Like Rally..

3) Usage by Operators is hard in case of N projects.

So you should have some kind of

Operators would like to have 1 button that will say (does cloud work or
not). And they don't want to combine all gabbi files from all projects and
run test.

>From other side there should be a way to write such code in-projects-tree
(so new features are directly tested) and then moved to some common place
that is run on every patch (without breaking gates)

4) Using subunit format is not good for functional testing.

It doesn't allow you to collect detailed information about execution of
test. Like for benchmarking it will be quite interesting to collect
durations of every API call.



Best regards,
Boris Pavlovic


On Mon, Jan 12, 2015 at 10:54 PM, Eoghan Glynn <eglynn at redhat.com> wrote:

>
>
> > After some discussion with Sean Dague and a few others it became
> > clear that it would be a good idea to introduce a new tool I've been
> > working on to the list to get a sense of its usefulness generally,
> > work towards getting it into global requirements, and get the
> > documentation fleshed out so that people can actually figure out how
> > to use it well.
> >
> > tl;dr: Help me make this interesting tool useful to you and your
> > HTTP testing by reading this message and following some of the links
> > and asking any questions that come up.
> >
> > The tool is called gabbi
> >
> >      https://github.com/cdent/gabbi
> >      http://gabbi.readthedocs.org/
> >      https://pypi.python.org/pypi/gabbi
> >
> > It describes itself as a tool for running HTTP tests where requests
> > and responses are represented in a declarative form. Its main
> > purpose is to allow testing of APIs where the focus of test writing
> > (and reading!) is on the HTTP requests and responses, not on a bunch of
> > Python (that obscures the HTTP).
> >
> > The tests are written in YAML and the simplest test file has this form:
> >
> > ```
> > tests:
> > - name: a test
> >    url: /
> > ```
> >
> > This test will pass if the response status code is '200'.
> >
> > The test file is loaded by a small amount of python code which transforms
> > the file into an ordered sequence of TestCases in a TestSuite[1].
> >
> > ```
> > def load_tests(loader, tests, pattern):
> >      """Provide a TestSuite to the discovery process."""
> >          test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)
> >          return driver.build_tests(test_dir, loader, host=None,
> >                                    intercept=SimpleWsgi,
> >                                    fixture_module=sys.modules[__name__])
> > ```
> >
> > The loader provides either:
> >
> > * a host to which real over-the-network requests are made
> > * a WSGI app which is wsgi-intercept-ed[2]
> >
> > If an individual TestCase is asked to be run by the testrunner, those
> tests
> > that are prior to it in the same file are run first, as prerequisites.
> >
> > Each test file can declare a sequence of nested fixtures to be loaded
> > from a configured (in the loader) module. Fixtures are context managers
> > (they establish the fixture upon __enter__ and destroy it upon
> > __exit__).
> >
> > With a proper group_regex setting in .testr.conf each YAML file can
> > run in its own process in a concurrent test runner.
> >
> > The docs contain information on the format of the test files:
> >
> >      http://gabbi.readthedocs.org/en/latest/format.html
> >
> > Each test can state request headers and bodies and evaluate both response
> > headers and response bodies. Request bodies can be strings in the
> > YAML, files read from disk, or JSON created from YAML structures.
> > Response verifcation can use JSONPath[3] to inspect the details of
> > response bodies. Response header validation may use regular
> > expressions.
> >
> > There is limited support for refering to the previous request
> > to construct URIs, potentially allowing traversal of a full HATEOAS
> > compliant API.
> >
> > At the moment the most complete examples of how things work are:
> >
> > * Ceilometer's pending use of gabbi:
> >    https://review.openstack.org/#/c/146187/
> > * Gabbi's testing of gabbi:
> >    https://github.com/cdent/gabbi/tree/master/gabbi/gabbits_intercept
> >    (the loader and faked WSGI app for those yaml files is in:
> >    https://github.com/cdent/gabbi/blob/master/gabbi/test_intercept.py)
> >
> > One obvious thing that will need to happen is a suite of concrete
> > examples on how to use the various features. I'm hoping that
> > feedback will help drive that.
> >
> > In my own experimentation with gabbi I've found it very useful. It's
> > helped me explore and learn the ceilometer API in a way that existing
> > test code has completely failed to do. It's also helped reveal
> > several warts that will be very useful to fix. And it is fast. To
> > run and to write. I hope that with some work it can be useful to you
> > too.
>
> Thanks for the write-up Chris,
>
> Needless to say, we're sold on the utility of this on the ceilometer
> side, in terms of crafting readable, self-documenting tests that reveal
> the core aspects of an API in a easily consumable way.
>
> I'd be interested in hearing the api-wg viewpoint, specifically whether
> that working group intends to recommend any best practices around the
> approach to API testing.
>
> If so, I think gabbi would be a worthy candidate for consideration.
>
> Cheers,
> Eoghan
>
> > Thanks.
> >
> > [1] Getting gabbi to play well with PyUnit style tests and
> >      with infrastructure like subunit and testrepository was one of
> >      the most challenging parts of the build, but the result has been
> >      a lot of flexbility.
> >
> > [2] https://pypi.python.org/pypi/wsgi_intercept
> > [3] https://pypi.python.org/pypi/jsonpath-rw
> >
> > --
> > Chris Dent tw:@anticdent freenode:cdent
> > https://tank.peermore.com/tanks/cdent
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150113/7cea299c/attachment.html>