<div dir="ltr">Chris, <div><br></div><div>The Idea is brilliant. I may steal it! =)</div><div><br></div><div>But there are some issues that will be faced: </div><div><br></div><div>1) Using as a base unittest: </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">python -m subunit.run discover -f gabbi | subunit2pyunit</blockquote><div><br></div><div>So rally team won't be able to reuse it for load testing (if we directly integrate it) because we will have huge overhead (of discover stuff)</div><div><br></div><div>2) Load testing. </div><div><br></div><div>Using unittest for functional testing adds a lot of troubles: </div><div>2.1) It makes things complicated: </div><div>Like reusing fixtures via input YAML will be painfull</div><div>2.2) It adds a lot of functionality that is not required </div>2.3) It makes it hardly integratabtle with other tools. Like Rally.. <div><span style="color:rgb(119,119,119);font-family:'Helvetica Neue',Helvetica,'Segoe UI',Arial,freesans,sans-serif;font-size:16px;line-height:25.6000003814697px"><br></span></div><div>3) Usage by Operators is hard in case of N projects. </div><div><br></div><div>So you should have some kind of </div><div><br></div><div>Operators would like to have 1 button that will say (does cloud work or not). And they don't want to combine all gabbi files from all projects and run test. </div><div><br></div><div>From other side there should be a way to write such code in-projects-tree (so new features are directly tested) and then moved to some common place that is run on every patch (without breaking gates) </div><div><br></div><div>4) Using subunit format is not good for functional testing.</div><div> </div><div>It doesn't allow you to collect detailed information about execution of test. Like for benchmarking it will be quite interesting to collect durations of every API call. </div><div><br></div><div><br></div><div><br></div><div>Best regards,</div><div>Boris Pavlovic </div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 12, 2015 at 10:54 PM, Eoghan Glynn <span dir="ltr"><<a href="mailto:eglynn@redhat.com" target="_blank">eglynn@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
<br>
> After some discussion with Sean Dague and a few others it became<br>
> clear that it would be a good idea to introduce a new tool I've been<br>
> working on to the list to get a sense of its usefulness generally,<br>
> work towards getting it into global requirements, and get the<br>
> documentation fleshed out so that people can actually figure out how<br>
> to use it well.<br>
><br>
> tl;dr: Help me make this interesting tool useful to you and your<br>
> HTTP testing by reading this message and following some of the links<br>
> and asking any questions that come up.<br>
><br>
> The tool is called gabbi<br>
><br>
> <a href="https://github.com/cdent/gabbi" target="_blank">https://github.com/cdent/gabbi</a><br>
> <a href="http://gabbi.readthedocs.org/" target="_blank">http://gabbi.readthedocs.org/</a><br>
> <a href="https://pypi.python.org/pypi/gabbi" target="_blank">https://pypi.python.org/pypi/gabbi</a><br>
><br>
> It describes itself as a tool for running HTTP tests where requests<br>
> and responses are represented in a declarative form. Its main<br>
> purpose is to allow testing of APIs where the focus of test writing<br>
> (and reading!) is on the HTTP requests and responses, not on a bunch of<br>
> Python (that obscures the HTTP).<br>
><br>
> The tests are written in YAML and the simplest test file has this form:<br>
><br>
> ```<br>
> tests:<br>
> - name: a test<br>
> url: /<br>
> ```<br>
><br>
> This test will pass if the response status code is '200'.<br>
><br>
> The test file is loaded by a small amount of python code which transforms<br>
> the file into an ordered sequence of TestCases in a TestSuite[1].<br>
><br>
> ```<br>
> def load_tests(loader, tests, pattern):<br>
> """Provide a TestSuite to the discovery process."""<br>
> test_dir = os.path.join(os.path.dirname(__file__), TESTS_DIR)<br>
> return driver.build_tests(test_dir, loader, host=None,<br>
> intercept=SimpleWsgi,<br>
> fixture_module=sys.modules[__name__])<br>
> ```<br>
><br>
> The loader provides either:<br>
><br>
> * a host to which real over-the-network requests are made<br>
> * a WSGI app which is wsgi-intercept-ed[2]<br>
><br>
> If an individual TestCase is asked to be run by the testrunner, those tests<br>
> that are prior to it in the same file are run first, as prerequisites.<br>
><br>
> Each test file can declare a sequence of nested fixtures to be loaded<br>
> from a configured (in the loader) module. Fixtures are context managers<br>
> (they establish the fixture upon __enter__ and destroy it upon<br>
> __exit__).<br>
><br>
> With a proper group_regex setting in .testr.conf each YAML file can<br>
> run in its own process in a concurrent test runner.<br>
><br>
> The docs contain information on the format of the test files:<br>
><br>
> <a href="http://gabbi.readthedocs.org/en/latest/format.html" target="_blank">http://gabbi.readthedocs.org/en/latest/format.html</a><br>
><br>
> Each test can state request headers and bodies and evaluate both response<br>
> headers and response bodies. Request bodies can be strings in the<br>
> YAML, files read from disk, or JSON created from YAML structures.<br>
> Response verifcation can use JSONPath[3] to inspect the details of<br>
> response bodies. Response header validation may use regular<br>
> expressions.<br>
><br>
> There is limited support for refering to the previous request<br>
> to construct URIs, potentially allowing traversal of a full HATEOAS<br>
> compliant API.<br>
><br>
> At the moment the most complete examples of how things work are:<br>
><br>
> * Ceilometer's pending use of gabbi:<br>
> <a href="https://review.openstack.org/#/c/146187/" target="_blank">https://review.openstack.org/#/c/146187/</a><br>
> * Gabbi's testing of gabbi:<br>
> <a href="https://github.com/cdent/gabbi/tree/master/gabbi/gabbits_intercept" target="_blank">https://github.com/cdent/gabbi/tree/master/gabbi/gabbits_intercept</a><br>
> (the loader and faked WSGI app for those yaml files is in:<br>
> <a href="https://github.com/cdent/gabbi/blob/master/gabbi/test_intercept.py" target="_blank">https://github.com/cdent/gabbi/blob/master/gabbi/test_intercept.py</a>)<br>
><br>
> One obvious thing that will need to happen is a suite of concrete<br>
> examples on how to use the various features. I'm hoping that<br>
> feedback will help drive that.<br>
><br>
> In my own experimentation with gabbi I've found it very useful. It's<br>
> helped me explore and learn the ceilometer API in a way that existing<br>
> test code has completely failed to do. It's also helped reveal<br>
> several warts that will be very useful to fix. And it is fast. To<br>
> run and to write. I hope that with some work it can be useful to you<br>
> too.<br>
<br>
</div></div>Thanks for the write-up Chris,<br>
<br>
Needless to say, we're sold on the utility of this on the ceilometer<br>
side, in terms of crafting readable, self-documenting tests that reveal<br>
the core aspects of an API in a easily consumable way.<br>
<br>
I'd be interested in hearing the api-wg viewpoint, specifically whether<br>
that working group intends to recommend any best practices around the<br>
approach to API testing.<br>
<br>
If so, I think gabbi would be a worthy candidate for consideration.<br>
<br>
Cheers,<br>
Eoghan<br>
<div class="HOEnZb"><div class="h5"><br>
> Thanks.<br>
><br>
> [1] Getting gabbi to play well with PyUnit style tests and<br>
> with infrastructure like subunit and testrepository was one of<br>
> the most challenging parts of the build, but the result has been<br>
> a lot of flexbility.<br>
><br>
> [2] <a href="https://pypi.python.org/pypi/wsgi_intercept" target="_blank">https://pypi.python.org/pypi/wsgi_intercept</a><br>
> [3] <a href="https://pypi.python.org/pypi/jsonpath-rw" target="_blank">https://pypi.python.org/pypi/jsonpath-rw</a><br>
><br>
> --<br>
> Chris Dent tw:@anticdent freenode:cdent<br>
> <a href="https://tank.peermore.com/tanks/cdent" target="_blank">https://tank.peermore.com/tanks/cdent</a><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>