<div dir="ltr">Sean, <div><br></div><div><br style="font-size:13px"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span style="font-size:13px">So I'd say let's focus on that problem right now, and get some traction<br></span><span style="font-size:13px">on this as part of functional test suites in OpenStack. Genericizing it<br></span><span style="font-size:13px">too much just turns this back into a version of every other full stack<br></span><span style="font-size:13px">testing tool, which we know isn't sufficient for having quality<br></span><span style="font-size:13px">components in OpenStack.</span></blockquote><div><br></div><div>Please be more specific about what tools were tested? </div></div><div>It will be nice to see overview. At least what tool were tested </div><div>and why they can't be used for testing-in-tree. </div><div><br></div><div><br></div><div>Best regards,</div><div>Boris Pavlovic </div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 13, 2015 at 1:37 AM, Anne Gentle <span dir="ltr"><<a href="mailto:anne@openstack.org" target="_blank">anne@openstack.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Mon, Jan 12, 2015 at 1:20 PM, Chris Dent <span dir="ltr"><<a href="mailto:chdent@redhat.com" target="_blank">chdent@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
After some discussion with Sean Dague and a few others it became<br>
clear that it would be a good idea to introduce a new tool I've been<br>
working on to the list to get a sense of its usefulness generally,<br>
work towards getting it into global requirements, and get the<br>
documentation fleshed out so that people can actually figure out how<br>
to use it well.<br>
<br>
tl;dr: Help me make this interesting tool useful to you and your<br>
HTTP testing by reading this message and following some of the links<br>
and asking any questions that come up.<br>
<br>
The tool is called gabbi<br>
<br>
<a href="https://github.com/cdent/gabbi" target="_blank">https://github.com/cdent/gabbi</a><br>
<a href="http://gabbi.readthedocs.org/" target="_blank">http://gabbi.readthedocs.org/</a><br>
<a href="https://pypi.python.org/pypi/gabbi" target="_blank">https://pypi.python.org/pypi/<u></u>gabbi</a><br>
<br>
It describes itself as a tool for running HTTP tests where requests<br>
and responses are represented in a declarative form. Its main<br>
purpose is to allow testing of APIs where the focus of test writing<br>
(and reading!) is on the HTTP requests and responses, not on a bunch of<br>
Python (that obscures the HTTP).<br>
<br></blockquote><div><br></div></span><div>Hi Chris,</div><div><br></div><div>I'm interested, sure. What did you use to write the HTTP tests, as in, what was the source of truth for what the requests and responses should be?</div><div><br></div><div>Thanks,</div><div>Anne</div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The tests are written in YAML and the simplest test file has this form:<br>
<br>
```<br>
tests:<br>
- name: a test<br>
url: /<br>
```<br>
<br>
This test will pass if the response status code is '200'.<br>
<br>
The test file is loaded by a small amount of python code which transforms<br>
the file into an ordered sequence of TestCases in a TestSuite[1].<br>
<br>
```<br>
def load_tests(loader, tests, pattern):<br>
"""Provide a TestSuite to the discovery process."""<br>
test_dir = os.path.join(os.path.dirname(_<u></u>_file__), TESTS_DIR)<br>
return driver.build_tests(test_dir, loader, host=None,<br>
intercept=SimpleWsgi,<br>
fixture_module=sys.modules[__<u></u>name__])<br>
```<br>
<br>
The loader provides either:<br>
<br>
* a host to which real over-the-network requests are made<br>
* a WSGI app which is wsgi-intercept-ed[2]<br>
<br>
If an individual TestCase is asked to be run by the testrunner, those tests<br>
that are prior to it in the same file are run first, as prerequisites.<br>
<br>
Each test file can declare a sequence of nested fixtures to be loaded<br>
from a configured (in the loader) module. Fixtures are context managers<br>
(they establish the fixture upon __enter__ and destroy it upon<br>
__exit__).<br>
<br>
With a proper group_regex setting in .testr.conf each YAML file can<br>
run in its own process in a concurrent test runner.<br>
<br>
The docs contain information on the format of the test files:<br>
<br>
<a href="http://gabbi.readthedocs.org/en/latest/format.html" target="_blank">http://gabbi.readthedocs.org/<u></u>en/latest/format.html</a><br>
<br>
Each test can state request headers and bodies and evaluate both response<br>
headers and response bodies. Request bodies can be strings in the<br>
YAML, files read from disk, or JSON created from YAML structures.<br>
Response verifcation can use JSONPath[3] to inspect the details of<br>
response bodies. Response header validation may use regular<br>
expressions.<br>
<br>
There is limited support for refering to the previous request<br>
to construct URIs, potentially allowing traversal of a full HATEOAS<br>
compliant API.<br>
<br>
At the moment the most complete examples of how things work are:<br>
<br>
* Ceilometer's pending use of gabbi:<br>
<a href="https://review.openstack.org/#/c/146187/" target="_blank">https://review.openstack.org/#<u></u>/c/146187/</a><br>
* Gabbi's testing of gabbi:<br>
<a href="https://github.com/cdent/gabbi/tree/master/gabbi/gabbits_intercept" target="_blank">https://github.com/cdent/<u></u>gabbi/tree/master/gabbi/<u></u>gabbits_intercept</a><br>
(the loader and faked WSGI app for those yaml files is in:<br>
<a href="https://github.com/cdent/gabbi/blob/master/gabbi/test_intercept.py" target="_blank">https://github.com/cdent/<u></u>gabbi/blob/master/gabbi/test_<u></u>intercept.py</a>)<br>
<br>
One obvious thing that will need to happen is a suite of concrete<br>
examples on how to use the various features. I'm hoping that<br>
feedback will help drive that.<br>
<br>
In my own experimentation with gabbi I've found it very useful. It's<br>
helped me explore and learn the ceilometer API in a way that existing<br>
test code has completely failed to do. It's also helped reveal<br>
several warts that will be very useful to fix. And it is fast. To<br>
run and to write. I hope that with some work it can be useful to you<br>
too.<br>
<br>
Thanks.<br>
<br>
[1] Getting gabbi to play well with PyUnit style tests and<br>
with infrastructure like subunit and testrepository was one of<br>
the most challenging parts of the build, but the result has been<br>
a lot of flexbility.<br>
<br>
[2] <a href="https://pypi.python.org/pypi/wsgi_intercept" target="_blank">https://pypi.python.org/pypi/<u></u>wsgi_intercept</a><br>
[3] <a href="https://pypi.python.org/pypi/jsonpath-rw" target="_blank">https://pypi.python.org/pypi/<u></u>jsonpath-rw</a><span><font color="#888888"><br>
<br>
-- <br>
Chris Dent tw:@anticdent freenode:cdent<br>
<a href="https://tank.peermore.com/tanks/cdent" target="_blank">https://tank.peermore.com/<u></u>tanks/cdent</a><br>
<br>
______________________________<u></u>______________________________<u></u>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.<u></u>openstack.org?subject:<u></u>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</font></span></blockquote></div></div></div><br></div></div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>