Open Stack

On Wed, 21 Jun 2017, Shewale, Bhagyashri wrote:

> I  would like to write functional tests to check the exact req/resp for each placement API for all supported versions similar
> to what is already done for other APIs under nova/tests/functional/api_sample_tests/api_samples/*.
> These request/response json samples can be used by the api.openstack.org and in the manuals.
>
> There are already functional tests written for placement APIs under nova/tests/functional/api/openstack/placement,
> but these tests doesn't check the entire HTTP response for each API for all supported versions.
>
> I think adding such functional tests for checking response for each placement API would be beneficial to the project.
> If there is an interest to create such functional tests, I can file a new blueprint for this activity.

At Matt points out elsewhere, we made a choice to not use the
api_samples format when developing placement. There were a few
different reasons for this:

* we wanted to limit the amount of nova code used in placement, to
   ease the eventual extraction of placement to its own code
   repository

* some of us wanted to use gabbi [1] as the testing framework as it is
   nicely declarative [2] and keeps the request and response in the same
   place

* we were building the api framework from scratch and doing what
   amounts to test driven development [2] using functional tests and
   gabbi works well for that

* testing the full response isn't actually a great way to test an
   API in a granular way; the info is useful to have but it isn't a
   useful test (from a development standpoint)

But, as you've noted, this means there isn't a single place to go to
see a collection of a full request and response bodies. That
information can be extracted from the gabbi tests, but it's a) not
immediately obvious, b) requires interpretation.

Quite some time ago I started a gabbi-based full request and
response suite of tests [3] but it was never finished and now is
very out of date.

If the end goal is to have a set of documents that pair all the
possible requests (with bodies) with all possible responses (with
bodies), gabbi could easily create this in its "verbose" mode [4]
when run as functional tests or with the gabbi-run [5] command that
can run against a running service.

So I would suggest that we more completely explain the goal or goals
that you're trying to satisfy and then see how we can use the
existing tooling to fill them. Some questions related to that:

* who or what needs to consume these JSON samples?
* do they need to be tests against current code, or are they
   primarily reference info?
* what are the value propositions associated with fully validating
   the structure and content of the response bodies?

We can relatively easily figure out some way to drive gabbi to
produce the desired information, but first we want to make sure that
the information produced is going to be the right info (that is,
will satisfy the needs of whoever wants it).

I am, as Matt mentioned, on holiday at the moment so my response to
any future messages may be delayed, but I'll catch up as I'm able.

[1] https://gabbi.readthedocs.io/en/latest/
[2] https://github.com/openstack/nova/tree/master/nova/tests/functional/api/openstack/placement/gabbits
[3] https://review.openstack.org/#/c/370204/
[4] verbose mode can print out request and response headers, bodies,
     or both. If the bodies are JSON, it will be pretty printed.
[5] https://gabbi.readthedocs.io/en/latest/runner.html

-- 
Chris Dent                  ┬──┬◡ﾉ(° -°ﾉ)       https://anticdent.org/
freenode: cdent                                         tw: @anticdent