Open Stack

>> * who or what needs to consume these JSON samples?
The users of placement API can rely on the request/response for different supported placement versions based on some tests running on the OpenStack CI infrastructure. 
Right now, most of the placement APIs are well documented and others are in progress but there are no tests to verify these APIs.
We would like to write new functional test to consume these json samples to verify each placement API for all supported versions.

>>do they need to be tests against current code, or are they  primarily reference info?
Yes, the request/response json files for each supported version should be run against the current master code.

>>what are the value propositions associated with fully validating the structure and content of the response bodies?
It's kind of validation and verification mechanism available on OpenStack CI which can be fully trusted by the users of placement API and some of the other value propositions are mentioned by Rochelle Grober in his reply.

Thanks for your response.

Regards,
Bhagyashri Shewale

-----Original Message-----
From: Chris Dent [mailto:cdent+os at anticdent.org] 
Sent: Thursday, June 22, 2017 3:00 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova][placement] Add API Sample tests for Placement APIs

On Wed, 21 Jun 2017, Shewale, Bhagyashri wrote:

> I  would like to write functional tests to check the exact req/resp 
> for each placement API for all supported versions similar to what is already done for other APIs under nova/tests/functional/api_sample_tests/api_samples/*.
> These request/response json samples can be used by the api.openstack.org and in the manuals.
>
> There are already functional tests written for placement APIs under 
> nova/tests/functional/api/openstack/placement,
> but these tests doesn't check the entire HTTP response for each API for all supported versions.
>
> I think adding such functional tests for checking response for each placement API would be beneficial to the project.
> If there is an interest to create such functional tests, I can file a new blueprint for this activity.

At Matt points out elsewhere, we made a choice to not use the api_samples format when developing placement. There were a few different reasons for this:

* we wanted to limit the amount of nova code used in placement, to
   ease the eventual extraction of placement to its own code
   repository

* some of us wanted to use gabbi [1] as the testing framework as it is
   nicely declarative [2] and keeps the request and response in the same
   place

* we were building the api framework from scratch and doing what
   amounts to test driven development [2] using functional tests and
   gabbi works well for that

* testing the full response isn't actually a great way to test an
   API in a granular way; the info is useful to have but it isn't a
   useful test (from a development standpoint)

But, as you've noted, this means there isn't a single place to go to see a collection of a full request and response bodies. That information can be extracted from the gabbi tests, but it's a) not immediately obvious, b) requires interpretation.

Quite some time ago I started a gabbi-based full request and response suite of tests [3] but it was never finished and now is very out of date.

If the end goal is to have a set of documents that pair all the possible requests (with bodies) with all possible responses (with bodies), gabbi could easily create this in its "verbose" mode [4] when run as functional tests or with the gabbi-run [5] command that can run against a running service.

So I would suggest that we more completely explain the goal or goals that you're trying to satisfy and then see how we can use the existing tooling to fill them. Some questions related to that:

* who or what needs to consume these JSON samples?
* do they need to be tests against current code, or are they
   primarily reference info?
* what are the value propositions associated with fully validating
   the structure and content of the response bodies?

We can relatively easily figure out some way to drive gabbi to produce the desired information, but first we want to make sure that the information produced is going to be the right info (that is, will satisfy the needs of whoever wants it).

I am, as Matt mentioned, on holiday at the moment so my response to any future messages may be delayed, but I'll catch up as I'm able.

[1] https://gabbi.readthedocs.io/en/latest/
[2] https://github.com/openstack/nova/tree/master/nova/tests/functional/api/openstack/placement/gabbits
[3] https://review.openstack.org/#/c/370204/
[4] verbose mode can print out request and response headers, bodies,
     or both. If the bodies are JSON, it will be pretty printed.
[5] https://gabbi.readthedocs.io/en/latest/runner.html

-- 
Chris Dent                  ┬──┬◡ﾉ(° -°ﾉ)       https://anticdent.org/
freenode: cdent                                         tw: @anticdent

______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.