Open Stack

Thanks for your opinion and I totally agree with your suggestion to generate docs based on the verbose functionality for accuracy.
but for one thing that I have observed is:
In the current gabbi tests If nova is returning extra parameters for a specific micro-version, then the tests won't fail because it is only validating required attributes from the response.

For example:
Imagine one api with version x returning 5 attributes in a response and micro-version x.1 returning 6 attributes in a response and for some reasons there is a regression
and the micro-version x starts returning 6 attributes instead of 5, then I expect the tests should fail but as per the current tests,
 it won't be detected as we are only verifying only the required attributes from that specific micro-version.

Maybe that the only case left out which I believe can be addressed if we decide to valid the actual response with the response.json.
Any thoughts about how to handle this case in gabbit tests?

Regards,
Bhagyashri Shewale

-----Original Message-----
From: Chris Dent [mailto:cdent+os at anticdent.org] 
Sent: Wednesday, July 5, 2017 5:03 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova][placement] Add API Sample tests for Placement APIs


Thanks for providing a bit more information on your goals, that helps shape my answers a bit more usefully. See within.

On Wed, 5 Jul 2017, Shewale, Bhagyashri wrote:

>>> So I'd say the api is verified. What is missing, and could be useful, is using those tests to get accurate and up to date representations of the JSON request and response bodies. If that's something we'd like to pursue as I said in my other message the 'verbose' functionality that can be provided in gabbi->>based tests should be able to help.
>
> I have checked with the 'verbose' functionality, but that doesn't verify the entire response, it's simply prints header and response on console.

That's correct. My suggestion to use the 'verbose' functionality was if your main goal was to create samples to be used in documentation:
a suite of gabbi tests can run which output those samples as needed, wherever they need to go.

> On the current master code, few of the gabbi tests verify the entire response and the remaining verify’s only specific attributes from the response.

This is mostly intentional. It doesn't make sense to me that we would verify serialization code at the level of API tests.

The difference you see in the extent to which each tests validates a response is the result of different people writing the tests.

> So instead of verifying each and every key-value pair from the 
> response object, it would nice if gabbit can add a support to accept response.json file and during execution of the test, it can verify whether all key-value pairs response.json file are matching with the actual response.

While I disagree with doing this (it makes the tests more fragile, requires more pieces to be changed when things are changed, and tests the serialization code in the api tests, rather than the API[1]) the functionality that you're describing can easily exist in gabbi so if you and others decide that it is useful it can be done, either in gabbi itself or in a custom content handler[2]. Is it your intent to compare the abstract nestable object represented by the JSON, or the string? If it's the latter there's existing code out there that does that[3] but that's probably not a good choice as it will break as soon as the format of the JSON changes for some reason.

(I'm happy to write the code that does full object comparison if we decide it's desirable or help land a pull request.)

[1] From https://gabbi.readthedocs.io/en/latest/jsonpath.html
     This is not a technique that should be used frequently as it can
     lead to difficult to read tests and it also indicates that your
     gabbi tests are being used to test your serializers and data
     models, not just your API interactions.

[2] https://gabbi.readthedocs.io/en/latest/handlers.html
     And an example:

[3] https://github.com/hogarthww/gabbi-tools/blob/master/src/gabbi_tools/response_handlers.py

-- 
Chris Dent                  ┬──┬◡ﾉ(° -°ﾉ)       https://anticdent.org/
freenode: cdent                                         tw: @anticdent

______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.