Open Stack

Thanks for providing a bit more information on your goals, that
helps shape my answers a bit more usefully. See within.

On Wed, 5 Jul 2017, Shewale, Bhagyashri wrote:

>>> So I'd say the api is verified. What is missing, and could be useful, is using those tests to get accurate and up to date representations of the JSON request and response bodies. If that's something we'd like to pursue as I said in my other message the 'verbose' functionality that can be provided in gabbi->>based tests should be able to help.
>
> I have checked with the 'verbose' functionality, but that doesn't verify the entire response, it's simply prints header and response on console.

That's correct. My suggestion to use the 'verbose' functionality was
if your main goal was to create samples to be used in documentation:
a suite of gabbi tests can run which output those samples as needed,
wherever they need to go.

> On the current master code, few of the gabbi tests verify the entire response and the remaining verify’s only specific attributes from the response.

This is mostly intentional. It doesn't make sense to me that we
would verify serialization code at the level of API tests.

The difference you see in the extent to which each tests validates a
response is the result of different people writing the tests.

> So instead of verifying each and every key-value pair from the response object, it would nice if gabbit can add a support to
> accept response.json file and during execution of the test, it can verify whether all key-value pairs response.json file are matching with the actual response.

While I disagree with doing this (it makes the tests more fragile,
requires more pieces to be changed when things are changed, and tests
the serialization code in the api tests, rather than the API[1]) the
functionality that you're describing can easily exist in gabbi so if
you and others decide that it is useful it can be done, either in
gabbi itself or in a custom content handler[2]. Is it your
intent to compare the abstract nestable object represented by the
JSON, or the string? If it's the latter there's existing code out
there that does that[3] but that's probably not a good choice as it
will break as soon as the format of the JSON changes for some
reason.

(I'm happy to write the code that does full object comparison if we
decide it's desirable or help land a pull request.)

[1] From https://gabbi.readthedocs.io/en/latest/jsonpath.html
     This is not a technique that should be used frequently as it can
     lead to difficult to read tests and it also indicates that your
     gabbi tests are being used to test your serializers and data
     models, not just your API interactions.

[2] https://gabbi.readthedocs.io/en/latest/handlers.html
     And an example:

[3] https://github.com/hogarthww/gabbi-tools/blob/master/src/gabbi_tools/response_handlers.py

-- 
Chris Dent                  ┬──┬◡ﾉ(° -°ﾉ)       https://anticdent.org/
freenode: cdent                                         tw: @anticdent