Open Stack

>>So I'd say the api is verified. What is missing, and could be useful, is using those tests to get accurate and up to date representations of the JSON request and response bodies. If that's something we'd like to pursue as I said in my other message the 'verbose' functionality that can be provided in gabbi->>based tests should be able to help.

I have checked with the 'verbose' functionality, but that doesn't verify the entire response, it's simply prints header and response on console.

On the current master code, few of the gabbi tests verify the entire response and the remaining verify’s only specific attributes from the response.

For example:

1. The test "update a resource provider" which is under "nova/tests/functional/api/openstack/placement/gabbits/resource-provider.yaml" does not check all the key-value pairs of "links" attribute, it only checks the "links[?rel = "self"].href".

Response return after update
{
  "links": [
    {
      "href": "/resource_providers/8e8b14f1-1360-4b61-8ff0-aeefc73a1808",
      "rel": "self"
    },
    {
      "href": "/resource_providers/8e8b14f1-1360-4b61-8ff0-aeefc73a1808/inventories",
      "rel": "inventories"
    },
    {
      "href": "/resource_providers/8e8b14f1-1360-4b61-8ff0-aeefc73a1808/usages",
      "rel": "usages"
    }
  ],
  "generation": 0,
  "uuid": "8e8b14f1-1360-4b61-8ff0-aeefc73a1808",
  "name": "new name"
}

2. The test "list one resource providers" under nova/tests/functional/api/openstack/placement/gabbits/resource-provider.yaml checks all the key-value pairs of response object.

Response return in case of list one resource providers

{
  "resource_providers": [
    {
      "links": [
        {
          "href": "/resource_providers/958f0cb7-8fa2-4521-95df-e4cff1cc0d07",
          "rel": "self"
        },
        {
          "href": "/resource_providers/958f0cb7-8fa2-4521-95df-e4cff1cc0d07/inventories",
          "rel": "inventories"
        },
        {
          "href": "/resource_providers/958f0cb7-8fa2-4521-95df-e4cff1cc0d07/usages",
          "rel": "usages"
        }
      ],
      "name": "0e6980e1-501a-468c-9d9a-bce6caecffb3",
      "generation": 0,
      "uuid": "958f0cb7-8fa2-4521-95df-e4cff1cc0d07"
    }
  ]
}

So instead of verifying each and every key-value pair from the response object, it would nice if gabbit can add a support to
accept response.json file and during execution of the test, it can verify whether all key-value pairs response.json file are matching with the actual response.

In the current master code, for a specific micro-version, if nova is returning extra parameters, then the gabbit test won't fail because it is only validating required attributes from the response.
I think, this issue can be handled once we do strict validation of actual response with the response.json.

Please let me know your thoughts of adding support in gabbi project to accept response.json file for each test.

Regards,
Bhagyashri Shewale

-----Original Message-----
From: Chris Dent [mailto:cdent+os at anticdent.org] 
Sent: Friday, June 23, 2017 5:35 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [nova][placement] Add API Sample tests for Placement APIs

On Thu, 22 Jun 2017, Shewale, Bhagyashri wrote:

>>> * who or what needs to consume these JSON samples?
> The users of placement API can rely on the request/response for different supported placement versions based on some tests running on the OpenStack CI infrastructure.
> Right now, most of the placement APIs are well documented and others are in progress but there are no tests to verify these APIs.

Either we are misunderstanding each other, or you're not aware of what the gabbi tests are doing. They verify the placement API and provide extensive coverage of the entire placement HTTP framework, including the accuracy of response codes in edge cases not on the "happy path". Coverage is well over 90% for the group of files in nova/api/openstack/placement (except for the wsgi deployment script
itself) when the
nova/tests/functional/api/openstack/placement/test_placement_api.py
functionl tests runs all the gabbi files in nova/tests/functional/api/openstack/placement/gabbits/.

So I'd say the api is verified. What is missing, and could be useful, is using those tests to get accurate and up to date representations of the JSON request and response bodies. If that's something we'd like to pursue as I said in my other message the 'verbose' functionality that can be provided in gabbi-based tests should be able to help.

> We would like to write new functional test to consume these json samples to verify each placement API for all supported versions.

Those gabbi files also test functionality at micorversion boundaries.

-- 
Chris Dent                  ┬──┬◡ﾉ(° -°ﾉ)       https://anticdent.org/
freenode: cdent                                         tw: @anticdent

______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.