Open Stack

On 7/14/2016 4:40 PM, Matt Riedemann wrote:
> On 7/14/2016 3:11 AM, GHANSHYAM MANN wrote:
>>>> 1. Always add a schema change to Tempest if a microversion changes a
>>>> response.
>>>
>>> The problem with this is we shouldn't land a schema change by itself
>>> in tempest.
>>> Until we have something using the schema we have no verification that
>>> they
>>> actually work. We can and will land incorrect schemas if we did this.
>>> That's why
>>> there is a pretty strong policy of only landing code that is run in
>>> CI somewhere
>>> for Tempest.
>>
>> +1, yes we should not add those without testing.
>>
>
> OK, good point on not landing changes that aren't tested. That's pretty
> obvious.
>
> For something like this though:
>
> https://review.openstack.org/#/c/339559/
>
> The gate-tempest-dsvm-neutron-full-ssh is testing it indirectly, so I'm
> assuming that's OK even though we don't have an explicit test for the
> 2.3 microversion?
>
> I know the patch needs to be updated for the other extended server
> attributes in that microversion, but it's the immediate thing I want to
> get fixed so we can get on with making the
> gate-tempest-dsvm-neutron-full-ssh job voting.
>

We talked about this topic at the nova midcycle and these are the 
notes/decisions I took:

* We can't have schema changes in Tempest that aren't tested - this is 
already the Tempest policy and makes sense.
* When adding tests for a new microversion, if there is a gap in Tempest 
response schema validation it should be filled in that patch.
* After feature freeze we should fill any gaps between nova's latest 
microversion and the schema coverage in Tempest. Right now we have a 
backlog so we're playing catch up, but that shouldn't happen once we get 
caught up.
* It's fine to have microversion tests in Tempest even if they only hit 
the nova API/DB because we want the response schema validation (and to 
avoid these gaps in coverage).

So I think we can move forward on filling the gap in Tempest (there are 
several open changes for review).

I'll also push a docs change to nova [1] to mention that a Tempest test 
needs to be added for any microversion which changes the response schema.

[1] 
http://docs.openstack.org/developer/nova/code-review.html#microversion-api

-- 

Thanks,

Matt Riedemann