Open Stack

On 04/08/2015 05:24 AM, Sean Dague wrote:
> On 04/08/2015 07:38 AM, Dmitry Tantsur wrote:
>> On 04/08/2015 12:53 PM, Sean Dague wrote:
>>> On 04/08/2015 03:58 AM, Dmitry Tantsur wrote:
>>>> On 04/08/2015 06:23 AM, Ken'ichi Ohmichi wrote:
>>>>> Hi,
>>>>>
>>>>> Now Nova and Ironic have implemented API microversions in Kilo.
>>>>> Nova's microversions are v2.1 - v2.3.
>>>>> Ironic's microversions are v1.1 - v1.6.
>>>>>
>>>>> Now Tempest is testing the lowest microversion on the gate, and
>>>>> Ironic's microversions test patch[1] is on the gerrit.
>>>>> Before merging the patch, I'd like to propose consistent test way for
>>>>> microversions of Nova and Ironic.
>>>>>
>>>>> My suggestion is the test target microversions are:
>>>>> * the lowest microversion
>>>>> * the biggest microversion, but don't use the keyword "latest" on a
>>>>> header and these microversions tests are operated on different gate
>>>>> jobs.
>>>>>
>>>>> The lowest microversion is already tested on check-tempest-dsvm-full
>>>>> or something, so this proposes just to add the biggest microversion
>>>>> job like check-tempest-dsvm-full-big-microversion.
>>>>>
>>>>> [background]
>>>>> In long-term, these microversions continue increasing and it is
>>>>> difficult to run Tempest for all microversions on the gate because of
>>>>> test workload. So I feel we need to select microversions which are
>>>>> tested on the gate for efficient testing.
>>>>>
>>>>> [the lowest microversion]
>>>>> On microversion mechanism, if a client *doesn't* specify favorite
>>>>> microversion in its request header, a Nova/Ironic server considers the
>>>>> request as the lowest microversion. So the lowest microversion is
>>>>> default behavior and important. I think we need to test it at least.
>>>>>
>>>>> [the biggest microversion]
>>>>> On microversion mechanism, if a client specify the keyword "latest" in
>>>>> its request header instead of microversion, a Nova/Ironic server works
>>>>> on the biggest microversion behavior.
>>>>> During the development, there is time lag between each project dev and
>>>>> Tempest dev. After adding a new API on a project, corresponding tests
>>>>> are added to Tempest in most cases. So if specifying the keyword
>>>>> "latest", Tempest would not handle the request/response and fail,
>>>>> because Tempest can not catch the latest API changes until
>>>>> corresponding Tempest patch is merged.
>>>>> So it is necessary to have the target microversion config option in
>>>>> Tempest and pass specific biggest microversion to Tempest with
>>>>> openstack-infra/project-config.
>>>>>
>>>>> Any thoughts?
>>>>
>>>> Hi! I've already stated this point in #openstack-ironic and I'd like to
>>>> reiterate: if we test only the lowest and the highest microversions
>>>> essentially means (or at least might mean) that the other are broken. At
>>>> least in Ironic only some unit tests actually touch code paths for
>>>> versions 1.2-1.5. As we really can't test too many versions, I suggest
>>>> we stop producing a microversion for every API feature feature change in
>>>> L. No idea what to do with 1.2-1.5 now except for politely asking people
>>>> not to use them :D
>>>
>>> Tempest shouldn't be the *only* test for a project API. The projects
>>> themselves need to take some ownership for their API getting full
>>> coverage with in tree testing, including whatever microversion strategy
>>> they are employing.
>>
>> Agreed, but in-tree testing is also not feasible with too many version.
>> Even now we have 7 (1.0-1.6), if it continues, we'll have not less than
>> 12 after L, 18 after M, etc. And we have to test every one of them for
>> regressions at least occasionally, provided that we don't start to
>> aggressively deprecated microversions. If we do start, then we'll start
>> breaking people even more often, than we should. E.g. if someone writes
>> a tool targeted at 1.1, and we deprecated 1.1 in M cycle, the tool will
>> break, though maybe it can actually work with new API.
>
> I do not understand how in tree testing is not feasible. In tree you
> have insights into all the branching that occurs within code so can very
> clearly understand what paths aren't possible. It should be a lot more
> straight forward than external black box testing where that can't be assume.

Exactly.

The whole *point* of microversions was to allow the APIs to evolve in a 
backwards-compatible, structured and advertised way. The evolution of 
the APIs response and request payloads should be tested fully for each 
microversion added to the codebase -- in tree.

-jay