Open Stack

On Wed, Nov 29, 2017 at 9:51 PM, Zane Bitter <zbitter at redhat.com> wrote:

> On 19/11/17 03:08, Rabi Mishra wrote:
>
>> Hi All,
>>
>> As part of community goal[1] for Queens, we've completed the repo split
>> and created the new project[2].
>>
>> The next objective to is to use the new plugin in our jobs. As we've
>> merged some changes after the split, I've synced them and fixed some minor
>> issues before using it in the gate jobs.
>>
>> These are very small changes and I would suggest we review/land them on
>> priority (we don't have to keep syncing them again and again for broken
>> jobs). We should probably *not approve* any changes to integration tests in
>> heat before these go in.
>>
>> - Changes in heat-tempest-plugin (sync  missing patches and fixes)
>> https://review.openstack.org/#/q/project:openstack/heat-temp
>> est-plugin+topic:sync_from_heat
>>
>> - Use heat-tempest-plugin for integration jobs
>> https://review.openstack.org/#/c/508112/
>>
>> - Use heat-tempest-plugin for grenade job
>> https://review.openstack.org/#/c/521246/
>>
>> - Add heat integration jobs to heat-tempest-plugin check/gate queue
>> https://review.openstack.org/#/c/521340/
>>    I guess this would result in some issues, where we can't add any
>> changes to heat that breaks the existing tests, as the changes for both
>> projects would have a circular dependency (not sure how it works atm with
>> other plugins!).
>>
>> - Remove plugin an integration tests from heat (This has -1 atm as
>> releasenote job is broken, waiting for infra to fix it)
>> https://review.openstack.org/#/c/521263/ <https://review.openstack.org/
>> #/c/521263/>
>>
>> I've also created an etherpad[3] to track these.
>>
>> Also, the plugin project is expected to be branchless (we may not
>> backport these job changes to stable branches soon though), we've to find
>> a  way to run additional tests for any new feature only on >
>> <release/branch>. AFAIK, other projects check api microversions supported
>> and without microversions in heat, may be we've find an alternate way.
>>
>
> So from discussion on IRC today I learned that the plan agreed at the PTG
> was *not* to move all of heat_integrationtests to a separate repo, but just
> those tests that test the API and are likely to be used in the trademark
> program for verifying clouds.
>
>
Well, the etherpad[1] for the session clearly mentions "Create the new repo
and import API and *scenario* tests" and *not* "that tests the API and
likely to be used in trademark programs".  What you mentioned above is
probably the conclusion from all the discussion we had on IRC yesterday.

In my view that effectively means just the Gabbi tests.
>
>
However, what is actually happening is that *all* of the
> integration/scenario tests are moving to the separate branchless repo. This
> looks to me like it will be a disaster for developer and reviewer
> productivity, and project quality[1]. (Other folks seem inclined to agree.)
>

Just to make it clear, there had been several discussions on 'what to
move'/'branchless or not' in meetings[2] and IRC discussions[3] and the
change of plan to move functional tests was based on the following.

- All our tests are API driven. There is probably not much difference
between functional and scenario (i.e. if we only move scenario tests, we
would have the same kind of issues as we would with functional tests).

- All our tests run with tempest as test runner and use the same tempest
configuration. Unless we run remaining in-tree tests without tempest, we
would end-up with the same set problems which the goal wanted to address (
issues with in-tree tempest plugins).

- Current coverage of gabbi tests is very poor and we've never been serious
to land the patches to complete the API coverage. One of the patches have
been languishing in review queue since a year[4].

- As heat is the orchestration service, I would think testing heat from
user/operator point of view is not only testing API and interoperability,
but how it works with other services deployed and many of our functional
tests[5] do that.

As per my understanding, the issue of having tests in a separate repo and
it's impact on developer productivity was discussed by the all concerned
before accepting this as a community goal. Our problem is probably not
different from some other projects[6].

It's little unfortunate that we've started discussing about these issues
after most of stuff has been done. Having said that, it's never late to do
the right thing that saves us from a disaster, if that's what it means:)

Given the option, I would prefer:

- We keep all tests in-tree and run them with tempest for the time being,
which means we would not complete the goal.

- Complete the coverage for API tests, probably identify
functional/scenario tests that are expected to be stable across branches
and then move them out along with the API tests.

- Change the remaining in-tree tests to run without tempest at the gate,
unless there is a solution to your question below.

[1] https://etherpad.openstack.org/p/heat-queens-ptg-tempest-plugin
[2]
http://eavesdrop.openstack.org/meetings/heat/2017/heat.2017-08-16-15.00.log.txt
[3]
http://eavesdrop.openstack.org/irclogs/%23heat/%23heat.2017-09-29.log.html
[4] https://review.openstack.org/#/c/421914/
[5]
https://github.com/openstack/heat/blob/master/heat_integrationtests/functional/test_lbaasv2.py
[6]
https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_tempest_plugin/scenario/test_floatingip.py#L118

The QA team assures us that the project-wide goal does not preclude keeping
> tests that are designed for Heat CI purposes (rather than cloud
> verification purposes) in-tree - even continuing to use tempest as a
> testrunner. However, it's not at all clear how to reconcile that with the
> goal description listing the evils of having an in-tree tempest plugin.[2]
> This appears to be the reason for the change in plan.
>
> Does anybody remember what the mechanism for dealing with this that was
> agreed at the PTG was?
>
> Basically we need a way to:
>
> * Run a bunch of integration tests in Heat CI jobs
> * against a full OpenStack cloud
> * preferably using tempest as a testrunner, but this is optional
> * without any danger of e.g. breaking tempest test discovery just because
> Heat was installed on the same machine as tempest
>
> How do we do it?
>
> thanks,
> Zane.
>
> [1] http://eavesdrop.openstack.org/irclogs/%23heat/%23heat.2017-
> 11-29.log.html#t2017-11-29T15:17:00
> [2] https://governance.openstack.org/tc/goals/queens/split-tempe
> st-plugins.html#split-tempest-plugins-into-separate-repos-projects
>
> --
>> Regards,
>> Rabi Mishra
>>
>> [1] https://governance.openstack.org/tc/goals/queens/split-tempe
>> st-plugins.html
>> [2] https://git.openstack.org/cgit/openstack/heat-tempest-plugin
>> [3] https://etherpad.openstack.org/p/heat-tempest-plugin
>>
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Regards,
Rabi Mishra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171130/95599767/attachment.html>