[nova][qa][cinder] CI job changes
I'd like to propose some changes primarily to the CI jobs that run on nova changes, but also impact cinder and tempest. 1. Drop the nova-multiattach job and move test coverage to other jobs This is actually an old thread [1] and I had started the work but got hung up on a bug that was teased out of one of the tests when running in the multi-node tempest-slow job [2]. For now I've added a conditional skip on that test if running in a multi-node job. The open changes are here [3]. 2. Only run compute.api and scenario tests in nova-next job and run under python3 only The nova-next job is a place to test new or advanced nova features like placement and cells v2 when those were still optional in Newton. It currently runs with a few changes from the normal tempest-full job: * configures service user tokens * configures nova console proxy to use TLS * disables the resource provider association refresh interval * it runs the post_test_hook which runs some commands like archive_delete_rows, purge, and looks for leaked resource allocations [4] Like tempest-full, it runs the non-slow tempest API tests concurrently and then the scenario tests serially. I'm proposing that we: a) change that job to only run tempest compute API tests and scenario tests to cut down on the number of tests to run; since the job is really only about testing nova features, we don't need to spend time running glance/keystone/cinder/neutron tests which don't touch nova. b) run it with python3 [5] which is the direction all jobs are moving anyway 3. Drop the integrated-gate (py2) template jobs (from nova) Nova currently runs with both the integrated-gate and integrated-gate-py3 templates, which adds a set of tempest-full and grenade jobs each to the check and gate pipelines. I don't think we need to be gating on both py2 and py3 at this point when it comes to tempest/grenade changes. Tempest changes are still gating on both so we have coverage there against breaking changes, but I think anything that's py2 specific would be caught in unit and functional tests (which we're running on both py27 and py3*). Who's with me? [1] http://lists.openstack.org/pipermail/openstack-dev/2018-October/135299.html [2] https://bugs.launchpad.net/tempest/+bug/1807723 [3] https://review.openstack.org/#/q/topic:drop-multiattach-job+(status:open+OR+...) [4] https://github.com/openstack/nova/blob/5283b464b/gate/post_test_hook.sh [5] https://review.openstack.org/#/c/634739/ -- Thanks, Matt
---- On Tue, 05 Feb 2019 23:35:20 +0900 Matt Riedemann <mriedemos@gmail.com> wrote ----
I'd like to propose some changes primarily to the CI jobs that run on nova changes, but also impact cinder and tempest.
1. Drop the nova-multiattach job and move test coverage to other jobs
This is actually an old thread [1] and I had started the work but got hung up on a bug that was teased out of one of the tests when running in the multi-node tempest-slow job [2]. For now I've added a conditional skip on that test if running in a multi-node job. The open changes are here [3].
+1. The only question I commented on review - this test is skipped in all jobs now. If that all ok as of now then I am +2.
2. Only run compute.api and scenario tests in nova-next job and run under python3 only
The nova-next job is a place to test new or advanced nova features like placement and cells v2 when those were still optional in Newton. It currently runs with a few changes from the normal tempest-full job:
* configures service user tokens * configures nova console proxy to use TLS * disables the resource provider association refresh interval * it runs the post_test_hook which runs some commands like archive_delete_rows, purge, and looks for leaked resource allocations [4]
Like tempest-full, it runs the non-slow tempest API tests concurrently and then the scenario tests serially. I'm proposing that we:
a) change that job to only run tempest compute API tests and scenario tests to cut down on the number of tests to run; since the job is really only about testing nova features, we don't need to spend time running glance/keystone/cinder/neutron tests which don't touch nova.
b) run it with python3 [5] which is the direction all jobs are moving anyway
+1. It make sense to run only compute test in this job.
3. Drop the integrated-gate (py2) template jobs (from nova)
Nova currently runs with both the integrated-gate and integrated-gate-py3 templates, which adds a set of tempest-full and grenade jobs each to the check and gate pipelines. I don't think we need to be gating on both py2 and py3 at this point when it comes to tempest/grenade changes. Tempest changes are still gating on both so we have coverage there against breaking changes, but I think anything that's py2 specific would be caught in unit and functional tests (which we're running on both py27 and py3*).
IMO, we should keep running integrated-gate py2 templates on the project gate also along with Tempest. Jobs in integrated-gate-* templates cover a large amount of code so running that for both versions make sure we keep our code running on py2 also. Rest other job like tempest-slow, nova-next etc are good to run only py3 on project side (Tempest gate keep running py2 version also). I am not sure if unit/functional jobs cover all code coverage and it is safe to ignore the py version consideration from integration CI. As per TC resolution, python2 can be dropped during begning of U cycle [1]. You have good point of having the integrated-gate py2 coverage on Tempest gate only is enough but it has risk of merging the py2 breaking code on project side which will block the Tempest gate. I agree that such chances are rare but still it can happen. Other point is that we need integrated-gate template running when Stein and Train become stable branch (means on stable/stein and stable/train gate). Otherwise there are chance when py2 broken code from U (because we will test only py3 in U) is backported to stable/Train or stable/stein. My opinion on this proposal is to wait till we officially drop py2 which is starting of U. [1] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation... -gmann
Who's with me?
[1] http://lists.openstack.org/pipermail/openstack-dev/2018-October/135299.html [2] https://bugs.launchpad.net/tempest/+bug/1807723 [3] https://review.openstack.org/#/q/topic:drop-multiattach-job+(status:open+OR+...) [4] https://github.com/openstack/nova/blob/5283b464b/gate/post_test_hook.sh [5] https://review.openstack.org/#/c/634739/
--
Thanks,
Matt
On 2/5/2019 11:09 PM, Ghanshyam Mann wrote:
3. Drop the integrated-gate (py2) template jobs (from nova)
Nova currently runs with both the integrated-gate and integrated-gate-py3 templates, which adds a set of tempest-full and grenade jobs each to the check and gate pipelines. I don't think we need to be gating on both py2 and py3 at this point when it comes to tempest/grenade changes. Tempest changes are still gating on both so we have coverage there against breaking changes, but I think anything that's py2 specific would be caught in unit and functional tests (which we're running on both py27 and py3*).
IMO, we should keep running integrated-gate py2 templates on the project gate also along with Tempest. Jobs in integrated-gate-* templates cover a large amount of code so running that for both versions make sure we keep our code running on py2 also. Rest other job like tempest-slow, nova-next etc are good to run only py3 on project side (Tempest gate keep running py2 version also).
I am not sure if unit/functional jobs cover all code coverage and it is safe to ignore the py version consideration from integration CI. As per TC resolution, python2 can be dropped during begning of U cycle [1].
You have good point of having the integrated-gate py2 coverage on Tempest gate only is enough but it has risk of merging the py2 breaking code on project side which will block the Tempest gate. I agree that such chances are rare but still it can happen.
Other point is that we need integrated-gate template running when Stein and Train become stable branch (means on stable/stein and stable/train gate). Otherwise there are chance when py2 broken code from U (because we will test only py3 in U) is backported to stable/Train or stable/stein.
My opinion on this proposal is to wait till we officially drop py2 which is starting of U.
[1]https://governance.openstack.org/tc/resolutions/20180529-python2-deprecation...
-gmann
We talked about this during the nova meeting today [1]. My main concern right now is efficiency and avoid running redundant test coverage when there is otherwise not much of a difference in the configured environment, which is what we have between the py2 and py3 integrated-gate templates. This is also driving my push to drop the nova-multiattach job and fold those tests into the integrated gate and slim down the number of tests we run in the nova-next job. I understand the concern of dropping the integrated-gate template from nova is a risk to break something in those jobs unknowingly. However, I assume that most py2-specific issues in nova will be caught in unit and functional test jobs which we continue to run. Also, nova is also running a few integration jobs that run on py27 (devstack-plugin-ceph-tempest and neutron-grenade-multinode), so we still have py2 test coverage. We're not dropping py27 support and we're still testing it, but it's a lower priority with everything moving to python3 and I think our test coverage should reflect that. I think we should try this [2] and if it does become a major issue we can revisit adding the integrated-gate py2 template jobs in nova until the U release. [1] http://eavesdrop.openstack.org/meetings/nova/2019/nova.2019-02-07-14.00.log.... [2] https://review.openstack.org/#/c/634949/ -- Thanks, Matt
Updates inline. On 2/5/2019 8:35 AM, Matt Riedemann wrote:
I'd like to propose some changes primarily to the CI jobs that run on nova changes, but also impact cinder and tempest.
1. Drop the nova-multiattach job and move test coverage to other jobs
This is actually an old thread [1] and I had started the work but got hung up on a bug that was teased out of one of the tests when running in the multi-node tempest-slow job [2]. For now I've added a conditional skip on that test if running in a multi-node job. The open changes are here [3].
Done: https://review.openstack.org/#/q/status:merged+topic:drop-multiattach-job
2. Only run compute.api and scenario tests in nova-next job and run under python3 only
The nova-next job is a place to test new or advanced nova features like placement and cells v2 when those were still optional in Newton. It currently runs with a few changes from the normal tempest-full job:
* configures service user tokens * configures nova console proxy to use TLS * disables the resource provider association refresh interval * it runs the post_test_hook which runs some commands like archive_delete_rows, purge, and looks for leaked resource allocations [4]
Like tempest-full, it runs the non-slow tempest API tests concurrently and then the scenario tests serially. I'm proposing that we:
a) change that job to only run tempest compute API tests and scenario tests to cut down on the number of tests to run; since the job is really only about testing nova features, we don't need to spend time running glance/keystone/cinder/neutron tests which don't touch nova.
Proposed: https://review.openstack.org/#/c/636459/ (+2 from Stephen)
b) run it with python3 [5] which is the direction all jobs are moving anyway
Done: https://review.openstack.org/#/c/634739/
3. Drop the integrated-gate (py2) template jobs (from nova)
Nova currently runs with both the integrated-gate and integrated-gate-py3 templates, which adds a set of tempest-full and grenade jobs each to the check and gate pipelines. I don't think we need to be gating on both py2 and py3 at this point when it comes to tempest/grenade changes. Tempest changes are still gating on both so we have coverage there against breaking changes, but I think anything that's py2 specific would be caught in unit and functional tests (which we're running on both py27 and py3*).
Proposed: https://review.openstack.org/#/c/634949/ (+2 from Stephen) -- Thanks, Matt
participants (2)
-
Ghanshyam Mann
-
Matt Riedemann