[nova][qa] Does nova need to run the nv tempest-multinode-full job?
Nova runs the tempest-multinode-full job as non-voting for some reason: https://github.com/openstack/nova/blob/c993d4fe2ffaadea76d5ae76eedfddc3a4c71... I'm assuming we were running that job initially for live migration testing, but we get that with the voting nova-live-migration and nova-grenade-live-migration jobs. The tempest-multinode-full job is essentially just the tempest-full job with 2 nodes. It doesn't run slow tests. But we have the tempest-slow job which is multinode and runs slow tests only. So it seems we have a lot of overlap between the nova-*live-migration multinode jobs for live migration testing (and evacuate gets tested in the nova-live-migration job as well), the tempest-full single node job, the nova-next single-node job, the tempest-slow job and the non-voting tempest-multinode-slow job, so can we just drop that last one? If someone were really ambitious they could do a diff of the tests that actually get run between say tempest-full/nova-live-migration/tempest-slow and tempest-multinode-full to see if we'd be missing any test coverage. -- Thanks, Matt
On 3/21/2019 10:16 PM, Matt Riedemann wrote:
Nova runs the tempest-multinode-full job as non-voting for some reason:
Looks like dougbot did it: https://github.com/openstack/nova/commit/05ab017907376cc262887104556b01fee18... Given that, it seems to me like we should just drop it from nova since we likely already have sufficient test coverage in other existing voting jobs. -- Thanks, Matt
On 3/21/2019 10:18 PM, Matt Riedemann wrote:
Given that, it seems to me like we should just drop it from nova since we likely already have sufficient test coverage in other existing voting jobs.
https://review.openstack.org/#/c/645456/ -- Thanks, Matt
---- On Thu, 21 Mar 2019 22:16:59 -0500 Matt Riedemann <mriedemos@gmail.com> wrote ----
Nova runs the tempest-multinode-full job as non-voting for some reason:
https://github.com/openstack/nova/blob/c993d4fe2ffaadea76d5ae76eedfddc3a4c71...
I'm assuming we were running that job initially for live migration testing, but we get that with the voting nova-live-migration and nova-grenade-live-migration jobs.
The tempest-multinode-full job is essentially just the tempest-full job with 2 nodes. It doesn't run slow tests. But we have the tempest-slow job which is multinode and runs slow tests only.
So it seems we have a lot of overlap between the nova-*live-migration multinode jobs for live migration testing (and evacuate gets tested in the nova-live-migration job as well), the tempest-full single node job, the nova-next single-node job, the tempest-slow job and the non-voting tempest-multinode-slow job, so can we just drop that last one?
If someone were really ambitious they could do a diff of the tests that actually get run between say tempest-full/nova-live-migration/tempest-slow and tempest-multinode-full to see if we'd be missing any test coverage.
Not all multinode tests tun on tempest-slow and nova-live-migration jobs. I did diff of those jobs tests run and below are the coverage nova is going to lose with removing the tempest-multinode-full. - test_server_multinode : https://github.com/openstack/tempest/blob/a955095f8369e2e15c97a536edc1a9c077... - test_cold_migration, test_revert_cold_migration: https://github.com/openstack/tempest/blob/16e4005dfcbd6b1aa68a1a1c0187b0af9c... - test_servers_on_multinodes: https://github.com/openstack/tempest/blob/7a588ded216f74ddd0015c3065d4fae10d... - Main concern of removing this job is each time we are going to make any mulitnode tests from slow->non-slow nova going to lose the coverage. I will suggest not to remove it. I have recently made this job running for stable branch also so that we do not regress there too. If we are going to cover most or full coverage of integrated tests, I will say we need to run these 3 tempest jobs 1. tempest-full-py3, 2. tempest-slow-py3 3. tempest-multinode-full When I made tempest-slow job I merged tempest-full-multinode also in that which means run all multinode test there but later tempest-slow job modified to run only slow tests. What we can improve is to make tempest-multinode-full to run only multinode tests (need to sort out the mechanish for that) but we need to check if there is no benefit of running single node test on multinode setup. -gmann
--
Thanks,
Matt
On 3/24/2019 8:50 PM, Ghanshyam Mann wrote:
I did diff of those jobs tests run and below are the coverage nova is going to lose with removing the tempest-multinode-full.
- test_server_multinode :https://github.com/openstack/tempest/blob/a955095f8369e2e15c97a536edc1a9c077...
- test_cold_migration, test_revert_cold_migration:https://github.com/openstack/tempest/blob/16e4005dfcbd6b1aa68a1a1c0187b0af9c...
- test_servers_on_multinodes:https://github.com/openstack/tempest/blob/7a588ded216f74ddd0015c3065d4fae10d...
I'm not sure if there is anything in those tests that we don't already have sufficient in-tree functional test coverage for the mechanics of the operation being tested, in other words, if those tests don't rely on specifics in the virt driver being used, we should have similar coverage already in-tree. -- Thanks, Matt
---- On Mon, 25 Mar 2019 12:27:22 -0500 Matt Riedemann <mriedemos@gmail.com> wrote ----
On 3/24/2019 8:50 PM, Ghanshyam Mann wrote:
I did diff of those jobs tests run and below are the coverage nova is going to lose with removing the tempest-multinode-full.
- test_server_multinode :https://github.com/openstack/tempest/blob/a955095f8369e2e15c97a536edc1a9c077...
- test_cold_migration, test_revert_cold_migration:https://github.com/openstack/tempest/blob/16e4005dfcbd6b1aa68a1a1c0187b0af9c...
- test_servers_on_multinodes:https://github.com/openstack/tempest/blob/7a588ded216f74ddd0015c3065d4fae10d...
I'm not sure if there is anything in those tests that we don't already have sufficient in-tree functional test coverage for the mechanics of the operation being tested, in other words, if those tests don't rely on specifics in the virt driver being used, we should have similar coverage already in-tree.
If we have the same coverage in functional test then question comes why we have such heavy tests as integration tests ? I think these were added for nova specific testing only. My other concern is that after removing the this job, nova gate would not have integration testing running on multinode env even tests are multinode specific or single node one and any future non-slow multinode tests will not be running. I see the benefit of running single nodes operation on multinode env for nova multinode arch. If the job is not stable then, we can work on making it stable but that's not the case seems. That job is voting on tempest side for long time. -gmann
--
Thanks,
Matt
On 3/26/2019 9:53 AM, Ghanshyam Mann wrote:
If we have the same coverage in functional test then question comes why we have such heavy tests as integration tests ? I think these were added for nova specific testing only.
My other concern is that after removing the this job, nova gate would not have integration testing running on multinode env even tests are multinode specific or single node one and any future non-slow multinode tests will not be running. I see the benefit of running single nodes operation on multinode env for nova multinode arch.
If the job is not stable then, we can work on making it stable but that's not the case seems. That job is voting on tempest side for long time.
My primary issue is the number of jobs and resources it takes to run through a nova change in the gate. I agree the job itself is probably stable, I'm not sure why it was non-voting but it was brought in that way with the aforementioned mechanical change. You've probably noticed a trend where I'm trying to reduce the amount of redundant overlapping tests in the jobs we run so we can trim down how many resources it takes in the gate to test a nova change and also cut back on how many different jobs with racy tests can prevent a nova change from landing. So if there are just 3 or 4 unique tests in that job which are now not covered by what nova does run, I would rather find a way to just run those tests rather than those tests plus the 98% other same tests we get in the tempest-full job. Maybe that's making a change such that single-node jobs no longer run those tests and the multinode job only runs tests that require multiple nodes, but that's quite a bit more work since it's going to affect all projects using the integrated-gate template. -- Thanks, Matt
participants (2)
-
Ghanshyam Mann
-
Matt Riedemann