[openstack-dev] [neutron] CI jobs take pretty long, can we improve that?

Clark Boylan cboylan at sapwetik.org
Mon Mar 21 18:49:00 UTC 2016


On Mon, Mar 21, 2016, at 11:08 AM, Clark Boylan wrote:
> On Mon, Mar 21, 2016, at 09:32 AM, Armando M. wrote: 
> > Do you have an a better insight of job runtimes vs jobs in other
> > projects?
> > Most of the time in the job runtime is actually spent setting the
> > infrastructure up, and I am not sure we can do anything about it, unless
> > we
> > take this with Infra.
> 
> I haven't done a comparison yet buts lets break down the runtime of a
> recent successful neutron full run against neutron master [0].

And now for some comparative data from the gate-tempest-dsvm-full job
[0]. This job also ran against a master change that merged and ran in
the same cloud and region as the neutron job.

Basic host setup takes 63 seconds. Start of job to 2016-03-21
16:46:41.058 [1]
Workspace setup takes 380 seconds. 2016-03-21 16:46:41.058 [1] to
2016-03-21 16:53:01.754 [2]
Devstack takes 890 seconds. 2016-03-21 16:53:19.235 [3] to 2016-03-21
17:08:10.082 [4]
Loading old tempest subunit streams takes 63 seconds. 2016-03-21
17:08:10.111 [5] to 2016-03-21 17:09:13.454 [6]
Tempest takes 1347 seconds. 2016-03-21 17:09:13.587 [7] to 2016-03-21
17:31:40.885 [8]
Then we spend the rest of the test time (52 seconds) cleaning up.
2016-03-21 17:31:40.885 [8] to end of job.

[0]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/
[1]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_16_46_41_058
[2]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_16_53_01_754
[3]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_16_53_19_235
[4]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_08_10_082
[5]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_08_10_111
[6]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_09_13_454
[7]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_09_13_587
[8]
http://logs.openstack.org/48/294548/1/gate/gate-tempest-dsvm-full/d94901e/console.html#_2016-03-21_17_31_40_885

Generally each step of this job was quicker. There were big differences
in devstack and tempest run time though. Is devstack much slower to
setup neutron when compared to nova net? For tempest it looks like we
run ~1510 tests against neutron and only ~1269 against nova net. This
may account for the large difference there. I also recall that we run
ipv6 tempest tests against neutron deployments that were inefficient and
booted 2 qemu VMs per test (not sure if that is still the case but
illustrates that the tests themselves may not be very quick in the
neutron case).

Of course we may also be seeing differences in cloud VMs (though tried
to control for by looking at tests that ran in the same region). Hard to
say without more data. In any case this hopefully serves as a good
starting point for others to dig into the ~20 minute discrepancy between
nova net + tempest and neutron + tempest.



More information about the OpenStack-dev mailing list