[openstack-dev] [neutron] CI jobs take pretty long, can we improve that?

Armando M. armamig at gmail.com
Mon Mar 21 18:29:15 UTC 2016


On 21 March 2016 at 11:08, Clark Boylan <cboylan at sapwetik.org> wrote:

> On Mon, Mar 21, 2016, at 09:32 AM, Armando M. wrote:
> > Do you have an a better insight of job runtimes vs jobs in other
> > projects?
> > Most of the time in the job runtime is actually spent setting the
> > infrastructure up, and I am not sure we can do anything about it, unless
> > we
> > take this with Infra.
>
> I haven't done a comparison yet buts lets break down the runtime of a
> recent successful neutron full run against neutron master [0].
>
> Basic host setup takes 65 seconds. Start of job to 2016-03-17
> 22:14:27.397 [1]
> Workspace setup takes 520 seconds. 2016-03-17 22:14:27.397 [1] to
> 2016-03-17 22:23:07.429 [2]
> Devstack takes 1205 seconds. 2016-03-17 22:23:18.760 [3] to 2016-03-17
> 22:43:23.339 [4]
> Loading old tempest subunit streams takes 155 seconds. 2016-03-17
> 22:43:23.340 [5] to 2016-03-17 22:45:58.061 [6]
> Tempest takes 1982 seconds. 2016-03-17 22:45:58.201 [7] to 2016-03-17
> 23:19:00.117 [8]
> Then we spend the rest of the test time (76 seconds) cleaning up.
> 2016-03-17 23:19:00.117 [8] to end of job.
>
> Note that I haven't accounted for all of the time used and instead of
> focused on the major steps that use the most time. Also it is Monday
> morning and some of my math may be off.


> [0]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/
> [1]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_14_27_397
> [2]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_07_429
> [3]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_18_760
> [4]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_339
> [5]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_340
> [6]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_061
> [7]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_201
> [8]
>
> http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_23_19_00_117
>
> One big takeaway from this is that the vast majority of the time is
> spent in devstack and tempest not in the infrastructure setup. You
> should be able to dig into both the devstack setup and tempest test
> runtimes and hopefully speed things up.
>
> Hopefully this gives you enough information to get started into digging
> on this.
>

Clark: thanks for this insightful response.

I should clarify my comment about infrastructure setup (it is Monday for me
too :)): what I meant was the there is a good portion of time spent to get
to a point where tests can be run. That includes node setup as well as
stacking. That is obviously less than 50%, but even >30% feels like a
substantial overhead. I am not sure what we can do about it, but looping
you in this discussion seemed like the least this thread should do.

That said, there are many tempest tests that take over 30 seconds to
complete and those do not even touch Neutron. For those that do, then we
should clearly identify where the slowness comes from and I think that's
where, as a Neutron team, our focus should be.

IMO, before we go on and talk about evicting jobs, I think we should take a
closer look (i.e. profiling) where time is spent so that we can make each
test run leaner.

[1]
http://status.openstack.org//openstack-health/#/job/gate-tempest-dsvm-neutron-full?groupKey=project&resolutionKey=hour&end=2016-03-21T18:14:19.534Z

>
>
> Clark
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160321/43db9853/attachment.html>


More information about the OpenStack-dev mailing list