<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 21 March 2016 at 11:08, Clark Boylan <span dir="ltr"><<a href="mailto:cboylan@sapwetik.org" target="_blank">cboylan@sapwetik.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">On Mon, Mar 21, 2016, at 09:32 AM, Armando M. wrote:<br>
> Do you have an a better insight of job runtimes vs jobs in other<br>
> projects?<br>
> Most of the time in the job runtime is actually spent setting the<br>
> infrastructure up, and I am not sure we can do anything about it, unless<br>
> we<br>
> take this with Infra.<br>
<br>
</span>I haven't done a comparison yet buts lets break down the runtime of a<br>
recent successful neutron full run against neutron master [0].<br>
<br>
Basic host setup takes 65 seconds. Start of job to 2016-03-17<br>
22:14:27.397 [1]<br>
Workspace setup takes 520 seconds. 2016-03-17 22:14:27.397 [1] to<br>
2016-03-17 22:23:07.429 [2]<br>
Devstack takes 1205 seconds. 2016-03-17 22:23:18.760 [3] to 2016-03-17<br>
22:43:23.339 [4]<br>
Loading old tempest subunit streams takes 155 seconds. 2016-03-17<br>
22:43:23.340 [5] to 2016-03-17 22:45:58.061 [6]<br>
Tempest takes 1982 seconds. 2016-03-17 22:45:58.201 [7] to 2016-03-17<br>
23:19:00.117 [8]<br>
Then we spend the rest of the test time (76 seconds) cleaning up.<br>
2016-03-17 23:19:00.117 [8] to end of job.<br>
<br>
Note that I haven't accounted for all of the time used and instead of<br>
focused on the major steps that use the most time. Also it is Monday<br>
morning and some of my math may be off. </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
[0]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/</a><br>
[1]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_14_27_397" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_14_27_397</a><br>
[2]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_07_429" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_07_429</a><br>
[3]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_18_760" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_23_18_760</a><br>
[4]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_339" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_339</a><br>
[5]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_340" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_43_23_340</a><br>
[6]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_061" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_061</a><br>
[7]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_201" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_22_45_58_201</a><br>
[8]<br>
<a href="http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_23_19_00_117" rel="noreferrer" target="_blank">http://logs.openstack.org/18/294018/3/check/gate-tempest-dsvm-neutron-full/1cce1e8/console.html.gz#_2016-03-17_23_19_00_117</a><br>
<br>
One big takeaway from this is that the vast majority of the time is<br>
spent in devstack and tempest not in the infrastructure setup. You<br>
should be able to dig into both the devstack setup and tempest test<br>
runtimes and hopefully speed things up.<br>
<br>
Hopefully this gives you enough information to get started into digging<br>
on this.<br></blockquote><br>Clark: thanks for this insightful response.<br><br>I should clarify my comment about infrastructure setup (it is Monday for me too :)): what I meant was the there is a good portion of time spent to get to a point where tests can be run. That includes node setup as well as stacking. That is obviously less than 50%, but even >30% feels like a substantial overhead. I am not sure what we can do about it, but looping you in this discussion seemed like the least this thread should do.<br><br>That said, there are many tempest tests that take over 30 seconds to complete and those do not even touch Neutron. For those that do, then we should clearly identify where the slowness comes from and I think that's where, as a Neutron team, our focus should be. </div><div class="gmail_quote"><br></div><div class="gmail_quote">IMO, before we go on and talk about evicting jobs, I think we should take a closer look (i.e. profiling) where time is spent so that we can make each test run leaner.</div><div class="gmail_quote"><br>[1] <a href="http://status.openstack.org//openstack-health/#/job/gate-tempest-dsvm-neutron-full?groupKey=project&resolutionKey=hour&end=2016-03-21T18:14:19.534Z">http://status.openstack.org//openstack-health/#/job/gate-tempest-dsvm-neutron-full?groupKey=project&resolutionKey=hour&end=2016-03-21T18:14:19.534Z</a><br> <blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span class=""><font color="#888888"><br>
Clark<br>
</font></span><div class=""><div class="h5"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div><div class="h5"><br></div></div></blockquote></div><br></div></div>