[openstack-dev] [openstack-ansible] Debugging slow Xenial gate

Major Hayden major at mhtx.net
Wed Nov 2 13:51:27 UTC 2016


On 10/28/2016 04:02 AM, Major Hayden wrote:
> On the topic of threads, the sysbench output from both Trusty and Xenial are nearly identical with the exception of threads.  Trusty is usually about 15-20% faster on that benchmark than Xenial.

I spoke with a few other people and it seems like the culprit could be a CPU scheduler difference and/or a glibc change.  After messing around with perf for a long time, I found that context switches and CPU migrations were slightly higher on Xenial than Trusty, but by a negligible amount (< 10% at worst).

I tossed up a horribly written hack[0] to change some CPU scheduler settings back to the Trusty settings.  My initial tests were great!  Also, the first test in OpenStack CI was really good --  62 minutes for trusty and 65 minutes for xenial.  However, that seems to be a fluke since the second test had a 30 minute gap between the test durations. :(

Those scheduler changes for busy_factor, min_interval, and max_interval appear to have been made in the upstream Linux kernel, and they're present on various distributions like Ubuntu, CentOS, and Fedora.

At this point, I'm still trying to test some additional theories. Does anyone have any other ideas?

[0] https://review.openstack.org/392316

--
Major Hayden

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20161102/76074148/attachment.pgp>


More information about the OpenStack-dev mailing list