<div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote">On Thu, Aug 22, 2013 at 7:48 PM, James E. Blair <span dir="ltr"><<a href="mailto:jeblair@openstack.org" target="_blank">jeblair@openstack.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">Monty Taylor <<a href="mailto:mordred@inaugust.com">mordred@inaugust.com</a>> writes:<br>
<br>
> The infra team has done a lot of work in prep for our favorite time of<br>
> year, and we've actually landed several upgrades to the gate without<br>
> which we'd be in particularly bad shape right now. (I'll let Jim write<br>
> about some of them later when he's not battling the current operational<br>
> issues - they're pretty spectacular) As with many scaling issues, some<br>
> of these upgrades have resulted in moving the point of pain further<br>
> along the stack. We're working on solutions to the current pain points.<br>
> (Or, I should say they are, because I'm on a plane headed to Burning Man<br>
> and not useful for much other than writing emails.)<br>
<br>
</div>Hi!<br>
<br>
The good news is that a lot of the operational problems over the past<br>
few days have been corrected, we are now pretty close to the noise floor<br>
of infrastructure issues in the gate, and over the next few days we'll<br>
work to get rid of the remaining bugs.<br>
<br>
As I'm sure everyone knows, we've seen a huge growth in the project, the<br>
number of changes, and the number of tests we run. That is both<br>
wonderful, and a little terrifying! But we haven't been idle: we have<br>
made some significant improvements and innovations to the project<br>
infrastructure to deal with our growing load, especially during these<br>
peak times.<br>
<br>
About a year ago, we realized that the growing number of jobs run (and<br>
number of test machines on which we run those jobs) was going to cause<br>
scaling issues with Jenkins. So with the help of Khai Do, we created<br>
the gearman-plugin[1] for Jenkins, and then we modified Zuul to use it.<br>
That means that Zuul isn't directly tied to Jenkins anymore, and can<br>
distribute the jobs it needs to run to anything that can run them via<br>
Gearman.<br>
<br>
A few weeks ago we took advantage of that by adding two new Jenkins<br>
masters to our system, giving us one of the first (if not the first)<br>
multi-master Jenkins systems. Since then, all of the test jobs have<br>
been run on nodes attached to either <a href="http://jenkins01.openstack.org" target="_blank">jenkins01.openstack.org</a> or<br>
<a href="http://jenkins02.openstack.org" target="_blank">jenkins02.openstack.org</a> (which you may have seen linked to from the Zuul<br>
status page). That has given us the ability to upgrade Jenkins and its<br>
plugins with no interruption due to the active-active nature of the<br>
system. And we can add hundreds of test nodes to each of these systems<br>
and continue to scale them horizontally as our load increases.<br>
<br>
With Jenkins now able to scale, the next bottleneck was the number of<br>
test nodes. Until recently, we had a handful of special Jenkins jobs<br>
which would launch and destroy the single-use nodes that are used for<br>
devstack tests. We were seeing issues with Jenkins running those jobs,<br>
as well as their ability to keep up with demand. So we started the<br>
Nodepool project[2] to create a daemon that could keep up with the<br>
demand for test nodes, be much more responsive, and eliminate some of<br>
the occasional errors that we would see in the old Rube-Goldberg system<br>
we had for managing nodes.<br>
<br>
In anticipation of the rush of patches for the feature freeze, we rolled<br>
that out over the weekend so it was ready to go Monday. And it worked!<br>
<br>
In fact, it's extremely responsive. It immediately utilized our entire<br>
capacity to supply test nodes. Which was great, except that a lot of<br>
our tests are configured to use the git repos from Gerrit, which is why<br>
Gerrit was very slow early in the week. Fortunately, Elizabeth Krumbach<br>
Joseph has been working on setting up a new Git server. That alone is<br>
pretty exciting, and she's going to send an announcement about it soon.<br>
Since it was ready to go, we moved the test load from Gerrit to the new<br>
git server, which has made Gerrit much more responsive again.<br>
Unfortunately, the new git server still wasn't quite able to keep up<br>
with the test load, so Clark Boylan, Elizabeth and I have spent some<br>
time tuning it as well as load-balancing it across several hosts.<br>
<br>
That is now in place, and the new system seems able to cope with the<br>
load from the current rush of patches.<br>
<br>
We're still seeing an occasional issue where a job is reported as LOST<br>
because Jenkins is apparently unaware that it can't talk to the test<br>
node. We have some workarounds in progress that we hope to have in<br>
place soon.<br>
<br>
Our goal is to have the most robust and accurate test system possible,<br>
that can run all of the tests we can think to throw at it. I think the<br>
improvements we've made recently are going to help tremendously and I'm<br>
pretty excited! As always, if you'd like to pitch in, stop by<br>
#openstack-infra on Freenode and see what we're up to.<br></blockquote><div><br></div><div>Wow, nice work! Thank you, infra!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
-Jim<br>
<br>
[1] <a href="http://git.openstack.org/cgit/openstack-infra/gearman-plugin/" target="_blank">http://git.openstack.org/cgit/openstack-infra/gearman-plugin/</a><br>
[2] <a href="http://git.openstack.org/cgit/openstack-infra/nodepool/" target="_blank">http://git.openstack.org/cgit/openstack-infra/nodepool/</a><br>
<div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><br></div>-Dolph
</div></div>