[openstack-dev] [TripleO] CI outage

Derek Higgins derekh at redhat.com
Mon Mar 30 13:49:02 UTC 2015

Tl;dr tripleo ci is back up and running, see below for more

On 21/03/15 01:41, Dan Prince wrote:
> Short version:
> The RH1 CI region has been down since yesterday afternoon.
> We have a misbehaving switch and have file a support ticket with the
> vendor to troubleshoot things further. We hope to know more this
> weekend, or Monday at the latest.
> Long version:
> Yesterday afternoon we started seeing issues in scheduling jobs on the
> RH1 CI cloud. We haven't made any OpenStack configuration changes
> recently, and things have been quite stable for some time now (our
> uptime was 365 days on the controller).
> Initially we found a misconfigured Keystone URL which was preventing
> some diagnostic queries via OS clients external to the rack. This
> setting hadn't been recently changed however and didn't seem to bother
> nodepool before so I don't think it is the cause of the outage...
> MySQL also got a bounce. It seemed happy enough after a restart as well.
> After fixing the keystone setting and bouncing MySQL instances appears
> to go ACTIVE but we were still having connectivity issues getting
> floating IPs and DHCP working on overcloud instances. After a good bit
> of debugging we started looking at the switches. Turns out one of them
> has a high CPU usuage (above the warning threshold) and MAC address are
> also unstable (ports are moving around).
> Until this is resolved RH1 is unavailable to host jobs CI jobs. Will
> post back here with an update once we have more information.

RH1 has been running as expected since last Thursday afternoon which 
means the cloud was down for almost a week, I'm left not entirely sure 
what some problems were, at various times during the week we tried a 
number of different interventions which may have caused (or exposed) 
some of our problems, e.g.

at one stage we restarted openvswitch in an attempt to ensure nothing 
had gone wrong with our ovs tunnels, around the same time (and possible 
caused by the restart), we started getting progressively worse 
connections to some of our servers. With lots of entries like this on 
our bastion server
Mar 20 13:22:49 host01-rack01 kernel: bond0.5: received packet with own 
address as source address

Not linking the restart with the looping packets message and instead 
thinking we may have a problem with the switch we put in a call with our 
switch vendor.

Continuing to chase down a problem on our own servers we noticed that 
tcpdump was reporting at times about 100,000 ARP packets per second 
(sometimes more).

Various interventions stopped the excess broadcast traffic e.g.
   Shutting down most of the compute nodes stopped the excess traffic, 
but the problem wasn't linked to any one particular compute node
   Running the tripleo os-refresh-config script on each compute node 
stopped the excess traffic

But restarting the controller node caused the excess traffic to return

Eventually we got the cloud running without the flood of broadcast 
traffic, with a small number of compute nodes, but instances still 
weren't getting IP address, with nova and neutron in debug mode we saw 
an error where nova failing to mount the qcow image (iirc it was 
attempting to resize the image).

Unable to figure out why this was working in the past but now isn't we 
redeployed this single compute node using the original image that was 
used (over a year ago), instances on this compute node we're booting but 
failing to get an IP address, we noticed this was because of a 
difference between the time on the controller when compared to the 
compute node. After resetting the time, now instances were booting and 
networking was working as expected (this was now Wednesday evening).

Looking back at the error while mounting the qcow image, I believe this 
was a red herring, it looks like this problem was always present on our 
system but we didn't have scary looking tracebacks in the logs until we 
switched to debug mode.

Now pretty confident we can get back to a running system by starting up 
all the compute nodes again and ensuring the os-refresh-config scripts 
were run then ensuring the times were all set on each host properly we 
decided to remove any entropy the may have built up while debugging 
problems on each computes node so we redeployed all of our compute nodes 
from scratch. This all went as expected but was a little time consuming 
as we spent time to verify each step as we went along, the steps went 
something like this

o with the exception of the overcloud controller, "nova delete" all of 
the hosts on the undercloud (31 hosts)

o we now have a problem, in tripleo the controller and compute nodes are 
tied together in a single heat template, so we need the heat template 
that was used a year ago to deploy the whole overcloud along with the 
parameters that were passed into it, we had actually done this before 
when adding new compute nodes to the cloud so it wasn't new territory.
    o Use "heat template-show ci-overcloud" to get the original heat 
template (a json version) that was used and remove anything referring to 
the controller
    o Edit a version of devtest overcloud to use this template and to 
skip various other steps

o Our overcloud had 3 VM instances used by CI that now needed to be 
replaced, each needed specific IP addresses and to be on both the 
default and test network
    o squid     - nova boot the image we had for this
    o bandersnatch -  nova boot the image we had for this
    o te-broker - we didn't have an image for this, we booted a vanilla 
Fedora image and manually installed geard

o testenv hosts - we also redeployed the test env hosts

o set the times on all of the hosts so that they are in sync

Its thursday lunchtime and our cloud is now back up and running, at this 
point we remove the IP tables rule preventing nodepool from talking to 
our cloud

We still have problems, regressions have been committed to various non 
tripleo repositories causing our tests to fail in various ways (4 
regressions in total)
1. Fedora instances started by nodepool were failing to boot, we 
eventually tracked this down to an update to the scripts nodpool uses to 
build this image, this update was only needed for a specific cloud, so 
our fix here was to only run it on that cloud [1]
2. A neutron regression [2]
3. A keystone regression [3]
4. A horizon regression[4]

[1] https://review.openstack.org/#/c/168196/
[2] https://bugs.launchpad.net/tripleo/+bug/1437116
[3] https://bugs.launchpad.net/tripleo/+bug/1437032
[4] https://bugs.launchpad.net/horizon/+bug/1438133

With these four problems now fixed,reverted or temporarily worked around 
the tripleo CI is back running and jobs are passing.

I'm pretty confident we'll never be sure what the initial problem was 
but here is what I believe

At the start of all this our cloud had only a single symptomatic problem 
being caused by time drifting on compute nodes. This was causing a high 
percentage of our instances to fail to boot (depending on when various 
agents last reported back to the controller), this problem was probably 
getting progressively worse over the last while but nodepool was 
handling it by deleting instances and restarting new ones until in got 
an instance that worked (infra may be able to verify this and if the 
situation has improved).

The change to the nodepool script which broke our fedora image caused us 
to start looking at the cloud for problems, while debugging the cloud 
our test instances we're not getting IP addresses (due to time drift) 
which set us down the path of a real issue but not the same issue that 
prompted use to start looking for problems in the first place.

All of the other problems were either caused by us trying to debug or 
fix the cloud or were red herrings.

This cloud was deployed using tripleo over a year ago, at the time ntp 
wasn't part of the default install, it is now installed by default. At 
some stage in the future when we redeploy it using ironic (its currently 
nova-bm) we should ensure it is used to avoid hitting this again. Also 
when redeploying we should take advantage of some monitoring that is now 
part of tripleo or add some in places where it is lacking.

The rh1 cloud has been up and running for just over a year without any 
major problems/outages (at least that we know of), given that its run by 
developers in the tripleo community, I think this is a reasonable time 
between outages, although I would hope we could debug and fix any future 
problems if they arise, in a shorter time frame.

Feel free to ask questions if you want me to elaborate on anything, I 
hope I didn't ramble too much,


> Dan
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

More information about the OpenStack-dev mailing list