<p dir="ltr">Looking into it now.</p>
<div class="gmail_quote">On 27 Feb 2014 15:56, "Derek Higgins" <<a href="mailto:derekh@redhat.com">derekh@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On 25/02/14 00:08, Robert Collins wrote:<br>
> Today we had an outage of the tripleo test cloud :(.<br>
><br>
> tl;dr:<br>
> - we were down for 14 hours<br>
> - we don't know the fundamental cause<br>
> - infra were not inconvenienced - yaaay<br>
> - its all ok now.<br>
Looks like we've hit the same problem again tonight, I've<br>
o rebooted the server<br>
o fixed up the hostname<br>
o restarted nova and neutron services on the controller<br>
<br>
VM's still not getting IP's, I'm not seeing dhcp requests from them<br>
coming into dnsmasq, spent some time trying to figure out the problem,<br>
no luck, I'll pick up in this in a few hours if nobody else has before then.<br>
<br>
><br>
> Read on for more information, what little we have.<br>
><br>
> We don't know exactly why it happened yet, but the control plane<br>
> dropped off the network. Console showed node still had a correct<br>
> networking configuration, including openflow rules and bridges. The<br>
> node was arpingable, and could arping out, but could not be pinged.<br>
> Tcpdump showed the node sending a ping reply on it's raw ethernet<br>
> device, but other machines on the same LAN did not see the packet.<br>
><br>
> From syslog we can see<br>
> Feb 24 06:28:31 ci-overcloud-notcompute0-gxezgcvv4v2q kernel:<br>
> [1454708.543053] hpsa 0000:06:00.0: cmd_alloc returned NULL!<br>
> events<br>
><br>
> around the time frame that the drop-off would have happened, but they<br>
> go back many hours before and after that.<br>
><br>
> After exhausting everything that came to mind we rebooted the machine,<br>
> which promptly spat an NMI trace into the console:<br>
><br>
> [1502354.552431] [<ffffffff810fdf98>] rcu_eqs_enter_common.isra.43+0x208/0x220<br>
> [1502354.552491] [<ffffffff810ff9ed>] rcu_irq_exit+0x5d/0x90<br>
> [1502354.552549] [<ffffffff81067670>] irq_exit+0x80/0xc0<br>
> [1502354.552605] [<ffffffff816f9605>] smp_apic_timer_interrupt+0x45/0x60<br>
> [1502354.552665] [<ffffffff816f7f9d>] apic_timer_interrupt+0x6d/0x80<br>
> [1502354.552722] <EOI> <NMI> [<ffffffff816e1384>] ? panic+0x193/0x1d7<br>
> [1502354.552880] [<ffffffffa02d18e5>] hpwdt_pretimeout+0xe5/0xe5 [hpwdt]<br>
> [1502354.552939] [<ffffffff816efc88>] nmi_handle.isra.3+0x88/0x180<br>
> [1502354.552997] [<ffffffff816eff11>] do_nmi+0x191/0x330<br>
> [1502354.553053] [<ffffffff816ef201>] end_repeat_nmi+0x1e/0x2e<br>
> [1502354.553111] [<ffffffff813d46c2>] ? intel_idle+0xc2/0x120<br>
> [1502354.553168] [<ffffffff813d46c2>] ? intel_idle+0xc2/0x120<br>
> [1502354.553226] [<ffffffff813d46c2>] ? intel_idle+0xc2/0x120<br>
> [1502354.553282] <<EOE>> [<ffffffff8159fe90>] cpuidle_enter_state+0x40/0xc0<br>
> [1502354.553408] [<ffffffff8159ffd9>] cpuidle_idle_call+0xc9/0x210<br>
> [1502354.553466] [<ffffffff8101bafe>] arch_cpu_idle+0xe/0x30<br>
> [1502354.553523] [<ffffffff810b54c5>] cpu_startup_entry+0xe5/0x280<br>
> [1502354.553581] [<ffffffff816d64b7>] rest_init+0x77/0x80<br>
> [1502354.553638] [<ffffffff81d26ef7>] start_kernel+0x40a/0x416<br>
> [1502354.553695] [<ffffffff81d268f6>] ? repair_env_string+0x5c/0x5c<br>
> [1502354.553753] [<ffffffff81d26120>] ? early_idt_handlers+0x120/0x120<br>
> [1502354.553812] [<ffffffff81d265de>] x86_64_start_reservations+0x2a/0x2c<br>
> [1502354.553871] [<ffffffff81d266e8>] x86_64_start_kernel+0x108/0x117<br>
> [1502354.553929] ---[ end trace 166b62e89aa1f54b ]---<br>
><br>
> 'yay'. After that, a power reset in the console, it came up ok, just<br>
> needed a minor nudge to refresh it's heat configuration and we were up<br>
> and running again.<br>
><br>
> For some reason, neutron decided to rename it's agents at this point<br>
> and we had to remove and reattach the l3 agent before VM connectivity<br>
> was restored.<br>
> <a href="https://bugs.launchpad.net/tripleo/+bug/1284354" target="_blank">https://bugs.launchpad.net/tripleo/+bug/1284354</a><br>
><br>
> However, about 90 nodepool nodes were stuck in states like ACTIVE<br>
> deleting, and did not clear until we did a rolling restart of every<br>
> nova compute process.<br>
> <a href="https://bugs.launchpad.net/tripleo/+bug/1284356" target="_blank">https://bugs.launchpad.net/tripleo/+bug/1284356</a><br>
><br>
> Cheers,<br>
> Rob<br>
><br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div>