<div dir="ltr">I found the exact messages in both servers and they match with the dates we had registered as reboots. So, I think this is my reboot problem. Thanks guys! </div><div class="gmail_extra"><br><br><div class="gmail_quote">
2014-07-24 15:20 GMT-03:00 Juan José Pavlik Salles <span dir="ltr"><<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Arne, you might be hitting the right nail here, I installed ipmitool and look at this:<div><br><div><div>root@cebolla:~# ipmitool sel elist</div></div><div>...</div><div><div> 47d | 07/15/2014 | 12:53:29 | System Event #0x83 | Timestamp Clock Sync | Asserted</div>
<div> 47e | 07/15/2014 | 12:53:30 | System Event #0x83 | Timestamp Clock Sync | Asserted</div><div> 47f | 07/15/2014 | 12:54:10 | System Event #0x83 | OEM System boot event | Asserted</div></div><div><div><b> 480 | 07/24/2014 | 06:04:08 | Memory Mmry ECC Sensor | Uncorrectable ECC | Asserted</b></div>
<div><b> 481 | 07/24/2014 | 06:05:10 | System Event #0x83 | Timestamp Clock Sync | Asserted</b></div><div><b> 482 | 07/24/2014 | 06:05:28 | System Event #0x83 | Timestamp Clock Sync | Asserted</b></div><div><b> 483 | 07/24/2014 | 06:06:05 | System Event #0x83 | OEM System boot event | Asserted</b></div>
<div>root@cebolla:~# </div></div></div><div><br></div><div>Going back in the logs, I see that the exact same 4 messages in the previous reboot dates. So it seems that one of my DIMM are dying.</div></div><div class="gmail_extra">
<br><br><div class="gmail_quote">2014-07-24 15:07 GMT-03:00 Arne Wiebalck <span dir="ltr"><<a href="mailto:Arne.Wiebalck@cern.ch" target="_blank">Arne.Wiebalck@cern.ch</a>></span>:<div><div class="h5"><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p dir="ltr">Does that mean that</p>
<p dir="ltr">$ ipmitool sel elist</p>
<p dir="ltr">returns nothing?</p>
<p dir="ltr">Arne</p>
<div>Am 24.07.2014 19:29 schrieb =?ISO-8859-1?Q?Juan_Jos=E9_Pavlik_Salles?= <<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>>:<br type="attribution">
</div><div><div>
<div>
<div dir="ltr">No problem Arne. I checked the ipmi config:
<div><br>
</div>
<div>
<div><i>root@cebolla:~# grep -v "^#" /etc/default/openipmi</i></div>
<div><i>IPMI_SI=yes</i></div>
<div><i>DEV_IPMI=yes</i></div>
<div><i>IPMI_WATCHDOG=no</i></div>
<div><i>IPMI_WATCHDOG_OPTIONS="timeout=60"</i></div>
<div><i>IPMI_POWEROFF=no</i></div>
<div><i>IPMI_POWERCYCLE=no</i></div>
<div><i>IPMI_IMB=no</i></div>
<div><i>root@cebolla:~# </i></div>
</div>
<div><br>
</div>
<div>Even though the IPMI interface is on, the watchdog is disabled. I'd like to try with another hardware just to check, but right now I haven't got any. </div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2014-07-24 13:30 GMT-03:00 Arne Wiebalck <span dir="ltr">
<<a href="mailto:Arne.Wiebalck@cern.ch" target="_blank">Arne.Wiebalck@cern.ch</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<p dir="ltr">Oops, I apparently wasn't reading carefully enough and mixed your issue with hosts and mine with guests.</p>
<p dir="ltr">Sorry for the noise!<br>
Arne</p>
<div>Am 24.07.2014 17:42 schrieb Tim Bell <<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>>:<br type="attribution">
</div>
</div>
<div>
<div><font><span style="font-size:10pt">
<div><br>
If it is the hypervisors rebooting, a possible scenario would be if you have a BMC and enabled watchdog. This will reboot the server if it does not call home to the BMC every 'n' seconds.<br>
<br>
If you have a very busy hypervisor, you may need to tune the watchdog timeout.<br>
<br>
I suspect something would be logged in the BMC ipmi sel logs but not sure.<br>
<br>
Tim<br>
<br>
> -----Original Message-----<br>
> From: Arne Wiebalck [<a href="mailto:Arne.Wiebalck@cern.ch" target="_blank">mailto:Arne.Wiebalck@cern.ch</a>]<br>
> Sent: 24 July 2014 17:10<br>
> To: Juan José Pavlik Salles<br>
> Cc: <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a><br>
> Subject: Re: [Openstack-operators] Compute nodes reboot periodically by their<br>
> own<br>
> <br>
> Hi,<br>
> <br>
> Your compute nodes reboot or are shut off?<br>
> <br>
> I am currently looking at some cases where VMs seem to spontaneously shut<br>
> themselves off. At least from the nova logs’ perspective there is no difference to<br>
> a normal shutdown, VM owners however confirm that they did not touch their<br>
> VMs. So far I was unable to explain this.<br>
> <br>
> This is with Havana on a RHEL6 derivative, though.<br>
> <br>
> Cheers,<br>
> Arne<br>
> <br>
> --<br>
> Arne Wiebalck<br>
> CERN IT<br>
> <br>
> On 24 Jul 2014, at 16:46, Juan José Pavlik Salles <<a href="mailto:jjpavlik@gmail.com" target="_blank">jjpavlik@gmail.com</a>> wrote:<br>
> <br>
> > Hello guys, We have got a small Grizzly cloud running since the begging of<br>
> 2013 with Ubuntu 12.04. 2 compute nodes, a storage node and a controller,<br>
> nothing too fancy. Everything works just fine, but... the compute nodes reboot<br>
> themselves periodically, sometimes every 2 weeks, some times once a month.<br>
> I've done almost everything I can think of: memory checks, analysed the logs,<br>
> moved all the VMs to one node, and I just can't find the problem.<br>
> ><br>
> > Have you ever heard this kind of behaviour on compute nodes? Any ideas<br>
> where I should look for the problem?<br>
> ><br>
> > Thanks in advance.<br>
> ><br>
> > --<br>
> > Pavlik Salles Juan José<br>
> > Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a><br>
> > _______________________________________________<br>
> > OpenStack-operators mailing list<br>
> > <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operator</a><br>
> > s<br>
> <br>
> <br>
> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</div>
</span></font></div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Pavlik Salles Juan José
<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div>
</div>
</div></div></div>
</blockquote></div></div></div><div><div class="h5"><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div>
</div>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr">Pavlik Salles Juan José<div>Blog - <a href="http://viviendolared.blogspot.com" target="_blank">http://viviendolared.blogspot.com</a></div></div>
</div>