<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi,<br>
</p>
<p>I have an Openstack Juno setup with one controller node and 3
compute nodes. I installed it using packstack on centOS 7. One of
the compute node crashed twice. First crash was two days back and
second one today. It automatically restarts after the crash. The
crash log in /var/crash were different in both the instances.</p>
<p>Crash 1 log:</p>
<pre class="prettyprint"><code><span class="pln">CPU</span><span class="pun">:</span><span class="pln"> </span><span class="lit">5</span><span class="pln"> PID</span><span class="pun">:</span><span class="pln"> </span><span class="lit">16617</span><span class="pln"> </span><span class="typ">Comm</span><span class="pun">:</span><span class="pln"> pickup </span><span class="typ">Not</span><span class="pln"> tainted </span><span class="lit">3.10</span><span class="pun">.</span><span class="lit">0</span><span class="pun">-</span><span class="lit">123.13</span><span class="pun">.</span><span class="lit">2.el7.x86</span><span class="pln">_64 </span><span class="com">#1</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208387</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Hardware</span><span class="pln"> name</span><span class="pun">:</span><span class="pln"> </span><span class="typ">Intel</span><span class="pln"> </span><span class="typ">Corporation</span><span class="pln"> S2600CP</span><span class="pun">/</span><span class="pln">S2600CP</span><span class="pun">,</span><span class="pln"> BIOS SE5C600</span><span class="pun">.</span><span class="lit">86B</span><span class="pun">.</span><span class="lit">02.02</span><span class="pun">.</span><span class="lit">0002.122320131210</span><span class="pln"> </span><span class="lit">12</span><span class="pun">/</span><span class="lit">23</span><span class="pun">/</span><span class="lit">2013</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208389</span><span class="pun">]</span><span class="pln"> ffff8807de2b1cc8 </span><span class="lit">00000000a2deb902</span><span class="pln"> ffff8807de2b1c80 ffffffff815e232c
</span><span class="pun">[</span><span class="lit">11971.208392</span><span class="pun">]</span><span class="pln"> ffff8807de2b1cb8 ffffffff8105dee1 ffff8807f8e85f50 ffff8807f8e85f40
</span><span class="pun">[</span><span class="lit">11971.208394</span><span class="pun">]</span><span class="pln"> ffff8807f90db0c0 ffff8807f8e85f50 ffff8807f90db0c0 ffff8807de2b1d20
</span><span class="pun">[</span><span class="lit">11971.208396</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Call</span><span class="pln"> </span><span class="typ">Trace</span><span class="pun">:</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208401</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff815e232c</span><span class="pun">>]</span><span class="pln"> dump_stack</span><span class="pun">+</span><span class="lit">0x19</span><span class="pun">/</span><span class="lit">0x1b</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208405</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff8105dee1</span><span class="pun">>]</span><span class="pln"> warn_slowpath_common</span><span class="pun">+</span><span class="lit">0x61</span><span class="pun">/</span><span class="lit">0x80</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208408</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff8105df5c</span><span class="pun">>]</span><span class="pln"> warn_slowpath_fmt</span><span class="pun">+</span><span class="lit">0x5c</span><span class="pun">/</span><span class="lit">0x80</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208410</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff812cff82</span><span class="pun">>]</span><span class="pln"> __list_del_entry</span><span class="pun">+</span><span class="lit">0x82</span><span class="pun">/</span><span class="lit">0xd0</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208412</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff812cffdd</span><span class="pun">>]</span><span class="pln"> list_del</span><span class="pun">+</span><span class="lit">0xd</span><span class="pun">/</span><span class="lit">0x30</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208415</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff811776a3</span><span class="pun">>]</span><span class="pln"> unlink_anon_vmas</span><span class="pun">+</span><span class="lit">0x93</span><span class="pun">/</span><span class="lit">0x180</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208418</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff81168b88</span><span class="pun">>]</span><span class="pln"> free_pgtables</span><span class="pun">+</span><span class="lit">0xa8</span><span class="pun">/</span><span class="lit">0x120</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208420</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff81173556</span><span class="pun">>]</span><span class="pln"> exit_mmap</span><span class="pun">+</span><span class="lit">0xc6</span><span class="pun">/</span><span class="lit">0x1a0</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208422</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff8105b187</span><span class="pun">>]</span><span class="pln"> mmput</span><span class="pun">+</span><span class="lit">0x67</span><span class="pun">/</span><span class="lit">0xf0</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208424</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff81063dac</span><span class="pun">>]</span><span class="pln"> do_exit</span><span class="pun">+</span><span class="lit">0x28c</span><span class="pun">/</span><span class="lit">0xa60</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208426</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff810645ff</span><span class="pun">>]</span><span class="pln"> do_group_exit</span><span class="pun">+</span><span class="lit">0x3f</span><span class="pun">/</span><span class="lit">0xa0</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208428</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff81064674</span><span class="pun">>]</span><span class="pln"> </span><span class="typ">SyS_exit_group</span><span class="pun">+</span><span class="lit">0x14</span><span class="pun">/</span><span class="lit">0x20</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208431</span><span class="pun">]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff815f2a19</span><span class="pun">>]</span><span class="pln"> system_call_fastpath</span><span class="pun">+</span><span class="lit">0x16</span><span class="pun">/</span><span class="lit">0x1b</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208432</span><span class="pun">]</span><span class="pln"> </span><span class="pun">---[</span><span class="pln"> </span><span class="kwd">end</span><span class="pln"> trace ebed116bce4ce8eb </span><span class="pun">]---</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208437</span><span class="pun">]</span><span class="pln"> BUG</span><span class="pun">:</span><span class="pln"> unable to handle kernel NULL pointer dereference at </span><span class="pun">(</span><span class="kwd">null</span><span class="pun">)</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208471</span><span class="pun">]</span><span class="pln"> IP</span><span class="pun">:</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffff81177663</span><span class="pun">>]</span><span class="pln"> unlink_anon_vmas</span><span class="pun">+</span><span class="lit">0x53</span><span class="pun">/</span><span class="lit">0x180</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208493</span><span class="pun">]</span><span class="pln"> PGD </span><span class="lit">0</span><span class="pln">
</span><span class="pun">[</span><span class="lit">11971.208502</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Oops</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000</span><span class="pln"> </span><span class="pun">[</span><span class="com">#1] SMP</span></code></pre>
<p>Crash 2 log:<br>
</p>
<pre class="prettyprint"><code><span class="pun">[</span><span class="lit">321808.123092</span><span class="pun">]</span><span class="pln"> BUG</span><span class="pun">:</span><span class="pln"> unable to handle kernel paging request at ffffc90017456008
</span><span class="pun">[</span><span class="lit">321808.123122</span><span class="pun">]</span><span class="pln"> IP</span><span class="pun">:</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffffa03a8521</span><span class="pun">>]</span><span class="pln"> get_counters</span><span class="pun">+</span><span class="lit">0x91</span><span class="pun">/</span><span class="lit">0xd0</span><span class="pln"> </span><span class="pun">[</span><span class="pln">ip_tables</span><span class="pun">]</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123146</span><span class="pun">]</span><span class="pln"> PGD </span><span class="lit">81d437067</span><span class="pln"> PUD </span><span class="lit">81d4a4067</span><span class="pln"> PMD </span><span class="lit">7ec309067</span><span class="pln"> PTE </span><span class="lit">0</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123167</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Oops</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0002</span><span class="pln"> </span><span class="pun">[</span><span class="com">#1] SMP</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123179</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Modules</span><span class="pln"> linked </span><span class="kwd">in</span><span class="pun">:</span><span class="pln"> dummy vhost_net macvtap macvlan tun iptable_nat nf_nat_ipv4 nf_nat iptable_raw iptable_filter ip_tables nf_conntrack_ipv6 nf_defrag_ipv6 xt_mac xt_physdev xt_set ip_set_hash_ip ip_set nfnetlink veth ip6table_filter ip6_tables ebtable_nat ebtables openvswitch vxlan ip_tunnel gre sg ipt_REJECT xt_comment xt_conntrack xt_multiport nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack coretemp kvm_intel kvm crct10dif_pclmul iTCO_wdt iTCO_vendor_support crc32_pclmul crc32c_intel ghash_clmulni_intel sb_edac edac_core pcspkr lpc_ich ioatdma i2c_i801 mfd_core aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mei_me mei shpchp wmi acpi_cpufreq mperf nfsd auth_rpcgss nfs_acl lockd sunrpc bridge stp llc xfs libcrc32c sd_mod crc_t10di
f crct10d
if_common mgag200 syscopyarea sysfillrect
</span><span class="pun">[</span><span class="lit">321808.123452</span><span class="pun">]</span><span class="pln"> sysimgblt drm_kms_helper ttm isci igb ahci drm libsas libahci ptp scsi_transport_sas pps_core dca libata i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod </span><span class="pun">[</span><span class="kwd">last</span><span class="pln"> unloaded</span><span class="pun">:</span><span class="pln"> ip_tables</span><span class="pun">]</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123521</span><span class="pun">]</span><span class="pln"> CPU</span><span class="pun">:</span><span class="pln"> </span><span class="lit">10</span><span class="pln"> PID</span><span class="pun">:</span><span class="pln"> </span><span class="lit">110268</span><span class="pln"> </span><span class="typ">Comm</span><span class="pun">:</span><span class="pln"> iptables</span><span class="pun">-</span><span class="pln">save </span><span class="typ">Not</span><span class="pln"> tainted </span><span class="lit">3.10</span><span class="pun">.</span><span class="lit">0</span><span class="pun">-</span><span class="lit">123.13</span><span class="pun">.</span><span class="lit">2.el7.x86</span><span class="pln">_64 </span><span class="com">#1</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123547</span><span class="pun">]</span><span class="pln"> </span><span class="typ">Hardware</span><span class="pln"> name</span><span class="pun">:</span><span class="pln"> </span><span class="typ">Intel</span><span class="pln"> </span><span class="typ">Corporation</span><span class="pln"> S2600CP</span><span class="pun">/</span><span class="pln">S2600CP</span><span class="pun">,</span><span class="pln"> BIOS SE5C600</span><span class="pun">.</span><span class="lit">86B</span><span class="pun">.</span><span class="lit">02.02</span><span class="pun">.</span><span class="lit">0002.122320131210</span><span class="pln"> </span><span class="lit">12</span><span class="pun">/</span><span class="lit">23</span><span class="pun">/</span><span class="lit">2013</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123577</span><span class="pun">]</span><span class="pln"> task</span><span class="pun">:</span><span class="pln"> ffff88080a8f38e0 ti</span><span class="pun">:</span><span class="pln"> ffff8807ed398000 task</span><span class="pun">.</span><span class="pln">ti</span><span class="pun">:</span><span class="pln"> ffff8807ed398000
</span><span class="pun">[</span><span class="lit">321808.123599</span><span class="pun">]</span><span class="pln"> RIP</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0010</span><span class="pun">:[<</span><span class="pln">ffffffffa03a8521</span><span class="pun">>]</span><span class="pln"> </span><span class="pun">[<</span><span class="pln">ffffffffa03a8521</span><span class="pun">>]</span><span class="pln"> get_counters</span><span class="pun">+</span><span class="lit">0x91</span><span class="pun">/</span><span class="lit">0xd0</span><span class="pln"> </span><span class="pun">[</span><span class="pln">ip_tables</span><span class="pun">]</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123627</span><span class="pun">]</span><span class="pln"> RSP</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0018</span><span class="pun">:</span><span class="pln">ffff8807ed399da8 EFLAGS</span><span class="pun">:</span><span class="pln"> </span><span class="lit">00010286</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123643</span><span class="pun">]</span><span class="pln"> RAX</span><span class="pun">:</span><span class="pln"> ffffc90014ab72e8 RBX</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000000000010380</span><span class="pln"> RCX</span><span class="pun">:</span><span class="pln"> ffffc90017456000
</span><span class="pun">[</span><span class="lit">321808.123664</span><span class="pun">]</span><span class="pln"> RDX</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000000000000054</span><span class="pln"> RSI</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000000000000000</span><span class="pln"> RDI</span><span class="pun">:</span><span class="pln"> ffff88081e290380
</span><span class="pun">[</span><span class="lit">321808.123685</span><span class="pun">]</span><span class="pln"> RBP</span><span class="pun">:</span><span class="pln"> ffff8807ed399dc8 R08</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000000000000101</span><span class="pln"> R09</span><span class="pun">:</span><span class="pln"> ffff8807ed96caa0
</span><span class="pun">[</span><span class="lit">321808.123706</span><span class="pun">]</span><span class="pln"> R10</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000000000000000</span><span class="pln"> R11</span><span class="pun">:</span><span class="pln"> ffffffff8117b2e8 R12</span><span class="pun">:</span><span class="pln"> ffffffff819e4aa0
</span><span class="pun">[</span><span class="lit">321808.123727</span><span class="pun">]</span><span class="pln"> R13</span><span class="pun">:</span><span class="pln"> ffff8807ed96c800 R14</span><span class="pun">:</span><span class="pln"> ffffc90017455000 R15</span><span class="pun">:</span><span class="pln"> ffff88080c961ba0
</span><span class="pun">[</span><span class="lit">321808.123748</span><span class="pun">]</span><span class="pln"> FS</span><span class="pun">:</span><span class="pln"> </span><span class="lit">00007fdcc2078740</span><span class="pun">(</span><span class="lit">0000</span><span class="pun">)</span><span class="pln"> GS</span><span class="pun">:</span><span class="pln">ffff88081d940000</span><span class="pun">(</span><span class="lit">0000</span><span class="pun">)</span><span class="pln"> knlGS</span><span class="pun">:</span><span class="lit">0000000000000000</span><span class="pln">
</span><span class="pun">[</span><span class="lit">321808.123772</span><span class="pun">]</span><span class="pln"> CS</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0010</span><span class="pln"> DS</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000</span><span class="pln"> ES</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000</span><span class="pln"> CR0</span><span class="pun">:</span><span class="pln"> </span><span class="lit">0000000080050033</span><span class="pln">
</span><span class="pun"></span></code><code>[321808.123789] CR2: ffffc90017456008 CR3: 00000007ec30e000 CR4: 00000000001427e0
[321808.123810] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[321808.123831] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
</code>
</pre>
<p>I am not able to debug the problem. The setup was working fine
till now. I am unable to understand what is cuasing such a
behaviour. Please help.<br>
</p>
<p><br>
Thanks,<br>
</p>
<p>Priyanka<br>
</p>
</body>
</html>