<div dir="ltr"><div>We may also need crush dump and System.map to get us more clear.<br><br></div> - Kinjo<br><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 3, 2015 at 2:24 PM, David S. <span dir="ltr"><<a href="mailto:david@zeromail.us" target="_blank">david@zeromail.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Priyanka,<div><br></div><div>Can you post dmesg and boot.log from your host?</div><div><br></div><div>Thank you</div></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><br><br>
Best regards,<br>David S.<br>
------------------------------------------------<br>
e. <a href="mailto:david@zeromail.us" target="_blank">david@zeromail.us</a><br>
w. <a href="http://blog.pnyet.web.id" target="_blank">http://blog.pnyet.web.id</a></div></div></div>
<br><div class="gmail_quote">On Wed, Jun 3, 2015 at 10:22 AM, Priyanka <span dir="ltr"><<a href="mailto:ppnaik@cse.iitb.ac.in" target="_blank">ppnaik@cse.iitb.ac.in</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
The hardware as well as software configurations are the same on all
nodes.<br>
<br>
Thanks,<br>
<br>
Priyanka<div><div><br>
<br>
<br>
<div>On Tuesday 02 June 2015 05:49 PM,
Shinobu Kinjo wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Hi,<br>
</div>
<div><br>
When were the node crashed?<br>
How about other two nodes?<br>
</div>
Is there any difference between crashed node and others in
terms of not only hardware but also software?<br>
<br>
</div>
- Kinjo<br>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jun 2, 2015 at 3:31 PM,
Priyanka <span dir="ltr"><<a href="mailto:ppnaik@cse.iitb.ac.in" target="_blank">ppnaik@cse.iitb.ac.in</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Sir,<br>
<br>
uname -r<br>
<br>
3.10.0-123.13.2.el7.x86_64<br>
<br>
I have stopped iptables services since the start. So,
should I still follow what they have mentioned in the
5th comment?<br>
<br>
Thanks,<br>
<br>
Priyanka<br>
<br>
<br>
On Tuesday 02 June 2015 11:49 AM, Matt Taylor wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Priyanka,<br>
<br>
Are you using an old kernel ('uname -r' please)?
I've seen issues with unlink_anon_vmas in Fedora, so
this is rather interesting.<br>
<br>
In regards to the 2nd kernel panic, it indicates
that it's due to iptables and get_counters.<br>
<br>
It's possibly related to this:<br>
<br>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1089569" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1089569</a><br>
<br>
Might be worth trying what was mentioned in the 5th
comment.<br>
<br>
Regards,<br>
Matt.<br>
<br>
On 2/06/2015 15:14, Priyanka wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
I have an Openstack Juno setup with one controller
node and 3 compute<br>
nodes. I installed it using packstack on centOS 7.
One of the compute<br>
node crashed twice. First crash was two days back
and second one today.<br>
It automatically restarts after the crash. The
crash log in /var/crash<br>
were different in both the instances.<br>
<br>
Crash 1 log:<br>
<br>
|CPU: 5 PID: 16617 Comm: pickupNot
tainted3.10.0-123.13.2.el7.x86_64#1<br>
[11971.208387] Hardware name: Intel
Corporation S2600CP/S2600CP, BIOS
SE5C600.86B.02.02.0002.122320131210 12/23/2013<br>
[11971.208389] ffff8807de2b1cc800000000a2deb902
ffff8807de2b1c80 ffffffff815e232c<br>
[11971.208392] ffff8807de2b1cb8 ffffffff8105dee1
ffff8807f8e85f50 ffff8807f8e85f40<br>
[11971.208394] ffff8807f90db0c0 ffff8807f8e85f50
ffff8807f90db0c0 ffff8807de2b1d20<br>
[11971.208396] Call Trace:<br>
[11971.208401] [<ffffffff815e232c>]
dump_stack+0x19/0x1b<br>
[11971.208405] [<ffffffff8105dee1>]
warn_slowpath_common+0x61/0x80<br>
[11971.208408] [<ffffffff8105df5c>]
warn_slowpath_fmt+0x5c/0x80<br>
[11971.208410] [<ffffffff812cff82>]
__list_del_entry+0x82/0xd0<br>
[11971.208412] [<ffffffff812cffdd>]
list_del+0xd/0x30<br>
[11971.208415] [<ffffffff811776a3>]
unlink_anon_vmas+0x93/0x180<br>
[11971.208418] [<ffffffff81168b88>]
free_pgtables+0xa8/0x120<br>
[11971.208420] [<ffffffff81173556>]
exit_mmap+0xc6/0x1a0<br>
[11971.208422] [<ffffffff8105b187>]
mmput+0x67/0xf0<br>
[11971.208424] [<ffffffff81063dac>]
do_exit+0x28c/0xa60<br>
[11971.208426] [<ffffffff810645ff>]
do_group_exit+0x3f/0xa0<br>
[11971.208428] [<ffffffff81064674>]
SyS_exit_group+0x14/0x20<br>
[11971.208431] [<ffffffff815f2a19>]
system_call_fastpath+0x16/0x1b<br>
[11971.208432] ---[ end trace
ebed116bce4ce8eb]---<br>
[11971.208437] BUG: unable to handle kernel NULL
pointer dereference at(null)<br>
[11971.208471] IP: [<ffffffff81177663>]
unlink_anon_vmas+0x53/0x180<br>
[11971.208493] PGD0<br>
[11971.208502] Oops: 0000 [#1] SMP|<br>
<br>
Crash 2 log:<br>
<br>
|[321808.123092] BUG: unable to handle kernel
paging request at ffffc90017456008<br>
[321808.123122] IP: [<ffffffffa03a8521>]
get_counters+0x91/0xd0 [ip_tables]<br>
[321808.123146] PGD81d437067 PUD81d4a4067
PMD7ec309067 PTE0<br>
[321808.123167] Oops: 0002 [#1] SMP<br>
[321808.123179] Modules linkedin: dummy
vhost_net macvtap macvlan tun iptable_nat
nf_nat_ipv4 nf_nat iptable_raw iptable_filter
ip_tables nf_conntrack_ipv6 nf_defrag_ipv6 xt_mac
xt_physdev xt_set ip_set_hash_ip ip_set nfnetlink
veth ip6table_filter ip6_tables ebtable_nat
ebtables openvswitch vxlan ip_tunnel gre sg
ipt_REJECT xt_comment xt_conntrack xt_multiport
nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack
coretemp kvm_intel kvm crct10dif_pclmul iTCO_wdt
iTCO_vendor_support crc32_pclmul crc32c_intel
ghash_clmulni_intel sb_edac edac_core pcspkr
lpc_ich ioatdma i2c_i801 mfd_core aesni_intel lrw
gf128mul glue_helper ablk_helper cryptd mei_me mei
shpchp wmi acpi_cpufreq mperf nfsd auth_rpcgss
nfs_acl lockd sunrpc bridge stp llc xfs libcrc32c
sd_mod crc_t10di<br>
f crct10d<br>
<br>
if_common mgag200 syscopyarea sysfillrect<br>
[321808.123452] sysimgblt drm_kms_helper ttm
isci igb ahci drm libsas libahci ptp
scsi_transport_sas pps_core dca libata
i2c_algo_bit i2c_core dm_mirror dm_region_hash
dm_log dm_mod[last unloaded: ip_tables]<br>
[321808.123521] CPU: 10 PID: 110268 Comm:
iptables-saveNot
tainted3.10.0-123.13.2.el7.x86_64#1<br>
[321808.123547] Hardware name: Intel
Corporation S2600CP/S2600CP, BIOS
SE5C600.86B.02.02.0002.122320131210 12/23/2013<br>
[321808.123577] task: ffff88080a8f38e0 ti:
ffff8807ed398000 task.ti: ffff8807ed398000<br>
[321808.123599] RIP:
0010:[<ffffffffa03a8521>]
[<ffffffffa03a8521>]
get_counters+0x91/0xd0 [ip_tables]<br>
[321808.123627] RSP: 0018:ffff8807ed399da8
EFLAGS: 00010286<br>
[321808.123643] RAX: ffffc90014ab72e8 RBX:
0000000000010380 RCX: ffffc90017456000<br>
[321808.123664] RDX: 0000000000000054 RSI:
0000000000000000 RDI: ffff88081e290380<br>
[321808.123685] RBP: ffff8807ed399dc8 R08:
0000000000000101 R09: ffff8807ed96caa0<br>
[321808.123706] R10: 0000000000000000 R11:
ffffffff8117b2e8 R12: ffffffff819e4aa0<br>
[321808.123727] R13: ffff8807ed96c800 R14:
ffffc90017455000 R15: ffff88080c961ba0<br>
[321808.123748] FS: 00007fdcc2078740(0000)
GS:ffff88081d940000(0000) knlGS:0000000000000000<br>
[321808.123772] CS: 0010 DS: 0000 ES: 0000
CR0: 0000000080050033<br>
||[321808.123789] CR2: ffffc90017456008 CR3:
00000007ec30e000 CR4: 00000000001427e0<br>
[321808.123810] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000<br>
[321808.123831] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400<br>
|<br>
<br>
I am not able to debug the problem. The setup was
working fine till now.<br>
I am unable to understand what is cuasing such a
behaviour. Please help.<br>
<br>
<br>
Thanks,<br>
<br>
Priyanka<br>
<br>
<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br>
</blockquote>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</blockquote>
<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
</blockquote>
</div>
<br>
<br clear="all"><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div><a href="http://i-shinobu.hatenablog.com/" target="_blank">Life w/ Linux</a><br>
</div>
</div>
</div>
</div>
</div>
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
</font></span></div></div></div><span class="HOEnZb"><font color="#888888">
<br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></font></span></blockquote></div><br></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><a href="http://i-shinobu.hatenablog.com/" target="_blank">Life w/ Linux</a><br></div></div></div></div></div>
</div></div></div></div></div>