Open Stack

Hi,

I am having Juno multinode setup with 1Controller, 1 Network and 10 Compute
nodes. There are 12 VMs running on that setup.

After some time one of my VM starts printing some strange messages on
console repeatedly, which says:

[78943.168505] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78946.592786] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78946.594413] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78947.617555] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78948.642004] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78949.600762] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78950.689530] Dead loop on virtual device vnf_sig_sp, fix it urgently!

vnf_sig_sp is one interface created internally on that VM (not by neutron).

After this VM runs in Kernel panic and then VM goes for a cyclic reboot.
I am attaching the logs as well.

Can anyone please suggest what should be done to fix this.

Thanks in advance!

BR,
Varun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150728/1394c274/attachment.html>
-------------- next part --------------
[78943.168505] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78946.592786] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78946.594413] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78947.617555] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78948.642004] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78949.600762] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78950.689530] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78952.290076] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78954.660765] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78955.592326] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78956.920754] Dead loop on virtual device vnf_sig_sp, fix it urgently!
[78984.036141] Stack:
[78984.036846] Call Trace:
[78984.037802] Code: 19 e1 df ff 5b c3 0f 1f 80 00 00 00 00 53 48 89 fb e8 77 34 c0 ff b8 00 00 01 00 f0 0f c1 03 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 53 48 89 
[78984.040002] Kernel panic - not syncing: softlockup: hung tasks
[78984.040002] Pid: 1342, comm: java Tainted: PF          ENX 3.0.101-0.47.52-default #1
[78984.040002] Call Trace:
[78984.040002]  [<ffffffff81004b95>] dump_trace+0x75/0x300
[78984.040002]  [<ffffffff81461d43>] dump_stack+0x69/0x6f
[78984.040002]  [<ffffffff81461ddc>] panic+0x93/0x201
[78984.040002]  [<ffffffff810c8261>] watchdog_timer_fn+0x181/0x190
[78984.040002]  [<ffffffff8108842e>] __run_hrtimer+0xbe/0x1a0
[78984.040002]  [<ffffffff81088739>] hrtimer_interrupt+0xd9/0x210
[78984.040002]  [<ffffffff81026f43>] smp_apic_timer_interrupt+0x63/0xa0
[78984.040002]  [<ffffffff8146d573>] apic_timer_interrupt+0x13/0x20
[78984.040002]  [<ffffffff81464e6e>] _raw_spin_lock_bh+0x1e/0x30
[78984.060008] BUG: soft lockup - CPU#1 stuck for 22s! [java:1345]
[78984.060009] Modules linked in: multimmap(PEN) tcp_diag inet_diag sch_htb xt_DROPDST(EN) ip_set_rule_ipportipportproto(EN) xt_set(E) ip_set_hash_ip(E) ip_set(E) ip_vs_rr(EN) ip6t_REJECT ipt_REJECT af_packet xt_tcpudp evip_sctpd(PFEN) nf_conntrack_ipv6(E) nf_defrag_ipv6 xfrm_user xt_multiport xt_connmark xt_state xt_mark xt_conntrack ip6table_mangle ip6table_filter iptable_filter ip_vs(EN) crc32c libcrc32c nf_conntrack_netlink nfnetlink ip6_tunnel(E) tunnel6 macvlan ev_drv(PEN) nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_HMARK(EN) ip6_tables(E) iptable_mangle ip_tables x_tables binfmt_misc mperf tipc(EX) softdog fuse loop dm_mod joydev usbhid hid sg sr_mod cdrom ata_generic ata_piix libata ipv6 ipv6_lib uhci_hcd rtc_cmos ehci_hcd xen_platform_pci acpiphp pci_hotplug processor thermal_sys hwmon serio_raw pcspkr scsi_mod usbcore button i2c_piix4 i2c_core usb_common nfs lockd fscache auth_rpcgss nfs_acl sunrpc virtio_balloon virtio_blk virtio_net virtio_pci virtio_ring virtio intel_agp intel_gtt floppy [last unloaded: multimmap]
[78984.060044] Supported: No, Proprietary and Unsupported modules are loaded
[78984.060045] CPU 1 
[78984.060046] Modules linked in: multimmap(PEN) tcp_diag inet_diag sch_htb xt_DROPDST(EN) ip_set_rule_ipportipportproto(EN) xt_set(E) ip_set_hash_ip(E) ip_set(E) ip_vs_rr(EN) ip6t_REJECT ipt_REJECT af_packet xt_tcpudp evip_sctpd(PFEN) nf_conntrack_ipv6(E) nf_defrag_ipv6 xfrm_user xt_multiport xt_connmark xt_state xt_mark xt_conntrack ip6table_mangle ip6table_filter iptable_filter ip_vs(EN) crc32c libcrc32c nf_conntrack_netlink nfnetlink ip6_tunnel(E) tunnel6 macvlan ev_drv(PEN) nf_conntrack_ipv4 nf_conntrack nf_defrag_ipv4 xt_HMARK(EN) ip6_tables(E) iptable_mangle ip_tables x_tables binfmt_misc mperf tipc(EX) softdog fuse loop dm_mod joydev usbhid hid sg sr_mod cdrom ata_generic ata_piix libata ipv6 ipv6_lib uhci_hcd rtc_cmos ehci_hcd xen_platform_pci acpiphp pci_hotplug processor thermal_sys hwmon serio_raw pcspkr scsi_mod usbcore button i2c_piix4 i2c_core usb_common nfs lockd fscache auth_rpcgss nfs_acl sunrpc virtio_balloon virtio_blk virtio_net virtio_pci virtio_ring virtio intel_agp intel_gtt floppy [last unloaded: multimmap]
[78984.060072] Supported: No, Proprietary and Unsupported modules are loaded
[78984.060073] 
[78984.060074] Pid: 1345, comm: java Tainted: PF          ENX 3.0.101-0.47.52-default #1 Fedora Project OpenStack Nova
[78984.060077] RIP: 0010:[<ffffffff81464e6e>]  [<ffffffff81464e6e>] _raw_spin_lock_bh+0x1e/0x30
[78984.060081] RSP: 0018:ffff88033999f7e8  EFLAGS: 00000297
[78984.060082] RAX: 000000000000cdcc RBX: dead000000200200 RCX: 0000000000001020
[78984.060083] RDX: 000000000000cdc6 RSI: ffffffff81a73580 RDI: ffffffffa0327b08
[78984.060085] RBP: 0000000000000206 R08: 0000000000000000 R09: 0000000000000002
[78984.060086] R10: ffff88033999f95c R11: ffff88033999fb38 R12: ffffffff8146d56e
[78984.060087] R13: 00a800003999f81c R14: ffffffffa02f5da4 R15: ffff8804217f1240
[78984.060089] FS:  00007f1bf27ca700(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000
[78984.060091] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[78984.060092] CR2: 00007fca964bd000 CR3: 0000000391653000 CR4: 00000000001406e0
[78984.060097] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[78984.060101] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[78984.060102] Process java (pid: 1345, threadinfo ffff88033999e000, task ffff88033999c540)
[78984.060103] Stack:
[78984.060104]  ffffffff81a73580 ffffffffa02f5dfc ffffffff81a73a00 ffffffff81a73a00
[78984.060106]  ffffffff81a73580 ffffffffa02ef706 000000043999f8d8 ffff88033999fb4c
[78984.060108]  ffffffff81a73a00 ffff880422106000 0000000000000000 0000000000000000
[78984.060111] Call Trace:
[78984.060122]  [<ffffffffa02f5dfc>] fib6_run_gc+0x1c/0xe0 [ipv6_lib]
[78984.060141]  [<ffffffffa02ef706>] ip6_dst_gc+0x96/0x120 [ipv6_lib]
[78984.060151]  [<ffffffff813aeed3>] dst_alloc+0x173/0x190
[78984.060159]  [<ffffffffa02f2bef>] ip6_rt_copy+0x2f/0x260 [ipv6_lib]
[78984.060174]  [<ffffffffa02f38ac>] rt6_alloc_cow+0x1c/0x230 [ipv6_lib]
[78984.060189]  [<ffffffffa02f3e0a>] ip6_pol_route+0x34a/0x380 [ipv6_lib]
[78984.060206]  [<ffffffffa0313813>] fib6_rule_action+0xb3/0x180 [ipv6_lib]
[78984.060246]  [<ffffffffa0405646>] my_fib6_rule_action+0x46/0xb0 [ev_drv]
[78984.060253]  [<ffffffff813bea26>] fib_rules_lookup+0xc6/0x150
[78984.060253]  [<ffffffffa031392e>] fib6_rule_lookup+0x3e/0x70 [ipv6_lib]
[78984.060253]  [<ffffffffa02e37eb>] ip6_dst_lookup_tail+0xcb/0xe0 [ipv6_lib]
[78984.060253]  [<ffffffffa02e3a2a>] ip6_dst_lookup_flow+0x3a/0xa0 [ipv6_lib]
[78984.060253]  [<ffffffffa03112ce>] inet6_csk_xmit+0x1be/0x240 [ipv6_lib]
[78984.060253]  [<ffffffff813f50b0>] tcp_transmit_skb+0x510/0x710
[78984.060253]  [<ffffffff813f7af8>] tcp_write_xmit+0x1e8/0x510
[78984.060253]  [<ffffffff813f7e85>] __tcp_push_pending_frames+0x25/0x60
[78984.060253]  [<ffffffff813ea3b0>] tcp_sendmsg+0x7f0/0xa00
[78984.060253]  [<ffffffff8138fce4>] sock_aio_write+0x114/0x130
[78984.060253]  [<ffffffff8115d3c8>] do_sync_write+0xc8/0x110
[78984.060253]  [<ffffffff8115da5c>] vfs_write+0x12c/0x140
[78984.060253]  [<ffffffff8115db73>] sys_write+0x53/0xa0
[78984.060253]  [<ffffffff8146cb92>] system_call_fastpath+0x16/0x1b
[78984.060253]  [<00007f1c2e2a979d>] 0x7f1c2e2a979c
[78984.060253] Code: 19 e1 df ff 5b c3 0f 1f 80 00 00 00 00 53 48 89 fb e8 77 34 c0 ff b8 00 00 01 00 f0 0f c1 03 0f b7 d0 c1 e8 10 39 c2 74 07 f3 90 <0f> b7 13 eb f5 5b c3 66 66 2e 0f 1f 84 00 00 00 00 00 53 48 89 
[78984.060253] Call Trace:
[78984.060253]  [<ffffffffa02f5dfc>] fib6_run_gc+0x1c/0xe0 [ipv6_lib]
[78984.060253]  [<ffffffffa02ef706>] ip6_dst_gc+0x96/0x120 [ipv6_lib]
[78984.060253]  [<ffffffff813aeed3>] dst_alloc+0x173/0x190
[78984.060253]  [<ffffffffa02f2bef>] ip6_rt_copy+0x2f/0x260 [ipv6_lib]
[78984.060253]  [<ffffffffa02f38ac>] rt6_alloc_cow+0x1c/0x230 [ipv6_lib]
[78984.060253]  [<ffffffffa02f3e0a>] ip6_pol_route+0x34a/0x380 [ipv6_lib]
[78984.060253]  [<ffffffffa0313813>] fib6_rule_action+0xb3/0x180 [ipv6_lib]
[78984.060253]  [<ffffffffa0405646>] my_fib6_rule_action+0x46/0xb0 [ev_drv]
[78984.060253]  [<ffffffff813bea26>] fib_rules_lookup+0xc6/0x150
[78984.060253]  [<ffffffffa031392e>] fib6_rule_lookup+0x3e/0x70 [ipv6_lib]
[78984.060253]  [<ffffffffa02e37eb>] ip6_dst_lookup_tail+0xcb/0xe0 [ipv6_lib]
[78984.060253]  [<ffffffffa02e3a2a>] ip6_dst_lookup_flow+0x3a/0xa0 [ipv6_lib]
[78984.060253]  [<ffffffffa03112ce>] inet6_csk_xmit+0x1be/0x240 [ipv6_lib]
[78984.060253]  [<ffffffff813f50b0>] tcp_transmit_skb+0x510/0x710
[78984.060253]  [<ffffffff813f7af8>] tcp_write_xmit+0x1e8/0x510
[78984.060253]  [<ffffffff813f7e85>] __tcp_push_pending_frames+0x25/0x60
[78984.060253]  [<ffffffff813ea3b0>] tcp_sendmsg+0x7f0/0xa00
[78984.060253]  [<ffffffff8138fce4>] sock_aio_write+0x114/0x130
[78984.060253]  [<ffffffff8115d3c8>] do_sync_write+0xc8/0x110
[78984.060253]  [<ffffffff8115da5c>] vfs_write+0x12c/0x140
[78984.060253]  [<ffffffff8115db73>] sys_write+0x53/0xa0
[78984.060253]  [<ffffffff8146cb92>] system_call_fastpath+0x16/0x1b
[78984.060253]  [<00007f1c2e2a979d>] 0x7f1c2e2a979c
[78984.060253] Kernel panic - not syncing: softlockup: hung tasks
[78984.060253] Pid: 1345, comm: java Tainted: PF          ENX 3.0.101-0.47.52-default #1
[78984.060253] Call Trace:
[78984.060253]  [<ffffffff81004b95>] dump_trace+0x75/0x300
[78984.060253]  [<ffffffff81461d43>] dump_stack+0x69/0x6f
[78984.060253]  [<ffffffff81461ddc>] panic+0x93/0x201
[78984.060253]  [<ffffffff810c8261>] watchdog_timer_fn+0x181/0x190
[78984.060253]  [<ffffffff8108842e>] __run_hrtimer+0xbe/0x1a0
[78984.060253]  [<ffffffff81088739>] hrtimer_interrupt+0xd9/0x210
[78984.060253]  [<ffffffff81026f43>] smp_apic_timer_interrupt+0x63/0xa0
[78984.060253]  [<ffffffff8146d573>] apic_timer_interrupt+0x13/0x20
[78984.060253]  [<ffffffff81464e6e>] _raw_spin_lock_bh+0x1e/0x30
[78984.060253]  [<ffffffffa02f5dfc>] fib6_run_gc+0x1c/0xe0 [ipv6_lib]
[78984.060253]  [<ffffffffa02ef706>] ip6_dst_gc+0x96/0x120 [ipv6_lib]
[78984.060253]  [<ffffffff813aeed3>] dst_alloc+0x173/0x190
[78984.060253]  [<ffffffffa02f2bef>] ip6_rt_copy+0x2f/0x260 [ipv6_lib]
[78984.060253]  [<ffffffffa02f38ac>] rt6_alloc_cow+0x1c/0x230 [ipv6_lib]
[78984.060253]  [<ffffffffa02f3e0a>] ip6_pol_route+0x34a/0x380 [ipv6_lib]
[78984.060253]  [<ffffffffa0313813>] fib6_rule_action+0xb3/0x180 [ipv6_lib]
[78984.060253]  [<ffffffffa0405646>] my_fib6_rule_action+0x46/0xb0 [ev_drv]
[78984.060253]  [<ffffffff813bea26>] fib_rules_lookup+0xc6/0x150
[78984.060253]  [<ffffffffa031392e>] fib6_rule_lookup+0x3e/0x70 [ipv6_lib]
[78984.060253]  [<ffffffffa02e37eb>] ip6_dst_lookup_tail+0xcb/0xe0 [ipv6_lib]
[78984.060253]  [<ffffffffa02e3a2a>] ip6_dst_lookup_flow+0x3a/0xa0 [ipv6_lib]
[78984.060253]  [<ffffffffa03112ce>] inet6_csk_xmit+0x1be/0x240 [ipv6_lib]
[78984.060253]  [<ffffffff813f50b0>] tcp_transmit_skb+0x510/0x710
[78984.060253]  [<ffffffff813f7af8>] tcp_write_xmit+0x1e8/0x510
[78984.060253]  [<ffffffff813f7e85>] __tcp_push_pending_frames+0x25/0x60
[78984.060253]  [<ffffffff813ea3b0>] tcp_sendmsg+0x7f0/0xa00
[78984.060253]  [<ffffffff8138fce4>] sock_aio_write+0x114/0x130
[78984.060253]  [<ffffffff8115d3c8>] do_sync_write+0xc8/0x110
[78984.060253]  [<ffffffff8115da5c>] vfs_write+0x12c/0x140
[78984.060253]  [<ffffffff8115db73>] sys_write+0x53/0xa0
[78984.060253]  [<ffffffff8146cb92>] system_call_fastpath+0x16/0x1b
[78984.060253]  [<00007f1c2e2a979d>] 0x7f1c2e2a979c
[78984.040002]  [<ffffffffa02f5dfc>] fib6_run_gc+0x1c/0xe0 [ipv6_lib]
[78984.040002]  [<ffffffffa02ef706>] ip6_dst_gc+0x96/0x120 [ipv6_lib]
[78984.040002]  [<ffffffff813aeed3>] dst_alloc+0x173/0x190
[78984.040002]  [<ffffffffa02f2bef>] ip6_rt_copy+0x2f/0x260 [ipv6_lib]
[78984.040002]  [<ffffffffa02f38ac>] rt6_alloc_cow+0x1c/0x230 [ipv6_lib]
[78984.040002]  [<ffffffffa02f3e0a>] ip6_pol_route+0x34a/0x380 [ipv6_lib]
[78984.040002]  [<ffffffffa0313813>] fib6_rule_action+0xb3/0x180 [ipv6_lib]
[78984.040002]  [<ffffffffa0405646>] my_fib6_rule_action+0x46/0xb0 [ev_drv]
[78984.040002]  [<ffffffff813bea26>] fib_rules_lookup+0xc6/0x150
[78984.040002]  [<ffffffffa031392e>] fib6_rule_lookup+0x3e/0x70 [ipv6_lib]
[78984.040002]  [<ffffffffa02e37eb>] ip6_dst_lookup_tail+0xcb/0xe0 [ipv6_lib]
[78984.040002]  [<ffffffffa02e3a2a>] ip6_dst_lookup_flow+0x3a/0xa0 [ipv6_lib]
[78984.040002]  [<ffffffffa03112ce>] inet6_csk_xmit+0x1be/0x240 [ipv6_lib]
[78984.040002]  [<ffffffff813f50b0>] tcp_transmit_skb+0x510/0x710
[78984.040002]  [<ffffffff813f7af8>] tcp_write_xmit+0x1e8/0x510
[78984.040002]  [<ffffffff813f7e85>] __tcp_push_pending_frames+0x25/0x60
[78984.040002]  [<ffffffff813ea3b0>] tcp_sendmsg+0x7f0/0xa00
[78984.040002]  [<ffffffff8138fce4>] sock_aio_write+0x114/0x130
[78984.040002]  [<ffffffff8115d3c8>] do_sync_write+0xc8/0x110
[78984.040002]  [<ffffffff8115da5c>] vfs_write+0x12c/0x140
[78984.040002]  [<ffffffff8115db73>] sys_write+0x53/0xa0
[78984.040002]  [<ffffffff8146cb92>] system_call_fastpath+0x16/0x1b
[78984.040002]  [<00007f1c2e2a979d>] 0x7f1c2e2a979c
[78984.060253] Rebooting in 5 seconds..
[78984.040002] Rebooting in 5 seconds..