[openstack-qa] [OpenStack-Infra] tgt restart fails in Cinder startup "start: job failed to start"

Roey Chen roeyc at mellanox.com
Tue Mar 11 10:06:47 UTC 2014

Forwarding the answer to the relevant mailing lists:



Hope this could help,

I've encountered this issue myself not to long ago on Ubuntu 12.04 host,
it didn't happen again after messing with the Kernel Semaphore Limits parameters [1]:

Adding this [2] line to `/etc/sysctl.conf` seems to do the trick.

- Roey

[1] http://paste.openstack.org/show/73086/
[2] http://paste.openstack.org/show/73082/

From: Sukhdev Kapur [mailto:sukhdevkapur at gmail.com]
Sent: Monday, March 10, 2014 5:56 PM
To: Dane Leblanc (leblancd)
Cc: OpenStack Development Mailing List (not for usage questions); openstack-infra at lists.openstack.org; openstack-qa at lists.openstack.org
Subject: Re: [OpenStack-Infra] tgt restart fails in Cinder startup "start: job failed to start"

I see the same issue. This issue has crept in during the latest flurry of check-ins. I started noticing this issue a day or two before the Icehouse Feature Freeze deadline.

I tried restarting tgt as well, but, it does not help.

However, rebooting the VM helps clear it up.

Has anybody else seen it as well? Does anybody have a solution for it?


On Mon, Mar 10, 2014 at 8:37 AM, Dane Leblanc (leblancd) <leblancd at cisco.com<mailto:leblancd at cisco.com>> wrote:
I don't know if anyone can give me some troubleshooting advice with this issue.

I'm seeing an occasional problem whereby after several DevStack unstack.sh/stack.sh<http://unstack.sh/stack.sh> cycles, the tgt daemon (tgtd) fails to start during Cinder startup.  Here's a snippet from the stack.sh log:

2014-03-10 07:09:45.214 | Starting Cinder
2014-03-10 07:09:45.215 | + return 0
2014-03-10 07:09:45.216 | + sudo rm -f /etc/tgt/conf.d/stack.conf
2014-03-10 07:09:45.217 | + _configure_tgt_for_config_d
2014-03-10 07:09:45.218 | + [[ ! -d /etc/tgt/stack.d/ ]]
2014-03-10 07:09:45.219 | + is_ubuntu
2014-03-10 07:09:45.220 | + [[ -z deb ]]
2014-03-10 07:09:45.221 | + '[' deb = deb ']'
2014-03-10 07:09:45.222 | + sudo service tgt restart
2014-03-10 07:09:45.223 | stop: Unknown instance:
2014-03-10 07:09:45.619 | start: Job failed to start
jenkins at neutronpluginsci:~/devstack$ 2014-03-10 07:09:45.621 | + exit_trap
2014-03-10 07:09:45.622 | + local r=1
2014-03-10 07:09:45.623 | ++ jobs -p
2014-03-10 07:09:45.624 | + jobs=
2014-03-10 07:09:45.625 | + [[ -n '' ]]
2014-03-10 07:09:45.626 | + exit 1

If I try to restart tgt manually without success:

jenkins at neutronpluginsci:~$ sudo service tgt restart
stop: Unknown instance:
start: Job failed to start
jenkins at neutronpluginsci:~$ sudo tgtd
librdmacm: couldn't read ABI version.
librdmacm: assuming: 4
CMA: unable to get RDMA device list
(null): iser_ib_init(3263) Failed to initialize RDMA; load kernel modules?
(null): fcoe_init(214) (null)
(null): fcoe_create_interface(171) no interface specified.
jenkins at neutronpluginsci:~$

The config in /etc/tgt is:

jenkins at neutronpluginsci:/etc/tgt$ ls -l
total 8
drwxr-xr-x 2 root root 4096 Mar 10 07:03 conf.d
lrwxrwxrwx 1 root root   30 Mar 10 06:50 stack.d -> /opt/stack/data/cinder/volumes
-rw-r--r-- 1 root root   58 Mar 10 07:07 targets.conf
jenkins at neutronpluginsci:/etc/tgt$ cat targets.conf
include /etc/tgt/conf.d/*.conf
include /etc/tgt/stack.d/*
jenkins at neutronpluginsci:/etc/tgt$ ls conf.d
jenkins at neutronpluginsci:/etc/tgt$ ls /opt/stack/data/cinder/volumes
jenkins at neutronpluginsci:/etc/tgt$

I don't know if there's any missing Cinder config in my DevStack localrc files. Here's one that I'm using:

enable_service mysql
disable_service n-net
enable_service q-svc
enable_service q-agt
enable_service q-l3
enable_service q-dhcp
enable_service q-meta
enable_service q-lbaas
enable_service neutron
enable_service tempest
declare -a Q_CISCO_PLUGIN_SUBPLUGINS=(openvswitch nexus)
declare -A Q_CISCO_PLUGIN_SWITCH_INFO=([]=admin:Cisco12345:22:neutronpluginsci:1/9)

Here are links to a log showing another localrc file that I use, and the corresponding stack.sh log:

Does anyone have any advice on how to debug this, or recover from this (beyond rebooting the node)? Or am I missing any Cinder config?

Thanks in advance for any help on this!!!

OpenStack-Infra mailing list
OpenStack-Infra at lists.openstack.org<mailto:OpenStack-Infra at lists.openstack.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-qa/attachments/20140311/872d6221/attachment-0001.html>

More information about the openstack-qa mailing list