<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Forwarding the answer to the relevant mailing lists:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">---<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Hope this could help,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">I've encountered this issue myself not to long ago on Ubuntu 12.04 host,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">it didn't happen again after messing with the Kernel Semaphore Limits parameters [1]:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Adding this [2] line to `/etc/sysctl.conf` seems to do the trick.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">- Roey<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">[1] <a href="http://paste.openstack.org/show/73086/" target="_blank">http://paste.openstack.org/show/73086/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">[2] <a href="http://paste.openstack.org/show/73082/" target="_blank">http://paste.openstack.org/show/73082/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Sukhdev Kapur [mailto:sukhdevkapur@gmail.com]
<br>
<b>Sent:</b> Monday, March 10, 2014 5:56 PM<br>
<b>To:</b> Dane Leblanc (leblancd)<br>
<b>Cc:</b> OpenStack Development Mailing List (not for usage questions); openstack-infra@lists.openstack.org; openstack-qa@lists.openstack.org<br>
<b>Subject:</b> Re: [OpenStack-Infra] tgt restart fails in Cinder startup "start: job failed to start"<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I see the same issue. This issue has crept in during the latest flurry of check-ins. I started noticing this issue a day or two before the Icehouse Feature Freeze deadline.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I tried restarting tgt as well, but, it does not help. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">However, rebooting the VM helps clear it up.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Has anybody else seen it as well? Does anybody have a solution for it? <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">-Sukhdev<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Mon, Mar 10, 2014 at 8:37 AM, Dane Leblanc (leblancd) <<a href="mailto:leblancd@cisco.com" target="_blank">leblancd@cisco.com</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal">I don't know if anyone can give me some troubleshooting advice with this issue.<br>
<br>
I'm seeing an occasional problem whereby after several DevStack <a href="http://unstack.sh/stack.sh" target="_blank">
unstack.sh/stack.sh</a> cycles, the tgt daemon (tgtd) fails to start during Cinder startup. Here's a snippet from the stack.sh log:<br>
<br>
2014-03-10 07:09:45.214 | Starting Cinder<br>
2014-03-10 07:09:45.215 | + return 0<br>
2014-03-10 07:09:45.216 | + sudo rm -f /etc/tgt/conf.d/stack.conf<br>
2014-03-10 07:09:45.217 | + _configure_tgt_for_config_d<br>
2014-03-10 07:09:45.218 | + [[ ! -d /etc/tgt/stack.d/ ]]<br>
2014-03-10 07:09:45.219 | + is_ubuntu<br>
2014-03-10 07:09:45.220 | + [[ -z deb ]]<br>
2014-03-10 07:09:45.221 | + '[' deb = deb ']'<br>
2014-03-10 07:09:45.222 | + sudo service tgt restart<br>
2014-03-10 07:09:45.223 | stop: Unknown instance:<br>
2014-03-10 07:09:45.619 | start: Job failed to start<br>
jenkins@neutronpluginsci:~/devstack$ 2014-03-10 07:09:45.621 | + exit_trap<br>
2014-03-10 07:09:45.622 | + local r=1<br>
2014-03-10 07:09:45.623 | ++ jobs -p<br>
2014-03-10 07:09:45.624 | + jobs=<br>
2014-03-10 07:09:45.625 | + [[ -n '' ]]<br>
2014-03-10 07:09:45.626 | + exit 1<br>
<br>
If I try to restart tgt manually without success:<br>
<br>
jenkins@neutronpluginsci:~$ sudo service tgt restart<br>
stop: Unknown instance:<br>
start: Job failed to start<br>
jenkins@neutronpluginsci:~$ sudo tgtd<br>
librdmacm: couldn't read ABI version.<br>
librdmacm: assuming: 4<br>
CMA: unable to get RDMA device list<br>
(null): iser_ib_init(3263) Failed to initialize RDMA; load kernel modules?<br>
(null): fcoe_init(214) (null)<br>
(null): fcoe_create_interface(171) no interface specified.<br>
jenkins@neutronpluginsci:~$<br>
<br>
The config in /etc/tgt is:<br>
<br>
jenkins@neutronpluginsci:/etc/tgt$ ls -l<br>
total 8<br>
drwxr-xr-x 2 root root 4096 Mar 10 07:03 conf.d<br>
lrwxrwxrwx 1 root root 30 Mar 10 06:50 stack.d -> /opt/stack/data/cinder/volumes<br>
-rw-r--r-- 1 root root 58 Mar 10 07:07 targets.conf<br>
jenkins@neutronpluginsci:/etc/tgt$ cat targets.conf<br>
include /etc/tgt/conf.d/*.conf<br>
include /etc/tgt/stack.d/*<br>
jenkins@neutronpluginsci:/etc/tgt$ ls conf.d<br>
jenkins@neutronpluginsci:/etc/tgt$ ls /opt/stack/data/cinder/volumes<br>
jenkins@neutronpluginsci:/etc/tgt$<br>
<br>
I don't know if there's any missing Cinder config in my DevStack localrc files. Here's one that I'm using:<br>
<br>
MYSQL_PASSWORD=nova<br>
RABBIT_PASSWORD=nova<br>
SERVICE_TOKEN=nova<br>
SERVICE_PASSWORD=nova<br>
ADMIN_PASSWORD=nova<br>
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,rabbit<br>
enable_service mysql<br>
disable_service n-net<br>
enable_service q-svc<br>
enable_service q-agt<br>
enable_service q-l3<br>
enable_service q-dhcp<br>
enable_service q-meta<br>
enable_service q-lbaas<br>
enable_service neutron<br>
enable_service tempest<br>
VOLUME_BACKING_FILE_SIZE=2052M<br>
Q_PLUGIN=cisco<br>
declare -a Q_CISCO_PLUGIN_SUBPLUGINS=(openvswitch nexus)<br>
declare -A Q_CISCO_PLUGIN_SWITCH_INFO=([10.0.100.243]=admin:Cisco12345:22:neutronpluginsci:1/9)<br>
NCCLIENT_REPO=git://<a href="http://github.com/CiscoSystems/ncclient.git" target="_blank">github.com/CiscoSystems/ncclient.git</a><br>
PHYSICAL_NETWORK=physnet1<br>
OVS_PHYSICAL_BRIDGE=br-eth1<br>
TENANT_VLAN_RANGE=810:819<br>
ENABLE_TENANT_VLANS=True<br>
API_RATE_LIMIT=False<br>
VERBOSE=True<br>
DEBUG=True<br>
LOGFILE=/opt/stack/logs/stack.sh.log<br>
USE_SCREEN=True<br>
SCREEN_LOGDIR=/opt/stack/logs<br>
<br>
Here are links to a log showing another localrc file that I use, and the corresponding stack.sh log:<br>
<br>
<a href="http://128.107.233.28:8080/job/neutron/1390/artifact/vpnaas_console_log.txt" target="_blank">http://128.107.233.28:8080/job/neutron/1390/artifact/vpnaas_console_log.txt</a><br>
<a href="http://128.107.233.28:8080/job/neutron/1390/artifact/vpnaas_stack_sh_log.txt" target="_blank">http://128.107.233.28:8080/job/neutron/1390/artifact/vpnaas_stack_sh_log.txt</a><br>
<br>
Does anyone have any advice on how to debug this, or recover from this (beyond rebooting the node)? Or am I missing any Cinder config?<br>
<br>
Thanks in advance for any help on this!!!<br>
Dane<br>
<br>
<br>
<br>
_______________________________________________<br>
OpenStack-Infra mailing list<br>
<a href="mailto:OpenStack-Infra@lists.openstack.org">OpenStack-Infra@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>