[Openstack-operators] cinder-volume isn't starting - cause and cure

Jeff Silverman jeff at sweetlabs.com
Wed Aug 13 00:17:48 UTC 2014


I am working from
http://docs.openstack.org/icehouse/install-guide/install/yum/content/cinder-verify.html
step 3.  When I go to create a new volume, I get an error in the cinder
list command.

I checked the directory /var/log/cinder and it is empty.  I found a
suggestion that it had to be owned by user cinder so I tried chown
cinder:cinder /var/log/cinder.  No joy.

I did some more checking.  The daemon dies within a few seconds of starting.

root at storage5-prod.storage5-prod:~# service openstack-cinder-volume start
Starting openstack-cinder-volume:                          [  OK  ]
root at storage5-prod.storage5-prod:~# service openstack-cinder-volume status
openstack-cinder-volume dead but pid file exists
root at storage5-prod.storage5-prod:~#


Looking in /var/log/messages, I see:

Aug 12 16:47:24 storage5-prod abrt: detected unhandled Python exception in
'/usr/bin/cinder-volume'
Aug 12 16:47:24 storage5-prod abrtd: New client connected
Aug 12 16:47:24 storage5-prod abrt-server[7029]: Saved Python crash dump of
pid 7017 to /var/spool/abrt/pyhook-2014-08-12-16:47:24-7017
Aug 12 16:47:24 storage5-prod abrtd: Directory
'pyhook-2014-08-12-16:47:24-7017' creation detected
Aug 12 16:47:24 storage5-prod abrtd: Package 'openstack-cinder' isn't
signed with proper key
Aug 12 16:47:24 storage5-prod abrtd: 'post-create' on
'/var/spool/abrt/pyhook-2014-08-12-16:47:24-7017' exited with 1
Aug 12 16:47:24 storage5-prod abrtd: Deleting problem directory
'/var/spool/abrt/pyhook-2014-08-12-16:47:24-7017'

I wish the abrtd wouldn't delete the problem directory - that makes the
problem tough to track down.

I diagnosed the problem by modifying /etc/init.d/openstack-cinder-volume:

root at storage5-prod.storage5-prod:/etc/init.d# diff openstack-cinder-volume
openstack-cinder-volume_ORIGINAL
43c43
<     daemon --user cinder --pidfile $pidfile "$exec --config-file
$distconfig --config-file $config --logfile $logfile
&>>/tmp/openstack-cinder-volume.txt & echo \$! > $pidfile"
---
>     daemon --user cinder --pidfile $pidfile "$exec --config-file
$distconfig --config-file $config --logfile $logfile &>/dev/null & echo \$!
> $pidfile"
root at storage5-prod.storage5-prod:/etc/init.d#

then loooked at /tmp/openstack-cinder-volume.txt and found:
root at storage5-prod.storage5-prod:/etc/init.d# cat
/tmp/openstack-cinder-volume.txt
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57:
PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using
libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to
avoid timing attack vulnerability.", PowmInsecureWarning)
Traceback (most recent call last):
  File "/usr/bin/cinder-volume", line 58, in <module>
    version=version.version_string())
  File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1638, in
__call__
    raise ConfigFilesNotFoundError(self._namespace.files_not_found)
oslo.config.cfg.ConfigFilesNotFoundError: Failed to read some config files:
/etc/cinder/cinder.conf

Ah hah!

I solved the problem by chown cinder:cinder /etc/cinder/cinder.conf

root at storage5-prod.storage5-prod:~# ls -ld /etc/cinder/
drwxr-xr-x 3 cinder cinder 4096 Aug 12 16:41 /etc/cinder/
root at storage5-prod.storage5-prod:~# ls -ld /etc/cinder/cinder.conf
-rw-r----- 1 root root 60893 Aug 12 15:55 /etc/cinder/cinder.conf
root at storage5-prod.storage5-prod:~# chown cinder:cinder
/etc/cinder/cinder.conf
root at storage5-prod.storage5-prod:~# ls -l /etc/cinder/cinder.conf
-rw-r----- 1 cinder cinder 60893 Aug 12 15:55 /etc/cinder/cinder.conf
You have new mail in /var/spool/mail/root
root at storage5-prod.storage5-prod:~#


root at storage5-prod.storage5-prod:~# service openstack-cinder-volume start
Starting openstack-cinder-volume:                          [  OK  ]
root at storage5-prod.storage5-prod:~# service openstack-cinder-volume status
openstack-cinder-volume (pid  7370) is running...
root at storage5-prod.storage5-prod:~# service openstack-cinder-volume status
openstack-cinder-volume (pid  7370) is running...
root at storage5-prod.storage5-prod:~# sleep 5; service
openstack-cinder-volume status
openstack-cinder-volume (pid  7370) is running...
root at storage5-prod.storage5-prod:~#

​The bottom line is that I would like to make a suggestion: openstack is
very complicated and has a lot of moving parts.  Furthermore, a lot of
people are using it in enterprise critical applications.  Put some thought
into troubleshooting techniques.  In particular, don't redirect stderr to
/dev/null, instead it write it somewhere.


While I have your undivided attention, there is another error message
coming out of stderr that you ought to think about:

​/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57:
PowmInsecureWarning: Not using mpz_powm_sec.  You should rebuild using
libgmp >= 5 to avoid timing attack vulnerability.
  _warn("Not using mpz_powm_sec.  You should rebuild using libgmp >= 5 to
avoid timing attack vulnerability.", PowmInsecureWarning)


​Thank you


Jeff
​



-- 
*Jeff Silverman*
Systems Engineer
(253) 459-2318 (c)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140812/118a8840/attachment.html>


More information about the OpenStack-operators mailing list