[Openstack-operators] cinder-volume isn't starting - cause and cure
Jeff Silverman
jeff at sweetlabs.com
Wed Aug 13 00:17:48 UTC 2014
I am working from
http://docs.openstack.org/icehouse/install-guide/install/yum/content/cinder-verify.html
step 3. When I go to create a new volume, I get an error in the cinder
list command.
I checked the directory /var/log/cinder and it is empty. I found a
suggestion that it had to be owned by user cinder so I tried chown
cinder:cinder /var/log/cinder. No joy.
I did some more checking. The daemon dies within a few seconds of starting.
root at storage5-prod.storage5-prod:~# service openstack-cinder-volume start
Starting openstack-cinder-volume: [ OK ]
root at storage5-prod.storage5-prod:~# service openstack-cinder-volume status
openstack-cinder-volume dead but pid file exists
root at storage5-prod.storage5-prod:~#
Looking in /var/log/messages, I see:
Aug 12 16:47:24 storage5-prod abrt: detected unhandled Python exception in
'/usr/bin/cinder-volume'
Aug 12 16:47:24 storage5-prod abrtd: New client connected
Aug 12 16:47:24 storage5-prod abrt-server[7029]: Saved Python crash dump of
pid 7017 to /var/spool/abrt/pyhook-2014-08-12-16:47:24-7017
Aug 12 16:47:24 storage5-prod abrtd: Directory
'pyhook-2014-08-12-16:47:24-7017' creation detected
Aug 12 16:47:24 storage5-prod abrtd: Package 'openstack-cinder' isn't
signed with proper key
Aug 12 16:47:24 storage5-prod abrtd: 'post-create' on
'/var/spool/abrt/pyhook-2014-08-12-16:47:24-7017' exited with 1
Aug 12 16:47:24 storage5-prod abrtd: Deleting problem directory
'/var/spool/abrt/pyhook-2014-08-12-16:47:24-7017'
I wish the abrtd wouldn't delete the problem directory - that makes the
problem tough to track down.
I diagnosed the problem by modifying /etc/init.d/openstack-cinder-volume:
root at storage5-prod.storage5-prod:/etc/init.d# diff openstack-cinder-volume
openstack-cinder-volume_ORIGINAL
43c43
< daemon --user cinder --pidfile $pidfile "$exec --config-file
$distconfig --config-file $config --logfile $logfile
&>>/tmp/openstack-cinder-volume.txt & echo \$! > $pidfile"
---
> daemon --user cinder --pidfile $pidfile "$exec --config-file
$distconfig --config-file $config --logfile $logfile &>/dev/null & echo \$!
> $pidfile"
root at storage5-prod.storage5-prod:/etc/init.d#
then loooked at /tmp/openstack-cinder-volume.txt and found:
root at storage5-prod.storage5-prod:/etc/init.d# cat
/tmp/openstack-cinder-volume.txt
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57:
PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using
libgmp >= 5 to avoid timing attack vulnerability.
_warn("Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to
avoid timing attack vulnerability.", PowmInsecureWarning)
Traceback (most recent call last):
File "/usr/bin/cinder-volume", line 58, in <module>
version=version.version_string())
File "/usr/lib/python2.6/site-packages/oslo/config/cfg.py", line 1638, in
__call__
raise ConfigFilesNotFoundError(self._namespace.files_not_found)
oslo.config.cfg.ConfigFilesNotFoundError: Failed to read some config files:
/etc/cinder/cinder.conf
Ah hah!
I solved the problem by chown cinder:cinder /etc/cinder/cinder.conf
root at storage5-prod.storage5-prod:~# ls -ld /etc/cinder/
drwxr-xr-x 3 cinder cinder 4096 Aug 12 16:41 /etc/cinder/
root at storage5-prod.storage5-prod:~# ls -ld /etc/cinder/cinder.conf
-rw-r----- 1 root root 60893 Aug 12 15:55 /etc/cinder/cinder.conf
root at storage5-prod.storage5-prod:~# chown cinder:cinder
/etc/cinder/cinder.conf
root at storage5-prod.storage5-prod:~# ls -l /etc/cinder/cinder.conf
-rw-r----- 1 cinder cinder 60893 Aug 12 15:55 /etc/cinder/cinder.conf
You have new mail in /var/spool/mail/root
root at storage5-prod.storage5-prod:~#
root at storage5-prod.storage5-prod:~# service openstack-cinder-volume start
Starting openstack-cinder-volume: [ OK ]
root at storage5-prod.storage5-prod:~# service openstack-cinder-volume status
openstack-cinder-volume (pid 7370) is running...
root at storage5-prod.storage5-prod:~# service openstack-cinder-volume status
openstack-cinder-volume (pid 7370) is running...
root at storage5-prod.storage5-prod:~# sleep 5; service
openstack-cinder-volume status
openstack-cinder-volume (pid 7370) is running...
root at storage5-prod.storage5-prod:~#
The bottom line is that I would like to make a suggestion: openstack is
very complicated and has a lot of moving parts. Furthermore, a lot of
people are using it in enterprise critical applications. Put some thought
into troubleshooting techniques. In particular, don't redirect stderr to
/dev/null, instead it write it somewhere.
While I have your undivided attention, there is another error message
coming out of stderr that you ought to think about:
/usr/lib64/python2.6/site-packages/Crypto/Util/number.py:57:
PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using
libgmp >= 5 to avoid timing attack vulnerability.
_warn("Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to
avoid timing attack vulnerability.", PowmInsecureWarning)
Thank you
Jeff
--
*Jeff Silverman*
Systems Engineer
(253) 459-2318 (c)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140812/118a8840/attachment.html>
More information about the OpenStack-operators
mailing list