Open Stack

From the traceback it looks like nova-compute is running out of a venv.

You need to activate the venv, most likely via: source /openstack/venvs/nova-12.0.16/.venv/bin/activate then run: pip freeze.  If you don’t see the RBD stuff – then that is your issue.  You might be able to fix via: pip install rbd.

Venv’s are self-contained python installs, so they do not use the system level python packages at all.

I would also ask for some help in the #openstack-ansible channel on irc as well.

___________________________________________________________________
Kris Lindgren
Senior Linux Systems Engineer
GoDaddy

From: Grant Morley <grant at absolutedevops.io>
Date: Friday, October 21, 2016 at 6:14 AM
To: OpenStack Operators <openstack-operators at lists.openstack.org>
Cc: "ian.banks at serverchoice.com" <ian.banks at serverchoice.com>
Subject: [Openstack-operators] Instances failing to launch when rbd backed (ansible Liberty setup)


Hi all,

We have a openstack-ansible setup and have ceph installed for the backend. However whenever we try and launch a new instance it fails to launch and we get the following error:

2016-10-21 12:08:06.241 70661 INFO nova.virt.libvirt.driver [req-79811c40-8394-4e33-b16d-ff5fa7341b6a 41c60f65ae914681b6a6ca27a42ff780 324844c815084205995aff10b03a85e1 - - -] [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f] Creating image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [req-79811c40-8394-4e33-b16d-ff5fa7341b6a 41c60f65ae914681b6a6ca27a42ff780 324844c815084205995aff10b03a85e1 - - -] [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f] Instance failed to spawn
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f] Traceback (most recent call last):
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 2156, in _build_resources
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     yield resources
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     block_device_info=block_device_info)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2527, in spawn
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     admin_pass=admin_password)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2939, in _create_image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     backend = image('disk')
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2884, in image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     fname + suffix, image_type)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 967, in image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     return backend(instance=instance, disk_name=disk_name)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 748, in __init__
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     rbd_user=self.rbd_user)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 117, in __init__
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     raise RuntimeError(_('rbd python libraries not found'))
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f] RuntimeError: rbd python libraries not found

It moans about the rbd python libraries not being found, however all of the rbd libraries appear to be installed fine via apt. ( We are running Ubuntu)

Compute host packages:

dpkg -l | grep ceph
ii  ceph-common                          10.2.3-1trusty                        amd64        common utilities to mount and interact with a ceph storage cluster
ii  libcephfs1                           10.2.3-1trusty                        amd64        Ceph distributed file system client library
ii  python-ceph                          10.2.3-1trusty                        amd64        Meta-package for python libraries for the Ceph libraries
ii  python-cephfs                        10.2.3-1trusty                        amd64        Python libraries for the Ceph libcephfs library

 dpkg -l | grep rbd
ii  librbd1                              10.2.3-1trusty                        amd64        RADOS block device client library
ii  python-rbd                           10.2.3-1trusty                        amd64        Python libraries for the Ceph librbd library

Has anyone come across this before? Ceph is working fine for Glance, it just seems to be with the nova compute hosts.

Many thanks,

--

[cid:image001.png at 01D22B7F.7756BDA0]
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
www.absolutedevops.io<http://www.absolutedevops.io/> grant at absolutedevops.io<mailto:grant at absolutedevops.i> 0845 874 0580
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161021/58de30e7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4370 bytes
Desc: image001.png
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161021/58de30e7/attachment-0001.png>