Open Stack

Hi Grant/Chris,

Can you verify whether there is a symlink from inside the venv to the location of the RBD python libs in the system packages? If there isn’t one then you should be able to re-execute the os-nova-install.yml playbook as it gets executed here: https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/roles/os_nova/tasks/nova_compute_kvm_install.yml#L103-L123

If you can file a bug with some information regarding what actions led you to get to this point it’d be great for us to try and figure out what went wrong so that we can improve things for the future.

Thanks,

Jesse
IRC: odyssey4me

From: Grant Morley <grant at absolutedevops.io>
Date: Friday, October 21, 2016 at 3:23 PM
To: Chris Sarginson <csargiso at gmail.com>, OpenStack Operators <openstack-operators at lists.openstack.org>
Cc: "ian.banks at serverchoice.com" <ian.banks at serverchoice.com>
Subject: Re: [Openstack-operators] Instances failing to launch when rbd backed (ansible Liberty setup)


Hi Chris,

That bug suggests there has been a fix in 12.0.8. Hopefully 12.0.16 should have that fixed still with a bit of luck , unless the fix hasn't been carried over.

I can see that the bug report says there is also a fix 12.0.9 and 12.0.11.

Do you know if there is a workaround we can try for this? As ideally I don't want to be downgrading the openstack_ansible version.

Regards,

On 21/10/16 14:09, Chris Sarginson wrote:
It seems like it may be an occurrence of this bug, as you look to be using python venvs:

https://bugs.launchpad.net/openstack-ansible/+bug/1509837

2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 117, in __init__

Chris

On Fri, 21 Oct 2016 at 13:19 Grant Morley <grant at absolutedevops.io<mailto:grant at absolutedevops.io>> wrote:

Hi all,

We have a openstack-ansible setup and have ceph installed for the backend. However whenever we try and launch a new instance it fails to launch and we get the following error:

2016-10-21 12:08:06.241 70661 INFO nova.virt.libvirt.driver [req-79811c40-8394-4e33-b16d-ff5fa7341b6a 41c60f65ae914681b6a6ca27a42ff780 324844c815084205995aff10b03a85e1 - - -] [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f] Creating image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [req-79811c40-8394-4e33-b16d-ff5fa7341b6a 41c60f65ae914681b6a6ca27a42ff780 324844c815084205995aff10b03a85e1 - - -] [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f] Instance failed to spawn
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f] Traceback (most recent call last):
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 2156, in _build_resources
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     yield resources
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     block_device_info=block_device_info)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2527, in spawn
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     admin_pass=admin_password)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2939, in _create_image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     backend = image('disk')
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2884, in image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     fname + suffix, image_type)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 967, in image
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     return backend(instance=instance, disk_name=disk_name)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 748, in __init__
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     rbd_user=self.rbd_user)
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]   File "/openstack/venvs/nova-12.0.16/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 117, in __init__
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f]     raise RuntimeError(_('rbd python libraries not found'))
2016-10-21 12:08:06.242 70661 ERROR nova.compute.manager [instance: 5633d98e-5f79-4c13-8d45-7544069f0e6f] RuntimeError: rbd python libraries not found

It moans about the rbd python libraries not being found, however all of the rbd libraries appear to be installed fine via apt. ( We are running Ubuntu)

Compute host packages:

dpkg -l | grep ceph
ii  ceph-common                          10.2.3-1trusty                        amd64        common utilities to mount and interact with a ceph storage cluster
ii  libcephfs1                           10.2.3-1trusty                        amd64        Ceph distributed file system client library
ii  python-ceph                          10.2.3-1trusty                        amd64        Meta-package for python libraries for the Ceph libraries
ii  python-cephfs                        10.2.3-1trusty                        amd64        Python libraries for the Ceph libcephfs library

 dpkg -l | grep rbd
ii  librbd1                              10.2.3-1trusty                        amd64        RADOS block device client library
ii  python-rbd                           10.2.3-1trusty                        amd64        Python libraries for the Ceph librbd library

Has anyone come across this before? Ceph is working fine for Glance, it just seems to be with the nova compute hosts.

Many thanks,

--
[cid:part1.9296627C.98B6FDF0 at absolutedevops.io]
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
www.absolutedevops.io<http://www.absolutedevops.io/> grant at absolutedevops.io<mailto:grant at absolutedevops.i> 0845 874 0580
_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

--

[cid:image001.png at 01D22BB1.3B2EE320]
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
www.absolutedevops.io<http://www.absolutedevops.io/> grant at absolutedevops.io<mailto:grant at absolutedevops.i> 0845 874 0580

________________________________
Rackspace Limited is a company registered in England & Wales (company registered number 03897010) whose registered office is at 5 Millington Road, Hyde Park Hayes, Middlesex UB3 4AZ. Rackspace Limited privacy policy can be viewed at www.rackspace.co.uk/legal/privacy-policy - This e-mail message may contain confidential or privileged information intended for the recipient. Any dissemination, distribution or copying of the enclosed material is prohibited. If you receive this transmission in error, please notify us immediately by e-mail at abuse at rackspace.com and delete the original message. Your cooperation is appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161021/5bd5926d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 4370 bytes
Desc: image001.png
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161021/5bd5926d/attachment.png>