Open Stack

Hi Everybody,

I'm attempting to get Ceph working for CentOS 6.4 running RDO Havana for Cinder volume storage and boot-from-volume, and I keep bumping into a very unhelpful errors on my nova-compute test node and my cinder controller node.

Here is what I see on my cinder-volume controller (Node #1) when I try to attach a RBD-backed Cinder volume to a Nova VM using either the GUI or nova volume-attach (/var/log/cinder/volume.log):

2013-10-20 18:21:05.880 13668 ERROR cinder.openstack.common.rpc.amqp [req-bd62cb07-42e7-414a-86dc-f26f7a569de6 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] Exception during message handling
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     **args)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 624, in initialize_connection
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     conn_info = self.driver.initialize_connection(volume, connector)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 665, in initialize_connection
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     hosts, ports = self._get_mon_addrs()
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 312, in _get_mon_addrs
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     out, _ = self._execute(*args)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 158, in execute
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     shell=shell)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/green/subprocess.py", line 25, in __init__
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     errread, errwrite)
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     raise child_exception
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory
2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp 
2013-10-20 18:21:05.883 13668 ERROR cinder.openstack.common.rpc.common [req-bd62cb07-42e7-414a-86dc-f26f7a569de6 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] Returning exception [Errno 2] No such file or directory to caller


Here is what I see on my nova-compute node (Node #2) when I try to boot from volume (/var/log/nova/compute.log):

ERROR nova.compute.manager [req-ced59268-4766-4f57-9cdb-4ba451b0faaa 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] Traceback (most recent call last):
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1028, in _build_instance
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     context, instance, bdms)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1393, in _prep_block_device
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     instance=instance)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1371, in _prep_block_device
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     self._await_block_device_map_created) +
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 283, in attach_block_devices
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     block_device_mapping)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 170, in attach
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     connector)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 176, in wrapper
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     res = method(self, ctx, volume_id, *args, **kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 274, in initialize_connection
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     connector)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 306, in initialize_connection
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     {'connector': connector})[1]['connection_info']
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 237, in _action
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     return self.api.client.post(url, body=body)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 210, in post
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     return self._cs_request(url, 'POST', **kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 174, in _cs_request
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     **kwargs)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 157, in request
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     raise exceptions.from_response(resp, body)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)
2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] 


More info on my setup:
I'm running the most recent release of Dumpling libraries (0.67.4) and python-ceph, but I do not have ceph itself installed as it should not be needed on Havana
cephx is in use, and I set up pools and keyrings per the guide at http://ceph.com/docs/next/rbd/rbd-openstack/
I'm using the ceph-extras repo to install the backported QEMU packages as described at http://openstack.redhat.com/Using_Ceph_for_Block_Storage_with_RDO
I'm also using Neutron+OVS and thus edited my qemu.conf according to this libvirt wiki page http://wiki.libvirt.org/page/Guest_won't_start_-_warning:_could_not_open_/dev/net/tun_('generic_ethernet'_interface)
I am presently not configuring Nova to put its ephemeral disk image on RBD (new option in Havana)

Things I've been able to do so far:
I stored an image to Glance backed by RBD
I used that image to create a Cinder volume backed by RBD
I instantiated a working ephemeral VM with Nova based on the image from Glance backed by RBD
I created a libvirt VM manually with virsh on the same compute node that attached a volume from the Cinder RBD pool
I created a VM with Nova (ephemeral local boot) that I was then able to use virsh to successfully attach a Cinder volume backed by RBD
I can use qemu-img to create volumes in the cinder RBD pool, but only if I have the client.admin keyring installed on the compute node

The error traced above is what happens every time I try to boot from volume using that RBD-backed Cinder volume spawned from the RBD-backed Glance image.  What did work led me to believe that QEMU was the problem, so I tried the following:
I changed the user and group qemu runs as from root to nova and from nova to qemu to see if their permissions had any affect; no change
I tried the above tests with matching permissions on the contents of /etc/ceph/ (ceph.conf and the keyrings for admin, cinder, and glance); no change

It seems like Nova is somehow failing to get from Cinder that Cinder is using RBD for its backend, but I can't understand why.  All my configs align with every instance of documentation I've been able to find for making OpenStack work with Ceph.  Has anyone done what I'm trying to do on CentOS 6 or even on some version of Fedora?  I am cross-posting this on the Ceph Users listserv as well.

Thanks for your time,

Andy

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131021/f3a2e148/attachment.html>