<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hi Everybody,<div><br></div><div>I'm attempting to get Ceph working for CentOS 6.4 running RDO Havana for Cinder volume storage and boot-from-volume, and I keep bumping into a very unhelpful errors on my nova-compute test node and my cinder controller node.</div><div><br></div><div>Here is what I see on my cinder-volume controller (Node #1) when I try to attach a RBD-backed Cinder volume to a Nova VM using either the GUI or nova volume-attach (/var/log/cinder/volume.log):</div><div><br></div><div><div><font face="Courier">2013-10-20 18:21:05.880 13668 ERROR cinder.openstack.common.rpc.amqp [req-bd62cb07-42e7-414a-86dc-f26f7a569de6 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] Exception during message handling</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp **args)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 624, in initialize_connection</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp conn_info = self.driver.initialize_connection(volume, connector)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 665, in initialize_connection</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp hosts, ports = self._get_mon_addrs()</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 312, in _get_mon_addrs</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp out, _ = self._execute(*args)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp return processutils.execute(*cmd, **kwargs)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 158, in execute</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp shell=shell)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.6/site-packages/eventlet/green/subprocess.py", line 25, in __init__</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp errread, errwrite)</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp raise child_exception</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory</font></div><div><font face="Courier">2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp </font></div><div><font face="Courier">2013-10-20 18:21:05.883 13668 ERROR cinder.openstack.common.rpc.common [req-bd62cb07-42e7-414a-86dc-f26f7a569de6 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] Returning exception [Errno 2] No such file or directory to caller</font></div></div><div><br></div><div><br></div><div>Here is what I see on my nova-compute node (Node #2) when I try to boot from volume (/var/log/nova/compute.log):</div><div><br></div><div><div><font face="Courier">ERROR nova.compute.manager [req-ced59268-4766-4f57-9cdb-4ba451b0faaa 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] Traceback (most recent call last):</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1028, in _build_instance</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] context, instance, bdms)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1393, in _prep_block_device</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] instance=instance)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1371, in _prep_block_device</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] self._await_block_device_map_created) +</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 283, in attach_block_devices</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] block_device_mapping)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 170, in attach</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] connector)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 176, in wrapper</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] res = method(self, ctx, volume_id, *args, **kwargs)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 274, in initialize_connection</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] connector)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 306, in initialize_connection</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] {'connector': connector})[1]['connection_info']</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 237, in _action</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] return self.api.client.post(url, body=body)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 210, in post</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] return self._cs_request(url, 'POST', **kwargs)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 174, in _cs_request</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] **kwargs)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 157, in request</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] raise exceptions.from_response(resp, body)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)</font></div><div><font face="Courier">2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] </font></div></div><div><div apple-content-edited="true"><span class="Apple-style-span" style="border-collapse: separate; border-spacing: 0px; "><div><br></div><div><br></div><div>More info on my setup:</div><div><ul class="MailOutline"><li>I'm running the most recent release of Dumpling libraries (0.67.4) and python-ceph, but I do not have ceph itself installed as it should not be needed on Havana</li><li>cephx is in use, and I set up pools and keyrings per the guide at <a href="http://ceph.com/docs/next/rbd/rbd-openstack/">http://ceph.com/docs/next/rbd/rbd-openstack/</a></li><li>I'm using the ceph-extras repo to install the backported QEMU packages as described at <a href="http://openstack.redhat.com/Using_Ceph_for_Block_Storage_with_RDO">http://openstack.redhat.com/Using_Ceph_for_Block_Storage_with_RDO</a></li><li>I'm also using Neutron+OVS and thus edited my qemu.conf according to this libvirt wiki page <a href="http://wiki.libvirt.org/page/Guest_won't_start_-_warning:_could_not_open_/dev/net/tun_('generic_ethernet'_interface)">http://wiki.libvirt.org/page/Guest_won't_start_-_warning:_could_not_open_/dev/net/tun_('generic_ethernet'_interface)</a></li><li>I am presently not configuring Nova to put its ephemeral disk image on RBD (new option in Havana)</li></ul><div><br></div></div><div>Things I've been able to do so far:</div><div><ul class="MailOutline"><li>I stored an image to Glance backed by RBD</li><li>I used that image to create a Cinder volume backed by RBD</li><li>I instantiated a working ephemeral VM with Nova based on the image from Glance backed by RBD</li><li>I created a libvirt VM manually with virsh on the same compute node that attached a volume from the Cinder RBD pool</li><li>I created a VM with Nova (ephemeral local boot) that I was then able to use virsh to successfully attach a Cinder volume backed by RBD</li><li>I can use qemu-img to create volumes in the cinder RBD pool, but only if I have the client.admin keyring installed on the compute node</li></ul><div><br></div></div><div>The error traced above is what happens every time I try to boot from volume using that RBD-backed Cinder volume spawned from the RBD-backed Glance image. What did work led me to believe that QEMU was the problem, so I tried the following:</div><div><ul class="MailOutline"><li>I changed the user and group qemu runs as from root to nova and from nova to qemu to see if their permissions had any affect; no change</li><li>I tried the above tests with matching permissions on the contents of /etc/ceph/ (ceph.conf and the keyrings for admin, cinder, and glance); no change</li></ul></div><div><br></div><div>It seems like Nova is somehow failing to get from Cinder that Cinder is using RBD for its backend, but I can't understand why. All my configs align with every instance of documentation I've been able to find for making OpenStack work with Ceph. Has anyone done what I'm trying to do on CentOS 6 or even on some version of Fedora? I am cross-posting this on the Ceph Users listserv as well.</div><div><br></div><div>Thanks for your time,</div><div><br></div><div>Andy</div><div><br></div></span></div></div></body></html>