Open Stack

Responding to my own post with the solution from Josh Durgin via the Ceph Users list that solved my problem, which turned out to be that I had not installed the ceph command line utility since I thought Cinder in Havana only needed python-ceph:

"This is where the error is coming from - although most of the rbd driver was converted to use librbd, there are a couple things that still rely on the cli tools, since they had some functionality that wasn't present in librados in cuttlefish or bobtail.  Specifically, the attach is failing because cinder is trying to use the 'ceph' command to get a list of all monitors to pass to nova. Installing ceph-common on the node running cinder-volume should fix it."

I then asked if I needed to export "CEPH_ARGS='--id volumes'" in my cinder init script for the sake of cephx like I had to do in Grizzly:

"No, that's no longer necessary."

A quick "yum install ceph" and a bounce of my cinder services had everything working.

Thanks,
Andy

On Oct 21, 2013, at 12:04 PM, Andrew Richards <andrew.richards at keepertech.com> wrote:

> Hi Everybody,
> 
> I'm attempting to get Ceph working for CentOS 6.4 running RDO Havana for Cinder volume storage and boot-from-volume, and I keep bumping into a very unhelpful errors on my nova-compute test node and my cinder controller node.
> 
> Here is what I see on my cinder-volume controller (Node #1) when I try to attach a RBD-backed Cinder volume to a Nova VM using either the GUI or nova volume-attach (/var/log/cinder/volume.log):
> 
> 2013-10-20 18:21:05.880 13668 ERROR cinder.openstack.common.rpc.amqp [req-bd62cb07-42e7-414a-86dc-f26f7a569de6 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] Exception during message handling
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     **args)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     return getattr(proxyobj, method)(ctxt, **kwargs)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 808, in wrapper
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     return func(self, *args, **kwargs)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 624, in initialize_connection
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     conn_info = self.driver.initialize_connection(volume, connector)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 665, in initialize_connection
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     hosts, ports = self._get_mon_addrs()
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/volume/drivers/rbd.py", line 312, in _get_mon_addrs
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     out, _ = self._execute(*args)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/utils.py", line 142, in execute
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/processutils.py", line 158, in execute
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     shell=shell)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/green/subprocess.py", line 25, in __init__
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/subprocess.py", line 642, in __init__
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     errread, errwrite)
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp   File "/usr/lib64/python2.6/subprocess.py", line 1234, in _execute_child
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp     raise child_exception
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp OSError: [Errno 2] No such file or directory
> 2013-10-20 18:21:05.880 13668 TRACE cinder.openstack.common.rpc.amqp 
> 2013-10-20 18:21:05.883 13668 ERROR cinder.openstack.common.rpc.common [req-bd62cb07-42e7-414a-86dc-f26f7a569de6 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] Returning exception [Errno 2] No such file or directory to caller
> 
> 
> Here is what I see on my nova-compute node (Node #2) when I try to boot from volume (/var/log/nova/compute.log):
> 
> ERROR nova.compute.manager [req-ced59268-4766-4f57-9cdb-4ba451b0faaa 9bfee22cd15b4dc0a2e203d7c151edbc 8431635821f84285afdd0f5faf1ce1aa] [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] Traceback (most recent call last):
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1028, in _build_instance
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     context, instance, bdms)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1393, in _prep_block_device
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     instance=instance)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1371, in _prep_block_device
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     self._await_block_device_map_created) +
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 283, in attach_block_devices
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     block_device_mapping)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 170, in attach
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     connector)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 176, in wrapper
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     res = method(self, ctx, volume_id, *args, **kwargs)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/nova/volume/cinder.py", line 274, in initialize_connection
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     connector)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 306, in initialize_connection
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     {'connector': connector})[1]['connection_info']
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 237, in _action
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     return self.api.client.post(url, body=body)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 210, in post
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     return self._cs_request(url, 'POST', **kwargs)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 174, in _cs_request
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     **kwargs)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]   File "/usr/lib/python2.6/site-packages/cinderclient/client.py", line 157, in request
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0]     raise exceptions.from_response(resp, body)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-44557bfa-6777-41a6-8183-e08dedf0611b)
> 2013-10-17 15:01:45.060 18546 TRACE nova.compute.manager [instance: c80a053f-b84c-401c-8e29-022d4c6f56a0] 
> 
> 
> More info on my setup:
> I'm running the most recent release of Dumpling libraries (0.67.4) and python-ceph, but I do not have ceph itself installed as it should not be needed on Havana
> cephx is in use, and I set up pools and keyrings per the guide at http://ceph.com/docs/next/rbd/rbd-openstack/
> I'm using the ceph-extras repo to install the backported QEMU packages as described at http://openstack.redhat.com/Using_Ceph_for_Block_Storage_with_RDO
> I'm also using Neutron+OVS and thus edited my qemu.conf according to this libvirt wiki page http://wiki.libvirt.org/page/Guest_won't_start_-_warning:_could_not_open_/dev/net/tun_('generic_ethernet'_interface)
> I am presently not configuring Nova to put its ephemeral disk image on RBD (new option in Havana)
> 
> Things I've been able to do so far:
> I stored an image to Glance backed by RBD
> I used that image to create a Cinder volume backed by RBD
> I instantiated a working ephemeral VM with Nova based on the image from Glance backed by RBD
> I created a libvirt VM manually with virsh on the same compute node that attached a volume from the Cinder RBD pool
> I created a VM with Nova (ephemeral local boot) that I was then able to use virsh to successfully attach a Cinder volume backed by RBD
> I can use qemu-img to create volumes in the cinder RBD pool, but only if I have the client.admin keyring installed on the compute node
> 
> The error traced above is what happens every time I try to boot from volume using that RBD-backed Cinder volume spawned from the RBD-backed Glance image.  What did work led me to believe that QEMU was the problem, so I tried the following:
> I changed the user and group qemu runs as from root to nova and from nova to qemu to see if their permissions had any affect; no change
> I tried the above tests with matching permissions on the contents of /etc/ceph/ (ceph.conf and the keyrings for admin, cinder, and glance); no change
> 
> It seems like Nova is somehow failing to get from Cinder that Cinder is using RBD for its backend, but I can't understand why.  All my configs align with every instance of documentation I've been able to find for making OpenStack work with Ceph.  Has anyone done what I'm trying to do on CentOS 6 or even on some version of Fedora?  I am cross-posting this on the Ceph Users listserv as well.
> 
> Thanks for your time,
> 
> Andy
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20131021/13d8695a/attachment.html>