[Openstack] Integrating Ceph Jewel as Storage Backend for OpenStack Newton

Esteban Freire caroubans at yahoo.es
Thu Mar 2 09:13:11 UTC 2017


Hello all,

I am testing OpenStack Newton on CentOS 7 and now I already have a OpenStack cloud infrastructure working, I would like to integrate Ceph with Cinder, Glance and Nova services.

I have found some information about how to performance this on previous releases for Openstack and Ceph:

    http://docs.ceph.com/docs/jewel/rbd/rbd-openstack/

And I tried to update the variables according to https://docs.openstack.org/newton/config-reference/block-storage/drivers/ceph-rbd-volume-driver.html

This is my current glance-api.conf and which is working:

    [glance_store]
    stores = file,http
    default_store = file
    filesystem_store_datadir = /var/lib/glance/images/


At this is what I tried (at the moment, I have only tried it with the glance service but I did not get success):

    * Install python-rbd and python-rados from centos-ceph-jewel repo on the controller node.
    * Create a ceph user and add it to sudoers.
    * On ceph admin node:
        sudo ceph osd pool create images 150
        sudo ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
        sudo ceph auth get-or-create client.glance | ssh ceph at controller-node1 sudo tee /etc/ceph/ceph.client.glance.keyring
        ssh ceph at controller-node1 sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
    
    * On the controller node, I edited the glance-api.conf file with the following variables:

        [glance_store]
        stores = file,http,rbd
        default_store = rbd
        rbd_pool = images
        rbd_user = glance
        rbd_ceph_conf = /etc/ceph/ceph.conf
        rbd_store_chunk_size = 8
        (!) I also have tried with stores = rbd but without success. 
        
    * And restart the service, systemctl restart openstack-glance-api

But when I try to create a new image, I get the following issue:

    [openstackadmin at controller-node1 ~]$ openstack image create "cirros ceph" --file /home/openstackadmin/cirros-0.3.4-x86_64-disk.raw --disk-format raw --container-format bare --public
    500 Internal Server Error
    The server has either erred or is incapable of performing the requested operation.
        (HTTP 500)

Is there any documentation about how to integrate Ceph jewel with OpenStack newton (Cinder, Glance and Nova services)? If it is so, could you please provide me the link?         

On the other hand, Is there any way to chose the store when creating an image? I mean, to choose for example if save the image on /var/lib/glance/images/ or on Ceph. 

I would appreciate if you could help me to set up this integration. 

This is the most relevant info I can see on the logs and as far as I see from it, I have a permissions error but I am not sure what I need to modify. It is my first installation with OpenStack by the way and I am trying at home to see how it works :)

{{{
/var/log/glance/api.log

2017-03-01 23:43:41.419 4197 INFO eventlet.wsgi.server [req-eea55909-c963-4158-b30c-3f2779fd78c6 c41043a1ddc14ffba1b45c0a3287e0bf b2d1547f0e734f87a84feea75ccd6453 - default default] 192.168.56.2 - - [01/Mar/2017 23:43:41] "GET /v2/schemas/image HTTP/1.1" 200 4352 0.421281
2017-03-01 23:43:41.570 4197 INFO eventlet.wsgi.server [req-f5ef0b43-0ddf-4698-8459-2f75fb1822a3 c41043a1ddc14ffba1b45c0a3287e0bf b2d1547f0e734f87a84feea75ccd6453 - default default] 192.168.56.2 - - [01/Mar/2017 23:43:41] "POST /v2/images HTTP/1.1" 201 859 0.110681
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data [req-98c406fb-7bc0-47de-bace-2b0a2097b699 c41043a1ddc14ffba1b45c0a3287e0bf b2d1547f0e734f87a84feea75ccd6453 - default default] Failed to upload image data due to internal error
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data Traceback (most recent call last):
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     image.set_data(data, size)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     self.base.set_data(data, size)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     _send_notification(notify_error, 'image.upload', msg)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     self.force_reraise()
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     six.reraise(self.type_, self.value, self.tb)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     self.repo.set_data(data, size)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 185, in set_data
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     return self.image.set_data(*args, **kwargs)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     self.image.set_data(data, size=size)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     verifier=verifier)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 447, in add_to_backend
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     verifier)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 420, in store_add_to_backend
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     verifier=verifier)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     return store_op_fun(store, *args, **kwargs)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 447, in add
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     rados_id=self.user) as conn:
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     return self.gen.next()
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 270, in get_connection
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data     client = rados.Rados(conffile=conffile, rados_id=rados_id)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "rados.pyx", line 525, in rados.Rados.__init__ (rados.c:5719)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "rados.pyx", line 568, in rados.Rados.__setup (rados.c:6448)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data   File "rados.pyx", line 631, in rados.Rados.conf_read_file (rados.c:7384)
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data PermissionDeniedError: error calling conf_read_file
2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data 
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi [req-98c406fb-7bc0-47de-bace-2b0a2097b699 c41043a1ddc14ffba1b45c0a3287e0bf b2d1547f0e734f87a84feea75ccd6453 - default default] Caught error: error calling conf_read_file
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi Traceback (most recent call last):
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1113, in __call__
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     request, **action_args)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1152, in dispatch
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     return method(*args, **kwargs)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 363, in wrapped
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 250, in upload
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     self._restore(image_repo, image)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     self.force_reraise()
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in upload
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     image.set_data(data, size)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in set_data
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     self.base.set_data(data, size)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     self.force_reraise()
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     self.repo.set_data(data, size)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 185, in set_data
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in set_data
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     self.image.set_data(data, size=size)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     verifier=verifier)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 447, in add_to_backend
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     verifier)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 420, in store_add_to_backend
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     verifier=verifier)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225, in op_checker
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 447, in add
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     rados_id=self.user) as conn:
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     return self.gen.next()
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 270, in get_connection
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi     client = rados.Rados(conffile=conffile, rados_id=rados_id)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "rados.pyx", line 525, in rados.Rados.__init__ (rados.c:5719)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "rados.pyx", line 568, in rados.Rados.__setup (rados.c:6448)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "rados.pyx", line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi   File "rados.pyx", line 631, in rados.Rados.conf_read_file (rados.c:7384)
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi PermissionDeniedError: error calling conf_read_file
2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi 
}}}

Thanks in advance,
Esteban
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170302/4ec1b057/attachment.html>


More information about the Openstack mailing list