[Openstack] Integrating Ceph Jewel as Storage Backend for OpenStack Newton
Evan Bollig PhD
boll0107 at umn.edu
Mon Mar 6 17:08:03 UTC 2017
Hey Esteban,
I got that same error at one point. Check your file permissions on the
/etc/ceph directory and contents. In particular, make sure the glance
user can access its keyring and the ceph.conf is readable for the
group as well. Here's an example:
drwxr-xr-x. 2 root root 4.0K Jan 17 12:25 .
drwxr-xr-x. 93 root root 8.0K Mar 3 14:27 ..
-rw-------. 1 root root 63 Jan 11 14:00 ceph.client.admin.keyring
-rw-------. 1 cinder cinder 71 Jan 11 13:59
ceph.client.cinder-backup.keyring
-r--------+ 1 cinder cinder 64 Jan 11 13:59 ceph.client.cinder.keyring
-rw-------. 1 glance glance 64 Jan 11 13:59 ceph.client.glance.keyring
-r--------+ 1 gnocchi gnocchi 65 Jan 17 12:25 ceph.client.gnocchi.keyring
-rw-r--r--. 1 root root 220 Jan 11 13:59 ceph.conf
Let me know how things in Newton are working; we're only as far as
Mitaka at the moment.
Cheers,
-E
--
Evan F. Bollig, PhD
Scientific Computing Consultant, Application Developer | Scientific
Computing Solutions (SCS)
Minnesota Supercomputing Institute | msi.umn.edu
University of Minnesota | umn.edu
boll0107 at umn.edu | 612-624-1447 | Walter Lib Rm 556
On Thu, Mar 2, 2017 at 3:13 AM, Esteban Freire <caroubans at yahoo.es> wrote:
> Hello all,
>
> I am testing OpenStack Newton on CentOS 7 and now I already have a OpenStack
> cloud infrastructure working, I would like to integrate Ceph with Cinder,
> Glance and Nova services.
>
> I have found some information about how to performance this on previous
> releases for Openstack and Ceph:
>
> http://docs.ceph.com/docs/jewel/rbd/rbd-openstack/
>
> And I tried to update the variables according to
> https://docs.openstack.org/newton/config-reference/block-storage/drivers/ceph-rbd-volume-driver.html
>
> This is my current glance-api.conf and which is working:
>
> [glance_store]
> stores = file,http
> default_store = file
> filesystem_store_datadir = /var/lib/glance/images/
>
>
> At this is what I tried (at the moment, I have only tried it with the glance
> service but I did not get success):
>
> * Install python-rbd and python-rados from centos-ceph-jewel repo on the
> controller node.
> * Create a ceph user and add it to sudoers.
> * On ceph admin node:
>
> sudo ceph osd pool create images 150
> sudo ceph auth get-or-create client.glance mon 'allow r' osd 'allow
> class-read object_prefix rbd_children, allow rwx pool=images'
> sudo ceph auth get-or-create client.glance | ssh
> ceph at controller-node1 sudo tee /etc/ceph/ceph.client.glance.keyring
> ssh ceph at controller-node1 sudo chown glance:glance
> /etc/ceph/ceph.client.glance.keyring
>
> * On the controller node, I edited the glance-api.conf file with the
> following variables:
>
> [glance_store]
> stores = file,http,rbd
> default_store = rbd
> rbd_pool = images
> rbd_user = glance
> rbd_ceph_conf = /etc/ceph/ceph.conf
> rbd_store_chunk_size = 8
>
> (!) I also have tried with stores = rbd but without success.
>
> * And restart the service, systemctl restart openstack-glance-api
>
> But when I try to create a new image, I get the following issue:
>
> [openstackadmin at controller-node1 ~]$ openstack image create "cirros
> ceph" --file /home/openstackadmin/cirros-0.3.4-x86_64-disk.raw --disk-format
> raw --container-format bare --public
> 500 Internal Server Error
> The server has either erred or is incapable of performing the requested
> operation.
> (HTTP 500)
>
> Is there any documentation about how to integrate Ceph jewel with OpenStack
> newton (Cinder, Glance and Nova services)? If it is so, could you please
> provide me the link?
>
> On the other hand, Is there any way to chose the store when creating an
> image? I mean, to choose for example if save the image on
> /var/lib/glance/images/ or on Ceph.
>
> I would appreciate if you could help me to set up this integration.
>
> This is the most relevant info I can see on the logs and as far as I see
> from it, I have a permissions error but I am not sure what I need to modify.
> It is my first installation with OpenStack by the way and I am trying at
> home to see how it works :)
>
> {{{
> /var/log/glance/api.log
>
> 2017-03-01 23:43:41.419 4197 INFO eventlet.wsgi.server
> [req-eea55909-c963-4158-b30c-3f2779fd78c6 c41043a1ddc14ffba1b45c0a3287e0bf
> b2d1547f0e734f87a84feea75ccd6453 - default default] 192.168.56.2 - -
> [01/Mar/2017 23:43:41] "GET /v2/schemas/image HTTP/1.1" 200 4352 0.421281
> 2017-03-01 23:43:41.570 4197 INFO eventlet.wsgi.server
> [req-f5ef0b43-0ddf-4698-8459-2f75fb1822a3 c41043a1ddc14ffba1b45c0a3287e0bf
> b2d1547f0e734f87a84feea75ccd6453 - default default] 192.168.56.2 - -
> [01/Mar/2017 23:43:41] "POST /v2/images HTTP/1.1" 201 859 0.110681
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> [req-98c406fb-7bc0-47de-bace-2b0a2097b699 c41043a1ddc14ffba1b45c0a3287e0bf
> b2d1547f0e734f87a84feea75ccd6453 - default default] Failed to upload image
> data due to internal error
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data Traceback (most
> recent call last):
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in
> upload
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> image.set_data(data, size)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in
> set_data
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> self.base.set_data(data, size)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> _send_notification(notify_error, 'image.upload', msg)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> self.force_reraise()
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> six.reraise(self.type_, self.value, self.tb)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> self.repo.set_data(data, size)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 185, in
> set_data
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data return
> self.image.set_data(*args, **kwargs)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in
> set_data
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> self.image.set_data(data, size=size)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> verifier=verifier)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 447, in
> add_to_backend
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data verifier)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 420, in
> store_add_to_backend
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> verifier=verifier)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225,
> in op_checker
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data return
> store_op_fun(store, *args, **kwargs)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 447,
> in add
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> rados_id=self.user) as conn:
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data return
> self.gen.next()
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 270,
> in get_connection
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data client =
> rados.Rados(conffile=conffile, rados_id=rados_id)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "rados.pyx", line 525, in rados.Rados.__init__ (rados.c:5719)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "rados.pyx", line 425, in rados.requires.wrapper.validate_func
> (rados.c:4106)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "rados.pyx", line 568, in rados.Rados.__setup (rados.c:6448)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "rados.pyx", line 425, in rados.requires.wrapper.validate_func
> (rados.c:4106)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data File
> "rados.pyx", line 631, in rados.Rados.conf_read_file (rados.c:7384)
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> PermissionDeniedError: error calling conf_read_file
> 2017-03-01 23:43:41.651 4197 ERROR glance.api.v2.image_data
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> [req-98c406fb-7bc0-47de-bace-2b0a2097b699 c41043a1ddc14ffba1b45c0a3287e0bf
> b2d1547f0e734f87a84feea75ccd6453 - default default] Caught error: error
> calling conf_read_file
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi Traceback (most recent
> call last):
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1113, in
> __call__
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi request,
> **action_args)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 1152, in
> dispatch
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi return
> method(*args, **kwargs)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 363, in
> wrapped
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi return func(self,
> req, *args, **kwargs)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 250, in
> upload
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> self._restore(image_repo, image)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> self.force_reraise()
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> six.reraise(self.type_, self.value, self.tb)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/api/v2/image_data.py", line 115, in
> upload
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> image.set_data(data, size)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 195, in
> set_data
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> self.base.set_data(data, size)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/notifier.py", line 479, in set_data
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> _send_notification(notify_error, 'image.upload', msg)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> self.force_reraise()
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> six.reraise(self.type_, self.value, self.tb)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/notifier.py", line 426, in set_data
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> self.repo.set_data(data, size)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 185, in
> set_data
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi return
> self.image.set_data(*args, **kwargs)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 304, in
> set_data
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> self.image.set_data(data, size=size)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance/location.py", line 440, in set_data
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi verifier=verifier)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 447, in
> add_to_backend
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi verifier)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance_store/backend.py", line 420, in
> store_add_to_backend
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi verifier=verifier)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance_store/capabilities.py", line 225,
> in op_checker
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi return
> store_op_fun(store, *args, **kwargs)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 447,
> in add
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> rados_id=self.user) as conn:
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi return
> self.gen.next()
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File
> "/usr/lib/python2.7/site-packages/glance_store/_drivers/rbd.py", line 270,
> in get_connection
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi client =
> rados.Rados(conffile=conffile, rados_id=rados_id)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File "rados.pyx",
> line 525, in rados.Rados.__init__ (rados.c:5719)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File "rados.pyx",
> line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File "rados.pyx",
> line 568, in rados.Rados.__setup (rados.c:6448)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File "rados.pyx",
> line 425, in rados.requires.wrapper.validate_func (rados.c:4106)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi File "rados.pyx",
> line 631, in rados.Rados.conf_read_file (rados.c:7384)
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi PermissionDeniedError:
> error calling conf_read_file
> 2017-03-01 23:43:41.709 4197 ERROR glance.common.wsgi
> }}}
>
> Thanks in advance,
> Esteban
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
More information about the Openstack
mailing list