In writing this all out the first time I had to delete it all, because I realise that in my config we only are using a single of our 3 ceph clusters for glance; 3 for cinder, more or less following the documentation, but only a single back end for glance which follows a similar setup, only that it isn’t a named group. This is the default supported arrangement for kolla-ansible deployments.

So, turns out, this was a case of blind-leading-blind.

That said, the configuration at https://docs.openstack.org/kolla-ansible/2024.1/reference/storage/external-ceph-guide.html is identical to how our configuration is set up – we just don’t have the other two ceph back ends configured for glance, only cinder.

Reading this documentation, it’s not clear to me that if I were to run

• openstack image create --volume my-volume my-new-image-name

I’m not sure how Openstack would decide which glance storage backend it should use; Openstack client does not appear to show any argument for specifying.

Perhaps only specified by metadata/properties when creating the image, based on information from https://docs.openstack.org/glance/latest/admin/multistores.html

Specifically, this document discusses that the glance-api.conf must have enabled_backends defined, a default backend must be set.

[DEFAULT]
enabled_backends = ceph:rbd
[glance_store]

default_backend = ceph

On that page there’s a reference to some meta data that might indicate that image metadata might be use to help it select which store it should live on, but it’s not explicitly documented.

Specific to kolla-ansible, backends for multistore are defined: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/glance/templates/glance-api.conf.j2

So, if you haven’t set them, you can probably set them via the appropriate override which appears to be /etc/kolla/config/glance-api.conf

The oversight with kolla-ansible out of the box is that it defaults to your first configured ceph backend:

/glance/defaults/main.yml:glance_default_backend: "{% if glance_backend_vmware | bool %}vmware{% elif glance_backend_ceph | bool %}{{ glance_ceph_backends[0].name }}{% elif glance_backend_swift | bool %}swift{% elif glance_backend_s3 | bool %}s3{% else %}file{% endif %}"

… and the glance ceph backend default template only had one called “rbd”, which is effectively hard coded. This could be worked around in kolla-ansible by modifying the template, but that’s where things will fall off the rails for most users.

Still, hopefully this points you in the right direction to get this working?

Kind Regards,

Joel McLean – Micron21 Pty Ltd