[kolla-ansible] nova ceph backend
Hi All, looking at https://docs.openstack.org/kolla-ansible/2025.1/reference/storage/external-c... To use ceph with my particular setup it seems I need to set this in globals.yml: nova_backend_ceph: "yes" ceph_nova_user: "nova" ceph_nova_pool_name: "ov-vms" And provide the keyring file in config/nova/ceph.client.nova.keyring I'm not configuring cells explicitly but when I run `kolla-ansible reconfigure --tags nova` I get: ASK [nova-cell : Check cinder keyring file] ************************************************************************** fatal: [kvmgen6-0.csail.mit.edu -> localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable.. ['{{ node_custom_config }}/nova/{{ inventory_hostname }}/{{ keyring }}', '{{ node_custom_config }}/nova/{{ keyring }}']: {{ nova_cell_ceph_backend['cluster'] }}.client.{{ nova_cell_ceph_backend['volumes']['user'] }}.keyring: 'dict object' has no attribute 'volumes'\n\nThe error appears to be in '/opt/kolla/venv/share/kolla-ansible/ansible/roles/nova-cell/tasks/external_ceph.yml': line 17, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Check cinder keyring file\n ^ here\n"} Digging through the code in roles/nova-cell/tasks/external_ceph.yml I kind of make it go by adding: nova_cell_ceph_backend: cluster: "ceph" vms: user: "{{ ceph_nova_user }}" pool: "{{ ceph_nova_pool_name }}" volumes: user: "{{ ceph_nova_user }}" pool: "{{ ceph_nova_pool_name }}" but I can find no reference in the docs to this and don't understand what in needs "volumes" for (And suspect it should be using some cinder stuff for that?) What have I done here? The nova backend "works" but I worry about how I got here since nova+cinder is very common and I shoudl not have needed to leave the documented path so I worry there's a deeper error in my config that caused this. Thanks, -Jon -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
G'day Jonathan, You've more or less done an okay thing here; it'll work but it's probably not the best way to solve the problem. The documentation suggests that the default user is ceph_cinder_user
"ceph_nova_user (by default it's the same as ceph_cinder_user)"
The ceph_cinder_user has the variables/dictionary defined that are brought in when this is the default, so that when deploying with the default ceph_cinder_user, the nova_cell_ceph_backend uses the ceph_cinder_user details. Since kolla-ansible is here just using these variables as a map, to name the files, and then refer to those files in a consistent way, what you have done is technically valid config; but it could cause headaches down the track when you want to upgrade your kolla-ansible, and find that you've got conflicts that you'll need to remember why they were there, and resolve them, which could be a big headache - unnecessary technical debt. We saw similar kinds of issues when we deviated from the defaults setting up cender_backend_ceph, so we just used the defaults to remedy the same kinds of errors - there wasn't a requirement for us to have a non-default cinder user. If you also have no problem using the defaults, then you should be able to get away with using the default and reverting your modifications. If you want to keep chasing the bouncing ball, however, you might find that the answer lies in ansible/roles/nova-cell/defaults/main.yml which defines: nova_cell_ceph_backend: cluster: "{{ ceph_cluster }}" vms: user: "{{ ceph_nova_user }}" pool: "{{ ceph_nova_pool_name }}" volumes: user: "{{ ceph_cinder_user }}" pool: "{{ ceph_cinder_pool_name }}" Since you did not have a ceph_cinder_user configured in globals.yml, thus the error "The task includes an option with an undefined variable" as volumes.user is currently NULL. You should be able to either set "ceph_cinder_user" in global.yml, or override nova_cell_ceph_backend.volumes.user to be {{ ceph_nova_user }} and the same for the pool. I hope this helps! Kind Regards, Joel McLean - Micron21 Pty Ltd -----Original Message----- From: Jonathan Proulx <jon@csail.mit.edu> Sent: Wednesday, 5 November 2025 5:30 AM To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [kolla-ansible] nova ceph backend Hi All, looking at https://docs.openstack.org/kolla-ansible/2025.1/reference/storage/external-c... To use ceph with my particular setup it seems I need to set this in globals.yml: nova_backend_ceph: "yes" ceph_nova_user: "nova" ceph_nova_pool_name: "ov-vms" And provide the keyring file in config/nova/ceph.client.nova.keyring I'm not configuring cells explicitly but when I run `kolla-ansible reconfigure --tags nova` I get: ASK [nova-cell : Check cinder keyring file] ************************************************************************** fatal: [kvmgen6-0.csail.mit.edu -> localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable.. ['{{ node_custom_config }}/nova/{{ inventory_hostname }}/{{ keyring }}', '{{ node_custom_config }}/nova/{{ keyring }}']: {{ nova_cell_ceph_backend['cluster'] }}.client.{{ nova_cell_ceph_backend['volumes']['user'] }}.keyring: 'dict object' has no attribute 'volumes'\n\nThe error appears to be in '/opt/kolla/venv/share/kolla-ansible/ansible/roles/nova-cell/tasks/external_ceph.yml': line 17, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Check cinder keyring file\n ^ here\n"} Digging through the code in roles/nova-cell/tasks/external_ceph.yml I kind of make it go by adding: nova_cell_ceph_backend: cluster: "ceph" vms: user: "{{ ceph_nova_user }}" pool: "{{ ceph_nova_pool_name }}" volumes: user: "{{ ceph_nova_user }}" pool: "{{ ceph_nova_pool_name }}" but I can find no reference in the docs to this and don't understand what in needs "volumes" for (And suspect it should be using some cinder stuff for that?) What have I done here? The nova backend "works" but I worry about how I got here since nova+cinder is very common and I shoudl not have needed to leave the documented path so I worry there's a deeper error in my config that caused this. Thanks, -Jon -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
Hi Joel, that's very helpful thank you. I'd definined multiple cinder backends and since a ceph user is specified in each of those I had left it undefined at the top level. Probably that is the root of my issue. If not your advice to just keep the defualts is sound. We created a user per openstack service for the new deployment, but this isn't a hard requirement. Thanks, -Jon On Thu, Nov 06, 2025 at 02:01:38AM +0000, Joel McLean wrote: :G'day Jonathan, : :You've more or less done an okay thing here; it'll work but it's probably not the best way to solve the problem. : :The documentation suggests that the default user is ceph_cinder_user : :> "ceph_nova_user (by default it's the same as ceph_cinder_user)" : :The ceph_cinder_user has the variables/dictionary defined that are brought in when this is the default, so that when deploying with the default ceph_cinder_user, the nova_cell_ceph_backend uses the ceph_cinder_user details. : :Since kolla-ansible is here just using these variables as a map, to name the files, and then refer to those files in a consistent way, what you have done is technically valid config; but it could cause headaches down the track when you want to upgrade your kolla-ansible, and find that you've got conflicts that you'll need to remember why they were there, and resolve them, which could be a big headache - unnecessary technical debt. : :We saw similar kinds of issues when we deviated from the defaults setting up cender_backend_ceph, so we just used the defaults to remedy the same kinds of errors - there wasn't a requirement for us to have a non-default cinder user. If you also have no problem using the defaults, then you should be able to get away with using the default and reverting your modifications. : :If you want to keep chasing the bouncing ball, however, you might find that the answer lies in ansible/roles/nova-cell/defaults/main.yml which defines: : :nova_cell_ceph_backend: : cluster: "{{ ceph_cluster }}" : vms: : user: "{{ ceph_nova_user }}" : pool: "{{ ceph_nova_pool_name }}" : volumes: : user: "{{ ceph_cinder_user }}" : pool: "{{ ceph_cinder_pool_name }}" : :Since you did not have a ceph_cinder_user configured in globals.yml, thus the error "The task includes an option with an undefined variable" as volumes.user is currently NULL. : :You should be able to either set "ceph_cinder_user" in global.yml, or override nova_cell_ceph_backend.volumes.user to be {{ ceph_nova_user }} and the same for the pool. : :I hope this helps! : :Kind Regards, : :Joel McLean - Micron21 Pty Ltd : :-----Original Message----- :From: Jonathan Proulx <jon@csail.mit.edu> :Sent: Wednesday, 5 November 2025 5:30 AM :To: OpenStack Discuss <openstack-discuss@lists.openstack.org> :Subject: [kolla-ansible] nova ceph backend : :Hi All, : :looking at https://docs.openstack.org/kolla-ansible/2025.1/reference/storage/external-c... : :To use ceph with my particular setup it seems I need to set this in globals.yml: : :nova_backend_ceph: "yes" :ceph_nova_user: "nova" :ceph_nova_pool_name: "ov-vms" : :And provide the keyring file in config/nova/ceph.client.nova.keyring : :I'm not configuring cells explicitly but when I run `kolla-ansible reconfigure --tags nova` I get: : :ASK [nova-cell : Check cinder keyring file] ************************************************************************** :fatal: [kvmgen6-0.csail.mit.edu -> localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable.. ['{{ node_custom_config }}/nova/{{ inventory_hostname }}/{{ keyring }}', '{{ node_custom_config }}/nova/{{ keyring }}']: {{ nova_cell_ceph_backend['cluster'] }}.client.{{ nova_cell_ceph_backend['volumes']['user'] }}.keyring: 'dict object' has no attribute 'volumes'\n\nThe error appears to be in '/opt/kolla/venv/share/kolla-ansible/ansible/roles/nova-cell/tasks/external_ceph.yml': line 17, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Check cinder keyring file\n ^ here\n"} : :Digging through the code in roles/nova-cell/tasks/external_ceph.yml I kind of make it go by adding: : :nova_cell_ceph_backend: : cluster: "ceph" : vms: : user: "{{ ceph_nova_user }}" : pool: "{{ ceph_nova_pool_name }}" : volumes: : user: "{{ ceph_nova_user }}" : pool: "{{ ceph_nova_pool_name }}" : :but I can find no reference in the docs to this and don't understand what in needs "volumes" for (And suspect it should be using some cinder stuff for that?) : :What have I done here? The nova backend "works" but I worry about how I got here since nova+cinder is very common and I shoudl not have needed to leave the documented path so I worry there's a deeper error in my config that caused this. : :Thanks, :-Jon : :-- :Jonathan Proulx (he/him) :Sr. Technical Architect :The Infrastructure Group :MIT CSAIL -- Jonathan Proulx (he/him) Sr. Technical Architect The Infrastructure Group MIT CSAIL
participants (2)
-
Joel McLean
-
Jonathan Proulx