I upgraded my Train test cluster to Ussuri following these instructions: OpenStack Docs: Operating Kolla<https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure> The upgrade completed successfully with no failures, and the existing VMs are fine, but new VM build fails with rados.Rados.connect\nrados.PermissionDeniedError: Ubuntu Pastebin<https://paste.ubuntu.com/p/MJhzgjYbW8/> I'm running external ceph so I looked at this document: OpenStack Docs: External Ceph<https://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html> It says that I need the following in /etc/kolla/config/glance/ceph.conf: auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx I didn't have that, so I added it and then redeployed, but still can't build VMs. I tried adding the same to all copies of ceph.conf and redeployed again, but that didn't help. Does anything else need to change in my ceph config when upgrading from Train to Ussuri? I see some cryptic talk about ceph in the release notes but it's not obvious what I'm being asked to change: OpenStack Docs: Ussuri Series Release Notes<https://docs.openstack.org/releasenotes/kolla-ansible/ussuri.html#relnotes-10-0-0-stable-ussuri-upgrade-notes> I read the bug that it refers to: Bug #1904062 "external ceph cinder volume config breaks volumes ..." : Bugs : kolla-ansible (launchpad.net)<https://bugs.launchpad.net/kolla-ansible/+bug/1904062> But I already have "backend_host=rbd:volumes" so I don't think I'm hitting that. Also I read these sections but I don't see anything obvious here that needs to be changed: * For cinder (cinder-volume and cinder-backup), glance-api and manila keyrings behavior has changed and Kolla Ansible deployment will not copy those keys using wildcards (ceph.*), instead will use newly introduced variables. Your environment may render unusable after an upgrade if your keys in /etc/kolla/config do not match default values for introduced variables. * The default behavior for generating the cinder.conf template has changed. An rbd-1 section will be generated when external Ceph functionality is used, i.e. cinder_backend_ceph is set to true. Previously it was only included when Kolla Ansible internal Ceph deployment mechanism was used. * The rbd section of nova.conf for nova-compute is now generated when nova_backend is set to "rbd". Previously it was only generated when both enable_ceph was "yes" and nova_backend was set to "rbd". My ceph keys have the default name and are in the default locations. I have cinder_backend_ceph: "yes". I don't have a nova_backend setting but I have nova_backend_ceph: "yes" I added nova_backend: "rbd" and redeployed and now I get a different error: rados.Rados.connect\nrados.ObjectNotFound Ubuntu Pastebin<https://paste.ubuntu.com/p/2kdKCW7npw/> I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.