RE: [kolla] VM build fails after Train-Ussuri upgrade
Can anyone help with this upgrade issue?
From: Braden, Albert Sent: Monday, April 19, 2021 8:20 AM To: openstack-discuss@lists.openstack.org Subject: [kolla] VM build fails after Train-Ussuri upgrade
I upgraded my Train test cluster to Ussuri following these instructions:
OpenStack Docs: Operating Kollahttps://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure
The upgrade completed successfully with no failures, and the existing VMs are fine, but new VM build fails with rados.Rados.connect\nrados.PermissionDeniedError:
Ubuntu Pastebinhttps://paste.ubuntu.com/p/MJhzgjYbW8/
I'm running external ceph so I looked at this document:
OpenStack Docs: External Cephhttps://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html
It says that I need the following in /etc/kolla/config/glance/ceph.conf:
auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx
I didn't have that, so I added it and then redeployed, but still can't build VMs. I tried adding the same to all copies of ceph.conf and redeployed again, but that didn't help. Does anything else need to change in my ceph config when upgrading from Train to Ussuri? I see some cryptic talk about ceph in the release notes but it's not obvious what I'm being asked to change:
OpenStack Docs: Ussuri Series Release Noteshttps://docs.openstack.org/releasenotes/kolla-ansible/ussuri.html#relnotes-10-0-0-stable-ussuri-upgrade-notes
I read the bug that it refers to: Bug #1904062 "external ceph cinder volume config breaks volumes ..." : Bugs : kolla-ansible (launchpad.net)https://bugs.launchpad.net/kolla-ansible/+bug/1904062
But I already have "backend_host=rbd:volumes" so I don't think I'm hitting that.
Also I read these sections but I don't see anything obvious here that needs to be changed:
* For cinder (cinder-volume and cinder-backup), glance-api and manila keyrings behavior has changed and Kolla Ansible deployment will not copy those keys using wildcards (ceph.*), instead will use newly introduced variables. Your environment may render unusable after an upgrade if your keys in /etc/kolla/config do not match default values for introduced variables.
* The default behavior for generating the cinder.conf template has changed. An rbd-1 section will be generated when external Ceph functionality is used, i.e. cinder_backend_ceph is set to true. Previously it was only included when Kolla Ansible internal Ceph deployment mechanism was used.
* The rbd section of nova.conf for nova-compute is now generated when nova_backend is set to "rbd". Previously it was only generated when both enable_ceph was "yes" and nova_backend was set to "rbd".
My ceph keys have the default name and are in the default locations. I have cinder_backend_ceph: "yes". I don't have a nova_backend setting but I have nova_backend_ceph: "yes"
I added nova_backend: "rbd" and redeployed and now I get a different error: rados.Rados.connect\nrados.ObjectNotFound
Ubuntu Pastebinhttps://paste.ubuntu.com/p/2kdKCW7npw/
I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails.
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Hi,
Did you check your keys files in nova, kvm and cinder generated by ceph to auth?
Example to generate key file and then copy to compute: ceph auth get-or-create compute mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow allow rwx pool=compute, allow allow rwx pool=volumes, allow rx pool=images' -o /ceph.client.compute.keyring
Hope this help you.
Regards.
De: Braden, Albert C-Albert.Braden@charter.com Enviado el: lunes, 26 de abril de 2021 15:46 Para: openstack-discuss@lists.openstack.org Asunto: RE: [kolla] VM build fails after Train-Ussuri upgrade
everis Security Awareness - This is an incoming mail from an EXTERNAL DOMAIN. Please verify sender before you open attachments or access links. Can anyone help with this upgrade issue?
From: Braden, Albert Sent: Monday, April 19, 2021 8:20 AM To: openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org Subject: [kolla] VM build fails after Train-Ussuri upgrade
I upgraded my Train test cluster to Ussuri following these instructions:
OpenStack Docs: Operating Kollahttps://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure
The upgrade completed successfully with no failures, and the existing VMs are fine, but new VM build fails with rados.Rados.connect\nrados.PermissionDeniedError:
Ubuntu Pastebinhttps://paste.ubuntu.com/p/MJhzgjYbW8/
I'm running external ceph so I looked at this document:
OpenStack Docs: External Cephhttps://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html
It says that I need the following in /etc/kolla/config/glance/ceph.conf:
auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx
I didn't have that, so I added it and then redeployed, but still can't build VMs. I tried adding the same to all copies of ceph.conf and redeployed again, but that didn't help. Does anything else need to change in my ceph config when upgrading from Train to Ussuri? I see some cryptic talk about ceph in the release notes but it's not obvious what I'm being asked to change:
OpenStack Docs: Ussuri Series Release Noteshttps://docs.openstack.org/releasenotes/kolla-ansible/ussuri.html#relnotes-10-0-0-stable-ussuri-upgrade-notes
I read the bug that it refers to: Bug #1904062 "external ceph cinder volume config breaks volumes ..." : Bugs : kolla-ansible (launchpad.net)https://bugs.launchpad.net/kolla-ansible/+bug/1904062
But I already have "backend_host=rbd:volumes" so I don't think I'm hitting that.
Also I read these sections but I don't see anything obvious here that needs to be changed:
* For cinder (cinder-volume and cinder-backup), glance-api and manila keyrings behavior has changed and Kolla Ansible deployment will not copy those keys using wildcards (ceph.*), instead will use newly introduced variables. Your environment may render unusable after an upgrade if your keys in /etc/kolla/config do not match default values for introduced variables. * The default behavior for generating the cinder.conf template has changed. An rbd-1 section will be generated when external Ceph functionality is used, i.e. cinder_backend_ceph is set to true. Previously it was only included when Kolla Ansible internal Ceph deployment mechanism was used.
* The rbd section of nova.conf for nova-compute is now generated when nova_backend is set to "rbd". Previously it was only generated when both enable_ceph was "yes" and nova_backend was set to "rbd".
My ceph keys have the default name and are in the default locations. I have cinder_backend_ceph: "yes". I don't have a nova_backend setting but I have nova_backend_ceph: "yes"
I added nova_backend: "rbd" and redeployed and now I get a different error: rados.Rados.connect\nrados.ObjectNotFound
Ubuntu Pastebinhttps://paste.ubuntu.com/p/2kdKCW7npw/
I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
Hi Ronal,
I'm not sure I understand your question. I used ceph-ansible to setup ceph when I built the cluster, and it created the keys. I didn't change anything key-related during the upgrade. Do I need to?
From: Ronal Mauricio Faraj Rodriguez ronal.mauricio.faraj.rodriguez@nttdata.com Sent: Monday, April 26, 2021 10:02 AM To: Braden, Albert C-Albert.Braden@charter.com Cc: openstack-discuss@lists.openstack.org Subject: [EXTERNAL] RE: [kolla] VM build fails after Train-Ussuri upgrade
CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. Hi,
Did you check your keys files in nova, kvm and cinder generated by ceph to auth?
Example to generate key file and then copy to compute: ceph auth get-or-create compute mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow allow rwx pool=compute, allow allow rwx pool=volumes, allow rx pool=images' -o /ceph.client.compute.keyring
Hope this help you.
Regards.
De: Braden, Albert <C-Albert.Braden@charter.commailto:C-Albert.Braden@charter.com> Enviado el: lunes, 26 de abril de 2021 15:46 Para: openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org Asunto: RE: [kolla] VM build fails after Train-Ussuri upgrade
everis Security Awareness - This is an incoming mail from an EXTERNAL DOMAIN. Please verify sender before you open attachments or access links. Can anyone help with this upgrade issue?
From: Braden, Albert Sent: Monday, April 19, 2021 8:20 AM To: openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org Subject: [kolla] VM build fails after Train-Ussuri upgrade
I upgraded my Train test cluster to Ussuri following these instructions:
OpenStack Docs: Operating Kollahttps://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html#upgrade-procedure
The upgrade completed successfully with no failures, and the existing VMs are fine, but new VM build fails with rados.Rados.connect\nrados.PermissionDeniedError:
Ubuntu Pastebinhttps://paste.ubuntu.com/p/MJhzgjYbW8/
I'm running external ceph so I looked at this document:
OpenStack Docs: External Cephhttps://docs.openstack.org/kolla-ansible/latest/reference/storage/external-ceph-guide.html
It says that I need the following in /etc/kolla/config/glance/ceph.conf:
auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx
I didn't have that, so I added it and then redeployed, but still can't build VMs. I tried adding the same to all copies of ceph.conf and redeployed again, but that didn't help. Does anything else need to change in my ceph config when upgrading from Train to Ussuri? I see some cryptic talk about ceph in the release notes but it's not obvious what I'm being asked to change:
OpenStack Docs: Ussuri Series Release Noteshttps://docs.openstack.org/releasenotes/kolla-ansible/ussuri.html#relnotes-10-0-0-stable-ussuri-upgrade-notes
I read the bug that it refers to: Bug #1904062 "external ceph cinder volume config breaks volumes ..." : Bugs : kolla-ansible (launchpad.net)https://bugs.launchpad.net/kolla-ansible/+bug/1904062
But I already have "backend_host=rbd:volumes" so I don't think I'm hitting that.
Also I read these sections but I don't see anything obvious here that needs to be changed:
* For cinder (cinder-volume and cinder-backup), glance-api and manila keyrings behavior has changed and Kolla Ansible deployment will not copy those keys using wildcards (ceph.*), instead will use newly introduced variables. Your environment may render unusable after an upgrade if your keys in /etc/kolla/config do not match default values for introduced variables. * The default behavior for generating the cinder.conf template has changed. An rbd-1 section will be generated when external Ceph functionality is used, i.e. cinder_backend_ceph is set to true. Previously it was only included when Kolla Ansible internal Ceph deployment mechanism was used.
* The rbd section of nova.conf for nova-compute is now generated when nova_backend is set to "rbd". Previously it was only generated when both enable_ceph was "yes" and nova_backend was set to "rbd".
My ceph keys have the default name and are in the default locations. I have cinder_backend_ceph: "yes". I don't have a nova_backend setting but I have nova_backend_ceph: "yes"
I added nova_backend: "rbd" and redeployed and now I get a different error: rados.Rados.connect\nrados.ObjectNotFound
Ubuntu Pastebinhttps://paste.ubuntu.com/p/2kdKCW7npw/
I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails.
The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
participants (2)
-
Braden, Albert
-
Ronal Mauricio Faraj Rodriguez