[keystone] What does a multi-region service catalog look like?
o/ The question in $subject came up in a discussion earlier today and I neither knew the answer nor was able to discover it by looking at a service catalogs for a few production deployments I have access to (also, the multi-region feature is currently broken in DevStack and likely has been for some time). Crucially, in a "standard" multi-region deployment where there is one Keystone instances and N Nova/Cinder/Glance/Neutron/... deployments, would I have something like the below, where there is only one identity endpoint but multiple e.g. compute endpoints, or would I expect there always to be as many identity endpoints as there are e.g. compute endpoints? { "token": { ... "catalog": [ { "endpoints": [ { "id": "fb757b386a23430d8f4d636b8b45fa89", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/identity", "region": "RegionOne" } ], "id": "50c4feff9d6149b898fce20ce23904a3", "type": "identity", "name": "keystone" }, { "endpoints": [ { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/compute/v2.1", "region": "RegionOne" }, { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionTwo", "url": "http://10.0.112.232/compute/v2.1", "region": "RegionOne" } ], "id": "87322637c5ab4e7b90bea7a8c1eb9525", "type": "compute", "name": "nova" }, ... ] } } I'm guessing the latter, but I'd like to confirm this. Stephen
Short answer Stephen, there can be both (and I've seen both), but typically there is single keystone for multiple non-identity regions. ---- typed from mobile, auto-correct typos assumed ---- On Tue, Oct 15, 2024, 13:33 Stephen Finucane <stephenfin@redhat.com> wrote:
o/
The question in $subject came up in a discussion earlier today and I neither knew the answer nor was able to discover it by looking at a service catalogs for a few production deployments I have access to (also, the multi-region feature is currently broken in DevStack and likely has been for some time). Crucially, in a "standard" multi-region deployment where there is one Keystone instances and N Nova/Cinder/Glance/Neutron/... deployments, would I have something like the below, where there is only one identity endpoint but multiple e.g. compute endpoints, or would I expect there always to be as many identity endpoints as there are e.g. compute endpoints?
{ "token": { ... "catalog": [ { "endpoints": [ { "id": "fb757b386a23430d8f4d636b8b45fa89", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/identity", "region": "RegionOne" } ], "id": "50c4feff9d6149b898fce20ce23904a3", "type": "identity", "name": "keystone" }, { "endpoints": [ { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/compute/v2.1", "region": "RegionOne" }, { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionTwo", "url": "http://10.0.112.232/compute/v2.1", "region": "RegionOne" } ], "id": "87322637c5ab4e7b90bea7a8c1eb9525", "type": "compute", "name": "nova" }, ... ] } }
I'm guessing the latter, but I'd like to confirm this.
Stephen
Hi Stephen, As Artem has pointed out, we’ve seen both. Some people also have a single keystone distributed across several regions using database replication. There’s pros and cons to all these deployments, however we personally use a centralised keystone and multiple regions with region specific endpoints. The downside of this is that loss of comms to the keystone makes multiple regions inaccessible however it doesn’t break customer compute workloads, so we absorb that as acceptable risk. Some others also do an A/B keystone behind keystone, I have also seen keystone presented more cleverly utilising anycast addressing and database replication, which avoids the aforementioned situation. Thanks, Karl Kloppenborg. Karl Kloppenborg Chief Technology Officer m: +61 437 239 565 resetdata.com<https://resetdata.com/> [cid:reset_69557fc2-1d63-4932-b5fd-93bd4f39ca7b.png] ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit; https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-state.pdf<https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-state.pdf> This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact@resetdata.com.au<mailto:contact@resetdata.com.au> From: Artem Goncharov <artem.goncharov@gmail.com> Date: Tuesday, 15 October 2024 at 10:36 pm To: Stephen Finucane <stephenfin@redhat.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [keystone] What does a multi-region service catalog look like? Short answer Stephen, there can be both (and I've seen both), but typically there is single keystone for multiple non-identity regions. ---- typed from mobile, auto-correct typos assumed ---- On Tue, Oct 15, 2024, 13:33 Stephen Finucane <stephenfin@redhat.com<mailto:stephenfin@redhat.com>> wrote: o/ The question in $subject came up in a discussion earlier today and I neither knew the answer nor was able to discover it by looking at a service catalogs for a few production deployments I have access to (also, the multi-region feature is currently broken in DevStack and likely has been for some time). Crucially, in a "standard" multi-region deployment where there is one Keystone instances and N Nova/Cinder/Glance/Neutron/... deployments, would I have something like the below, where there is only one identity endpoint but multiple e.g. compute endpoints, or would I expect there always to be as many identity endpoints as there are e.g. compute endpoints? { "token": { ... "catalog": [ { "endpoints": [ { "id": "fb757b386a23430d8f4d636b8b45fa89", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/identity", "region": "RegionOne" } ], "id": "50c4feff9d6149b898fce20ce23904a3", "type": "identity", "name": "keystone" }, { "endpoints": [ { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/compute/v2.1", "region": "RegionOne" }, { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionTwo", "url": "http://10.0.112.232/compute/v2.1", "region": "RegionOne" } ], "id": "87322637c5ab4e7b90bea7a8c1eb9525", "type": "compute", "name": "nova" }, ... ] } } I'm guessing the latter, but I'd like to confirm this. Stephen
That's great info. Thanks both. So to be clear, Karl, in your service catalog you see a single endpoint for Keystone and N endpoints (one per region) for Cinder, Nova etc? And tooling (OSC, SDK, etc.) is happy using one region for Keystone and potentially a different one for all other services? I would like to document this somewhere, so I'd like to make sure said docs reflect a real world scenario. If you had a service catalog you could share (with potentially sensitive info like URLs stripped out/anonymised) that would be helpful. Cheers, Stephen On Tue, 2024-10-15 at 22:19 +0000, Karl Kloppenborg wrote:
Hi Stephen, As Artem has pointed out, we’ve seen both. Some people also have a single keystone distributed across several regions using database replication. There’s pros and cons to all these deployments, however we personally use a centralised keystone and multiple regions with region specific endpoints. The downside of this is that loss of comms to the keystone makes multiple regions inaccessible however it doesn’t break customer compute workloads, so we absorb that as acceptable risk. Some others also do an A/B keystone behind keystone, I have also seen keystone presented more cleverly utilising anycast addressing and database replication, which avoids the aforementioned situation. Thanks, Karl Kloppenborg.
Karl Kloppenborg
Chief Technology Officer
m: +61 437 239 565 resetdata.com
reset.png
ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit; https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-stat...
This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact@resetdata.com.au
From:Artem Goncharov <artem.goncharov@gmail.com> Date: Tuesday, 15 October 2024 at 10:36 pm To: Stephen Finucane <stephenfin@redhat.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [keystone] What does a multi-region service catalog look like?
Short answer Stephen, there can be both (and I've seen both), but typically there is single keystone for multiple non-identity regions.
---- typed from mobile, auto-correct typos assumed ----
On Tue, Oct 15, 2024, 13:33 Stephen Finucane <stephenfin@redhat.com> wrote:
o/
The question in $subject came up in a discussion earlier today and I neither knew the answer nor was able to discover it by looking at a service catalogs for a few production deployments I have access to (also, the multi-region feature is currently broken in DevStack and likely has been for some time). Crucially, in a "standard" multi-region deployment where there is one Keystone instances and N Nova/Cinder/Glance/Neutron/... deployments, would I have something like the below, where there is only one identity endpoint but multiple e.g. compute endpoints, or would I expect there always to be as many identity endpoints as there are e.g. compute endpoints?
{ "token": { ... "catalog": [ { "endpoints": [ { "id": "fb757b386a23430d8f4d636b8b45fa89", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/identity", "region": "RegionOne" } ], "id": "50c4feff9d6149b898fce20ce23904a3", "type": "identity", "name": "keystone" }, { "endpoints": [ { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/compute/v2.1", "region": "RegionOne" }, { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionTwo", "url": "http://10.0.112.232/compute/v2.1", "region": "RegionOne" } ], "id": "87322637c5ab4e7b90bea7a8c1eb9525", "type": "compute", "name": "nova" }, ... ] } }
I'm guessing the latter, but I'd like to confirm this.
Stephen
Hi Stephen, In our case, we end up with multiple keystone endpoints (one for each region) but going to the same auth.<blah> address. Will try and send over a sample later. I think it would be a good addition to the core openstack documentation to talk more about multi-region deployments and what the big operators get up to with implementing this. It’s also worth asking the question, how does this change when SSO/Centralised AD etc authentication is put into practice? Thanks, Karl. Karl Kloppenborg Chief Technology Officer m: +61 437 239 565 resetdata.com<https://resetdata.com/> [cid:reset_69557fc2-1d63-4932-b5fd-93bd4f39ca7b.png] ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit; https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-state.pdf<https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-state.pdf> This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact@resetdata.com.au<mailto:contact@resetdata.com.au> From: Stephen Finucane <stephenfin@redhat.com> Date: Wednesday, 16 October 2024 at 8:40 pm To: Karl Kloppenborg <kkloppenborg@resetdata.com.au>, Artem Goncharov <artem.goncharov@gmail.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [keystone] What does a multi-region service catalog look like? That's great info. Thanks both. So to be clear, Karl, in your service catalog you see a single endpoint for Keystone and N endpoints (one per region) for Cinder, Nova etc? And tooling (OSC, SDK, etc.) is happy using one region for Keystone and potentially a different one for all other services? I would like to document this somewhere, so I'd like to make sure said docs reflect a real world scenario. If you had a service catalog you could share (with potentially sensitive info like URLs stripped out/anonymised) that would be helpful. Cheers, Stephen On Tue, 2024-10-15 at 22:19 +0000, Karl Kloppenborg wrote: Hi Stephen, As Artem has pointed out, we’ve seen both. Some people also have a single keystone distributed across several regions using database replication. There’s pros and cons to all these deployments, however we personally use a centralised keystone and multiple regions with region specific endpoints. The downside of this is that loss of comms to the keystone makes multiple regions inaccessible however it doesn’t break customer compute workloads, so we absorb that as acceptable risk. Some others also do an A/B keystone behind keystone, I have also seen keystone presented more cleverly utilising anycast addressing and database replication, which avoids the aforementioned situation. Thanks, Karl Kloppenborg. Karl Kloppenborg Chief Technology Officer m: +61 437 239 565 resetdata.com<https://resetdata.com/> [cid:reset_69557fc2-1d63-4932-b5fd-93bd4f39ca7b.png] ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit; https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-stat... This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact@resetdata.com.au<mailto:contact@resetdata.com.au> From:Artem Goncharov <artem.goncharov@gmail.com> Date: Tuesday, 15 October 2024 at 10:36 pm To: Stephen Finucane <stephenfin@redhat.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: Re: [keystone] What does a multi-region service catalog look like? Short answer Stephen, there can be both (and I've seen both), but typically there is single keystone for multiple non-identity regions. ---- typed from mobile, auto-correct typos assumed ---- On Tue, Oct 15, 2024, 13:33 Stephen Finucane <stephenfin@redhat.com<mailto:stephenfin@redhat.com>> wrote: o/ The question in $subject came up in a discussion earlier today and I neither knew the answer nor was able to discover it by looking at a service catalogs for a few production deployments I have access to (also, the multi-region feature is currently broken in DevStack and likely has been for some time). Crucially, in a "standard" multi-region deployment where there is one Keystone instances and N Nova/Cinder/Glance/Neutron/... deployments, would I have something like the below, where there is only one identity endpoint but multiple e.g. compute endpoints, or would I expect there always to be as many identity endpoints as there are e.g. compute endpoints? { "token": { ... "catalog": [ { "endpoints": [ { "id": "fb757b386a23430d8f4d636b8b45fa89", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/identity", "region": "RegionOne" } ], "id": "50c4feff9d6149b898fce20ce23904a3", "type": "identity", "name": "keystone" }, { "endpoints": [ { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionOne", "url": "http://10.0.110.125/compute/v2.1", "region": "RegionOne" }, { "id": "2ed0cb59adb0415c948f8351b6546d3e", "interface": "public", "region_id": "RegionTwo", "url": "http://10.0.112.232/compute/v2.1", "region": "RegionOne" } ], "id": "87322637c5ab4e7b90bea7a8c1eb9525", "type": "compute", "name": "nova" }, ... ] } } I'm guessing the latter, but I'd like to confirm this. Stephen
participants (3)
-
Artem Goncharov
-
Karl Kloppenborg
-
Stephen Finucane