[keystone][horizon] Two approaches to "multi-region" OpenStack deployment
Hi, I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds. The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc. Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica. This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails. The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources. While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks: * In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role) Additional thoughts Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon? In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense? In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require. -Chris
Hi Chris, On Tue, Mar 24, 2020, at 06:45, Krzysztof Klimonda wrote:
Hi,
I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds.
The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc.
Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica.
This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails.
The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources.
While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks:
* In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role)
Additional thoughts
Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon?
In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense?
In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require.
-Chris
An alternative to either replicating databases or using an external data syncing tool that the keystone team has been pushing is to federate your keystone deployments. With Keystone-to-Keystone federation, keystone instances act as identity providers to one another, and keystone instances are registered as service providers to one another - which allows horizon to recognize the other keystone instances as alternative sites and allow users to log into them ("switch endpoints" and get a new token) without having prior knowledge of them. The data is not replicated between keystone databases but mapping rules allow you to create identical authorization schemes that gives a uniform user experience on each site. More information can be found in our documentation: https://docs.openstack.org/keystone/latest/admin/federation/federated_identi... Hope this helps as a starting point, happy to answer further questions. Colleen (cmurphy)
Hi Colleen, Interesting - I’ve looked at K2K federation initially, but it didn’t seem to fit my requirements. My understanding is that in this setup we have a global keystone (which I’ve described in my initial post) and then all regions are federated to that keystone. Is that correct? Would that work with an existing SSO that we have in place (Keycloak) or does global keystone has to be IdP and not chain to another one? -Chris
On 24 Mar 2020, at 21:53, Colleen Murphy <colleen@gazlene.net> wrote:
Hi Chris,
On Tue, Mar 24, 2020, at 06:45, Krzysztof Klimonda wrote:
Hi,
I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds.
The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc.
Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica.
This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails.
The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources.
While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks:
* In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role)
Additional thoughts
Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon?
In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense?
In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require.
-Chris
An alternative to either replicating databases or using an external data syncing tool that the keystone team has been pushing is to federate your keystone deployments. With Keystone-to-Keystone federation, keystone instances act as identity providers to one another, and keystone instances are registered as service providers to one another - which allows horizon to recognize the other keystone instances as alternative sites and allow users to log into them ("switch endpoints" and get a new token) without having prior knowledge of them. The data is not replicated between keystone databases but mapping rules allow you to create identical authorization schemes that gives a uniform user experience on each site.
More information can be found in our documentation:
https://docs.openstack.org/keystone/latest/admin/federation/federated_identi...
Hope this helps as a starting point, happy to answer further questions.
Colleen (cmurphy)
On Wed, 25 Mar 2020 at 08:49, Krzysztof Klimonda <kklimonda@syntaxhighlighted.com> wrote:
Hi Colleen,
Interesting - I’ve looked at K2K federation initially, but it didn’t seem to fit my requirements.
My understanding is that in this setup we have a global keystone (which I’ve described in my initial post) and then all regions are federated to that keystone. Is that correct? Would that work with an existing SSO that we have in place (Keycloak) or does global keystone has to be IdP and not chain to another one?
Hi Chris, You might be interested to read our blog on federated Keystone using Keycloak: https://www.stackhpc.com/federation-and-identity-brokering-using-keycloak.ht.... Mark
-Chris
On 24 Mar 2020, at 21:53, Colleen Murphy <colleen@gazlene.net> wrote:
Hi Chris,
On Tue, Mar 24, 2020, at 06:45, Krzysztof Klimonda wrote:
Hi,
I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds.
The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc.
Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica.
This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails.
The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources.
While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks:
* In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role)
Additional thoughts
Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon?
In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense?
In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require.
-Chris
An alternative to either replicating databases or using an external data syncing tool that the keystone team has been pushing is to federate your keystone deployments. With Keystone-to-Keystone federation, keystone instances act as identity providers to one another, and keystone instances are registered as service providers to one another - which allows horizon to recognize the other keystone instances as alternative sites and allow users to log into them ("switch endpoints" and get a new token) without having prior knowledge of them. The data is not replicated between keystone databases but mapping rules allow you to create identical authorization schemes that gives a uniform user experience on each site.
More information can be found in our documentation:
https://docs.openstack.org/keystone/latest/admin/federation/federated_identi...
Hope this helps as a starting point, happy to answer further questions.
Colleen (cmurphy)
Hi Mark, Colleen, @Mark Thanks, I remember reading that blog post already :). Just to make sure, in the deployment described in the post you federate keystone with keycloak, but without K2K federation in the middle, right? Also, in the linked blog post there is no mention of regions, so my understanding is that this is a single keycloak->keystone federation. On a related note, in your blog post you show “chained federation” in Keycloak. Does this setup support generating Keystone tokens for API/CLI access? If so, how does the authentication flow look like? Is keystone federated with keycloak using OIDC, and `openstack token issue` opens a web browser to allow for selecting IdP configured in keycloak? “Chained Federation” didn’t seem to work with SAML2 + ECP profile. @Colleen I’m trying to visualise your proposed solution, does this diagram make sense: http://paste.openstack.org/show/791120/ <http://paste.openstack.org/show/791120/> ? In K2K federation, how are regions handled? Are they out of the picture, with federated keystone taking their place instead? I assume all keystone catalogs are separate, and users must use specific cloud auth url to authenticate. Authentication can be then delegated to central keystone, but what about another layer of SSO, i.e. Keycloak? -Chris
On 25 Mar 2020, at 10:27, Mark Goddard <mark@stackhpc.com> wrote:
On Wed, 25 Mar 2020 at 08:49, Krzysztof Klimonda <kklimonda@syntaxhighlighted.com> wrote:
Hi Colleen,
Interesting - I’ve looked at K2K federation initially, but it didn’t seem to fit my requirements.
My understanding is that in this setup we have a global keystone (which I’ve described in my initial post) and then all regions are federated to that keystone. Is that correct? Would that work with an existing SSO that we have in place (Keycloak) or does global keystone has to be IdP and not chain to another one?
Hi Chris,
You might be interested to read our blog on federated Keystone using Keycloak: https://www.stackhpc.com/federation-and-identity-brokering-using-keycloak.ht....
Mark
-Chris
On 24 Mar 2020, at 21:53, Colleen Murphy <colleen@gazlene.net> wrote:
Hi Chris,
On Tue, Mar 24, 2020, at 06:45, Krzysztof Klimonda wrote:
Hi,
I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds.
The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc.
Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica.
This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails.
The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources.
While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks:
* In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role)
Additional thoughts
Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon?
In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense?
In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require.
-Chris
An alternative to either replicating databases or using an external data syncing tool that the keystone team has been pushing is to federate your keystone deployments. With Keystone-to-Keystone federation, keystone instances act as identity providers to one another, and keystone instances are registered as service providers to one another - which allows horizon to recognize the other keystone instances as alternative sites and allow users to log into them ("switch endpoints" and get a new token) without having prior knowledge of them. The data is not replicated between keystone databases but mapping rules allow you to create identical authorization schemes that gives a uniform user experience on each site.
More information can be found in our documentation:
https://docs.openstack.org/keystone/latest/admin/federation/federated_identi...
Hope this helps as a starting point, happy to answer further questions.
Colleen (cmurphy)
Hi Chris, responses inline:
On Wed, 25 Mar 2020 at 08:49, Krzysztof Klimonda <kklimonda@syntaxhighlighted.com> wrote:
Hi Colleen,
Interesting - I’ve looked at K2K federation initially, but it didn’t seem to fit my requirements.
My understanding is that in this setup we have a global keystone (which I’ve described in my initial post) and then all regions are federated to that keystone. Is that correct? Would that work with an existing SSO that we have in place (Keycloak) or does global keystone has to be IdP and not chain to another one?
In the setup I described, there doesn't necessarily need to be a global or authoritative keystone. Rather, each keystone could act as both an IdP and an SP to one another. We usually simplify it in documentation to describe a keystone instance as being one or the other, but in practice there's no need for that restriction. In theory a keystone instance could work as a service provider to a Keycloak IdP while also acting as an IdP for other keystones. On Wed, Mar 25, 2020, at 04:00, Krzysztof Klimonda wrote: [snipped]
@Colleen I’m trying to visualise your proposed solution, does this diagram make sense: http://paste.openstack.org/show/791120/ ? In K2K federation, how are regions handled? Are they out of the picture, with federated keystone taking their place instead? I assume all keystone catalogs are separate, and users must use specific cloud auth url to authenticate. Authentication can be then delegated to central keystone, but what about another layer of SSO, i.e. Keycloak? [snipped]
Your diagram is a hierarchical tree where it doesn't necessarily need to be, it could be a bidirected graph. But having a single known entrypoint into the cluster of clouds as shown in the diagram may also be a good way to structure it. "Regions" in the OpenStack sense of the word would not really apply because regions are owned by a single keystone instance. This also does indeed mean that the catalogs are separate, so users could only discover other catalogs by knowing the endpoint of one keystone, discovering service providers for that keystone, and then querying for catalogs from the other keystone service providers. The lack of a global or federated catalog for this use case is something that we recognize as a gap. The other, simpler "federation" approach that has been mentioned in this thread is to omit keystone as an IdP and instead use an external non-OpenStack IdP like Keycloak as your source of identity for all sites. This is conceptually easier but still doesn't solve the problem of having no shared catalog between sites. Colleen (cmurphy)
On Wed, 25 Mar 2020 at 11:08, Krzysztof Klimonda <kklimonda@syntaxhighlighted.com> wrote:
@Mark Thanks, I remember reading that blog post already :). Just to make sure, in the deployment described in the post you federate keystone with keycloak, but without K2K federation in the middle, right?
Correct, as colleen confirmed, its without K2K federation. We use OIDC to link Keycloak with OpenStack. And also use OIDC between Keycloak and the acutal IdP, For context, we are using this pattern, using Keycloak as the proxy: https://aarc-project.eu/architecture/ I see that as hub-and-spoke when you compare it to other approaches here: https://wiki.geant.org/display/eduGAIN/Federation+Architectures I should be clear the requirements for this project ruled out K2K from the start. Rather than being multi-region, our aim was allowing people to login to OpenStack using the regular organisation credentials. (In the UK university context, we made use of the existing Edugain federation.) Howerver, we also needed something that also allowed user applications to also make use of the same AAAI system, in particular to project the users web apps like JuypterHub, OpenOnDemand, etc.
On a related note, in your blog post you show “chained federation” in Keycloak. Does this setup support generating Keystone tokens for API/CLI access? If so, how does the authentication flow look like?
We are using Application Credentials for all API and CLI access (including terraform): https://docs.openstack.org/keystone/train/user/application_credentials.html They are keystone region specific (which I see as a positive feature), but they work well. (Assuming your federation mapping doesn't use groups, and instead directly maps users to projects) I hope that helps, johnthetubaguy
We have a multi-region (20+ regions) deployment with Keystone. We use the old templated ini format for catalog and ldap for user backend. We then create all our projects using matching uuids (we added custom patch to allow you to provide your own uuid for projects). Initially we replicated the database globally, but some newer features (and/or changes) added to Keystone made this difficult. We also do something similar for Glance with a globally replicated Swift cluster. Best Regards, Erik Olof Gunnar Andersson -----Original Message----- From: Krzysztof Klimonda <kklimonda@syntaxhighlighted.com> Sent: Tuesday, March 24, 2020 6:46 AM To: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: [keystone][horizon] Two approaches to "multi-region" OpenStack deployment Hi, I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds. The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc. Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica. This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails. The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources. While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks: * In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role) Additional thoughts Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon? In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense? In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require. -Chris
Hi Chris, Following your email, I think it makes sense to mention the activities we have been doing initially within the FEMDC SiG [1] and more recently under the Edge Computing WG managed by the OSF., To make a long story short, we [2] tried both approaches you mentionned. Regarding the shared database first, we investigated and compared the performance of Galera and CockroachDB[3]. You can refer to [4][5] for futher details. Regarding the second approach you talked about, It looks rather similar to the OID prototype we introduced during the Berlin Hackathon/summit [6] and we will be more than happy to get your feedback on it. Briefly, our mechanism is larger than delivering collaborations/federations between mutiple instances of keystone, larger in the sense that it can be applied to any openstack service. Considering several OpenStacks running, the idea is to use a DSL to specify which instance of the service (i.e. on which Openstack) the user wants its request to be executed. For instance, to provision a VM on a Nova in Paris using an image from a Glance in New York. You may contact the API either in NewYork or in Paris and perform the following request: "openstack server create --image debian --scope { compute: Paris, image: New York }" A proof of concept is available on GitLab[7]. Obviously, our approach needs to be challenged and additional actions should be done to deliver a production ready system. For instance, we should define a clear semantic for the DSL. This is critical as we are extending the DSL with operators such as AND and OR. Using operators will enable Admin/DevOps to perform requests such as : list all VMs running in Paris, NewYork etc : "openstack server list --scope{compute: Paris & Newyork & Boston}" An academic paper presenting the concept overall has been submitted to the HotEdge topic. We don't know yet whether the paper will be accepted or not (due to the COVID-19 everything is unfortunately delayed). However, we hope to share it ASAP. Feedbacks/questions/comments welcome :-) Have a nice day. [1]https://wiki.openstack.org/wiki/Fog_Edge_Massively_Distributed_Clouds [2] https://beyondtheclouds.github.io/ [3] https://www.cockroachlabs.com/ [4] https://beyondtheclouds.github.io/blog/openstack/cockroachdb/2018/06/04/eval... This article was never entirely finished because we lack time, but you can find every results we had there. Note also that CockroachDB team did some amazing work to get better perfomance since, so the results would probably be better now. [5] https://www.openstack.org/videos/summits/vancouver-2018/keystone-in-the-cont... [6] https://www.openstack.org/summit/denver-2019/summit-schedule/events/23352/im... [7] https://gitlab.inria.fr/discovery/openstackoid Marie Delavergne ----- Mail original -----
De: "Krzysztof Klimonda" <kklimonda@syntaxhighlighted.com> À: "openstack-discuss" <openstack-discuss@lists.openstack.org> Envoyé: Mardi 24 Mars 2020 14:45:42 Objet: [keystone][horizon] Two approaches to "multi-region" OpenStack deployment
Hi,
I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds.
The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc.
Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica.
This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails.
The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources.
While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks:
* In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role)
Additional thoughts
Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon?
In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense?
In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require.
-Chris
participants (6)
-
Colleen Murphy
-
Erik Olof Gunnar Andersson
-
John Garbutt
-
Krzysztof Klimonda
-
Marie Delavergne
-
Mark Goddard