Open Stack

Hi Chris,

On Tue, Mar 24, 2020, at 06:45, Krzysztof Klimonda wrote:
> Hi,
> 
> I’ve been spending some time recently thinking about best approach to 
> some sort of multi-region deployment of OpenStack clouds.
> 
> The two approaches that I’m currently evaluating are shared keystone 
> database (with galera cluster spanning three locations) and 
> shared-nothing approach, where external component is responsible for 
> managing users, projects etc.
> 
> Shared keystone database seems fairly straightforward from OS point of 
> view (I’m ignoring galera replication over WAN woes for the purpose of 
> this discussion) until I hit 3 regions. Additional regions must either 
> reuse “global” keystone adding latency everywhere, or we need a way to 
> replicate data from “master” galera cluster to “slave” clusters, and 
> route all database write statements back to the master galera cluster, 
> while reading from local asynchronous replica.
> 
> This has me worried somewhat, as doing that that into 
> eventually-consistent deployment of sort. Services deployed in regions 
> with asynchronous replication can no longer depend on the fact that 
> once transaction is finished, consecutive reads will return up-to-date 
> state. I can imagine scenarios where, as an example, trust is setup for 
> heat, but that fact is not replicated back to the database by the time 
> heat tries to issue a token based on that trust and the process fails.
> 
> The other approach would be to keep keystone databases completely 
> separate, and have something external to openstack manage all those 
> resources.
> 
> While not sharing keystone database between regions sidesteps the issue 
> of scalability, and the entire setup seems to be more resilient to 
> failures, it’s not without its own drawbacks:
> 
> * In this setup Horizon can no longer switch between regions without 
> additional work (see notes below)
> * There is no longer single API entrypoint to the cloud
> * Some Keystone API operations would have to be removed from users via 
> custom policy - for example, managing user assignment to projects (for 
> users who have domain admin role)
> 
> Additional thoughts
> 
> Could horizon be modified to switch endpoints based on the region 
> selected in the UI? Is the token reissued when region is changed in 
> horizon, or is single token used? I’m assuming it’s the former given my 
> understanding that when projects are changed, a new token is issued - 
> but perhaps the initial token is always used to issue project-scoped 
> tokens for Horizon?
> 
> In the second scenario, with separate keystone databases, a backend for 
> keystone could be created that proxies some operations (like 
> aforementioned user assignment) back to the external manager so that it 
> can be propagated to other clouds. Does that even make sense?
> 
> In the end I’m reaching out in hope that someone could chime in based 
> on their experience - perhaps I’m missing a better approach, or making 
> wrong assumptions in my email, especially around asynchronous 
> replication of keystone database and its effect on services in regions 
> that may not have up-to-data view of the databas. Or perhaps trying ot 
> synchronize keystone state by external tool is not really worth the 
> additional effort that would require.
> 
> -Chris
>

An alternative to either replicating databases or using an external data syncing tool that the keystone team has been pushing is to federate your keystone deployments. With Keystone-to-Keystone federation, keystone instances act as identity providers to one another, and keystone instances are registered as service providers to one another - which allows horizon to recognize the other keystone instances as alternative sites and allow users to log into them ("switch endpoints" and get a new token) without having prior knowledge of them. The data is not replicated between keystone databases but mapping rules allow you to create identical authorization schemes that gives a uniform user experience on each site.

More information can be found in our documentation:

https://docs.openstack.org/keystone/latest/admin/federation/federated_identity.html

Hope this helps as a starting point, happy to answer further questions.

Colleen (cmurphy)