Open Stack

Hi Colleen,

Interesting - I’ve looked at K2K federation initially, but it didn’t seem to fit my requirements.

My understanding is that in this setup we have a global keystone (which I’ve described in my initial post)
and then all regions are federated to that keystone. Is that correct? Would that work with an existing
SSO that we have in place (Keycloak) or does global keystone has to be IdP and not chain to another one?

-Chris

> On 24 Mar 2020, at 21:53, Colleen Murphy <colleen at gazlene.net> wrote:
> 
> Hi Chris,
> 
> On Tue, Mar 24, 2020, at 06:45, Krzysztof Klimonda wrote:
>> Hi,
>> 
>> I’ve been spending some time recently thinking about best approach to 
>> some sort of multi-region deployment of OpenStack clouds.
>> 
>> The two approaches that I’m currently evaluating are shared keystone 
>> database (with galera cluster spanning three locations) and 
>> shared-nothing approach, where external component is responsible for 
>> managing users, projects etc.
>> 
>> Shared keystone database seems fairly straightforward from OS point of 
>> view (I’m ignoring galera replication over WAN woes for the purpose of 
>> this discussion) until I hit 3 regions. Additional regions must either 
>> reuse “global” keystone adding latency everywhere, or we need a way to 
>> replicate data from “master” galera cluster to “slave” clusters, and 
>> route all database write statements back to the master galera cluster, 
>> while reading from local asynchronous replica.
>> 
>> This has me worried somewhat, as doing that that into 
>> eventually-consistent deployment of sort. Services deployed in regions 
>> with asynchronous replication can no longer depend on the fact that 
>> once transaction is finished, consecutive reads will return up-to-date 
>> state. I can imagine scenarios where, as an example, trust is setup for 
>> heat, but that fact is not replicated back to the database by the time 
>> heat tries to issue a token based on that trust and the process fails.
>> 
>> The other approach would be to keep keystone databases completely 
>> separate, and have something external to openstack manage all those 
>> resources.
>> 
>> While not sharing keystone database between regions sidesteps the issue 
>> of scalability, and the entire setup seems to be more resilient to 
>> failures, it’s not without its own drawbacks:
>> 
>> * In this setup Horizon can no longer switch between regions without 
>> additional work (see notes below)
>> * There is no longer single API entrypoint to the cloud
>> * Some Keystone API operations would have to be removed from users via 
>> custom policy - for example, managing user assignment to projects (for 
>> users who have domain admin role)
>> 
>> Additional thoughts
>> 
>> Could horizon be modified to switch endpoints based on the region 
>> selected in the UI? Is the token reissued when region is changed in 
>> horizon, or is single token used? I’m assuming it’s the former given my 
>> understanding that when projects are changed, a new token is issued - 
>> but perhaps the initial token is always used to issue project-scoped 
>> tokens for Horizon?
>> 
>> In the second scenario, with separate keystone databases, a backend for 
>> keystone could be created that proxies some operations (like 
>> aforementioned user assignment) back to the external manager so that it 
>> can be propagated to other clouds. Does that even make sense?
>> 
>> In the end I’m reaching out in hope that someone could chime in based 
>> on their experience - perhaps I’m missing a better approach, or making 
>> wrong assumptions in my email, especially around asynchronous 
>> replication of keystone database and its effect on services in regions 
>> that may not have up-to-data view of the databas. Or perhaps trying ot 
>> synchronize keystone state by external tool is not really worth the 
>> additional effort that would require.
>> 
>> -Chris
>> 
> 
> An alternative to either replicating databases or using an external data syncing tool that the keystone team has been pushing is to federate your keystone deployments. With Keystone-to-Keystone federation, keystone instances act as identity providers to one another, and keystone instances are registered as service providers to one another - which allows horizon to recognize the other keystone instances as alternative sites and allow users to log into them ("switch endpoints" and get a new token) without having prior knowledge of them. The data is not replicated between keystone databases but mapping rules allow you to create identical authorization schemes that gives a uniform user experience on each site.
> 
> More information can be found in our documentation:
> 
> https://docs.openstack.org/keystone/latest/admin/federation/federated_identity.html
> 
> Hope this helps as a starting point, happy to answer further questions.
> 
> Colleen (cmurphy)
>