We have a multi-region (20+ regions) deployment with Keystone. We use the old templated ini format for catalog and ldap for user backend. We then create all our projects using matching uuids (we added custom patch to allow you to provide your own uuid for projects). Initially we replicated the database globally, but some newer features (and/or changes) added to Keystone made this difficult. We also do something similar for Glance with a globally replicated Swift cluster. Best Regards, Erik Olof Gunnar Andersson -----Original Message----- From: Krzysztof Klimonda <kklimonda@syntaxhighlighted.com> Sent: Tuesday, March 24, 2020 6:46 AM To: openstack-discuss <openstack-discuss@lists.openstack.org> Subject: [keystone][horizon] Two approaches to "multi-region" OpenStack deployment Hi, I’ve been spending some time recently thinking about best approach to some sort of multi-region deployment of OpenStack clouds. The two approaches that I’m currently evaluating are shared keystone database (with galera cluster spanning three locations) and shared-nothing approach, where external component is responsible for managing users, projects etc. Shared keystone database seems fairly straightforward from OS point of view (I’m ignoring galera replication over WAN woes for the purpose of this discussion) until I hit 3 regions. Additional regions must either reuse “global” keystone adding latency everywhere, or we need a way to replicate data from “master” galera cluster to “slave” clusters, and route all database write statements back to the master galera cluster, while reading from local asynchronous replica. This has me worried somewhat, as doing that that into eventually-consistent deployment of sort. Services deployed in regions with asynchronous replication can no longer depend on the fact that once transaction is finished, consecutive reads will return up-to-date state. I can imagine scenarios where, as an example, trust is setup for heat, but that fact is not replicated back to the database by the time heat tries to issue a token based on that trust and the process fails. The other approach would be to keep keystone databases completely separate, and have something external to openstack manage all those resources. While not sharing keystone database between regions sidesteps the issue of scalability, and the entire setup seems to be more resilient to failures, it’s not without its own drawbacks: * In this setup Horizon can no longer switch between regions without additional work (see notes below) * There is no longer single API entrypoint to the cloud * Some Keystone API operations would have to be removed from users via custom policy - for example, managing user assignment to projects (for users who have domain admin role) Additional thoughts Could horizon be modified to switch endpoints based on the region selected in the UI? Is the token reissued when region is changed in horizon, or is single token used? I’m assuming it’s the former given my understanding that when projects are changed, a new token is issued - but perhaps the initial token is always used to issue project-scoped tokens for Horizon? In the second scenario, with separate keystone databases, a backend for keystone could be created that proxies some operations (like aforementioned user assignment) back to the external manager so that it can be propagated to other clouds. Does that even make sense? In the end I’m reaching out in hope that someone could chime in based on their experience - perhaps I’m missing a better approach, or making wrong assumptions in my email, especially around asynchronous replication of keystone database and its effect on services in regions that may not have up-to-data view of the databas. Or perhaps trying ot synchronize keystone state by external tool is not really worth the additional effort that would require. -Chris