Good morning Nguyễn. Latency won’t be too much of an issue because if you have read replicas in each region, you can route your read requests to the read slaves and only have the writes pushed to the master. There’s a few different options in terms of how you route, we use proxysql.com for our systems. I’d say about 90% of traffic in your keystone environment will be read traffic, 10% write. So I really would not be too concerned about your latency. Also, keystone isn’t a very heavy use database. TXN replication or WAL latency shouldn’t be too much of a problem between the continents. The fact of the matter is, the way you’re architecting this will always need to have compromises. You’re basically hitting the issues described in CAP theorem. (Read more here: https://en.m.wikipedia.org/wiki/CAP_theorem) You’ll notice that most AWS and GCP, Azure etc whenever IAM or Keystone equivalents are written to, I.e a token made, permissions updated, there’s a delay in the response, this is usually artificial and designed to induce enough time into the request so that reader slaves have received the latest WAL replications. / or db version equivalent. The only alternative to this is if you can deploy something like cockroachDB or Yugabyte. However, this will be fraught with heavy and costly complexity. Thanks, Karl. On Tue, 25 Jul 2023 at 09:17, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello Karl, How are you? Thank you for your response.. Hope you are ok. Nguyen Huu Khoi
On Mon, Jul 24, 2023 at 10:46 AM Karl Kloppenborg < kkloppenborg@rwts.com.au> wrote:
Apologies I’ve been off sick.
However yes, this is the way we do it as well.
I would say this is also the most sane way to deal with this.
Thanks, Karl.
*From: *Arnaud Morin <arnaud.morin@gmail.com> *Date: *Sunday, 23 July 2023 at 10:56 pm *To: *Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> *Cc: *Karl Kloppenborg <kkloppenborg@rwts.com.au>, OpenStack Discuss < openstack-discuss@lists.openstack.org> *Subject: *Re: [openstack][largescale-sig] Openstack multi region deployment
We have this model also with only one keystone. We have multiple galera clusters synchronized together. Only one cluster is used for write requests (located in one region), others are read only / cache. Most of the calls done to our keystone are "read" or token validation requests, and this works fine with a read galera cluster / cache.
I know that we also have a custom way to invalidate cache across regions, but I dont remember the details, I can ask the team.
Anyway, this is do-able :)
I imagine it also depends on the usage you have, if you create a lot of users/projects/assignments, then it may be harder to achieve.
Cheers, Arnaud
On 19.07.23 - 14:03, Nguyễn Hữu Khôi wrote:
Hello, thank you very much.
But can I ask how we process if 1 region at ASIA and 2 regions in the USA?
Database latency will be our problem.
Nguyen Huu Khoi
On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg < kkloppenborg@rwts.com.au> wrote:
Hi Nguy,
We’ve deployed a large multi-region openstack deployment.
As a rule of thumb we’ve got a “keystone” region which is as best we can highly available and very redundant.
We then have all other regions talk back to this region, we just usually call it “keystone” or “core” and it’s hidden from the UI from users.
We then just run a large well kept Galara cluster to support it.
--Karl.
*From: *openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> *Date: *Tuesday, 18 July 2023 at 9:25 pm *To: *openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject: *openstack-discuss Digest, Vol 57, Issue 55
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez)
----------------------------------------------------------------------
Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: < CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3
nodes).
and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much.