[openstack][largescale-sig] Openstack multi region deployment
Hi Nguy, We’ve deployed a large multi-region openstack deployment. As a rule of thumb we’ve got a “keystone” region which is as best we can highly available and very redundant. We then have all other regions talk back to this region, we just usually call it “keystone” or “core” and it’s hidden from the UI from users. We then just run a large well kept Galara cluster to support it. --Karl. From: openstack-discuss-request@lists.openstack.org <openstack-discuss-request@lists.openstack.org> Date: Tuesday, 18 July 2023 at 9:25 pm To: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: openstack-discuss Digest, Vol 57, Issue 55 Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..." Today's Topics: 1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez) ---------------------------------------------------------------------- Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: <CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hello guys, I am going to deploy openstack multi regions and I know that keystone replication is the most challenging. I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes) When region 1 is node available, I will map region 2 to use standby edge galera cluster. I hope you give me some experience and advice with multi regions. Thank you very much.
Hello, thank you very much. But can I ask how we process if 1 region at ASIA and 2 regions in the USA? Database latency will be our problem. Nguyen Huu Khoi On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg <kkloppenborg@rwts.com.au> wrote:
Hi Nguy,
We’ve deployed a large multi-region openstack deployment.
As a rule of thumb we’ve got a “keystone” region which is as best we can highly available and very redundant.
We then have all other regions talk back to this region, we just usually call it “keystone” or “core” and it’s hidden from the UI from users.
We then just run a large well kept Galara cluster to support it.
--Karl.
*From: *openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> *Date: *Tuesday, 18 July 2023 at 9:25 pm *To: *openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject: *openstack-discuss Digest, Vol 57, Issue 55
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez)
----------------------------------------------------------------------
Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: < CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much.
We have this model also with only one keystone. We have multiple galera clusters synchronized together. Only one cluster is used for write requests (located in one region), others are read only / cache. Most of the calls done to our keystone are "read" or token validation requests, and this works fine with a read galera cluster / cache. I know that we also have a custom way to invalidate cache across regions, but I dont remember the details, I can ask the team. Anyway, this is do-able :) I imagine it also depends on the usage you have, if you create a lot of users/projects/assignments, then it may be harder to achieve. Cheers, Arnaud On 19.07.23 - 14:03, Nguyễn Hữu Khôi wrote:
Hello, thank you very much.
But can I ask how we process if 1 region at ASIA and 2 regions in the USA?
Database latency will be our problem.
Nguyen Huu Khoi
On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg <kkloppenborg@rwts.com.au> wrote:
Hi Nguy,
We’ve deployed a large multi-region openstack deployment.
As a rule of thumb we’ve got a “keystone” region which is as best we can highly available and very redundant.
We then have all other regions talk back to this region, we just usually call it “keystone” or “core” and it’s hidden from the UI from users.
We then just run a large well kept Galara cluster to support it.
--Karl.
*From: *openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> *Date: *Tuesday, 18 July 2023 at 9:25 pm *To: *openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject: *openstack-discuss Digest, Vol 57, Issue 55
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez)
----------------------------------------------------------------------
Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: < CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much.
Apologies I’ve been off sick. However yes, this is the way we do it as well. I would say this is also the most sane way to deal with this. Thanks, Karl. From: Arnaud Morin <arnaud.morin@gmail.com> Date: Sunday, 23 July 2023 at 10:56 pm To: Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> Cc: Karl Kloppenborg <kkloppenborg@rwts.com.au>, OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [openstack][largescale-sig] Openstack multi region deployment We have this model also with only one keystone. We have multiple galera clusters synchronized together. Only one cluster is used for write requests (located in one region), others are read only / cache. Most of the calls done to our keystone are "read" or token validation requests, and this works fine with a read galera cluster / cache. I know that we also have a custom way to invalidate cache across regions, but I dont remember the details, I can ask the team. Anyway, this is do-able :) I imagine it also depends on the usage you have, if you create a lot of users/projects/assignments, then it may be harder to achieve. Cheers, Arnaud On 19.07.23 - 14:03, Nguyễn Hữu Khôi wrote:
Hello, thank you very much.
But can I ask how we process if 1 region at ASIA and 2 regions in the USA?
Database latency will be our problem.
Nguyen Huu Khoi
On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg <kkloppenborg@rwts.com.au> wrote:
Hi Nguy,
We’ve deployed a large multi-region openstack deployment.
As a rule of thumb we’ve got a “keystone” region which is as best we can highly available and very redundant.
We then have all other regions talk back to this region, we just usually call it “keystone” or “core” and it’s hidden from the UI from users.
We then just run a large well kept Galara cluster to support it.
--Karl.
*From: *openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> *Date: *Tuesday, 18 July 2023 at 9:25 pm *To: *openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject: *openstack-discuss Digest, Vol 57, Issue 55
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez)
----------------------------------------------------------------------
Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: < CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much. -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718...
------------------------------
Message: 2 Date: Tue, 18 Jul 2023 09:34:35 +0200 From: Felix Huettner <felix.huettner@mail.schwarz> To: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> Cc: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [openstack][largescale-sig] Openstack multi region deployment Message-ID: <ZLZAS-99RzzXfyr8@SIT-SLAP8639.int.lidl.net> Content-Type: text/plain; charset=utf-8
Hi,
i think you have two options here: 1. you could span a single galera cluster over all of your regions. this might have some latency issues, but if your are not too write heavy that might be fine. I know some companies use that setup. 2. you use some kind of multiple galera clusters with replication. But i have not yet heard of anybody using this setup.
An alternative might be to have separate keystone setups with separate databases. This would probably reduce the error potential, but might not fit your usecase.
Thanks Felix
On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote:
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much. Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail.
Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz>.
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz><https://www.datenschutz.schwarz%3e>.
------------------------------
Message: 3 Date: Tue, 18 Jul 2023 15:36:11 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com>, OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [openstack][largescale-sig] Openstack multi region deployment Message-ID: <CABAODRcJr= CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hi. Thank you for your reply.
The first one has a problem because each region is too soft. If a member is down, so this region is gone.
It is so challenge with us.
Nguyen Huu Khoi
On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner <felix.huettner@mail.schwarz
wrote:
Hi,
i think you have two options here: 1. you could span a single galera cluster over all of your regions. this might have some latency issues, but if your are not too write heavy that might be fine. I know some companies use that setup. 2. you use some kind of multiple galera clusters with replication. But i have not yet heard of anybody using this setup.
An alternative might be to have separate keystone setups with separate databases. This would probably reduce the error potential, but might not fit your usecase.
Thanks Felix
On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote:
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much. Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r die Verwertung durch den vorgesehenen Empf?nger bestimmt. Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender bitte unverz?glich in Kenntnis und l?schen diese E Mail.
Hinweise zum Datenschutz finden Sie hier<https://www.datenschutz.schwarz .
This e-mail may contain confidential content and is intended only for the specified recipient/s. If you are not the intended recipient, please inform the sender immediately and delete this e-mail.
Information on data protection can be found here< https://www.datenschutz.schwarz><https://www.datenschutz.schwarz%3e>.
-------------- next part -------------- An HTML attachment was scrubbed... URL: < https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718...
------------------------------
Message: 4 Date: Tue, 18 Jul 2023 13:23:27 +0200 From: Rodolfo Alonso Hernandez <ralonsoh@redhat.com> To: Roberto Bartzen Acosta <roberto.acosta@luizalabs.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.org>, Terry Wilson <twilson@redhat.com>, Tiago Pires < tiago.pires@luizalabs.com> Subject: Re: [neutron] unmanaged router resources - OVN interconnect Message-ID: < CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Ok, this is being tortuous. First of all: define a strategy. If you are going to create the resources in Neutron, define how. I've provided a way to do this, find a formal strategy to ground it.
Second: (again) try to find a connection between resources, if you are going to use the strategy of creating the resources in Neutron. The "Logical_Router_Static_Route" belongs to a router univocally. If that router has been created by OpenStack, then you can modify the DB sync method to consider learned routes too.
In order to do this, you'll need a set of resources that are going to be needed in Neutron, the OVN counterparts and other resources (like "Logical_Router_Static_Route") that will be added and will be present in OVN and not in Neutron DB. Also you'll need to know how to relate all of them. -------------- next part -------------- An HTML attachment was scrubbed... URL: < https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718...
------------------------------
Subject: Digest Footer
_______________________________________________ openstack-discuss mailing list openstack-discuss@lists.openstack.org
------------------------------
End of openstack-discuss Digest, Vol 57, Issue 55 *************************************************
Hello Karl, How are you? Thank you for your response.. Hope you are ok. Nguyen Huu Khoi On Mon, Jul 24, 2023 at 10:46 AM Karl Kloppenborg <kkloppenborg@rwts.com.au> wrote:
Apologies I’ve been off sick.
However yes, this is the way we do it as well.
I would say this is also the most sane way to deal with this.
Thanks, Karl.
*From: *Arnaud Morin <arnaud.morin@gmail.com> *Date: *Sunday, 23 July 2023 at 10:56 pm *To: *Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> *Cc: *Karl Kloppenborg <kkloppenborg@rwts.com.au>, OpenStack Discuss < openstack-discuss@lists.openstack.org> *Subject: *Re: [openstack][largescale-sig] Openstack multi region deployment
We have this model also with only one keystone. We have multiple galera clusters synchronized together. Only one cluster is used for write requests (located in one region), others are read only / cache. Most of the calls done to our keystone are "read" or token validation requests, and this works fine with a read galera cluster / cache.
I know that we also have a custom way to invalidate cache across regions, but I dont remember the details, I can ask the team.
Anyway, this is do-able :)
I imagine it also depends on the usage you have, if you create a lot of users/projects/assignments, then it may be harder to achieve.
Cheers, Arnaud
On 19.07.23 - 14:03, Nguyễn Hữu Khôi wrote:
Hello, thank you very much.
But can I ask how we process if 1 region at ASIA and 2 regions in the USA?
Database latency will be our problem.
Nguyen Huu Khoi
On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg < kkloppenborg@rwts.com.au> wrote:
Hi Nguy,
We’ve deployed a large multi-region openstack deployment.
As a rule of thumb we’ve got a “keystone” region which is as best we can highly available and very redundant.
We then have all other regions talk back to this region, we just usually call it “keystone” or “core” and it’s hidden from the UI from users.
We then just run a large well kept Galara cluster to support it.
--Karl.
*From: *openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> *Date: *Tuesday, 18 July 2023 at 9:25 pm *To: *openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject: *openstack-discuss Digest, Vol 57, Issue 55
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez)
----------------------------------------------------------------------
Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: < CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much.
Good morning Nguyễn. Latency won’t be too much of an issue because if you have read replicas in each region, you can route your read requests to the read slaves and only have the writes pushed to the master. There’s a few different options in terms of how you route, we use proxysql.com for our systems. I’d say about 90% of traffic in your keystone environment will be read traffic, 10% write. So I really would not be too concerned about your latency. Also, keystone isn’t a very heavy use database. TXN replication or WAL latency shouldn’t be too much of a problem between the continents. The fact of the matter is, the way you’re architecting this will always need to have compromises. You’re basically hitting the issues described in CAP theorem. (Read more here: https://en.m.wikipedia.org/wiki/CAP_theorem) You’ll notice that most AWS and GCP, Azure etc whenever IAM or Keystone equivalents are written to, I.e a token made, permissions updated, there’s a delay in the response, this is usually artificial and designed to induce enough time into the request so that reader slaves have received the latest WAL replications. / or db version equivalent. The only alternative to this is if you can deploy something like cockroachDB or Yugabyte. However, this will be fraught with heavy and costly complexity. Thanks, Karl. On Tue, 25 Jul 2023 at 09:17, Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello Karl, How are you? Thank you for your response.. Hope you are ok. Nguyen Huu Khoi
On Mon, Jul 24, 2023 at 10:46 AM Karl Kloppenborg < kkloppenborg@rwts.com.au> wrote:
Apologies I’ve been off sick.
However yes, this is the way we do it as well.
I would say this is also the most sane way to deal with this.
Thanks, Karl.
*From: *Arnaud Morin <arnaud.morin@gmail.com> *Date: *Sunday, 23 July 2023 at 10:56 pm *To: *Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> *Cc: *Karl Kloppenborg <kkloppenborg@rwts.com.au>, OpenStack Discuss < openstack-discuss@lists.openstack.org> *Subject: *Re: [openstack][largescale-sig] Openstack multi region deployment
We have this model also with only one keystone. We have multiple galera clusters synchronized together. Only one cluster is used for write requests (located in one region), others are read only / cache. Most of the calls done to our keystone are "read" or token validation requests, and this works fine with a read galera cluster / cache.
I know that we also have a custom way to invalidate cache across regions, but I dont remember the details, I can ask the team.
Anyway, this is do-able :)
I imagine it also depends on the usage you have, if you create a lot of users/projects/assignments, then it may be harder to achieve.
Cheers, Arnaud
On 19.07.23 - 14:03, Nguyễn Hữu Khôi wrote:
Hello, thank you very much.
But can I ask how we process if 1 region at ASIA and 2 regions in the USA?
Database latency will be our problem.
Nguyen Huu Khoi
On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg < kkloppenborg@rwts.com.au> wrote:
Hi Nguy,
We’ve deployed a large multi-region openstack deployment.
As a rule of thumb we’ve got a “keystone” region which is as best we can highly available and very redundant.
We then have all other regions talk back to this region, we just usually call it “keystone” or “core” and it’s hidden from the UI from users.
We then just run a large well kept Galara cluster to support it.
--Karl.
*From: *openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> *Date: *Tuesday, 18 July 2023 at 9:25 pm *To: *openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject: *openstack-discuss Digest, Vol 57, Issue 55
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez)
----------------------------------------------------------------------
Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: < CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3
nodes).
and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much.
Hello Arnaud, Thank you for sharing. But how do you deal with db latency, example one region is in Asia and one is in America. Nguyen Huu Khoi On Sun, Jul 23, 2023 at 7:56 PM Arnaud Morin <arnaud.morin@gmail.com> wrote:
We have this model also with only one keystone. We have multiple galera clusters synchronized together. Only one cluster is used for write requests (located in one region), others are read only / cache. Most of the calls done to our keystone are "read" or token validation requests, and this works fine with a read galera cluster / cache.
I know that we also have a custom way to invalidate cache across regions, but I dont remember the details, I can ask the team.
Anyway, this is do-able :)
I imagine it also depends on the usage you have, if you create a lot of users/projects/assignments, then it may be harder to achieve.
Cheers, Arnaud
On 19.07.23 - 14:03, Nguyễn Hữu Khôi wrote:
Hello, thank you very much.
But can I ask how we process if 1 region at ASIA and 2 regions in the USA?
Database latency will be our problem.
Nguyen Huu Khoi
On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg < kkloppenborg@rwts.com.au> wrote:
Hi Nguy,
We’ve deployed a large multi-region openstack deployment.
As a rule of thumb we’ve got a “keystone” region which is as best we can highly available and very redundant.
We then have all other regions talk back to this region, we just usually call it “keystone” or “core” and it’s hidden from the UI from users.
We then just run a large well kept Galara cluster to support it.
--Karl.
*From: *openstack-discuss-request@lists.openstack.org < openstack-discuss-request@lists.openstack.org> *Date: *Tuesday, 18 July 2023 at 9:25 pm *To: *openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject: *openstack-discuss Digest, Vol 57, Issue 55
Send openstack-discuss mailing list submissions to openstack-discuss@lists.openstack.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
or, via email, send a message with subject or body 'help' to openstack-discuss-request@lists.openstack.org
You can reach the person managing the list at openstack-discuss-owner@lists.openstack.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of openstack-discuss digest..."
Today's Topics:
1. [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 2. Re: [openstack][largescale-sig] Openstack multi region deployment (Felix Huettner) 3. Re: [openstack][largescale-sig] Openstack multi region deployment (Nguy?n H?u Kh?i) 4. Re: [neutron] unmanaged router resources - OVN interconnect (Rodolfo Alonso Hernandez)
----------------------------------------------------------------------
Message: 1 Date: Tue, 18 Jul 2023 12:07:12 +0700 From: Nguy?n H?u Kh?i <nguyenhuukhoinw@gmail.com> To: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: [openstack][largescale-sig] Openstack multi region deployment Message-ID: < CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
Hello guys,
I am going to deploy openstack multi regions and I know that keystone replication is the most challenging.
I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes)
When region 1 is node available, I will map region 2 to use standby edge galera cluster.
I hope you give me some experience and advice with multi regions.
Thank you very much.
participants (3)
-
Arnaud Morin
-
Karl Kloppenborg
-
Nguyễn Hữu Khôi