Open Stack

Good morning Nguyễn.

Latency won’t be too much of an issue because if you have read replicas in
each region, you can route your read requests to the read slaves and only
have the writes pushed to the master.

There’s a few different options in terms of how you route, we use
proxysql.com for our systems.

I’d say about 90% of traffic in your keystone environment will be read
traffic, 10% write.

So I really would not be too concerned about your latency.

Also, keystone isn’t a very heavy use database. TXN replication or WAL
latency shouldn’t be too much of a problem between the continents.

The fact of the matter is, the way you’re architecting this will always
need to have compromises. You’re basically hitting the issues described in
CAP theorem. (Read more here:
https://en.m.wikipedia.org/wiki/CAP_theorem)

You’ll notice that most AWS and GCP, Azure etc whenever IAM or Keystone
equivalents are written to, I.e a token made, permissions updated, there’s
a delay in the response, this is usually artificial and designed to induce
enough time into the request so that reader slaves have received the latest
WAL replications. / or db version equivalent.

The only alternative to this is if you can deploy something like
cockroachDB or Yugabyte.

However, this will be fraught with heavy and costly complexity.


Thanks,
Karl.

On Tue, 25 Jul 2023 at 09:17, Nguyễn Hữu Khôi <nguyenhuukhoinw at gmail.com>
wrote:

> Hello Karl,
> How are you? Thank you for your response.. Hope you are ok.
> Nguyen Huu Khoi
>
>
> On Mon, Jul 24, 2023 at 10:46 AM Karl Kloppenborg <
> kkloppenborg at rwts.com.au> wrote:
>
>> Apologies I’ve been off sick.
>>
>>
>>
>> However yes, this is the way we do it as well.
>>
>> I would say this is also the most sane way to deal with this.
>>
>>
>>
>> Thanks,
>> Karl.
>>
>>
>>
>> *From: *Arnaud Morin <arnaud.morin at gmail.com>
>> *Date: *Sunday, 23 July 2023 at 10:56 pm
>> *To: *Nguyễn Hữu Khôi <nguyenhuukhoinw at gmail.com>
>> *Cc: *Karl Kloppenborg <kkloppenborg at rwts.com.au>, OpenStack Discuss <
>> openstack-discuss at lists.openstack.org>
>> *Subject: *Re: [openstack][largescale-sig] Openstack multi region
>> deployment
>>
>> We have this model also with only one keystone.
>> We have multiple galera clusters synchronized together.
>> Only one cluster is used for write requests (located in one region),
>> others are read only / cache.
>> Most of the calls done to our keystone are "read" or token validation
>> requests, and this works fine with a read galera cluster / cache.
>>
>> I know that we also have a custom way to invalidate cache across
>> regions, but I dont remember the details, I can ask the team.
>>
>> Anyway, this is do-able :)
>>
>> I imagine it also depends on the usage you have, if you create a lot of
>> users/projects/assignments, then it may be harder to achieve.
>>
>> Cheers,
>> Arnaud
>>
>> On 19.07.23 - 14:03, Nguyễn Hữu Khôi wrote:
>> > Hello, thank you very much.
>> >
>> > But can I ask how we process if 1 region at ASIA and  2 regions in the
>> USA?
>> >
>> > Database latency will be our problem.
>> >
>> > Nguyen Huu Khoi
>> >
>> >
>> > On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg <
>> kkloppenborg at rwts.com.au>
>> > wrote:
>> >
>> > > Hi Nguy,
>> > >
>> > >
>> > >
>> > > We’ve deployed a large multi-region openstack deployment.
>> > >
>> > > As a rule of thumb we’ve got a “keystone” region which is as best we
>> can
>> > > highly available and very redundant.
>> > >
>> > >
>> > >
>> > > We then have all other regions talk back to this region, we just
>> usually
>> > > call it “keystone” or “core” and it’s hidden from the UI from users.
>> > >
>> > >
>> > >
>> > > We then just run a large well kept Galara cluster to support it.
>> > >
>> > >
>> > >
>> > > --Karl.
>> > >
>> > >
>> > >
>> > > *From: *openstack-discuss-request at lists.openstack.org <
>> > > openstack-discuss-request at lists.openstack.org>
>> > > *Date: *Tuesday, 18 July 2023 at 9:25 pm
>> > > *To: *openstack-discuss at lists.openstack.org <
>> > > openstack-discuss at lists.openstack.org>
>> > > *Subject: *openstack-discuss Digest, Vol 57, Issue 55
>> > >
>> > > Send openstack-discuss mailing list submissions to
>> > >         openstack-discuss at lists.openstack.org
>> > >
>> > > To subscribe or unsubscribe via the World Wide Web, visit
>> > >
>> > >
>> https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss
>> > >
>> > > or, via email, send a message with subject or body 'help' to
>> > >         openstack-discuss-request at lists.openstack.org
>> > >
>> > > You can reach the person managing the list at
>> > >         openstack-discuss-owner at lists.openstack.org
>> > >
>> > > When replying, please edit your Subject line so it is more specific
>> > > than "Re: Contents of openstack-discuss digest..."
>> > >
>> > >
>> > > Today's Topics:
>> > >
>> > >    1. [openstack][largescale-sig] Openstack multi region deployment
>> > >       (Nguy?n H?u Kh?i)
>> > >    2. Re: [openstack][largescale-sig] Openstack multi region
>> > >       deployment (Felix Huettner)
>> > >    3. Re: [openstack][largescale-sig] Openstack multi region
>> > >       deployment (Nguy?n H?u Kh?i)
>> > >    4. Re: [neutron] unmanaged router resources - OVN interconnect
>> > >       (Rodolfo Alonso Hernandez)
>> > >
>> > >
>> > > ----------------------------------------------------------------------
>> > >
>> > > Message: 1
>> > > Date: Tue, 18 Jul 2023 12:07:12 +0700
>> > > From: Nguy?n H?u Kh?i <nguyenhuukhoinw at gmail.com>
>> > > To: OpenStack Discuss <openstack-discuss at lists.openstack.org>
>> > > Subject: [openstack][largescale-sig] Openstack multi region deployment
>> > > Message-ID:
>> > >         <
>> > > CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw at mail.gmail.com>
>> > > Content-Type: text/plain; charset="utf-8"
>> > >
>> > > Hello guys,
>> > >
>> > > I am going to deploy openstack multi regions and I know that keystone
>> > > replication is the most challenging.
>> > >
>> > > I plan to set up 2 regions which use centralize galera cluster(3
>> nodes).
>> > > and one standby edge galera cluster(3 nodes)
>> > >
>> > > When region 1 is node available, I will map region 2 to use standby
>> edge
>> > > galera cluster.
>> > >
>> > > I hope  you give me some experience and advice with multi regions.
>> > >
>> > > Thank you very much.
>> > > -------------- next part --------------
>> > > An HTML attachment was scrubbed...
>> > > URL: <
>> > >
>> https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm
>> > > >
>> > >
>> > > ------------------------------
>> > >
>> > > Message: 2
>> > > Date: Tue, 18 Jul 2023 09:34:35 +0200
>> > > From: Felix Huettner <felix.huettner at mail.schwarz>
>> > > To: Nguy?n H?u Kh?i <nguyenhuukhoinw at gmail.com>
>> > > Cc: OpenStack Discuss <openstack-discuss at lists.openstack.org>
>> > > Subject: Re: [openstack][largescale-sig] Openstack multi region
>> > >         deployment
>> > > Message-ID: <ZLZAS-99RzzXfyr8 at SIT-SLAP8639.int.lidl.net>
>> > > Content-Type: text/plain; charset=utf-8
>> > >
>> > > Hi,
>> > >
>> > > i think you have two options here:
>> > > 1. you could span a single galera cluster over all of your regions.
>> > >    this might have some latency issues, but if your are not too write
>> > >    heavy that might be fine. I know some companies use that setup.
>> > > 2. you use some kind of multiple galera clusters with replication.
>> > >    But i have not yet heard of anybody using this setup.
>> > >
>> > > An alternative might be to have separate keystone setups with separate
>> > > databases. This would probably reduce the error potential, but might
>> not
>> > > fit your usecase.
>> > >
>> > > Thanks
>> > > Felix
>> > >
>> > >
>> > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote:
>> > > > Hello guys,
>> > > >
>> > > > I am going to deploy openstack multi regions and I know that
>> keystone
>> > > > replication is the most challenging.
>> > > >
>> > > > I plan to set up 2 regions which use centralize galera cluster(3
>> nodes).
>> > > > and one standby edge galera cluster(3 nodes)
>> > > >
>> > > > When region 1 is node available, I will map region 2 to use standby
>> edge
>> > > > galera cluster.
>> > > >
>> > > > I hope  you give me some experience and advice with multi regions.
>> > > >
>> > > > Thank you very much.
>> > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur
>> f?r
>> > > die Verwertung durch den vorgesehenen Empf?nger bestimmt.
>> > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den
>> Absender
>> > > bitte unverz?glich in Kenntnis und l?schen diese E Mail.
>> > >
>> > > Hinweise zum Datenschutz finden Sie hier<
>> https://www.datenschutz.schwarz>.
>> > >
>> > >
>> > > This e-mail may contain confidential content and is intended only for
>> the
>> > > specified recipient/s.
>> > > If you are not the intended recipient, please inform the sender
>> > > immediately and delete this e-mail.
>> > >
>> > > Information on data protection can be found here<
>> > > https://www.datenschutz.schwarz>.
>> > >
>> > >
>> > >
>> > > ------------------------------
>> > >
>> > > Message: 3
>> > > Date: Tue, 18 Jul 2023 15:36:11 +0700
>> > > From: Nguy?n H?u Kh?i <nguyenhuukhoinw at gmail.com>
>> > > To: Nguy?n H?u Kh?i <nguyenhuukhoinw at gmail.com>, OpenStack Discuss
>> > >         <openstack-discuss at lists.openstack.org>
>> > > Subject: Re: [openstack][largescale-sig] Openstack multi region
>> > >         deployment
>> > > Message-ID:
>> > >         <CABAODRcJr=
>> > > CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow at mail.gmail.com>
>> > > Content-Type: text/plain; charset="utf-8"
>> > >
>> > > Hi.
>> > > Thank you for your reply.
>> > >
>> > > The first one has a problem because each region is too soft. If a
>> member is
>> > > down, so this region is gone.
>> > >
>> > > It is so challenge with us.
>> > >
>> > >
>> > > Nguyen Huu Khoi
>> > >
>> > >
>> > > On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner
>> <felix.huettner at mail.schwarz
>> > > >
>> > > wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > i think you have two options here:
>> > > > 1. you could span a single galera cluster over all of your regions.
>> > > >    this might have some latency issues, but if your are not too
>> write
>> > > >    heavy that might be fine. I know some companies use that setup.
>> > > > 2. you use some kind of multiple galera clusters with replication.
>> > > >    But i have not yet heard of anybody using this setup.
>> > > >
>> > > > An alternative might be to have separate keystone setups with
>> separate
>> > > > databases. This would probably reduce the error potential, but
>> might not
>> > > > fit your usecase.
>> > > >
>> > > > Thanks
>> > > > Felix
>> > > >
>> > > >
>> > > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote:
>> > > > > Hello guys,
>> > > > >
>> > > > > I am going to deploy openstack multi regions and I know that
>> keystone
>> > > > > replication is the most challenging.
>> > > > >
>> > > > > I plan to set up 2 regions which use centralize galera cluster(3
>> > > nodes).
>> > > > > and one standby edge galera cluster(3 nodes)
>> > > > >
>> > > > > When region 1 is node available, I will map region 2 to use
>> standby
>> > > edge
>> > > > > galera cluster.
>> > > > >
>> > > > > I hope  you give me some experience and advice with multi regions.
>> > > > >
>> > > > > Thank you very much.
>> > > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist
>> nur f?r
>> > > > die Verwertung durch den vorgesehenen Empf?nger bestimmt.
>> > > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den
>> Absender
>> > > > bitte unverz?glich in Kenntnis und l?schen diese E Mail.
>> > > >
>> > > > Hinweise zum Datenschutz finden Sie hier<
>> https://www.datenschutz.schwarz
>> > > >.
>> > > >
>> > > >
>> > > > This e-mail may contain confidential content and is intended only
>> for the
>> > > > specified recipient/s.
>> > > > If you are not the intended recipient, please inform the sender
>> > > > immediately and delete this e-mail.
>> > > >
>> > > > Information on data protection can be found here<
>> > > > https://www.datenschutz.schwarz>.
>> > > >
>> > > -------------- next part --------------
>> > > An HTML attachment was scrubbed...
>> > > URL: <
>> > >
>> https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm
>> > > >
>> > >
>> > > ------------------------------
>> > >
>> > > Message: 4
>> > > Date: Tue, 18 Jul 2023 13:23:27 +0200
>> > > From: Rodolfo Alonso Hernandez <ralonsoh at redhat.com>
>> > > To: Roberto Bartzen Acosta <roberto.acosta at luizalabs.com>
>> > > Cc: openstack-discuss <openstack-discuss at lists.openstack.org>,  Terry
>> > >         Wilson <twilson at redhat.com>, Tiago Pires <
>> > > tiago.pires at luizalabs.com>
>> > > Subject: Re: [neutron] unmanaged router resources - OVN interconnect
>> > > Message-ID:
>> > >         <
>> > > CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug at mail.gmail.com>
>> > > Content-Type: text/plain; charset="utf-8"
>> > >
>> > > Ok, this is being tortuous. First of all: define a strategy. If you
>> are
>> > > going to create the resources in Neutron, define how. I've provided a
>> way
>> > > to do this, find a formal strategy to ground it.
>> > >
>> > > Second: (again) try to find a connection between resources, if you are
>> > > going to use the strategy of creating the resources in Neutron. The
>> > > "Logical_Router_Static_Route" belongs to a router univocally. If that
>> > > router has been created by OpenStack, then you can modify the DB sync
>> > > method to consider learned routes too.
>> > >
>> > > In order to do this, you'll need a set of resources that are going to
>> be
>> > > needed in Neutron, the OVN counterparts and other resources (like
>> > > "Logical_Router_Static_Route") that will be added and will be present
>> in
>> > > OVN and not in Neutron DB. Also you'll need to know how to relate all
>> of
>> > > them.
>> > > -------------- next part --------------
>> > > An HTML attachment was scrubbed...
>> > > URL: <
>> > >
>> https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm
>> > > >
>> > >
>> > > ------------------------------
>> > >
>> > > Subject: Digest Footer
>> > >
>> > > _______________________________________________
>> > > openstack-discuss mailing list
>> > > openstack-discuss at lists.openstack.org
>> > >
>> > >
>> > > ------------------------------
>> > >
>> > > End of openstack-discuss Digest, Vol 57, Issue 55
>> > > *************************************************
>> > >
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230725/160f695d/attachment-0001.htm>