<div dir="ltr">Hello Karl,<div>How are you? Thank you for your response.. Hope you are ok.<br><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Nguyen Huu Khoi<br></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jul 24, 2023 at 10:46 AM Karl Kloppenborg <<a href="mailto:kkloppenborg@rwts.com.au">kkloppenborg@rwts.com.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg180595987271259253">
<div lang="EN-AU" style="overflow-wrap: break-word;">
<div class="m_180595987271259253WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:-webkit-standard;color:black">Apologies I’ve been off sick.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:-webkit-standard;color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:-webkit-standard;color:black">However yes, this is the way we do it as well.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:-webkit-standard;color:black">I would say this is also the most sane way to deal with this.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:-webkit-standard;color:black"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:-webkit-standard;color:black">Thanks,<br>
Karl.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt"><u></u> <u></u></span></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(181,196,223);padding:3pt 0cm 0cm">
<p class="MsoNormal" style="margin-bottom:12pt"><b><span style="font-size:12pt;color:black">From:
</span></b><span style="font-size:12pt;color:black">Arnaud Morin <<a href="mailto:arnaud.morin@gmail.com" target="_blank">arnaud.morin@gmail.com</a>><br>
<b>Date: </b>Sunday, 23 July 2023 at 10:56 pm<br>
<b>To: </b>Nguyễn Hữu Khôi <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>><br>
<b>Cc: </b>Karl Kloppenborg <<a href="mailto:kkloppenborg@rwts.com.au" target="_blank">kkloppenborg@rwts.com.au</a>>, OpenStack Discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [openstack][largescale-sig] Openstack multi region deployment<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt">We have this model also with only one keystone.<br>
We have multiple galera clusters synchronized together.<br>
Only one cluster is used for write requests (located in one region),<br>
others are read only / cache.<br>
Most of the calls done to our keystone are "read" or token validation<br>
requests, and this works fine with a read galera cluster / cache.<br>
<br>
I know that we also have a custom way to invalidate cache across<br>
regions, but I dont remember the details, I can ask the team.<br>
<br>
Anyway, this is do-able :)<br>
<br>
I imagine it also depends on the usage you have, if you create a lot of<br>
users/projects/assignments, then it may be harder to achieve.<br>
<br>
Cheers,<br>
Arnaud<br>
<br>
On 19.07.23 - 14:03, Nguyễn Hữu Khôi wrote:<br>
> Hello, thank you very much.<br>
> <br>
> But can I ask how we process if 1 region at ASIA and 2 regions in the USA?<br>
> <br>
> Database latency will be our problem.<br>
> <br>
> Nguyen Huu Khoi<br>
> <br>
> <br>
> On Tue, Jul 18, 2023 at 8:21 PM Karl Kloppenborg <<a href="mailto:kkloppenborg@rwts.com.au" target="_blank">kkloppenborg@rwts.com.au</a>><br>
> wrote:<br>
> <br>
> > Hi Nguy,<br>
> ><br>
> ><br>
> ><br>
> > We’ve deployed a large multi-region openstack deployment.<br>
> ><br>
> > As a rule of thumb we’ve got a “keystone” region which is as best we can<br>
> > highly available and very redundant.<br>
> ><br>
> ><br>
> ><br>
> > We then have all other regions talk back to this region, we just usually<br>
> > call it “keystone” or “core” and it’s hidden from the UI from users.<br>
> ><br>
> ><br>
> ><br>
> > We then just run a large well kept Galara cluster to support it.<br>
> ><br>
> ><br>
> ><br>
> > --Karl.<br>
> ><br>
> ><br>
> ><br>
> > *From: *<a href="mailto:openstack-discuss-request@lists.openstack.org" target="_blank">openstack-discuss-request@lists.openstack.org</a> <<br>
> > <a href="mailto:openstack-discuss-request@lists.openstack.org" target="_blank">openstack-discuss-request@lists.openstack.org</a>><br>
> > *Date: *Tuesday, 18 July 2023 at 9:25 pm<br>
> > *To: *<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a> <<br>
> > <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
> > *Subject: *openstack-discuss Digest, Vol 57, Issue 55<br>
> ><br>
> > Send openstack-discuss mailing list submissions to<br>
> > <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a><br>
> ><br>
> > To subscribe or unsubscribe via the World Wide Web, visit<br>
> ><br>
> > <a href="https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss" target="_blank">
https://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss</a><br>
> ><br>
> > or, via email, send a message with subject or body 'help' to<br>
> > <a href="mailto:openstack-discuss-request@lists.openstack.org" target="_blank">openstack-discuss-request@lists.openstack.org</a><br>
> ><br>
> > You can reach the person managing the list at<br>
> > <a href="mailto:openstack-discuss-owner@lists.openstack.org" target="_blank">openstack-discuss-owner@lists.openstack.org</a><br>
> ><br>
> > When replying, please edit your Subject line so it is more specific<br>
> > than "Re: Contents of openstack-discuss digest..."<br>
> ><br>
> ><br>
> > Today's Topics:<br>
> ><br>
> > 1. [openstack][largescale-sig] Openstack multi region deployment<br>
> > (Nguy?n H?u Kh?i)<br>
> > 2. Re: [openstack][largescale-sig] Openstack multi region<br>
> > deployment (Felix Huettner)<br>
> > 3. Re: [openstack][largescale-sig] Openstack multi region<br>
> > deployment (Nguy?n H?u Kh?i)<br>
> > 4. Re: [neutron] unmanaged router resources - OVN interconnect<br>
> > (Rodolfo Alonso Hernandez)<br>
> ><br>
> ><br>
> > ----------------------------------------------------------------------<br>
> ><br>
> > Message: 1<br>
> > Date: Tue, 18 Jul 2023 12:07:12 +0700<br>
> > From: Nguy?n H?u Kh?i <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>><br>
> > To: OpenStack Discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
> > Subject: [openstack][largescale-sig] Openstack multi region deployment<br>
> > Message-ID:<br>
> > <<br>
> > <a href="mailto:CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com" target="_blank">CABAODReJ6QW8A4OENEjmhFCiM-15B0qc2La_aMr1EKfaENq9iw@mail.gmail.com</a>><br>
> > Content-Type: text/plain; charset="utf-8"<br>
> ><br>
> > Hello guys,<br>
> ><br>
> > I am going to deploy openstack multi regions and I know that keystone<br>
> > replication is the most challenging.<br>
> ><br>
> > I plan to set up 2 regions which use centralize galera cluster(3 nodes).<br>
> > and one standby edge galera cluster(3 nodes)<br>
> ><br>
> > When region 1 is node available, I will map region 2 to use standby edge<br>
> > galera cluster.<br>
> ><br>
> > I hope you give me some experience and advice with multi regions.<br>
> ><br>
> > Thank you very much.<br>
> > -------------- next part --------------<br>
> > An HTML attachment was scrubbed...<br>
> > URL: <<br>
> > <a href="https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm" target="_blank">
https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/c95d3675/attachment-0001.htm</a><br>
> > ><br>
> ><br>
> > ------------------------------<br>
> ><br>
> > Message: 2<br>
> > Date: Tue, 18 Jul 2023 09:34:35 +0200<br>
> > From: Felix Huettner <felix.huettner@mail.schwarz><br>
> > To: Nguy?n H?u Kh?i <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>><br>
> > Cc: OpenStack Discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
> > Subject: Re: [openstack][largescale-sig] Openstack multi region<br>
> > deployment<br>
> > Message-ID: <<a href="mailto:ZLZAS-99RzzXfyr8@SIT-SLAP8639.int.lidl.net" target="_blank">ZLZAS-99RzzXfyr8@SIT-SLAP8639.int.lidl.net</a>><br>
> > Content-Type: text/plain; charset=utf-8<br>
> ><br>
> > Hi,<br>
> ><br>
> > i think you have two options here:<br>
> > 1. you could span a single galera cluster over all of your regions.<br>
> > this might have some latency issues, but if your are not too write<br>
> > heavy that might be fine. I know some companies use that setup.<br>
> > 2. you use some kind of multiple galera clusters with replication.<br>
> > But i have not yet heard of anybody using this setup.<br>
> ><br>
> > An alternative might be to have separate keystone setups with separate<br>
> > databases. This would probably reduce the error potential, but might not<br>
> > fit your usecase.<br>
> ><br>
> > Thanks<br>
> > Felix<br>
> ><br>
> ><br>
> > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote:<br>
> > > Hello guys,<br>
> > ><br>
> > > I am going to deploy openstack multi regions and I know that keystone<br>
> > > replication is the most challenging.<br>
> > ><br>
> > > I plan to set up 2 regions which use centralize galera cluster(3 nodes).<br>
> > > and one standby edge galera cluster(3 nodes)<br>
> > ><br>
> > > When region 1 is node available, I will map region 2 to use standby edge<br>
> > > galera cluster.<br>
> > ><br>
> > > I hope you give me some experience and advice with multi regions.<br>
> > ><br>
> > > Thank you very much.<br>
> > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r<br>
> > die Verwertung durch den vorgesehenen Empf?nger bestimmt.<br>
> > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender<br>
> > bitte unverz?glich in Kenntnis und l?schen diese E Mail.<br>
> ><br>
> > Hinweise zum Datenschutz finden Sie hier<<a href="https://www.datenschutz.schwarz" target="_blank">https://www.datenschutz.schwarz</a>>.<br>
> ><br>
> ><br>
> > This e-mail may contain confidential content and is intended only for the<br>
> > specified recipient/s.<br>
> > If you are not the intended recipient, please inform the sender<br>
> > immediately and delete this e-mail.<br>
> ><br>
> > Information on data protection can be found here<<br>
> > <a href="https://www.datenschutz.schwarz%3e" target="_blank">https://www.datenschutz.schwarz></a>.<br>
> ><br>
> ><br>
> ><br>
> > ------------------------------<br>
> ><br>
> > Message: 3<br>
> > Date: Tue, 18 Jul 2023 15:36:11 +0700<br>
> > From: Nguy?n H?u Kh?i <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>><br>
> > To: Nguy?n H?u Kh?i <<a href="mailto:nguyenhuukhoinw@gmail.com" target="_blank">nguyenhuukhoinw@gmail.com</a>>, OpenStack Discuss<br>
> > <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
> > Subject: Re: [openstack][largescale-sig] Openstack multi region<br>
> > deployment<br>
> > Message-ID:<br>
> > <CABAODRcJr=<br>
> > <a href="mailto:CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow@mail.gmail.com" target="_blank">CGBW1_bRkLQJAxLZxAx8V4qvbdBmTUQBUm2SRsow@mail.gmail.com</a>><br>
> > Content-Type: text/plain; charset="utf-8"<br>
> ><br>
> > Hi.<br>
> > Thank you for your reply.<br>
> ><br>
> > The first one has a problem because each region is too soft. If a member is<br>
> > down, so this region is gone.<br>
> ><br>
> > It is so challenge with us.<br>
> ><br>
> ><br>
> > Nguyen Huu Khoi<br>
> ><br>
> ><br>
> > On Tue, Jul 18, 2023 at 2:34?PM Felix Huettner <felix.huettner@mail.schwarz<br>
> > ><br>
> > wrote:<br>
> ><br>
> > > Hi,<br>
> > ><br>
> > > i think you have two options here:<br>
> > > 1. you could span a single galera cluster over all of your regions.<br>
> > > this might have some latency issues, but if your are not too write<br>
> > > heavy that might be fine. I know some companies use that setup.<br>
> > > 2. you use some kind of multiple galera clusters with replication.<br>
> > > But i have not yet heard of anybody using this setup.<br>
> > ><br>
> > > An alternative might be to have separate keystone setups with separate<br>
> > > databases. This would probably reduce the error potential, but might not<br>
> > > fit your usecase.<br>
> > ><br>
> > > Thanks<br>
> > > Felix<br>
> > ><br>
> > ><br>
> > > On Tue, Jul 18, 2023 at 12:07:12PM +0700, Nguy?n H?u Kh?i wrote:<br>
> > > > Hello guys,<br>
> > > ><br>
> > > > I am going to deploy openstack multi regions and I know that keystone<br>
> > > > replication is the most challenging.<br>
> > > ><br>
> > > > I plan to set up 2 regions which use centralize galera cluster(3<br>
> > nodes).<br>
> > > > and one standby edge galera cluster(3 nodes)<br>
> > > ><br>
> > > > When region 1 is node available, I will map region 2 to use standby<br>
> > edge<br>
> > > > galera cluster.<br>
> > > ><br>
> > > > I hope you give me some experience and advice with multi regions.<br>
> > > ><br>
> > > > Thank you very much.<br>
> > > Diese E Mail enth?lt m?glicherweise vertrauliche Inhalte und ist nur f?r<br>
> > > die Verwertung durch den vorgesehenen Empf?nger bestimmt.<br>
> > > Sollten Sie nicht der vorgesehene Empf?nger sein, setzen Sie den Absender<br>
> > > bitte unverz?glich in Kenntnis und l?schen diese E Mail.<br>
> > ><br>
> > > Hinweise zum Datenschutz finden Sie hier<<a href="https://www.datenschutz.schwarz" target="_blank">https://www.datenschutz.schwarz</a><br>
> > >.<br>
> > ><br>
> > ><br>
> > > This e-mail may contain confidential content and is intended only for the<br>
> > > specified recipient/s.<br>
> > > If you are not the intended recipient, please inform the sender<br>
> > > immediately and delete this e-mail.<br>
> > ><br>
> > > Information on data protection can be found here<<br>
> > > <a href="https://www.datenschutz.schwarz%3e" target="_blank">https://www.datenschutz.schwarz></a>.<br>
> > ><br>
> > -------------- next part --------------<br>
> > An HTML attachment was scrubbed...<br>
> > URL: <<br>
> > <a href="https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm" target="_blank">
https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/749440e3/attachment-0001.htm</a><br>
> > ><br>
> ><br>
> > ------------------------------<br>
> ><br>
> > Message: 4<br>
> > Date: Tue, 18 Jul 2023 13:23:27 +0200<br>
> > From: Rodolfo Alonso Hernandez <<a href="mailto:ralonsoh@redhat.com" target="_blank">ralonsoh@redhat.com</a>><br>
> > To: Roberto Bartzen Acosta <<a href="mailto:roberto.acosta@luizalabs.com" target="_blank">roberto.acosta@luizalabs.com</a>><br>
> > Cc: openstack-discuss <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>>, Terry<br>
> > Wilson <<a href="mailto:twilson@redhat.com" target="_blank">twilson@redhat.com</a>>, Tiago Pires <<br>
> > <a href="mailto:tiago.pires@luizalabs.com" target="_blank">tiago.pires@luizalabs.com</a>><br>
> > Subject: Re: [neutron] unmanaged router resources - OVN interconnect<br>
> > Message-ID:<br>
> > <<br>
> > <a href="mailto:CAECr9X7U7YsGBv9ajcmeOCxfdD%2BYLar8QyPwYBN0qaP10CzTug@mail.gmail.com" target="_blank">CAECr9X7U7YsGBv9ajcmeOCxfdD+YLar8QyPwYBN0qaP10CzTug@mail.gmail.com</a>><br>
> > Content-Type: text/plain; charset="utf-8"<br>
> ><br>
> > Ok, this is being tortuous. First of all: define a strategy. If you are<br>
> > going to create the resources in Neutron, define how. I've provided a way<br>
> > to do this, find a formal strategy to ground it.<br>
> ><br>
> > Second: (again) try to find a connection between resources, if you are<br>
> > going to use the strategy of creating the resources in Neutron. The<br>
> > "Logical_Router_Static_Route" belongs to a router univocally. If that<br>
> > router has been created by OpenStack, then you can modify the DB sync<br>
> > method to consider learned routes too.<br>
> ><br>
> > In order to do this, you'll need a set of resources that are going to be<br>
> > needed in Neutron, the OVN counterparts and other resources (like<br>
> > "Logical_Router_Static_Route") that will be added and will be present in<br>
> > OVN and not in Neutron DB. Also you'll need to know how to relate all of<br>
> > them.<br>
> > -------------- next part --------------<br>
> > An HTML attachment was scrubbed...<br>
> > URL: <<br>
> > <a href="https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm" target="_blank">
https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230718/90712e47/attachment.htm</a><br>
> > ><br>
> ><br>
> > ------------------------------<br>
> ><br>
> > Subject: Digest Footer<br>
> ><br>
> > _______________________________________________<br>
> > openstack-discuss mailing list<br>
> > <a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a><br>
> ><br>
> ><br>
> > ------------------------------<br>
> ><br>
> > End of openstack-discuss Digest, Vol 57, Issue 55<br>
> > *************************************************<br>
> ><u></u><u></u></span></p>
</div>
</div>
</div>
</div></blockquote></div>