<div dir="ltr">BP is at <a href="https://blueprints.launchpad.net/keystone/+spec/keystone-ha-multisite">https://blueprints.launchpad.net/keystone/+spec/keystone-ha-multisite</a> , spec will come later :)</div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 19, 2015 at 11:21 AM, Adam Young <span dir="ltr"><<a href="mailto:ayoung@redhat.com" target="_blank">ayoung@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span class="">
<div>On 03/18/2015 08:59 PM, joehuang wrote:<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">[Joe]: For reliability purpose, I suggest that
the keystone client should provide a fail-safe design:
primary KeyStone server, the second KeyStone server (or even
the third KeySont server) . If the primary KeyStone server
is out of service, then the KeyStone client will try the
second KeyStone server. Different KeyStone client may be
configured with different primary KeyStone server and the
second KeyStone server.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
[Adam]: Makes sense, but that can be handled outside of
Keystone using HA and Heartbear and awhole slew of
technologies. Each Keystone server can validate each
other's tokens.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">For cross-site KeyStone HA, the backend of HA
can leverage MySQL Galera cluster for multisite database
synchronous replication to provide high availability, but
for the KeyStone front-end the API server, it’s web service
and accessed through the endpoint address ( name, or domain
name, or ip address ) , like <a>http://</a>.... or ip address.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">AFAIK, the HA for web service will usually be
done through DNS based geo-load balancer in multi-site
scenario. The shortcoming for this HA is that the fault
recovery ( forward request to the health web service) will
take longer time, it's up to the configuration in the DNS
system. The other way is to put a load balancer like LVS
ahead of KeyStone web services in multi-site. Then either
the LVS is put in one site(so that KeyStone client only
configured with one IP address based endpoint item, but LVS
cross-site HA is lack), or in multisite site, and register
the multi-LVS’s IP to the DNS or Name server(so that
KeyStone client only configured with one Domain name or name
based endpoint item, same issue just mentioned).<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Therefore, I still think that keystone client
with a fail-safe design( primary KeyStone server, the second
KeyStone server ) will be a “very high gain but low invest”
multisite high availability solution. Just like MySQL
itself, we know there is some outbound high availability
solution (for example, PaceMaker+ColoSync+DRDB), but also
there is Galera like inbound cluster ware.</span></p>
</div>
</blockquote>
<br></span>
Write it up as a full spec, and we will discuss at the summit.<br>
<br>
<blockquote type="cite"><span class="">
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Best Regards<u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">Chaoyi Huang ( Joe Huang )<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext" lang="EN-US"> Adam Young [<a href="mailto:ayoung@redhat.com" target="_blank">mailto:ayoung@redhat.com</a>]
<br>
<b>Sent:</b> Tuesday, March 17, 2015 10:00 PM<br>
<b>To:</b> <a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a><br>
<b>Subject:</b> Re: [openstack-dev] [opnfv-tech-discuss]
[Keystone][Multisite] Huge token size<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On 03/17/2015 02:51
AM, joehuang wrote:<u></u><u></u></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">It’s not reality to deploy KeyStone service (
including backend store ) in each site if the number, for
example, is more than 10. The reason is that the stored
data including data related to revocation need to be
replicated to all sites in synchronization manner.
Otherwise, the API server might attempt to use the token
before it's able to be validated in the target site. </span><span lang="EN-US"><u></u><u></u></span></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
Replicating revocati9on data across 10 sites will be tricky,
but far better than replicating all of the token data.
Revocations should be relatively rare.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">When Fernet token is used in multisite
scenario, each API request will ask for token validation
from KeyStone. The cloud will be out of service if KeyStone
stop working, therefore KeyStone service need to run in
several sites.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
There will be multiple Keystone servers, so each should talk
to their local instance.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-US">For reliability purpose, I suggest that the
keystone client should provide a fail-safe design: primary
KeyStone server, the second KeyStone server (or even the
third KeySont server) . If the primary KeyStone server is
out of service, then the KeyStone client will try the second
KeyStone server. Different KeyStone client may be configured
with different primary KeyStone server and the second
KeyStone server.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
Makes sense, but that can be handled outside of Keystone
using HA and Heartbear and awhole slew of technologies.
Each Keystone server can validate each other's tokens.<u></u><u></u></span></p>
</div>
<br>
<fieldset></fieldset>
<br>
</span><span class=""><pre>__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</span></blockquote>
<br>
</div>
<br>__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">Zhipeng (Howard) Huang</div><div dir="ltr"><br></div><div dir="ltr">Standard Engineer</div><div>IT Standard & Patent/IT Prooduct Line</div><div dir="ltr">Huawei Technologies Co,. Ltd</div><div dir="ltr">Email: <a href="mailto:huangzhipeng@huawei.com" target="_blank">huangzhipeng@huawei.com</a></div><div dir="ltr">Office: Huawei Industrial Base, Longgang, Shenzhen</div><div dir="ltr"><br></div><div dir="ltr">(Previous)<br><div>Research Assistant</div><div>Mobile Ad-Hoc Network Lab, Calit2</div><div>University of California, Irvine</div><div>Email: <a href="mailto:zhipengh@uci.edu" target="_blank">zhipengh@uci.edu</a></div><div>Office: Calit2 Building Room 2402</div><div><br></div><div>OpenStack, OPNFV, OpenDaylight, OpenCompute Aficionado</div></div></div></div></div></div></div>
</div>