<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Glad to know you guys are talking about the key distribution and rotation for Fernet token. Hans and I did a prototype for multisite identity service
management, and have a similar issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">The use case is : a user should, using a single authentication point be able to manage virtual resources spread over multiple OpenStack regions
(<a href="https://etherpad.opnfv.org/p/multisite_identity_management">https://etherpad.opnfv.org/p/multisite_identity_management</a>)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">We did the prototype of Fernet token used in multi-KeyStone cluster for multi-OpenStack instances installed in multi-sites, “write” is only allowed
in the master KeyStone cluster, the slave KeyStone cluster is read only ( <a href="https://github.com/hafe/dockers">https://github.com/hafe/dockers</a>, remember that the slave Galera cluster should be configured with replicate_do_db=KeyStone, but not binlog_do_db=KeyStone,
Hans may haven’t update the script yet. The prototype is for candidate solution 2 )<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">From the prototype, we found that Fernet token validation could be successfully done by local KeyStone server with the async-replicated db. This
means if we have a lot of sites with OpenStack installed, we can deploy a fully distributed KeyStone service in each site, provide token validation in local site only to realize high performance and high availability.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">After the prototype, I think the candidate solution 3 would be better one solution for multisite identity service management.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">“Candidate solution 3”. KeyStone service(Distributed) with Fernet token + Async replication ( star-mode).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">one master KeyStone cluster with Fernet token in two sites (for site level high availability purpose), other sites will be installed with at least
2 slave nodes where the node is configured with DB async replication from the master cluster members, and one slave’s mater node in site1, another slave’s master node in site 2.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Only the master cluster nodes are allowed to write, other slave nodes waiting for replication from the master cluster ( very little delay) member.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Pros.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">1) Why cluster in the master sites? There are lots of master nodes in the cluster, in order to provide more slaves could be done async. replication
in parallel.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">2) Why two sites for the master cluster? to provide higher reliability (site level) for writing request.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">3) Why using multi-slaves in other sites. Slave has no knowledge of other slaves, so easy to manage multi-slaves in one site than a cluster, and
multi-slaves work independently but provide multi-instance redundancy(like a cluster, but independent).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Cons. The distribution/rotation of key management.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Appreciate the new introduced Fernet token very much in addressing the scenario of multi-site cloud identity management, but it brings a new challenge
that how to address the key distribution and rotation in multi-site cloud. Should the key distribution/rotation management be the responsibility of a new service or KeyStone itself? It’s tough to depends on script to manage multi-sites (lots of sites, not
only 3 or 5).<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Best Regards<o:p></o:p></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D">Chaoyi Huang ( Joe Huang )</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Dolph Mathews [mailto:dolph.mathews@gmail.com]
<br>
<b>Sent:</b> Tuesday, July 28, 2015 3:31 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] [Keystone][Fernet] HA SQL backend for Fernet keys<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, Jul 27, 2015 at 2:03 PM, Clint Byrum <<a href="mailto:clint@fewbar.com" target="_blank">clint@fewbar.com</a>> wrote:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Excerpts from Dolph Mathews's message of 2015-07-27 11:48:12 -0700:<o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">> On Mon, Jul 27, 2015 at 1:31 PM, Clint Byrum <<a href="mailto:clint@fewbar.com">clint@fewbar.com</a>> wrote:<br>
><br>
> > Excerpts from Alexander Makarov's message of 2015-07-27 10:01:34 -0700:<br>
> > > Greetings!<br>
> > ><br>
> > > I'd like to discuss pro's and contra's of having Fernet encryption keys<br>
> > > stored in a database backend.<br>
> > > The idea itself emerged during discussion about synchronizing rotated<br>
> > keys<br>
> > > in HA environment.<br>
> > > Now Fernet keys are stored in the filesystem that has some availability<br>
> > > issues in unstable cluster.<br>
> > > OTOH, making SQL highly available is considered easier than that for a<br>
> > > filesystem.<br>
> > ><br>
> ><br>
> > I don't think HA is the root of the problem here. The problem is<br>
> > synchronization. If I have 3 keystone servers (n+1), and I rotate keys on<br>
> > them, I must very carefully restart them all at the exact right time to<br>
> > make sure one of them doesn't issue a token which will not be validated<br>
> > on another. This is quite a real possibility because the validation<br>
> > will not come from the user, but from the service, so it's not like we<br>
> > can use simple persistence rules. One would need a layer 7 capable load<br>
> > balancer that can find the token ID and make sure it goes back to the<br>
> > server that issued it.<br>
> ><br>
><br>
> This is not true (or if it is, I'd love see a bug report). keystone-manage<br>
> fernet_rotate uses a three phase rotation strategy (staged -> primary -><br>
> secondary) that allows you to distribute a staged key (used only for token<br>
> validation) throughout your cluster before it becomes a primary key (used<br>
> for token creation and validation) anywhere. Secondary keys are only used<br>
> for token validation.<br>
><br>
> All you have to do is atomically replace the fernet key directory with a<br>
> new key set.<br>
><br>
> You also don't have to restart keystone for it to pickup new keys dropped<br>
> onto the filesystem beneath it.<br>
><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US">That's great news! Is this documented anywhere? I dug through the<br>
operators guides, security guide, install guide, etc. Nothing described<br>
this dance, which is impressive and should be written down!<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">(BTW, your original assumption would normally have been an accurate one!)<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I don't believe it's documented in any of those places, yet. The best explanation of the three phases in tree I'm aware of is probably this (which isn't particularly accessible..):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <a href="https://github.com/openstack/keystone/blob/6a6fcc2/keystone/cmd/cli.py#L208-L223">https://github.com/openstack/keystone/blob/6a6fcc2/keystone/cmd/cli.py#L208-L223</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Lance Bragstad and I also gave a small presentation at the Vancouver summit on the behavior and he mentions the same on one of his blog posts:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <a href="https://www.youtube.com/watch?v=duRBlm9RtCw&feature=youtu.be">https://www.youtube.com/watch?v=duRBlm9RtCw&feature=youtu.be</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <a href="http://lbragstad.com/?p=133">http://lbragstad.com/?p=133</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span lang="EN-US"><br>
I even tried to discern how it worked from the code but it actually<br>
looks like it does not work the way you describe on casual investigation.<o:p></o:p></span></p>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I don't blame you! I'll work to improve the user-facing docs on the topic.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span lang="EN-US"><br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></span></p>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</div>
</div>
</body>
</html>