<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
mso-fareast-language:EN-US;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal">While doing scale tests on our infrastructure, we observed some increase in the response times of our keystone servers.
<o:p></o:p></p>
<p class="MsoNormal">After further investigation we observed that we have a hot key in our cache configuration (this means than all keystone servers are checking this key quite frequently)<o:p></o:p></p>
<p class="MsoNormal">We are using a pool of memcache servers for hosting the cache and the solution does not seem ideal at this scale.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The key turns out to be the revocation tree, that is evaluated in every token validation. If the revocation tree object stored is big enough it can kill the network connectivity<o:p></o:p></p>
<p class="MsoNormal">on the cache server affecting the whole infrastructure as the identity servers needs to check the key before validating a token.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">On our scale tests after the cleanup, we have 250 requests/second for an object of 500KB that is a throughput of 1Gbit/sec that saturate the network link of the cache server.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We are checking other strategies like redis or mongo, but we would like to know if you have already seen this before? If so what you have done?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Kind regards,<o:p></o:p></p>
<p class="MsoNormal">Jose<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b><span style="font-size:12.0pt;mso-fareast-language:EN-GB">Jose Castro Leon<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;mso-fareast-language:EN-GB">CERN IT-CM-RPS tel: +41.22.76.74272<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;mso-fareast-language:EN-GB"> mob: +41.75.41.19222<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;mso-fareast-language:EN-GB"> fax: +41.22.76.67955<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;mso-fareast-language:EN-GB">Office: 31-1-026 CH-1211 Geneve 23<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;mso-fareast-language:EN-GB">email:
<span style="color:#1F497D"><a href="mailto:jose.castro.leon@cern.ch"><span style="color:#1F497D">jose.castro.leon@cern.ch</span></a><o:p></o:p></span></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</div>
</body>
</html>