<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">We’ve got several OpenStack clouds at CERN (details in
<a href="http://openstack-in-production.blogspot.fr/2013/09/a-tale-of-3-openstack-clouds-50000.html">
http://openstack-in-production.blogspot.fr/2013/09/a-tale-of-3-openstack-clouds-50000.html</a>).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">The CMS farm was the further ahead and encountered problems with the number of database connections at around 1300 hypervisors. Nova
conductor helps some of these.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Given that we’re heading towards 15K hypervisors for the central instance at CERN, I am not sure a single cell would handle it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">I’d be happy to hear experiences from others in this area.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Belmiro will be giving a summit talk on the deep dive including our experiences for those who are able to make it.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US">Tim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Joshua Harlow [mailto:harlowja@yahoo-inc.com]
<br>
<b>Sent:</b> 03 October 2013 20:32<br>
<b>To:</b> Subbu Allamaraju; Tim Bell<br>
<b>Cc:</b> openstack@lists.openstack.org<br>
<b>Subject:</b> Re: [Openstack] Cells use cases<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Hi Tim,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">I'd also like to know what happens above 1000 hypervisors that u think needs cells?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">From experience at y! we actually start to see the nova-scheduler (and the filter scheduler mainly) be the problem (at around ~400 hypervisors) and that seems
addressable without cells (yes it requires some smart/fast coding that the current scheduler is not designed for, but that seems manageable and achievable) via reviews like
<a href="https://review.openstack.org/#/c/46588">https://review.openstack.org/#/c/46588</a>,
<a href="https://review.openstack.org/#/c/45867">https://review.openstack.org/#/c/45867</a> (and others that are popping up). The filter scheduler appears to scale linearly with the number of hypervisors, and this is problematic since the filter-scheduler is
also single-CPU bound (due to eventlet) so that overall, makes for some nice suckage. We haven't seen the RPC layer be a problem at our current scale, but maybe u guys have hit this. The other issue that starts to happen around ~400 is the nova service group
code, which is not exactly performant when using the DB backend (we haven't tried the ZK backend yet, WIP!) due to frequent and repeated DB calls. <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">It'd be interesting to hear the kind of limitations u guys hit that cells resolved, instead of just fixing the underlying code itself to scale better.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">-Josh<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">From:
</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Subbu Allamaraju <<a href="mailto:subbu@subbu.org">subbu@subbu.org</a>><br>
<b>Date: </b>Thursday, October 3, 2013 10:23 AM<br>
<b>To: </b>Tim Bell <<a href="mailto:Tim.Bell@cern.ch">Tim.Bell@cern.ch</a>><br>
<b>Cc: </b>"<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>" <<a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [Openstack] Cells use cases<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Hi Tim,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Can you comment on scalability more? Are you referring to just the RPC layer in the control plane?<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Subbu<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><br>
On Oct 3, 2013, at 8:53 AM, Tim Bell <<a href="mailto:Tim.Bell@cern.ch">Tim.Bell@cern.ch</a>> wrote:<o:p></o:p></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">At CERN, we’re running cells for scalability. When you go over 1000 hypervisors or so, the general recommendation is to be in a cells configuration.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Cells are quite complex and the full functionality is not there yet so some parts will need to wait for Havana.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Tim</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="color:black"><o:p></o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"> Dmitry Ukov [<a href="mailto:dukov@mirantis.com">mailto:dukov@mirantis.com</a>]
<br>
<b>Sent:</b> 03 October 2013 16:38<br>
<b>To:</b> <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
<b>Subject:</b> [Openstack] Cells use cases</span><span style="color:black"><o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">Hello all,</span><span style="color:black"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">I've really interested in cells but unfortunately i can't find any useful use cases of them.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">For instance I have 4 DCs and I need single entry point for them. In this case cells are a bit complicated solution. It's better to use multiple regions in keystone
instead</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">The only one good reason for cells, which I've found, is to organize so-called failure domains, i.e. scheduling on another DCs in case of failures.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black"> </span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">Does anyone have different use cases or vision on cells usage?</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:black">Thanks in advance.</span><span style="color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="color:black">-- <br>
Kind regards<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="color:black">Dmitry Ukov<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">IT Engineer<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Mirantis, Inc.<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> <o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><o:p></o:p></span></p>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>