<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1503623945;
mso-list-type:hybrid;
mso-list-template-ids:-1308602392 -617729242 269025283 269025285 269025281 269025283 269025285 269025281 269025283 269025285;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:SimSun;
mso-bidi-font-family:"Times New Roman";}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Kris,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I think the proposal of hierarchical projects is out of the scope of magnum, and you might need to bring it up at keystone or cross-project meeting. I am going
to propose a walk-around that might work for you at existing tenancy model.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Suppose there is a department (department A) with two subteams (team 1 and team 2). You can create three projects:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Project A<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Project A-1<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Project A-2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Then you can assign users to projects in the following ways:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Assign team 1 members to both Project A and Project A-1<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-18.0pt;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Assign team 2 members to both Project A and Project A-2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Then you can create a bay at project A, which is shared by the whole department. In addition, each subteam can create their own bays at project A-X if they
want. Does it address your use cases?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hongbin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span lang="EN-US" style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Kris G. Lindgren [mailto:klindgren@godaddy.com]
<br>
<b>Sent:</b> September-30-15 7:26 PM<br>
<b>To:</b> openstack-dev@lists.openstack.org<br>
<b>Subject:</b> Re: [openstack-dev] [magnum]swarm + compose = k8s?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">We are looking at deploying magnum as an answer for how do we do containers company wide at Godaddy. I am going to agree with both you and josh.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">I agree that managing one large system is going to be a pain and pas experience tells me this wont be practical/scale, however from experience I also know exactly
the pain Josh is talking about.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">We currently have ~4k projects in our internal openstack cloud, about 1/4 of the projects are currently doing some form of containers on their own, with more
joining every day. If all of these projects were to convert of to the current magnum configuration we would suddenly be attempting to support/configure ~1k magnum clusters. Considering that everyone will want it HA, we are looking at a minimum of 2 kube
nodes per cluster + lbaas vips + floating ips. From a capacity standpoint this is an excessive amount of duplicated infrastructure to spinup in projects where people maybe running 10–20 containers per project. From an operator support perspective this is
a special level of hell that I do not want to get into. Even if I am off by 75%, 250 still sucks.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">From my point of view an ideal use case for companies like ours (yahoo/godaddy) would be able to support hierarchical projects in magnum. That way we could create
a project for each department, and then the subteams of those departments can have their own projects. We create a a bay per department. Sub-projects if they want to can support creation of their own bays (but support of the kube cluster would then fall
to that team). When a sub-project spins up a pod on a bay, minions get created inside that teams sub projects and the containers in that pod run on the capacity that was spun up under that project, the minions for each pod would be a in a scaling group and
as such grow/shrink as dictated by load.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">The above would make it so where we support a minimal, yet imho reasonable, number of kube clusters, give people who can't/don’t want to fall inline with the
provided resource a way to make their own and still offer a "good enough for a single company" level of multi-tenancy.<o:p></o:p></span></p>
</div>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;white-space:pre-wrap;widows: 1"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>Joshua,<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">><o:p> </o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>If you share resources, you give up multi-tenancy. No COE system has the<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>concept of multi-tenancy (kubernetes has some basic implementation but it<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>is totally insecure). Not only does multi-tenancy have to “look like” it<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>offers multiple tenants isolation, but it actually has to deliver the<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>goods.<br><br><o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">><o:p> </o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>I understand that at first glance a company like Yahoo may not want<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>separate bays for their various applications because of the perceived<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>administrative overhead. I would then challenge Yahoo to go deploy a COE<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>like kubernetes (which has no multi-tenancy or a very basic implementation<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>of such) and get it to work with hundreds of different competing<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>applications. I would speculate the administrative overhead of getting<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>all that to work would be greater then the administrative overhead of<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>simply doing a bay create for the various tenants.<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;white-space:pre-wrap"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">><o:p> </o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>Placing tenancy inside a COE seems interesting, but no COE does that<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>today. Maybe in the future they will. Magnum was designed to present an<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>integration point between COEs and OpenStack today, not five years down<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>the road. Its not as if we took shortcuts to get to where we are.<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;white-space:pre-wrap"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">><o:p> </o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>I will grant you that density is lower with the current design of Magnum<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>vs a full on integration with OpenStack within the COE itself. However,<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>that model which is what I believe you proposed is a huge design change to<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>each COE which would overly complicate the COE at the gain of increased<o:p></o:p></span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353">>density. I personally don’t feel that pain is worth the gain.<o:p></o:p></span></pre>
<pre style="mso-margin-top-alt:18.0pt;margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;white-space:pre-wrap;widows: 1"><span style="font-size:9.0pt;font-family:"Lucida Console";color:#535353"><o:p> </o:p></span></pre>
<div>
<div id="MAC_OUTLOOK_SIGNATURE">
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">___________________________________________________________________<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Kris Lindgren<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">Senior Linux Systems Engineer<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:"Calibri","sans-serif";color:black">GoDaddy<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</body>
</html>