<html><head></head><body><div style="color:#000; background-color:#fff; font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:13px"><div id="yui_3_16_0_1_1455302653460_10863" dir="ltr"><span style="font-family: Arial; font-size: small;" id="yui_3_16_0_1_1455302653460_10865" class="">Hongbin,</span><span></span></div><div></div><div id="yui_3_16_0_1_1455302653460_10862"><br></div><div id="yui_3_16_0_1_1455302653460_10862" dir="ltr">I am not sure that it's good idea, it looks you propose Magnum enter to "schedulers war" (personally I tired from these debates Mesos vs Kub vs Swarm).</div><div id="yui_3_16_0_1_1455302653460_10862" dir="ltr">If your concern is just utilization you can always run control plane at "agent/slave" nodes, there main reason why operators (at least in our case) keep them</div><div id="yui_3_16_0_1_1455302653460_10862" dir="ltr">separate because they need different attention (e.g. I almost don't care why/when "agent/slave" node died, but always double check that master node was </div><div id="yui_3_16_0_1_1455302653460_10862" dir="ltr">repaired or replaced). </div><div id="yui_3_16_0_1_1455302653460_10862"><br></div><div id="yui_3_16_0_1_1455302653460_10862" dir="ltr">One use case I see for shared COE (at least in our environment), when developers want run just docker container without installing anything locally </div><div id="yui_3_16_0_1_1455302653460_10862" dir="ltr">(e.g docker-machine). But in most cases it's just examples from internet or there own experiments ): </div><div id="yui_3_16_0_1_1455302653460_10862" dir="ltr"><br></div><div id="yui_3_16_0_1_1455302653460_10862">But we definitely should discuss it during midcycle next week. </div><div id="yui_3_16_0_1_1455302653460_10857"> </div><div class="signature" id="yui_3_16_0_1_1455302653460_10856">--- </div><div class="signature" id="yui_3_16_0_1_1455302653460_10856">Egor</div><div class="qtdSeparateBR" id="yui_3_16_0_1_1455302653460_10853"><br></div><div class="yahoo_quoted" id="yui_3_16_0_1_1455302653460_10837" style="display: block;"> <div style="font-family: Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 13px;" id="yui_3_16_0_1_1455302653460_10836"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;" id="yui_3_16_0_1_1455302653460_10835"> <div dir="ltr" id="yui_3_16_0_1_1455302653460_10834"> <font size="2" face="Arial" id="yui_3_16_0_1_1455302653460_10852"> <hr size="1" id="yui_3_16_0_1_1455302653460_12350"> <b><span style="font-weight:bold;">From:</span></b> Hongbin Lu <hongbin.lu@huawei.com><br> <b><span style="font-weight: bold;">To:</span></b> OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> <br> <b><span style="font-weight: bold;">Sent:</span></b> Thursday, February 11, 2016 8:50 PM<br> <b id="yui_3_16_0_1_1455302653460_12353"><span style="font-weight: bold;" id="yui_3_16_0_1_1455302653460_12352">Subject:</span></b> Re: [openstack-dev] [magnum]swarm + compose = k8s?<br> </font> </div> <div class="y_msg_container" id="yui_3_16_0_1_1455302653460_10879"><br><div id="yiv4408789594">
<style><!--
#yiv4408789594
_filtered #yiv4408789594 {font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
_filtered #yiv4408789594 {font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
_filtered #yiv4408789594 {font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
_filtered #yiv4408789594 {font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
_filtered #yiv4408789594 {font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
_filtered #yiv4408789594 {
panose-1:2 1 6 0 3 1 1 1 1 1;}
_filtered #yiv4408789594 {font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
_filtered #yiv4408789594 {font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
#yiv4408789594
#yiv4408789594 p.yiv4408789594MsoNormal, #yiv4408789594 li.yiv4408789594MsoNormal, #yiv4408789594 div.yiv4408789594MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman", "serif";}
#yiv4408789594 a:link, #yiv4408789594 span.yiv4408789594MsoHyperlink
{
color:blue;
text-decoration:underline;}
#yiv4408789594 a:visited, #yiv4408789594 span.yiv4408789594MsoHyperlinkFollowed
{
color:purple;
text-decoration:underline;}
#yiv4408789594 pre
{
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
#yiv4408789594 p.yiv4408789594MsoListParagraph, #yiv4408789594 li.yiv4408789594MsoListParagraph, #yiv4408789594 div.yiv4408789594MsoListParagraph
{
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman", "serif";}
#yiv4408789594 span.yiv4408789594HTMLPreformattedChar
{
font-family:Consolas;}
#yiv4408789594 span.yiv4408789594EmailStyle19
{
font-family:"Calibri", "sans-serif";
color:#1F497D;}
#yiv4408789594 .yiv4408789594MsoChpDefault
{
font-size:10.0pt;}
_filtered #yiv4408789594 {
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
#yiv4408789594 div.yiv4408789594WordSection1
{}
#yiv4408789594
_filtered #yiv4408789594 {
}
_filtered #yiv4408789594 {
font-family:Symbol;}
#yiv4408789594 ol
{margin-bottom:0cm;}
#yiv4408789594 ul
{margin-bottom:0cm;}
--></style>
<div id="yui_3_16_0_1_1455302653460_10883">
<div class="yiv4408789594WordSection1" id="yui_3_16_0_1_1455302653460_10882">
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_11156"><span style="font-size:11.0pt;color:#1F497D;" id="yui_3_16_0_1_1455302653460_12354">Hi team,</span></div>
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_11154"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_11334"><span style="font-size:11.0pt;color:#1F497D;" id="yui_3_16_0_1_1455302653460_11444">Sorry for bringing up this old thread, but a recent debate on container resource [1] reminded me the use case Kris mentioned below. I am going to propose a
preliminary idea to address the use case. Of course, we could continue the discussion in the team meeting or midcycle.</span></div>
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_10944"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_10881"><b><span style="font-size:11.0pt;color:#1F497D;">Idea</span></b><span style="font-size:11.0pt;color:#1F497D;" id="yui_3_16_0_1_1455302653460_10880">: Introduce a docker-native COE, which consists of only minion/worker/slave
nodes (no master nodes).</span></div>
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_11155"><b><span style="font-size:11.0pt;color:#1F497D;">Goal</span></b><span style="font-size:11.0pt;color:#1F497D;">: Eliminate duplicated IaaS resources (master node VMs, lbaas
vips, floating ips, etc.)</span></div>
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_10885"><b><span style="font-size:11.0pt;color:#1F497D;">Details</span></b><span style="font-size:11.0pt;color:#1F497D;" id="yui_3_16_0_1_1455302653460_10884">: Traditional COE (k8s/swarm/mesos) consists of master
nodes and worker nodes. In these COEs, control services (i.e. scheduler) run on master nodes, and containers run on worker nodes. If we can port the COE control services to Magnum control plate and share them with all tenants, we eliminate the need of master
nodes thus improving resource utilization. In the new COE, users create/manage containers through Magnum API endpoints. Magnum is responsible to spin tenant VMs, schedule containers to the VMs, and manage the life-cycle of those containers. Unlike other COEs,
containers created by this COE are considered as OpenStack-manage resources. That means they will be tracked in Magnum DB, and accessible by other OpenStack services (i.e. Horizon, Heat, etc.).</span></div>
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_10886"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div class="yiv4408789594MsoNormal" id="yui_3_16_0_1_1455302653460_10945"><span style="font-size:11.0pt;color:#1F497D;">What do you feel about this proposal? Let’s discuss.</span></div>
<div class="yiv4408789594MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div class="yiv4408789594MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">[1]
<a rel="nofollow" target="_blank" href="https://etherpad.openstack.org/p/magnum-native-api">https://etherpad.openstack.org/p/magnum-native-api</a></span></div>
<div class="yiv4408789594MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div class="yiv4408789594MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">Best regards,</span></div>
<div class="yiv4408789594MsoNormal"><span style="font-size:11.0pt;color:#1F497D;">Hongbin</span></div>
<div class="yiv4408789594MsoNormal"><span style="font-size:11.0pt;color:#1F497D;"> </span></div>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm;">
<div class="yiv4408789594MsoNormal"><b><span lang="EN-US" style="font-size:10.0pt;">From:</span></b><span lang="EN-US" style="font-size:10.0pt;"> Kris G. Lindgren [mailto:klindgren@godaddy.com]
<br>
<b>Sent:</b> September-30-15 7:26 PM<br>
<b>To:</b> openstack-dev@lists.openstack.org<br>
<b>Subject:</b> Re: [openstack-dev] [magnum]swarm + compose = k8s?</span></div>
</div>
</div>
<div class="yiv4408789594MsoNormal"> </div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">We are looking at deploying magnum as an answer for how do we do containers company wide at Godaddy. I am going to agree with both you and josh.</span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">I agree that managing one large system is going to be a pain and pas experience tells me this wont be practical/scale, however from experience I also know exactly
the pain Josh is talking about.</span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">We currently have ~4k projects in our internal openstack cloud, about 1/4 of the projects are currently doing some form of containers on their own, with more
joining every day. If all of these projects were to convert of to the current magnum configuration we would suddenly be attempting to support/configure ~1k magnum clusters. Considering that everyone will want it HA, we are looking at a minimum of 2 kube
nodes per cluster + lbaas vips + floating ips. From a capacity standpoint this is an excessive amount of duplicated infrastructure to spinup in projects where people maybe running 10–20 containers per project. From an operator support perspective this is
a special level of hell that I do not want to get into. Even if I am off by 75%, 250 still sucks.</span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">From my point of view an ideal use case for companies like ours (yahoo/godaddy) would be able to support hierarchical projects in magnum. That way we could create
a project for each department, and then the subteams of those departments can have their own projects. We create a a bay per department. Sub-projects if they want to can support creation of their own bays (but support of the kube cluster would then fall
to that team). When a sub-project spins up a pod on a bay, minions get created inside that teams sub projects and the containers in that pod run on the capacity that was spun up under that project, the minions for each pod would be a in a scaling group and
as such grow/shrink as dictated by load.</span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">The above would make it so where we support a minimal, yet imho reasonable, number of kube clusters, give people who can't/don’t want to fall inline with the
provided resource a way to make their own and still offer a "good enough for a single company" level of multi-tenancy.</span></div>
</div>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;white-space:pre-wrap;widows:1;"><span style="font-size:9.0pt;color:#535353;">>Joshua,</span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">> </span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>If you share resources, you give up multi-tenancy. No COE system has the</span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>concept of multi-tenancy (kubernetes has some basic implementation but it</span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>is totally insecure). Not only does multi-tenancy have to “look like” it</span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>offers multiple tenants isolation, but it actually has to deliver the</span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>goods.<br><br></span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">> </span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>I understand that at first glance a company like Yahoo may not want</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>separate bays for their various applications because of the perceived</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>administrative overhead. I would then challenge Yahoo to go deploy a COE</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>like kubernetes (which has no multi-tenancy or a very basic implementation</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>of such) and get it to work with hundreds of different competing</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>applications. I would speculate the administrative overhead of getting</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>all that to work would be greater then the administrative overhead of</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>simply doing a bay create for the various tenants.</span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;white-space:pre-wrap;"><span style="font-size:9.0pt;color:#535353;">> </span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>Placing tenancy inside a COE seems interesting, but no COE does that</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>today. Maybe in the future they will. Magnum was designed to present an</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>integration point between COEs and OpenStack today, not five years down</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>the road. Its not as if we took shortcuts to get to where we are.</span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;white-space:pre-wrap;"><span style="font-size:9.0pt;color:#535353;">> </span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>I will grant you that density is lower with the current design of Magnum</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>vs a full on integration with OpenStack within the COE itself. However,</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>that model which is what I believe you proposed is a huge design change to</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>each COE which would overly complicate the COE at the gain of increased</span></pre>
<pre style="line-height:13.5pt;background:white;vertical-align:baseline;"><span style="font-size:9.0pt;color:#535353;">>density. I personally don’t feel that pain is worth the gain.</span></pre>
<pre style="margin-right:0cm;margin-bottom:18.0pt;margin-left:0cm;line-height:13.5pt;background:white;vertical-align:baseline;white-space:pre-wrap;widows:1;"><span style="font-size:9.0pt;color:#535353;"> </span></pre>
<div>
<div id="yiv4408789594MAC_OUTLOOK_SIGNATURE">
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;"> </span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">___________________________________________________________________</span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">Kris Lindgren</span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">Senior Linux Systems Engineer</span></div>
</div>
<div>
<div class="yiv4408789594MsoNormal"><span style="font-size:10.5pt;color:black;">GoDaddy</span></div>
</div>
</div>
</div>
</div>
</div>
</div><br>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: <a ymailto="mailto:OpenStack-dev-request@lists.openstack.org" href="mailto:OpenStack-dev-request@lists.openstack.org">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br><br><br></div> </div> </div> </div></div></body></html>