<html><body><p>Hi HongBin and Egor,<br>I went through what you talked about, and thinking what's the great benefits for utilisation here.<br>For user cases, looks like following:<br><br> user A want to have a COE provision.<br> user B want to have a separate COE. (different tenant, non-share)<br> user C want to use existed COE (same tenant as User A, share)<br><br>When you talked about utilisation case, it seems you mentioned:<br> different tenant users want to use same control node to manage different nodes, it seems that try to make COE openstack tenant aware, it also means you want to introduce another control schedule layer above the COEs, we need to think about the if it is typical user case, and what's the benefit compared with containerisation. <br><br><br>And finally, it is a topic can be discussed in middle cycle meeting. <br><br><br>Thanks<br><br>Best Wishes,<br>--------------------------------------------------------------------------------<br>Kai Qiang Wu (Î⿪ǿ Kennan£©<br>IBM China System and Technology Lab, Beijing<br><br>E-mail: wkqwu@cn.ibm.com<br>Tel: 86-10-82451647<br>Address: Building 28(Ring Building), ZhongGuanCun Software Park, <br> No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193<br>--------------------------------------------------------------------------------<br>Follow your heart. You are miracle! <br><br><img width="16" height="16" src="cid:1__=8FBBF5CADF9AA7368f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for Hongbin Lu ---13/02/2016 11:02:13 am---Egor, Thanks for sharing your insights. I gave it more thought"><font color="#424282">Hongbin Lu ---13/02/2016 11:02:13 am---Egor, Thanks for sharing your insights. I gave it more thoughts. Maybe the goal can be achieved with</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Hongbin Lu <hongbin.lu@huawei.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">Guz Egor <guz_egor@yahoo.com>, "OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">13/02/2016 11:02 am</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [openstack-dev] [magnum]swarm + compose = k8s?</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font color="#1F497D" face="Calibri">Egor,</font><br><font color="#1F497D" face="Calibri"> </font><br><font color="#1F497D" face="Calibri">Thanks for sharing your insights. I gave it more thoughts. Maybe the goal can be achieved without implementing a shared COE. We could move all the master nodes out of user tenants, containerize them, and consolidate them into a set of VMs/Physical servers.</font><br><font color="#1F497D" face="Calibri"> </font><br><font color="#1F497D" face="Calibri">I think we could separate the discussion into two:</font><ul><ul><font color="#1F497D" face="Calibri">1. Should Magnum introduce a new bay type, in which master nodes are managed by Magnum (not users themselves)? Like what GCE [1] or ECS [2] does.</font><br><font color="#1F497D" face="Calibri">2. How to consolidate the control services that originally runs on master nodes of each cluster?</font></ul></ul><font color="#1F497D" face="Calibri"> </font><br><font color="#1F497D" face="Calibri">Note that the proposal is for adding a new COE (not for changing the existing COEs). That means users will continue to provision existing self-managed COE (k8s/swarm/mesos) if they choose to.</font><br><font color="#1F497D" face="Calibri"> </font><br><font color="#1F497D" face="Calibri">[1] </font><a href="https://cloud.google.com/container-engine/"><u><font color="#0000FF" face="Calibri">https://cloud.google.com/container-engine/</font></u></a><br><font color="#1F497D" face="Calibri">[2] </font><a href="http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html"><u><font color="#0000FF" face="Calibri">http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html</font></u></a><br><font color="#1F497D" face="Calibri"> </font><br><font color="#1F497D" face="Calibri">Best regards,</font><br><font color="#1F497D" face="Calibri">Hongbin</font><br><font color="#1F497D" face="Calibri"> </font><br><b><font face="Tahoma">From:</font></b><font face="Tahoma"> Guz Egor [</font><font face="Tahoma"><a href="mailto:guz_egor@yahoo.com">mailto:guz_egor@yahoo.com</a></font><font face="Tahoma">] </font><b><font face="Tahoma"><br>Sent:</font></b><font face="Tahoma"> February-12-16 2:34 PM</font><b><font face="Tahoma"><br>To:</font></b><font face="Tahoma"> OpenStack Development Mailing List (not for usage questions)</font><b><font face="Tahoma"><br>Cc:</font></b><font face="Tahoma"> Hongbin Lu</font><b><font face="Tahoma"><br>Subject:</font></b><font face="Tahoma"> Re: [openstack-dev] [magnum]swarm + compose = k8s?</font><br><font size="4" face="Times New Roman"> </font><br><font size="4" face="Arial">Hongbin,</font><br><font face="Helvetica"> </font><br><font face="Helvetica">I am not sure that it's good idea, it looks you propose Magnum enter to "schedulers war" (personally I tired from these debates Mesos vs Kub vs Swarm).</font><br><font face="Helvetica">If your concern is just utilization you can always run control plane at "agent/slave" nodes, there main reason why operators (at least in our case) keep them</font><br><font face="Helvetica">separate because they need different attention (e.g. I almost don't care why/when "agent/slave" node died, but always double check that master node was </font><br><font face="Helvetica">repaired or replaced). </font><br><font face="Helvetica"> </font><br><font face="Helvetica">One use case I see for shared COE (at least in our environment), when developers want run just docker container without installing anything locally </font><br><font face="Helvetica">(e.g docker-machine). But in most cases it's just examples from internet or there own experiments ): </font><br><font face="Helvetica"> </font><br><font face="Helvetica">But we definitely should discuss it during midcycle next week. </font><br><font face="Helvetica"> </font><br><font face="Helvetica">--- </font><br><font face="Helvetica">Egor</font><br><font face="Helvetica"> </font><div align="center"><hr width="100%" size="2" align="center"></div><b><font face="Arial">From:</font></b><font face="Arial"> Hongbin Lu <</font><a href="mailto:hongbin.lu@huawei.com"><u><font color="#0000FF" face="Arial">hongbin.lu@huawei.com</font></u></a><font face="Arial">></font><b><font face="Arial"><br>To:</font></b><font face="Arial"> OpenStack Development Mailing List (not for usage questions) <</font><a href="mailto:openstack-dev@lists.openstack.org"><u><font color="#0000FF" face="Arial">openstack-dev@lists.openstack.org</font></u></a><font face="Arial">> </font><b><font face="Arial"><br>Sent:</font></b><font face="Arial"> Thursday, February 11, 2016 8:50 PM</font><b><font face="Arial"><br>Subject:</font></b><font face="Arial"> Re: [openstack-dev] [magnum]swarm + compose = k8s?</font><br><font size="4" face="Helvetica"> </font><br><font color="#1F497D" face="Helvetica">Hi team,</font><br><font color="#1F497D" face="Helvetica"> </font><br><font color="#1F497D" face="Helvetica">Sorry for bringing up this old thread, but a recent debate on container resource [1] reminded me the use case Kris mentioned below. I am going to propose a preliminary idea to address the use case. Of course, we could continue the discussion in the team meeting or midcycle.</font><br><font color="#1F497D" face="Helvetica"> </font><br><b><font color="#1F497D" face="Helvetica">Idea</font></b><font color="#1F497D" face="Helvetica">: Introduce a docker-native COE, which consists of only minion/worker/slave nodes (no master nodes).</font><br><b><font color="#1F497D" face="Helvetica">Goal</font></b><font color="#1F497D" face="Helvetica">: Eliminate duplicated IaaS resources (master node VMs, lbaas vips, floating ips, etc.)</font><br><b><font color="#1F497D" face="Helvetica">Details</font></b><font color="#1F497D" face="Helvetica">: Traditional COE (k8s/swarm/mesos) consists of master nodes and worker nodes. In these COEs, control services (i.e. scheduler) run on master nodes, and containers run on worker nodes. If we can port the COE control services to Magnum control plate and share them with all tenants, we eliminate the need of master nodes thus improving resource utilization. In the new COE, users create/manage containers through Magnum API endpoints. Magnum is responsible to spin tenant VMs, schedule containers to the VMs, and manage the life-cycle of those containers. Unlike other COEs, containers created by this COE are considered as OpenStack-manage resources. That means they will be tracked in Magnum DB, and accessible by other OpenStack services (i.e. Horizon, Heat, etc.).</font><br><font color="#1F497D" face="Helvetica"> </font><br><font color="#1F497D" face="Helvetica">What do you feel about this proposal? Let¡¯s discuss.</font><br><font color="#1F497D" face="Helvetica"> </font><br><font color="#1F497D" face="Helvetica">[1] </font><a href="https://etherpad.openstack.org/p/magnum-native-api" target="_blank"><u><font color="#0000FF" face="Helvetica">https://etherpad.openstack.org/p/magnum-native-api</font></u></a><br><font color="#1F497D" face="Helvetica"> </font><br><font color="#1F497D" face="Helvetica">Best regards,</font><br><font color="#1F497D" face="Helvetica">Hongbin</font><br><font color="#1F497D" face="Helvetica"> </font><br><b><font face="Helvetica">From:</font></b><font face="Helvetica"> Kris G. Lindgren [</font><a href="mailto:klindgren@godaddy.com"><u><font color="#0000FF" face="Helvetica">mailto:klindgren@godaddy.com</font></u></a><font face="Helvetica">] </font><b><font face="Helvetica"><br>Sent:</font></b><font face="Helvetica"> September-30-15 7:26 PM</font><b><font face="Helvetica"><br>To:</font></b><font face="Helvetica"> </font><a href="mailto:openstack-dev@lists.openstack.org"><u><font color="#0000FF" face="Helvetica">openstack-dev@lists.openstack.org</font></u></a><b><font face="Helvetica"><br>Subject:</font></b><font face="Helvetica"> Re: [openstack-dev] [magnum]swarm + compose = k8s?</font><br><font size="4" face="Helvetica"> </font><br><font face="Helvetica">We are looking at deploying magnum as an answer for how do we do containers company wide at Godaddy. I am going to agree with both you and josh.</font><br><font face="Helvetica"> </font><br><font face="Helvetica">I agree that managing one large system is going to be a pain and pas experience tells me this wont be practical/scale, however from experience I also know exactly the pain Josh is talking about.</font><br><font face="Helvetica"> </font><br><font face="Helvetica">We currently have ~4k projects in our internal openstack cloud, about 1/4 of the projects are currently doing some form of containers on their own, with more joining every day. If all of these projects were to convert of to the current magnum configuration we would suddenly be attempting to support/configure ~1k magnum clusters. Considering that everyone will want it HA, we are looking at a minimum of 2 kube nodes per cluster + lbaas vips + floating ips. From a capacity standpoint this is an excessive amount of duplicated infrastructure to spinup in projects where people maybe running 10¨C20 containers per project. From an operator support perspective this is a special level of hell that I do not want to get into. Even if I am off by 75%, 250 still sucks.</font><br><font face="Helvetica"> </font><br><font face="Helvetica">From my point of view an ideal use case for companies like ours (yahoo/godaddy) would be able to support hierarchical projects in magnum. That way we could create a project for each department, and then the subteams of those departments can have their own projects. We create a a bay per department. Sub-projects if they want to can support creation of their own bays (but support of the kube cluster would then fall to that team). When a sub-project spins up a pod on a bay, minions get created inside that teams sub projects and the containers in that pod run on the capacity that was spun up under that project, the minions for each pod would be a in a scaling group and as such grow/shrink as dictated by load.</font><br><font face="Helvetica"> </font><br><font face="Helvetica">The above would make it so where we support a minimal, yet imho reasonable, number of kube clusters, give people who can't/don¡¯t want to fall inline with the provided resource a way to make their own and still offer a "good enough for a single company" level of multi-tenancy.</font><br><font size="2" color="#535353" face="Courier New">>Joshua,</font><br><font size="2" color="#535353" face="Courier New">> </font><br><font size="2" color="#535353" face="Courier New">>If you share resources, you give up multi-tenancy. No COE system has the</font><br><font size="2" color="#535353" face="Courier New">>concept of multi-tenancy (kubernetes has some basic implementation but it</font><br><font size="2" color="#535353" face="Courier New">>is totally insecure). Not only does multi-tenancy have to ¡°look like¡± it</font><br><font size="2" color="#535353" face="Courier New">>offers multiple tenants isolation, but it actually has to deliver the</font><br><font size="2" color="#535353" face="Courier New">>goods.</font><br><font size="2" color="#535353" face="Courier New">> </font><br><font size="2" color="#535353" face="Courier New">>I understand that at first glance a company like Yahoo may not want</font><br><font size="2" color="#535353" face="Courier New">>separate bays for their various applications because of the perceived</font><br><font size="2" color="#535353" face="Courier New">>administrative overhead. I would then challenge Yahoo to go deploy a COE</font><br><font size="2" color="#535353" face="Courier New">>like kubernetes (which has no multi-tenancy or a very basic implementation</font><br><font size="2" color="#535353" face="Courier New">>of such) and get it to work with hundreds of different competing</font><br><font size="2" color="#535353" face="Courier New">>applications. I would speculate the administrative overhead of getting</font><br><font size="2" color="#535353" face="Courier New">>all that to work would be greater then the administrative overhead of</font><br><font size="2" color="#535353" face="Courier New">>simply doing a bay create for the various tenants.</font><br><font size="2" color="#535353" face="Courier New">> </font><br><font size="2" color="#535353" face="Courier New">>Placing tenancy inside a COE seems interesting, but no COE does that</font><br><font size="2" color="#535353" face="Courier New">>today. Maybe in the future they will. Magnum was designed to present an</font><br><font size="2" color="#535353" face="Courier New">>integration point between COEs and OpenStack today, not five years down</font><br><font size="2" color="#535353" face="Courier New">>the road. Its not as if we took shortcuts to get to where we are.</font><br><font size="2" color="#535353" face="Courier New">> </font><br><font size="2" color="#535353" face="Courier New">>I will grant you that density is lower with the current design of Magnum</font><br><font size="2" color="#535353" face="Courier New">>vs a full on integration with OpenStack within the COE itself. However,</font><br><font size="2" color="#535353" face="Courier New">>that model which is what I believe you proposed is a huge design change to</font><br><font size="2" color="#535353" face="Courier New">>each COE which would overly complicate the COE at the gain of increased</font><br><font size="2" color="#535353" face="Courier New">>density. I personally don¡¯t feel that pain is worth the gain.</font><br><font size="2" color="#535353" face="Courier New"> </font><br><font face="Helvetica"> </font><br><font face="Helvetica">___________________________________________________________________</font><br><font face="Helvetica">Kris Lindgren</font><br><font face="Helvetica">Senior Linux Systems Engineer</font><br><font face="Helvetica">GoDaddy</font><br><font size="4" face="Helvetica"><br>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: </font><a href="mailto:OpenStack-dev-request@lists.openstack.org"><u><font size="4" color="#0000FF" face="Helvetica">OpenStack-dev-request@lists.openstack.org</font></u></a><font size="4" face="Helvetica">?subject:unsubscribe</font><u><font size="4" color="#0000FF" face="Helvetica"><br></font></u><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><u><font size="4" color="#0000FF" face="Helvetica">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></u></a><font size="4" face="Helvetica"><br></font><tt>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br></tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></tt><tt><br></tt><br><br><BR>
</body></html>