<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
Kris,
<div class=""><br class="">
<div>
<blockquote type="cite" class="">
<div class="">On Sep 30, 2015, at 4:26 PM, Kris G. Lindgren <<a href="mailto:klindgren@godaddy.com" class="">klindgren@godaddy.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
We are looking at deploying magnum as an answer for how do we do containers company wide at Godaddy. I am going to agree with both you and josh.</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
I agree that managing one large system is going to be a pain and pas experience tells me this wont be practical/scale, however from experience I also know exactly the pain Josh is talking about.</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
We currently have ~4k projects in our internal openstack cloud, about 1/4 of the projects are currently doing some form of containers on their own, with more joining every day. If all of these projects were to convert of to the current magnum configuration
we would suddenly be attempting to support/configure ~1k magnum clusters. Considering that everyone will want it HA, we are looking at a minimum of 2 kube nodes per cluster + lbaas vips + floating ips. From a capacity standpoint this is an excessive amount
of duplicated infrastructure to spinup in projects where people maybe running 10–20 containers per project. From an operator support perspective this is a special level of hell that I do not want to get into. Even if I am off by 75%, 250 still sucks.</div>
</div>
</blockquote>
<div><br class="">
</div>
Keep in mind that your magnum bays can use the same floating ip addresses that your containers do, and the containers hosts are shared between the COE nodes and the containers that make up the applications running in the bay. It is possible to use private address
space for that, and proxy public facing access through a proxy layer that uses names to route connections to the appropriate magnum bay. That’s how you can escape the problem of public IP addresses as a scarce resource. </div>
<div><br class="">
</div>
<div>Also, if you use Magnum to start all those bays, they can all look the same, rather than the ~1000 container environments you have today that probably don’t look very similar, one to the next. Upgrading becomes much more achievable when you have wider
consistency. There is a new feature currently in review called public baymodel that allows the cloud operator to define the bay model, but individual tenants can start bays based on that one common “template”. This is a way of centralizing most of your configuration.
This balances a lot of the operational concern.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
From my point of view an ideal use case for companies like ours (yahoo/godaddy) would be able to support hierarchical projects in magnum. That way we could create a project for each department, and then the subteams of those departments can have their own
projects. We create a a bay per department. Sub-projects if they want to can support creation of their own bays (but support of the kube cluster would then fall to that team). When a sub-project spins up a pod on a bay, minions get created inside that teams
sub projects and the containers in that pod run on the capacity that was spun up under that project, the minions for each pod would be a in a scaling group and as such grow/shrink as dictated by load.</div>
</div>
</blockquote>
<div><br class="">
</div>
You can do this today by sharing your TLS certs. In fact, you could make the cert signing a bit more sophisticated than it is today, and allow each subteam to have a unique TLS cert that can auth against a common bay.</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">
<div style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
The above would make it so where we support a minimal, yet imho reasonable, number of kube clusters, give people who can't/don’t want to fall inline with the provided resource a way to make their own and still offer a "good enough for a single company" level
of multi-tenancy.</div>
</div>
</blockquote>
<div><br class="">
</div>
This is different than what Joshua was asking for with identities in keystone, because today’s COE’s themselves don’t have modular identity solutions that are implemented with multi-tenancy.</div>
<div><br class="">
</div>
<div>Imagine for a moment that you don’t need to run your bays on Nova instances that are virtual machines. What if you had an additional host aggregate that could produce libvirt/lxc guests that you can use to form bays. They can actually be composed of nodes
that are sourced from BOTH your libvirt/lxc host aggregate (for hosting your COE’s) and your normal KVM (or the hypervisor) host aggregate for your apps to use. Then the effective consolidation ratio of your bays (what you referred to as “excessive amount
of duplicated infrastructure”) become processes running on a much smaller number of compute nodes. You could do this by specifying a different master_flavor_id and flavor_id such that these fall on different host aggregates. As long as you are “all one company”
and are not concerned primarily with security isolation between neighboring COE master nodes, that approach may actually be the right balance, and would not require an architectural shift or figuring out how to accomplish nested tenants.</div>
<div><br class="">
</div>
<div>Adrian<br class="">
<blockquote type="cite" class="">
<div class="">
<pre style="font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin-top: 1.5em; margin-bottom: 1.5em; padding: 0px; border: 0px; font-size: 12.0012px; font-family: 'andale mono', 'lucida console', monospace; vertical-align: baseline; white-space: pre-wrap; line-height: 18.0018px; color: rgb(83, 83, 83); widows: 1; background-color: rgb(255, 255, 255);" class="">>Joshua,
>
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>If you share resources, you give up multi-tenancy. No COE system has the
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>concept of multi-tenancy (kubernetes has some basic implementation but it
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>is totally insecure). Not only does multi-tenancy have to “look like” it
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>offers multiple tenants isolation, but it actually has to deliver the
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>goods.<br class=""><pre style="margin-top: 1.5em; margin-bottom: 1.5em; padding: 0px; border: 0px; font-size: 12.0012px; font-family: 'andale mono', 'lucida console', monospace; vertical-align: baseline; white-space: pre-wrap; line-height: 18.0018px;" class="">></pre><span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>I understand that at first glance a company like Yahoo may not want
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>separate bays for their various applications because of the perceived
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>administrative overhead. I would then challenge Yahoo to go deploy a COE
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>like kubernetes (which has no multi-tenancy or a very basic implementation
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>of such) and get it to work with hundreds of different competing
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>applications. I would speculate the administrative overhead of getting
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>all that to work would be greater then the administrative overhead of
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>simply doing a bay create for the various tenants.
<pre style="margin-top: 1.5em; margin-bottom: 1.5em; padding: 0px; border: 0px; font-size: 12.0012px; font-family: 'andale mono', 'lucida console', monospace; vertical-align: baseline; white-space: pre-wrap; line-height: 18.0018px;" class="">></pre><span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>Placing tenancy inside a COE seems interesting, but no COE does that
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>today. Maybe in the future they will. Magnum was designed to present an
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>integration point between COEs and OpenStack today, not five years down
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>the road. Its not as if we took shortcuts to get to where we are.
<pre style="margin-top: 1.5em; margin-bottom: 1.5em; padding: 0px; border: 0px; font-size: 12.0012px; font-family: 'andale mono', 'lucida console', monospace; vertical-align: baseline; white-space: pre-wrap; line-height: 18.0018px;" class="">></pre><span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>I will grant you that density is lower with the current design of Magnum
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>vs a full on integration with OpenStack within the COE itself. However,
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>that model which is what I believe you proposed is a huge design change to
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>each COE which would overly complicate the COE at the gain of increased
<span style="font-size: 12.0012px; line-height: 18.0018px;" class="">></span>density. I personally don’t feel that pain is worth the gain.</pre>
<pre style="font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; word-spacing: 0px; -webkit-text-stroke-width: 0px; margin-top: 1.5em; margin-bottom: 1.5em; padding: 0px; border: 0px; font-size: 12.0012px; font-family: 'andale mono', 'lucida console', monospace; vertical-align: baseline; white-space: pre-wrap; line-height: 18.0018px; color: rgb(83, 83, 83); widows: 1; background-color: rgb(255, 255, 255);" class=""><br class=""></pre>
<div style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<div id="MAC_OUTLOOK_SIGNATURE" class="">
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri"><br class="">
</font></font></div>
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri">___________________________________________________________________</font></font></div>
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri">Kris Lindgren</font></font></div>
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri">Senior Linux Systems Engineer</font></font></div>
<div class=""><font class="Apple-style-span"><font class="Apple-style-span" face="Calibri">GoDaddy</font></font></div>
</div>
</div>
<span style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">__________________________________________________________________________</span><br style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">OpenStack
Development Mailing List (not for usage questions)</span><br style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<span style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Unsubscribe:<span class="Apple-converted-space"> </span></span><a href="mailto:OpenStack-dev-request@lists.openstack.org" style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">OpenStack-dev-request@lists.openstack.org</a><span style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">?subject:unsubscribe</span><br style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="font-family: Calibri, sans-serif; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>