<div dir="ltr"><div class="gmail_default" style="font-family:tahoma,sans-serif">I'm also FWD'ing this convo onto the AppEco-WG mailing list for awareness by some of the dev re below...<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jul 3, 2016 at 4:30 AM, Clayton Coleman <span dir="ltr"><<a href="mailto:ccoleman@redhat.com" target="_blank">ccoleman@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div><div><div></div><div><br></div><div><br>On Jul 1, 2016, at 7:46 PM, Hongbin Lu <<a href="mailto:hongbin034@gmail.com" target="_blank">hongbin034@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 1, 2016 at 6:14 AM, Salvatore Orlando <span dir="ltr"><<a href="mailto:salv.orlando@gmail.com" target="_blank">salv.orlando@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>some comments from me as well.</div><div><br></div><div>Salvatore<br><div class="gmail_extra"><br><div class="gmail_quote"><span>On 1 July 2016 at 00:48, Chris Marino <span dir="ltr"><<a href="mailto:chris@romana.io" target="_blank">chris@romana.io</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Hi Hongbin, as a first step, I like the idea of tenants using<br>
OpenStack to provision VMs for their own private k8s pods. And with<br>
RBAC [1] and network policy [2] support in k8s 1.3 I think it probably<br>
can be built.<br></blockquote><div><br></div></span><div>I think this also requires that a node is exclusively assigned to a tenant. The kubernetes scheduler allows using node selectors or node affinity [3]; that is something this solution should leverage.</div><div>The flipside is that it must be ensured that a tenant-specific node only hosts pods belonging to that tenant. Therefore every pod must specify a node selector. If you are wrapping pod creation requests via magnum (or higgins???) APIs, you can surely do that.</div><div><br></div><div>[3] <a href="http://kubernetes.io/docs/user-guide/node-selection/" target="_blank">http://kubernetes.io/docs/user-guide/node-selection/</a></div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<br>
As for what might go upstream, that's hard to tell. But having this as<br>
an example could provide lots of use cases that need to be addressed.<br>
<br>
[1] <a href="https://coreos.com/blog/kubernetes-v1.3-preview.html" rel="noreferrer" target="_blank">https://coreos.com/blog/kubernetes-v1.3-preview.html</a><br>
[2] <a href="https://github.com/caseydavenport/kubernetes.github.io/blob/e232dae842529e17a17f5490a5f8db9e2f900408/docs/user-guide/networkpolicies.md" rel="noreferrer" target="_blank">https://github.com/caseydavenport/kubernetes.github.io/blob/e232dae842529e17a17f5490a5f8db9e2f900408/docs/user-guide/networkpolicies.md</a><br>
<br>
CM<br>
<div><div><br>
On Thu, Jun 30, 2016 at 12:47 PM, Hongbin Lu <<a href="mailto:hongbin034@gmail.com" target="_blank">hongbin034@gmail.com</a>> wrote:<br>
> Hi folks,<br>
><br>
> This is a continued discussion about multi-tenancy. In short, I am looking<br>
> for a solution (optimally an upstream solution) to make k8s be compatible<br>
> with the multi-tenancy model of cloud providers, in particular OpenStack.<br>
><br>
> Currently, there is no perfect solution for that. The most common solution<br>
> is to place the entire k8s cluster to a set of VMs within a single OpenStack<br>
> tenant. This works well in some cases, but it is not perfect. First, it<br>
> requires end-users to manage their clusters but they generally don't have<br>
> the expertise to do that. Second, the resource utilization is low if there<br>
> are many OpenStack tenants and each tenant/user wants to create its own k8s<br>
> cluster. Such problem was well described in this email [1].<br></div></div></blockquote><div><br></div></span><div>The beauty of this scenario is that we don't have to worry about it. It has plenty of problems but surely is the easiest way to run kubernetes on Openstack!</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div>
><br>
> A potential solution is to have a big k8s cluster hosted in an OpenStack<br>
> tenant, and re-distribute it to other tenants. This solution is not perfect<br>
> as well. An obstacle is the difficulties for IaaS providers to bill users<br>
> because all the resources are mixed in a single tenant. There are other<br>
> challenges as well. For example, how to isolate containers from different<br>
> tenants in the same host.<br></div></div></blockquote><div><br></div></span><div>It seems to me that you are implicitly assuming that a k8s cluster must run in OpenStack instances, which is not necessary in many cases.<br></div><div>This approach frees tenants from the burden of managing the cluster by themselves but introduces, in my opinion, a new set of issues that we probably don't want to deal with - mostly associated with redistribution of the cluster to other tenants.</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div>
><br>
> IMHO, an ideal solution is to have a centralized k8s control plane, and<br>
> allow end-users to provision minion nodes from their tenants, like AWS ECS.<br>
> This requires k8s to understand where each node is coming from. For example,<br>
> if OpenStack users want to create pods, they need to pre-create a set of VMs<br>
> in their tenants. K8s will use the Keystone plugin to authenticate the<br>
> users, and schedule their pods to their VMs. Is this a reasonable<br>
> requirement for k8s upstream?<br></div></div></blockquote><div><br></div></span><div>It seems that is kind of similar to consider a kubernetes cluster conceptually similar to a compute service, and the cluster nodes as the hypervisors it manages?</div><div>That would seem a decent thing to do imho, and from my 30k ft perspective surely appears feasible, especially if you're using a service like higgins (magnum???) to wrap k8s APIs.<br></div></div></div></div></div></blockquote><div><br></div><div>I am afraid wrapping k8s APIs is out of Magnum's scope. Higgins (renamed to Zun) might do it but it doesn't address the use cases that end-users use native tool (i.e. kubectl) to access the k8s cluster. Frankly, the approach of building an external API wrapper looks like a hack. It is better to have a built-in multi-tenancy solution rather than relying on external system to tell k8s how to enforce multi-tenancy policy.</div></div></div></div></div></blockquote><div><br></div></div></div><div>TL:DR - yes.</div><div><br></div><div>It has always been a goal to enable Kube for multi-tenancy, but to do so in a way that remains flexible enough so that many different use cases are possible. For example, OpenShift and (probably) Tectonic are interested in "innate multi-tenancy" - the cluster provides tenancy and policy in an integrated fashion, without requiring an external authorization store. Others such as GKE and OpenStack have investigated "1:1 tenancy" - each cluster is a tenant. Still others want something in the middle (what I call "programmable multi-tenancy") where K8S has enough tools that a higher level program can describe policy onto the cluster but Kube isn't aware of the tenancy. </div><div><br></div><div>Depending on which of these you are targeting, there is varying levels of readiness, and varying levels of work to be done. </div><div><br></div><div><span style="background-color:rgba(255,255,255,0)">1:1 tenancy is done (as noted).</span></div><div><br></div><div>OpenShift is fully MT (down to a fine grained level of including a user and oath model that correlates to security policy on the cluster) today, but it will likely take some time to introduce those features into Kube. PodSecurityPolicy, Quota/Limits the Authenticator/Authorizer interfaces, and the work CoreOS did to port the OpenShift Authz engine into Kube are mostly done, but lots remain.</div><div><br></div><div>Those features should be usable for "programmable multitenancy" today, but in the programmable model an external party (what we in Kube call a controller, but could be a management head or manual scripting) needs to manage that work. Instead of trying to mutate API calls, you'll want to set up core policy on Kube to enforce your wishes. That could be magnum or another custom component.</div><div><br></div><div>It would probably be best to start from what works today and help identify gaps - policy that cannot be enforced from outside the cluster without API mutation - and target those. The policy proposals from Apcera and others, field level access control, enhanced defaulting and quota controls (we'd like to be able to quota arbitrary fields eventually), and allowing external admission hooks are all examples of ongoing work that can benefit from participation.</div><blockquote type="cite"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><span><div><div></div></div></span></blockquote></div></div></div></div></blockquote></div></div></div>
</div></blockquote></div><span>
<p></p>
-- <br>
You received this message because you are subscribed to the Google Groups "kubernetes-sig-openstack" group.<br>
To unsubscribe from this group and stop receiving emails from it, send an email to <a href="mailto:kubernetes-sig-openstack+unsubscribe@googlegroups.com" target="_blank">kubernetes-sig-openstack+unsubscribe@googlegroups.com</a>.<br>
To post to this group, send email to <a href="mailto:kubernetes-sig-openstack@googlegroups.com" target="_blank">kubernetes-sig-openstack@googlegroups.com</a>.<br></span>
To view this discussion on the web visit <a href="https://groups.google.com/d/msgid/kubernetes-sig-openstack/-7868473893287563063%40unknownmsgid?utm_medium=email&utm_source=footer" target="_blank">https://groups.google.com/d/msgid/kubernetes-sig-openstack/-7868473893287563063%40unknownmsgid</a>.<div><div><br>
For more options, visit <a href="https://groups.google.com/d/optout" target="_blank">https://groups.google.com/d/optout</a>.<br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">=================<div>Twitter: <a href="https://twitter.com/dfflanders" target="_blank">@DFFlanders</a></div><div>Skype: david.flanders</div><div>Based in Melbourne, Australia</div><div><br></div><div><br></div></div></div>
</div></div></blockquote></div></div>
</div>