Open Stack

On Mon, 2020-03-23 at 12:17 -0400, Hongbin Lu wrote:
> 
> 
> On Mon, Mar 23, 2020 at 11:48 AM <mdulko at redhat.com> wrote:
> > On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
> > > Hi all,
> > > 
> > > As we are approaching the end of Ussuri cycle, I would like to take a
> > > chance to introduce a new feature the Zun team implemented in this
> > > cycle - CRI integration [1].

<snip!>

> > > Under the hook, a capsule is a podsandbox with one or more containers
> > > in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime
> > > has a better support for the pod concept so we chose it to implement
> > > capsule. A caveat is that CRI requires a CNI plugin for the
> > > networking, so we need to implement a CNI plugin for Zun (called zun-
> > > cni). The role of CNI plugin is similar as kuryr-libnetwork that we
> > > are using for Docker except it implements a different networking
> > > model (CNI). I summaries it as below:
> > 
> > Hi,
> > 
> > I noticed that Zun's CNI plugin [1] is basically a simplified version
> > of kuryr-kubernetes code. While it's totally fine you've copied that, I
> > wonder what modifications had been made to make it suit Zun? Is there a
> > chance to converge this to make Zun use kuryr-kubernetes directly so
> > that we won't develop two versions of that code in parallel?
> 
> Right. I also investigated the possibilities of reusing the kuryr-
> kubernetes codebase as well. Definitely, some codes are common among
> two projects. If we can move the common code to a library (i.e.
> kuryr-lib), Zun should be able to directly consume the code. In
> particular, I am interesting to directly consume the CNI binding code
> (kuryr_kubernetes/cni/binding/) and the VIF versioned object
> (kuryr_kubernetes/objects).
> 
> Most parts of kuryr-kubernetes code is coupling with the "list-and-
> watch" logic against k8s API. Zun is not able to reuse those part of
> code. However, I do advocate to move all the common code to kuryr-lib 
> so Zun can reuse it whenever it is appropriate.

Uhm, moving more code into kuryr.lib is something Kuryr team would like
to avoid. Our tendency is rather to stop depending from it, as kuryr-
kubernetes being a CNI plugin is normally consumed as a container image
and having any dependencies is a burden there.

That's why I was asking about modifications to kuryr-daemon code that
Zun required - to see if we can modify kuryr-daemon to be pluggable
enough to be consumed by Zun directly.

> > Thanks,
> > Michał
> > 
> > [1] https://github.com/openstack/zun/tree/master/zun/cni
> > 
> > > +--------------+------------------------+---------------+
> > > | Concept      | Container              | Capsule (Pod) |
> > > +--------------+------------------------+---------------+
> > > | API endpoint | /containers            | /capsules     |
> > > | Engine       | Docker                 | CRI runtime   |
> > > | Network      | kuryr-libnetwork (CNM) | zun-cni (CNI) |
> > > +--------------+------------------------+---------------+
> > > 
> > > Typically, a CRI runtime works well with Kata Container which
> > > provides hypervisor-based isolation for neighboring containers in the
> > > same node. As a result, it is secure to consolidate pods from
> > > different tenants into a single node which increases the resource
> > > utilization. For deployment, a typical stack looks like below:
> > > 
> > > +----------------------------------------------+
> > > | k8s control plane                            |
> > > +----------------------------------------------+
> > > | Virtual Kubelet (OpenStack provider)         |
> > > +----------------------------------------------+
> > > | OpenStack control plane (Zun, Neutron, etc.) |
> > > +----------------------------------------------+
> > > | OpenStack data plane                         |
> > > | (Zun compute agent, Neutron OVS agent, etc.) |
> > > +----------------------------------------------+
> > > | Containerd (with CRI plugin)                 |
> > > +----------------------------------------------+
> > > | Kata Container                               |
> > > +----------------------------------------------+
> > > 
> > > In this stack, if a user creates a deployment or pod in k8s, the k8s
> > > scheduler will schedule the pod to the virtual node registered by
> > > Virtual Kubelet. Virtual Kubelet will pick up the pod and let the
> > > configured cloud provider to handle it. The cloud provider invokes
> > > Zun API to create a capsule. Upon receiving the API request to create
> > > a capsule, Zun scheduler will schedule the capsule to a compute node.
> > > The Zun compute agent in that node will provision the capsule using a
> > > CRI runtime (containerd in this example). The Zun-CRI runtime
> > > communication is done via a gRPC protocol through a unix socket. The
> > > CRI runtime will first create the pod in Kata Container (or runc as
> > > an alternative) that realizes the pod using a lightweight VM.
> > > Furthermore, the CRI runtime will use a CNI plugin, which is the zun-
> > > cni binary, to setup the network. The zun-cni binary is a thin
> > > executable that dispatches the CNI command to a daemon service called
> > > zun-cni-daemon. The community is via HTTP within localhost. The zun-
> > > cni-daemon will look up the Neutron port information from DB and
> > > perform the port binding.
> > > 
> > > In conclusion, starting from Ussuri, Zun adds support for CRI-
> > > compatible runtime. Zun uses CRI runtime to realize the concept of
> > > pod. Using this feature together with Virtual Kubelet and Kata
> > > Container, we can offer "serverless kubernetes pod" service which is
> > > comparable with AWS EKS with Fargate.
> > > 
> > > [1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime
> > > [2] https://github.com/virtual-kubelet/virtual-kubelet
> > > [3] https://github.com/virtual-kubelet/openstack-zun
> > > [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes-pods-using-amazon-eks-and-aws-fargate/
> > > [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-available/
> > 
> >