On Tue, 2020-03-24 at 09:16 -0400, Hongbin Lu wrote:
On Tue, Mar 24, 2020 at 7:28 AM <mdulko@redhat.com> wrote:
On Mon, 2020-03-23 at 12:17 -0400, Hongbin Lu wrote:
On Mon, Mar 23, 2020 at 11:48 AM <mdulko@redhat.com> wrote:
On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
Hi all,
As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1].
<snip!>
Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun- cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below:
Hi,
I noticed that Zun's CNI plugin [1] is basically a simplified version of kuryr-kubernetes code. While it's totally fine you've copied that, I wonder what modifications had been made to make it suit Zun? Is there a chance to converge this to make Zun use kuryr-kubernetes directly so that we won't develop two versions of that code in parallel?
Right. I also investigated the possibilities of reusing the kuryr- kubernetes codebase as well. Definitely, some codes are common among two projects. If we can move the common code to a library (i.e. kuryr-lib), Zun should be able to directly consume the code. In particular, I am interesting to directly consume the CNI binding code (kuryr_kubernetes/cni/binding/) and the VIF versioned object (kuryr_kubernetes/objects).
Most parts of kuryr-kubernetes code is coupling with the "list-and- watch" logic against k8s API. Zun is not able to reuse those part of code. However, I do advocate to move all the common code to kuryr-lib so Zun can reuse it whenever it is appropriate.
Uhm, moving more code into kuryr.lib is something Kuryr team would like to avoid. Our tendency is rather to stop depending from it, as kuryr- kubernetes being a CNI plugin is normally consumed as a container image and having any dependencies is a burden there.
Kuyur-lib is already a dependency for kuryr-kubernetes: https://github.com/openstack/kuryr-kubernetes/blob/master/requirements.txt . Do you mean kuryr-kubernetes is going to remove kuryr-lib as a dependency? And I don't quite get the "container image" justification. Could you explain more?
Hi! Sorry for late reply, I must have missed that email when reading the list. Our plan was to move the bits from kuryr-lib that we use directly into kuryr-kubernetes. This is basically the segmetnation_type_drivers module and some utility functions. kuryr-kubernetes is built as part of OpenShift (OKD) platform and OKD's buildsystem doesn't handle dependencies too well. The problem is that if we'd start to depend more on kuryr-lib, we would probably need to fork it too as part of openshift's GitHub (note that we maintain fork kuryr-kubernetes at github.com/openshift/kuryr-kubernetes with the bits related to OKD builds).
That's why I was asking about modifications to kuryr-daemon code that Zun required - to see if we can modify kuryr-daemon to be pluggable enough to be consumed by Zun directly.
In theory, you can refactor the code and make it pluggable. Suppose you are able to do that, I would still suggest to move the whole framework out as a library. That is a prerequisite for Zun (or any other projects) to consume it, right?
Right, without a shared library it's not an ideal solution, but library is problematic for sure… Would it be possible for Zun to just run kuryr-daemon directly from a kuryr-kubernetes release?
Thanks, Michał
[1] https://github.com/openstack/zun/tree/master/zun/cni
+--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+
Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below:
+----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+
In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun- cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun- cni-daemon will look up the Neutron port information from DB and perform the port binding.
In conclusion, starting from Ussuri, Zun adds support for CRI- compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate.
[1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...