[k8s][zun] Introduce a new feature in Ussuri - CRI integration
Hi all, As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1]. As known by people, Zun is an OpenStack Container service. It provides API for users to create and manage application containers in an OpenStack cloud. The main concepts in Zun are "Container" and "Capsule". A container is a single container, while a capsule is a group of co-located and co-scheduled containers (basically the same as k8s pod). Probably, the "Container" concept is more widely used. People can use the /containers API endpoint to create and manage a single container. Under the hook, a container is a Docker container in a compute node. What is special is that each Docker container is given a Neutron port so the container is connected to a tenant network in Neutron. Kuryr-libnetwork is the Docker network plugin we use to perform the Neutron port binding which basically connects the container to the virtual switch managed by Neutron. As mentioned before, the concept of "Capsule" in Zun is basically the same as pod in k8s. We introduced this concept mainly for k8s integration. Roughly speaking, the Zun-k8s integration is achieved by (i) registering a special node in k8s, (ii) watching the k8s API for pods being scheduled to this node and (iii) invoking Zun's /capsules API endpoint to create a capsule for each incoming pod. The (i) and (ii) is done by a CNCF sandbox project called Virtual Kubelet [2]. The (iii) is achieved by providing an OpenStack provider [3] for Virtual Kubelet. The special node registered by Virtual Kubelet is called a virtual node because the node doesn't physically exist. Pods being scheduled to the virtual node is basically offloaded from the current k8s cluster, eventually landed on an external platform such as an OpenStack cloud. In high level, what is offered to end-users is a "serverless kubernetes pod" [4]. This term basically means the ability to run pods on demand without planing the capacity (i.e. nodes) upfront. An example of that is AWS EKS on Fargate [5]. In comparison, the traditional approach is to create an entire k8s cluster upfront in order to run the workload. Let's give a simple example. Suppose you want to run a pod, the traditional approach is to provision a k8s cluster with a worker node. Then, run the pod on the worker node. In contract, the "serverless" approach is to create a k8s cluster without any worker node and the pod is offloaded to a cloud provider that provisions the pods at runtime. This approach works well for applications who have fluctuated workloads so it is hard to provision a cluster with the right size for them. Furthermore, from cloud provider's perspective, if all tenant users offloads their pods to the cloud, the cloud provider might be able to pack the workload better (i.e. with fewer physical nodes) thus saving cost. Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun-cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below: +--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+ Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below: +----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+ In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun-cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun-cni-daemon will look up the Neutron port information from DB and perform the port binding. In conclusion, starting from Ussuri, Zun adds support for CRI-compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate. [1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime <https://github.com/virtual-kubelet/openstack-zun> [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...
Hongbin Lu wrote:
[...] In conclusion, starting from Ussuri, Zun adds support for CRI-compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate.
That's very exciting integration work, Hongbin! Don't forget to mention it in your cycle-highlights, as I expect it to be picked up in Ussuri release communications :) -- Thierry
On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
Hi all,
As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1].
As known by people, Zun is an OpenStack Container service. It provides API for users to create and manage application containers in an OpenStack cloud. The main concepts in Zun are "Container" and "Capsule". A container is a single container, while a capsule is a group of co-located and co-scheduled containers (basically the same as k8s pod).
Probably, the "Container" concept is more widely used. People can use the /containers API endpoint to create and manage a single container. Under the hook, a container is a Docker container in a compute node. What is special is that each Docker container is given a Neutron port so the container is connected to a tenant network in Neutron. Kuryr- libnetwork is the Docker network plugin we use to perform the Neutron port binding which basically connects the container to the virtual switch managed by Neutron.
As mentioned before, the concept of "Capsule" in Zun is basically the same as pod in k8s. We introduced this concept mainly for k8s integration. Roughly speaking, the Zun-k8s integration is achieved by (i) registering a special node in k8s, (ii) watching the k8s API for pods being scheduled to this node and (iii) invoking Zun's /capsules API endpoint to create a capsule for each incoming pod. The (i) and (ii) is done by a CNCF sandbox project called Virtual Kubelet [2]. The (iii) is achieved by providing an OpenStack provider [3] for Virtual Kubelet. The special node registered by Virtual Kubelet is called a virtual node because the node doesn't physically exist. Pods being scheduled to the virtual node is basically offloaded from the current k8s cluster, eventually landed on an external platform such as an OpenStack cloud.
In high level, what is offered to end-users is a "serverless kubernetes pod" [4]. This term basically means the ability to run pods on demand without planing the capacity (i.e. nodes) upfront. An example of that is AWS EKS on Fargate [5]. In comparison, the traditional approach is to create an entire k8s cluster upfront in order to run the workload. Let's give a simple example. Suppose you want to run a pod, the traditional approach is to provision a k8s cluster with a worker node. Then, run the pod on the worker node. In contract, the "serverless" approach is to create a k8s cluster without any worker node and the pod is offloaded to a cloud provider that provisions the pods at runtime. This approach works well for applications who have fluctuated workloads so it is hard to provision a cluster with the right size for them. Furthermore, from cloud provider's perspective, if all tenant users offloads their pods to the cloud, the cloud provider might be able to pack the workload better (i.e. with fewer physical nodes) thus saving cost.
Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun- cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below:
Hi, I noticed that Zun's CNI plugin [1] is basically a simplified version of kuryr-kubernetes code. While it's totally fine you've copied that, I wonder what modifications had been made to make it suit Zun? Is there a chance to converge this to make Zun use kuryr-kubernetes directly so that we won't develop two versions of that code in parallel? Thanks, Michał [1] https://github.com/openstack/zun/tree/master/zun/cni
+--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+
Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below:
+----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+
In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun- cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun- cni-daemon will look up the Neutron port information from DB and perform the port binding.
In conclusion, starting from Ussuri, Zun adds support for CRI- compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate.
[1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...
On Mon, Mar 23, 2020 at 11:48 AM <mdulko@redhat.com> wrote:
On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
Hi all,
As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1].
As known by people, Zun is an OpenStack Container service. It provides API for users to create and manage application containers in an OpenStack cloud. The main concepts in Zun are "Container" and "Capsule". A container is a single container, while a capsule is a group of co-located and co-scheduled containers (basically the same as k8s pod).
Probably, the "Container" concept is more widely used. People can use the /containers API endpoint to create and manage a single container. Under the hook, a container is a Docker container in a compute node. What is special is that each Docker container is given a Neutron port so the container is connected to a tenant network in Neutron. Kuryr- libnetwork is the Docker network plugin we use to perform the Neutron port binding which basically connects the container to the virtual switch managed by Neutron.
As mentioned before, the concept of "Capsule" in Zun is basically the same as pod in k8s. We introduced this concept mainly for k8s integration. Roughly speaking, the Zun-k8s integration is achieved by (i) registering a special node in k8s, (ii) watching the k8s API for pods being scheduled to this node and (iii) invoking Zun's /capsules API endpoint to create a capsule for each incoming pod. The (i) and (ii) is done by a CNCF sandbox project called Virtual Kubelet [2]. The (iii) is achieved by providing an OpenStack provider [3] for Virtual Kubelet. The special node registered by Virtual Kubelet is called a virtual node because the node doesn't physically exist. Pods being scheduled to the virtual node is basically offloaded from the current k8s cluster, eventually landed on an external platform such as an OpenStack cloud.
In high level, what is offered to end-users is a "serverless kubernetes pod" [4]. This term basically means the ability to run pods on demand without planing the capacity (i.e. nodes) upfront. An example of that is AWS EKS on Fargate [5]. In comparison, the traditional approach is to create an entire k8s cluster upfront in order to run the workload. Let's give a simple example. Suppose you want to run a pod, the traditional approach is to provision a k8s cluster with a worker node. Then, run the pod on the worker node. In contract, the "serverless" approach is to create a k8s cluster without any worker node and the pod is offloaded to a cloud provider that provisions the pods at runtime. This approach works well for applications who have fluctuated workloads so it is hard to provision a cluster with the right size for them. Furthermore, from cloud provider's perspective, if all tenant users offloads their pods to the cloud, the cloud provider might be able to pack the workload better (i.e. with fewer physical nodes) thus saving cost.
Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun- cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below:
Hi,
I noticed that Zun's CNI plugin [1] is basically a simplified version of kuryr-kubernetes code. While it's totally fine you've copied that, I wonder what modifications had been made to make it suit Zun? Is there a chance to converge this to make Zun use kuryr-kubernetes directly so that we won't develop two versions of that code in parallel?
Right. I also investigated the possibilities of reusing the kuryr-kubernetes codebase as well. Definitely, some codes are common among two projects. If we can move the common code to a library (i.e. kuryr-lib), Zun should be able to directly consume the code. In particular, I am interesting to directly consume the CNI binding code (kuryr_kubernetes/cni/binding/) and the VIF versioned object (kuryr_kubernetes/objects). Most parts of kuryr-kubernetes code is coupling with the "list-and-watch" logic against k8s API. Zun is not able to reuse those part of code. However, I do advocate to move all the common code to kuryr-lib so Zun can reuse it whenever it is appropriate.
Thanks, Michał
[1] https://github.com/openstack/zun/tree/master/zun/cni
+--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+
Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below:
+----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+
In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun- cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun- cni-daemon will look up the Neutron port information from DB and perform the port binding.
In conclusion, starting from Ussuri, Zun adds support for CRI- compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate.
[1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...
On Mon, 2020-03-23 at 12:17 -0400, Hongbin Lu wrote:
On Mon, Mar 23, 2020 at 11:48 AM <mdulko@redhat.com> wrote:
On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
Hi all,
As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1].
<snip!>
Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun- cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below:
Hi,
I noticed that Zun's CNI plugin [1] is basically a simplified version of kuryr-kubernetes code. While it's totally fine you've copied that, I wonder what modifications had been made to make it suit Zun? Is there a chance to converge this to make Zun use kuryr-kubernetes directly so that we won't develop two versions of that code in parallel?
Right. I also investigated the possibilities of reusing the kuryr- kubernetes codebase as well. Definitely, some codes are common among two projects. If we can move the common code to a library (i.e. kuryr-lib), Zun should be able to directly consume the code. In particular, I am interesting to directly consume the CNI binding code (kuryr_kubernetes/cni/binding/) and the VIF versioned object (kuryr_kubernetes/objects).
Most parts of kuryr-kubernetes code is coupling with the "list-and- watch" logic against k8s API. Zun is not able to reuse those part of code. However, I do advocate to move all the common code to kuryr-lib so Zun can reuse it whenever it is appropriate.
Uhm, moving more code into kuryr.lib is something Kuryr team would like to avoid. Our tendency is rather to stop depending from it, as kuryr- kubernetes being a CNI plugin is normally consumed as a container image and having any dependencies is a burden there. That's why I was asking about modifications to kuryr-daemon code that Zun required - to see if we can modify kuryr-daemon to be pluggable enough to be consumed by Zun directly.
Thanks, Michał
[1] https://github.com/openstack/zun/tree/master/zun/cni
+--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+
Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below:
+----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+
In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun- cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun- cni-daemon will look up the Neutron port information from DB and perform the port binding.
In conclusion, starting from Ussuri, Zun adds support for CRI- compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate.
[1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...
On Tue, Mar 24, 2020 at 7:28 AM <mdulko@redhat.com> wrote:
On Mon, 2020-03-23 at 12:17 -0400, Hongbin Lu wrote:
On Mon, Mar 23, 2020 at 11:48 AM <mdulko@redhat.com> wrote:
On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
Hi all,
As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1].
<snip!>
Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun- cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below:
Hi,
I noticed that Zun's CNI plugin [1] is basically a simplified version of kuryr-kubernetes code. While it's totally fine you've copied that, I wonder what modifications had been made to make it suit Zun? Is there a chance to converge this to make Zun use kuryr-kubernetes directly so that we won't develop two versions of that code in parallel?
Right. I also investigated the possibilities of reusing the kuryr- kubernetes codebase as well. Definitely, some codes are common among two projects. If we can move the common code to a library (i.e. kuryr-lib), Zun should be able to directly consume the code. In particular, I am interesting to directly consume the CNI binding code (kuryr_kubernetes/cni/binding/) and the VIF versioned object (kuryr_kubernetes/objects).
Most parts of kuryr-kubernetes code is coupling with the "list-and- watch" logic against k8s API. Zun is not able to reuse those part of code. However, I do advocate to move all the common code to kuryr-lib so Zun can reuse it whenever it is appropriate.
Uhm, moving more code into kuryr.lib is something Kuryr team would like to avoid. Our tendency is rather to stop depending from it, as kuryr- kubernetes being a CNI plugin is normally consumed as a container image and having any dependencies is a burden there.
Kuyur-lib is already a dependency for kuryr-kubernetes: https://github.com/openstack/kuryr-kubernetes/blob/master/requirements.txt . Do you mean kuryr-kubernetes is going to remove kuryr-lib as a dependency? And I don't quite get the "container image" justification. Could you explain more?
That's why I was asking about modifications to kuryr-daemon code that Zun required - to see if we can modify kuryr-daemon to be pluggable enough to be consumed by Zun directly.
In theory, you can refactor the code and make it pluggable. Suppose you are able to do that, I would still suggest to move the whole framework out as a library. That is a prerequisite for Zun (or any other projects) to consume it, right?
Thanks, Michał
[1] https://github.com/openstack/zun/tree/master/zun/cni
+--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+
Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below:
+----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+
In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun- cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun- cni-daemon will look up the Neutron port information from DB and perform the port binding.
In conclusion, starting from Ussuri, Zun adds support for CRI- compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate.
[1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...
On Tue, 2020-03-24 at 09:16 -0400, Hongbin Lu wrote:
On Tue, Mar 24, 2020 at 7:28 AM <mdulko@redhat.com> wrote:
On Mon, 2020-03-23 at 12:17 -0400, Hongbin Lu wrote:
On Mon, Mar 23, 2020 at 11:48 AM <mdulko@redhat.com> wrote:
On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
Hi all,
As we are approaching the end of Ussuri cycle, I would like to take a chance to introduce a new feature the Zun team implemented in this cycle - CRI integration [1].
<snip!>
Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun- cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below:
Hi,
I noticed that Zun's CNI plugin [1] is basically a simplified version of kuryr-kubernetes code. While it's totally fine you've copied that, I wonder what modifications had been made to make it suit Zun? Is there a chance to converge this to make Zun use kuryr-kubernetes directly so that we won't develop two versions of that code in parallel?
Right. I also investigated the possibilities of reusing the kuryr- kubernetes codebase as well. Definitely, some codes are common among two projects. If we can move the common code to a library (i.e. kuryr-lib), Zun should be able to directly consume the code. In particular, I am interesting to directly consume the CNI binding code (kuryr_kubernetes/cni/binding/) and the VIF versioned object (kuryr_kubernetes/objects).
Most parts of kuryr-kubernetes code is coupling with the "list-and- watch" logic against k8s API. Zun is not able to reuse those part of code. However, I do advocate to move all the common code to kuryr-lib so Zun can reuse it whenever it is appropriate.
Uhm, moving more code into kuryr.lib is something Kuryr team would like to avoid. Our tendency is rather to stop depending from it, as kuryr- kubernetes being a CNI plugin is normally consumed as a container image and having any dependencies is a burden there.
Kuyur-lib is already a dependency for kuryr-kubernetes: https://github.com/openstack/kuryr-kubernetes/blob/master/requirements.txt . Do you mean kuryr-kubernetes is going to remove kuryr-lib as a dependency? And I don't quite get the "container image" justification. Could you explain more?
Hi! Sorry for late reply, I must have missed that email when reading the list. Our plan was to move the bits from kuryr-lib that we use directly into kuryr-kubernetes. This is basically the segmetnation_type_drivers module and some utility functions. kuryr-kubernetes is built as part of OpenShift (OKD) platform and OKD's buildsystem doesn't handle dependencies too well. The problem is that if we'd start to depend more on kuryr-lib, we would probably need to fork it too as part of openshift's GitHub (note that we maintain fork kuryr-kubernetes at github.com/openshift/kuryr-kubernetes with the bits related to OKD builds).
That's why I was asking about modifications to kuryr-daemon code that Zun required - to see if we can modify kuryr-daemon to be pluggable enough to be consumed by Zun directly.
In theory, you can refactor the code and make it pluggable. Suppose you are able to do that, I would still suggest to move the whole framework out as a library. That is a prerequisite for Zun (or any other projects) to consume it, right?
Right, without a shared library it's not an ideal solution, but library is problematic for sure… Would it be possible for Zun to just run kuryr-daemon directly from a kuryr-kubernetes release?
Thanks, Michał
[1] https://github.com/openstack/zun/tree/master/zun/cni
+--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+
Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below:
+----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+
In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun- cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun- cni-daemon will look up the Neutron port information from DB and perform the port binding.
In conclusion, starting from Ussuri, Zun adds support for CRI- compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate.
[1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...
On Fri, Apr 3, 2020 at 1:01 PM <mdulko@redhat.com> wrote:
On Tue, 2020-03-24 at 09:16 -0400, Hongbin Lu wrote:
On Tue, Mar 24, 2020 at 7:28 AM <mdulko@redhat.com> wrote:
On Mon, 2020-03-23 at 12:17 -0400, Hongbin Lu wrote:
On Mon, Mar 23, 2020 at 11:48 AM <mdulko@redhat.com> wrote:
On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
Hi all,
As we are approaching the end of Ussuri cycle, I would like to
chance to introduce a new feature the Zun team implemented in
cycle - CRI integration [1].
<snip!>
Under the hook, a capsule is a podsandbox with one or more containers in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime has a better support for the pod concept so we chose it to implement capsule. A caveat is that CRI requires a CNI plugin for the networking, so we need to implement a CNI plugin for Zun (called zun- cni). The role of CNI plugin is similar as kuryr-libnetwork that we are using for Docker except it implements a different networking model (CNI). I summaries it as below:
Hi,
I noticed that Zun's CNI plugin [1] is basically a simplified version of kuryr-kubernetes code. While it's totally fine you've copied
wonder what modifications had been made to make it suit Zun? Is
take a this that, I there a
chance to converge this to make Zun use kuryr-kubernetes directly so that we won't develop two versions of that code in parallel?
Right. I also investigated the possibilities of reusing the kuryr- kubernetes codebase as well. Definitely, some codes are common among two projects. If we can move the common code to a library (i.e. kuryr-lib), Zun should be able to directly consume the code. In particular, I am interesting to directly consume the CNI binding code (kuryr_kubernetes/cni/binding/) and the VIF versioned object (kuryr_kubernetes/objects).
Most parts of kuryr-kubernetes code is coupling with the "list-and- watch" logic against k8s API. Zun is not able to reuse those part of code. However, I do advocate to move all the common code to kuryr-lib so Zun can reuse it whenever it is appropriate.
Uhm, moving more code into kuryr.lib is something Kuryr team would like to avoid. Our tendency is rather to stop depending from it, as kuryr- kubernetes being a CNI plugin is normally consumed as a container image and having any dependencies is a burden there.
Kuyur-lib is already a dependency for kuryr-kubernetes:
https://github.com/openstack/kuryr-kubernetes/blob/master/requirements.txt
. Do you mean kuryr-kubernetes is going to remove kuryr-lib as a dependency? And I don't quite get the "container image" justification. Could you explain more?
Hi! Sorry for late reply, I must have missed that email when reading the list.
Our plan was to move the bits from kuryr-lib that we use directly into kuryr-kubernetes. This is basically the segmetnation_type_drivers module and some utility functions.
kuryr-kubernetes is built as part of OpenShift (OKD) platform and OKD's buildsystem doesn't handle dependencies too well. The problem is that if we'd start to depend more on kuryr-lib, we would probably need to fork it too as part of openshift's GitHub (note that we maintain fork kuryr-kubernetes at github.com/openshift/kuryr-kubernetes with the bits related to OKD builds).
Right. In this case, adding a new library dependency is not ideal from your perspective because it increases the maintenance burden from your downstream. From Zun's perspective, this is not ideal because we are not able to directly reuse the common code (which is primarily the binding module). This is the reason we ends up with two copies of some codes.
That's why I was asking about modifications to kuryr-daemon code that Zun required - to see if we can modify kuryr-daemon to be pluggable enough to be consumed by Zun directly.
In theory, you can refactor the code and make it pluggable. Suppose you are able to do that, I would still suggest to move the whole framework out as a library. That is a prerequisite for Zun (or any other projects) to consume it, right?
Right, without a shared library it's not an ideal solution, but library is problematic for sure… Would it be possible for Zun to just run kuryr-daemon directly from a kuryr-kubernetes release?
First, I am not sure if it is possible from technical perspective. Second, to address the problem, an alternative approach is to have kuryr-kubernetes run zun-cni-deamon directly from a Zun release. In theory, we can make Zun pluggable so other projects can extend it. The first approach (Zun depends on Kuryr-kubernetes) is not ideal for Zun because Zun will have a heavy dependency on an external service. The second approach (Kuryr-kubernetes depends on Zun) is not ideal for Kuryr-kubernetes under the same reason. I think there is a middle ground that can balance the benefits of both projects. For example, refactoring the common code into a shared library would be such option.
Thanks, Michał
[1] https://github.com/openstack/zun/tree/master/zun/cni
+--------------+------------------------+---------------+ | Concept | Container | Capsule (Pod) | +--------------+------------------------+---------------+ | API endpoint | /containers | /capsules | | Engine | Docker | CRI runtime | | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | +--------------+------------------------+---------------+
Typically, a CRI runtime works well with Kata Container which provides hypervisor-based isolation for neighboring containers in the same node. As a result, it is secure to consolidate pods from different tenants into a single node which increases the resource utilization. For deployment, a typical stack looks like below:
+----------------------------------------------+ | k8s control plane | +----------------------------------------------+ | Virtual Kubelet (OpenStack provider) | +----------------------------------------------+ | OpenStack control plane (Zun, Neutron, etc.) | +----------------------------------------------+ | OpenStack data plane | | (Zun compute agent, Neutron OVS agent, etc.) | +----------------------------------------------+ | Containerd (with CRI plugin) | +----------------------------------------------+ | Kata Container | +----------------------------------------------+
In this stack, if a user creates a deployment or pod in k8s, the k8s scheduler will schedule the pod to the virtual node registered by Virtual Kubelet. Virtual Kubelet will pick up the pod and let the configured cloud provider to handle it. The cloud provider invokes Zun API to create a capsule. Upon receiving the API request to create a capsule, Zun scheduler will schedule the capsule to a compute node. The Zun compute agent in that node will provision the capsule using a CRI runtime (containerd in this example). The Zun-CRI runtime communication is done via a gRPC protocol through a unix socket. The CRI runtime will first create the pod in Kata Container (or runc as an alternative) that realizes the pod using a lightweight VM. Furthermore, the CRI runtime will use a CNI plugin, which is the zun- cni binary, to setup the network. The zun-cni binary is a thin executable that dispatches the CNI command to a daemon service called zun-cni-daemon. The community is via HTTP within localhost. The zun- cni-daemon will look up the Neutron port information from DB and perform the port binding.
In conclusion, starting from Ussuri, Zun adds support for CRI- compatible runtime. Zun uses CRI runtime to realize the concept of pod. Using this feature together with Virtual Kubelet and Kata Container, we can offer "serverless kubernetes pod" service which is comparable with AWS EKS with Fargate.
[1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime [2] https://github.com/virtual-kubelet/virtual-kubelet [3] https://github.com/virtual-kubelet/openstack-zun [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...
On Fri, 2020-04-03 at 20:29 -0400, Hongbin Lu wrote:
On Fri, Apr 3, 2020 at 1:01 PM <mdulko@redhat.com> wrote:
On Tue, 2020-03-24 at 09:16 -0400, Hongbin Lu wrote:
On Tue, Mar 24, 2020 at 7:28 AM <mdulko@redhat.com> wrote:
On Mon, 2020-03-23 at 12:17 -0400, Hongbin Lu wrote:
On Mon, Mar 23, 2020 at 11:48 AM <mdulko@redhat.com> wrote:
On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote: > Hi all, > > As we are approaching the end of Ussuri cycle, I would like to take a > chance to introduce a new feature the Zun team implemented in this > cycle - CRI integration [1].
<snip!>
> Under the hook, a capsule is a podsandbox with one or more containers > in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime > has a better support for the pod concept so we chose it to implement > capsule. A caveat is that CRI requires a CNI plugin for the > networking, so we need to implement a CNI plugin for Zun (called zun- > cni). The role of CNI plugin is similar as kuryr-libnetwork that we > are using for Docker except it implements a different networking > model (CNI). I summaries it as below:
Hi,
I noticed that Zun's CNI plugin [1] is basically a simplified version of kuryr-kubernetes code. While it's totally fine you've copied that, I wonder what modifications had been made to make it suit Zun? Is there a chance to converge this to make Zun use kuryr-kubernetes directly so that we won't develop two versions of that code in parallel?
Right. I also investigated the possibilities of reusing the kuryr- kubernetes codebase as well. Definitely, some codes are common among two projects. If we can move the common code to a library (i.e. kuryr-lib), Zun should be able to directly consume the code. In particular, I am interesting to directly consume the CNI binding code (kuryr_kubernetes/cni/binding/) and the VIF versioned object (kuryr_kubernetes/objects).
Most parts of kuryr-kubernetes code is coupling with the "list-and- watch" logic against k8s API. Zun is not able to reuse those part of code. However, I do advocate to move all the common code to kuryr-lib so Zun can reuse it whenever it is appropriate.
Uhm, moving more code into kuryr.lib is something Kuryr team would like to avoid. Our tendency is rather to stop depending from it, as kuryr- kubernetes being a CNI plugin is normally consumed as a container image and having any dependencies is a burden there.
Kuyur-lib is already a dependency for kuryr-kubernetes: https://github.com/openstack/kuryr-kubernetes/blob/master/requirements.txt . Do you mean kuryr-kubernetes is going to remove kuryr-lib as a dependency? And I don't quite get the "container image" justification. Could you explain more?
Hi! Sorry for late reply, I must have missed that email when reading the list.
Our plan was to move the bits from kuryr-lib that we use directly into kuryr-kubernetes. This is basically the segmetnation_type_drivers module and some utility functions.
kuryr-kubernetes is built as part of OpenShift (OKD) platform and OKD's buildsystem doesn't handle dependencies too well. The problem is that if we'd start to depend more on kuryr-lib, we would probably need to fork it too as part of openshift's GitHub (note that we maintain fork kuryr-kubernetes at github.com/openshift/kuryr-kubernetes with the bits related to OKD builds).
Right. In this case, adding a new library dependency is not ideal from your perspective because it increases the maintenance burden from your downstream. From Zun's perspective, this is not ideal because we are not able to directly reuse the common code (which is primarily the binding module). This is the reason we ends up with two copies of some codes.
That's why I was asking about modifications to kuryr-daemon code that Zun required - to see if we can modify kuryr-daemon to be pluggable enough to be consumed by Zun directly.
In theory, you can refactor the code and make it pluggable. Suppose you are able to do that, I would still suggest to move the whole framework out as a library. That is a prerequisite for Zun (or any other projects) to consume it, right?
Right, without a shared library it's not an ideal solution, but library is problematic for sure… Would it be possible for Zun to just run kuryr-daemon directly from a kuryr-kubernetes release?
First, I am not sure if it is possible from technical perspective.
Second, to address the problem, an alternative approach is to have kuryr-kubernetes run zun-cni-deamon directly from a Zun release. In theory, we can make Zun pluggable so other projects can extend it. The first approach (Zun depends on Kuryr-kubernetes) is not ideal for Zun because Zun will have a heavy dependency on an external service. The second approach (Kuryr-kubernetes depends on Zun) is not ideal for Kuryr-kubernetes under the same reason.
I think there is a middle ground that can balance the benefits of both projects. For example, refactoring the common code into a shared library would be such option.
Well, I got to agree with you. I'll check what we can do to have the kuryr.lib built on the OpenShift side too and see if it's feasible. Unfortunately for sure that's not something that can happen in Ussuri timeframe.
Thanks, Michał
[1] https://github.com/openstack/zun/tree/master/zun/cni
> +--------------+------------------------+---------------+ > | Concept | Container | Capsule (Pod) | > +--------------+------------------------+---------------+ > | API endpoint | /containers | /capsules | > | Engine | Docker | CRI runtime | > | Network | kuryr-libnetwork (CNM) | zun-cni (CNI) | > +--------------+------------------------+---------------+ > > Typically, a CRI runtime works well with Kata Container which > provides hypervisor-based isolation for neighboring containers in the > same node. As a result, it is secure to consolidate pods from > different tenants into a single node which increases the resource > utilization. For deployment, a typical stack looks like below: > > +----------------------------------------------+ > | k8s control plane | > +----------------------------------------------+ > | Virtual Kubelet (OpenStack provider) | > +----------------------------------------------+ > | OpenStack control plane (Zun, Neutron, etc.) | > +----------------------------------------------+ > | OpenStack data plane | > | (Zun compute agent, Neutron OVS agent, etc.) | > +----------------------------------------------+ > | Containerd (with CRI plugin) | > +----------------------------------------------+ > | Kata Container | > +----------------------------------------------+ > > In this stack, if a user creates a deployment or pod in k8s, the k8s > scheduler will schedule the pod to the virtual node registered by > Virtual Kubelet. Virtual Kubelet will pick up the pod and let the > configured cloud provider to handle it. The cloud provider invokes > Zun API to create a capsule. Upon receiving the API request to create > a capsule, Zun scheduler will schedule the capsule to a compute node. > The Zun compute agent in that node will provision the capsule using a > CRI runtime (containerd in this example). The Zun-CRI runtime > communication is done via a gRPC protocol through a unix socket. The > CRI runtime will first create the pod in Kata Container (or runc as > an alternative) that realizes the pod using a lightweight VM. > Furthermore, the CRI runtime will use a CNI plugin, which is the zun- > cni binary, to setup the network. The zun-cni binary is a thin > executable that dispatches the CNI command to a daemon service called > zun-cni-daemon. The community is via HTTP within localhost. The zun- > cni-daemon will look up the Neutron port information from DB and > perform the port binding. > > In conclusion, starting from Ussuri, Zun adds support for CRI- > compatible runtime. Zun uses CRI runtime to realize the concept of > pod. Using this feature together with Virtual Kubelet and Kata > Container, we can offer "serverless kubernetes pod" service which is > comparable with AWS EKS with Fargate. > > [1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime > [2] https://github.com/virtual-kubelet/virtual-kubelet > [3] https://github.com/virtual-kubelet/openstack-zun > [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes... > [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-ava...
participants (3)
-
Hongbin Lu
-
mdulko＠redhat.com
-
Thierry Carrez