Open Stack

On Tue, 2020-03-24 at 09:16 -0400, Hongbin Lu wrote:
> 
> 
> On Tue, Mar 24, 2020 at 7:28 AM <mdulko at redhat.com> wrote:
> > On Mon, 2020-03-23 at 12:17 -0400, Hongbin Lu wrote:
> > > 
> > > 
> > > On Mon, Mar 23, 2020 at 11:48 AM <mdulko at redhat.com> wrote:
> > > > On Sun, 2020-03-22 at 13:28 -0400, Hongbin Lu wrote:
> > > > > Hi all,
> > > > > 
> > > > > As we are approaching the end of Ussuri cycle, I would like to take a
> > > > > chance to introduce a new feature the Zun team implemented in this
> > > > > cycle - CRI integration [1].
> > 
> > <snip!>
> > 
> > > > > Under the hook, a capsule is a podsandbox with one or more containers
> > > > > in a CRI runtime (i.e. containerd). Compared to Docker, a CRI runtime
> > > > > has a better support for the pod concept so we chose it to implement
> > > > > capsule. A caveat is that CRI requires a CNI plugin for the
> > > > > networking, so we need to implement a CNI plugin for Zun (called zun-
> > > > > cni). The role of CNI plugin is similar as kuryr-libnetwork that we
> > > > > are using for Docker except it implements a different networking
> > > > > model (CNI). I summaries it as below:
> > > > 
> > > > Hi,
> > > > 
> > > > I noticed that Zun's CNI plugin [1] is basically a simplified version
> > > > of kuryr-kubernetes code. While it's totally fine you've copied that, I
> > > > wonder what modifications had been made to make it suit Zun? Is there a
> > > > chance to converge this to make Zun use kuryr-kubernetes directly so
> > > > that we won't develop two versions of that code in parallel?
> > > 
> > > Right. I also investigated the possibilities of reusing the kuryr-
> > > kubernetes codebase as well. Definitely, some codes are common among
> > > two projects. If we can move the common code to a library (i.e.
> > > kuryr-lib), Zun should be able to directly consume the code. In
> > > particular, I am interesting to directly consume the CNI binding code
> > > (kuryr_kubernetes/cni/binding/) and the VIF versioned object
> > > (kuryr_kubernetes/objects).
> > > 
> > > Most parts of kuryr-kubernetes code is coupling with the "list-and-
> > > watch" logic against k8s API. Zun is not able to reuse those part of
> > > code. However, I do advocate to move all the common code to kuryr-lib 
> > > so Zun can reuse it whenever it is appropriate.
> > 
> > Uhm, moving more code into kuryr.lib is something Kuryr team would like
> > to avoid. Our tendency is rather to stop depending from it, as kuryr-
> > kubernetes being a CNI plugin is normally consumed as a container image
> > and having any dependencies is a burden there.
> 
> Kuyur-lib is already a dependency for kuryr-kubernetes: 
> https://github.com/openstack/kuryr-kubernetes/blob/master/requirements.txt
>  . Do you mean kuryr-kubernetes is going to remove kuryr-lib as a
> dependency? And I don't quite get the "container image"
> justification. Could you explain more?

Hi! Sorry for late reply, I must have missed that email when reading
the list.

Our plan was to move the bits from kuryr-lib that we use directly into
kuryr-kubernetes. This is basically the segmetnation_type_drivers
module and some utility functions.

kuryr-kubernetes is built as part of OpenShift (OKD) platform and OKD's
buildsystem doesn't handle dependencies too well. The problem is that
if we'd start to depend more on kuryr-lib, we would probably need to
fork it too as part of openshift's GitHub (note that we maintain fork
kuryr-kubernetes at github.com/openshift/kuryr-kubernetes with the bits
related to OKD builds).

> > That's why I was asking about modifications to kuryr-daemon code that
> > Zun required - to see if we can modify kuryr-daemon to be pluggable
> > enough to be consumed by Zun directly.
> 
> In theory, you can refactor the code and make it pluggable. Suppose you are able to do that, I would still suggest to move the whole framework out as a library. That is a prerequisite for Zun (or any other projects) to consume it, right?

Right, without a shared library it's not an ideal solution, but library
is problematic for sure… Would it be possible for Zun to just run
kuryr-daemon directly from a kuryr-kubernetes release?

> > > > Thanks,
> > > > Michał
> > > > 
> > > > [1] https://github.com/openstack/zun/tree/master/zun/cni
> > > > 
> > > > > +--------------+------------------------+---------------+
> > > > > | Concept      | Container              | Capsule (Pod) |
> > > > > +--------------+------------------------+---------------+
> > > > > | API endpoint | /containers            | /capsules     |
> > > > > | Engine       | Docker                 | CRI runtime   |
> > > > > | Network      | kuryr-libnetwork (CNM) | zun-cni (CNI) |
> > > > > +--------------+------------------------+---------------+
> > > > > 
> > > > > Typically, a CRI runtime works well with Kata Container which
> > > > > provides hypervisor-based isolation for neighboring containers in the
> > > > > same node. As a result, it is secure to consolidate pods from
> > > > > different tenants into a single node which increases the resource
> > > > > utilization. For deployment, a typical stack looks like below:
> > > > > 
> > > > > +----------------------------------------------+
> > > > > | k8s control plane                            |
> > > > > +----------------------------------------------+
> > > > > | Virtual Kubelet (OpenStack provider)         |
> > > > > +----------------------------------------------+
> > > > > | OpenStack control plane (Zun, Neutron, etc.) |
> > > > > +----------------------------------------------+
> > > > > | OpenStack data plane                         |
> > > > > | (Zun compute agent, Neutron OVS agent, etc.) |
> > > > > +----------------------------------------------+
> > > > > | Containerd (with CRI plugin)                 |
> > > > > +----------------------------------------------+
> > > > > | Kata Container                               |
> > > > > +----------------------------------------------+
> > > > > 
> > > > > In this stack, if a user creates a deployment or pod in k8s, the k8s
> > > > > scheduler will schedule the pod to the virtual node registered by
> > > > > Virtual Kubelet. Virtual Kubelet will pick up the pod and let the
> > > > > configured cloud provider to handle it. The cloud provider invokes
> > > > > Zun API to create a capsule. Upon receiving the API request to create
> > > > > a capsule, Zun scheduler will schedule the capsule to a compute node.
> > > > > The Zun compute agent in that node will provision the capsule using a
> > > > > CRI runtime (containerd in this example). The Zun-CRI runtime
> > > > > communication is done via a gRPC protocol through a unix socket. The
> > > > > CRI runtime will first create the pod in Kata Container (or runc as
> > > > > an alternative) that realizes the pod using a lightweight VM.
> > > > > Furthermore, the CRI runtime will use a CNI plugin, which is the zun-
> > > > > cni binary, to setup the network. The zun-cni binary is a thin
> > > > > executable that dispatches the CNI command to a daemon service called
> > > > > zun-cni-daemon. The community is via HTTP within localhost. The zun-
> > > > > cni-daemon will look up the Neutron port information from DB and
> > > > > perform the port binding.
> > > > > 
> > > > > In conclusion, starting from Ussuri, Zun adds support for CRI-
> > > > > compatible runtime. Zun uses CRI runtime to realize the concept of
> > > > > pod. Using this feature together with Virtual Kubelet and Kata
> > > > > Container, we can offer "serverless kubernetes pod" service which is
> > > > > comparable with AWS EKS with Fargate.
> > > > > 
> > > > > [1] https://blueprints.launchpad.net/zun/+spec/add-support-cri-runtime
> > > > > [2] https://github.com/virtual-kubelet/virtual-kubelet
> > > > > [3] https://github.com/virtual-kubelet/openstack-zun
> > > > > [4] https://aws.amazon.com/about-aws/whats-new/2019/12/run-serverless-kubernetes-pods-using-amazon-eks-and-aws-fargate/
> > > > > [5] https://aws.amazon.com/blogs/aws/amazon-eks-on-aws-fargate-now-generally-available/
> > > > 
> > > > 
> > 
> > 
> > 
> >