[openstack-dev] [kuryr] Kuryr-Kubernetes devstack proposal for testing

Antoni Segura Puimedon celebdor at gmail.com
Tue Sep 6 19:52:00 UTC 2016


Hi Kuryrs!

Yesterday in the meeting we discussed about the need to design a way for the
functional testing to happen for the Kuryr-Kubernetes integration. I studied
the possibilities today and came up with the following proposal (After the
proposal I'll put more detailed explanations).

Prerequisites
=============
Usual services:
* Neutron and its agents (LBaaSv2 included)
* Keystone

Devstack plugin
===============

* Installs Docker just like kuryr-libnetwork's plugin
* Installs Docker compose
* Pulls gcr.io/google_containers/hyperkube-amd64:v1.3.6
* Pulls quay.io/coreos/etcd:v3.0.7
* Runs in --net=host:
  * coreos/etcd
  * google_containers/hyperkube /setup-files.sh
  * google_containers/hyperkube /hyperkube apiserver
  * google_containers/hyperkube /hyperkube controller-manager
  * google_containers/hyperkube /hyperkube scheduler
  * google_containers/hyperkube /hyperkube kubelet with the Kuryr CNI driver
    mounted as a volume
* Starts the Kuryr Watcher pointing to the apiserver as a devstack service

After the steps above, we can use a python kubernetes client to run the tests.

apiserver
---------
This is what the Kuryr watcher connects to. Example parameters for running the
hyperkube container::

    --service-cluster-ip-range=10.0.0.1/24 \
    --insecure-bind-address=0.0.0.0 \
    --insecure-port=8080 \
    --etcd-servers=http://${LOCAL_IP}:2379 \
    --admission-control=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota
\
    --client-ca-file=/srv/kubernetes/ca.crt \
    --basic-auth-file=/srv/kubernetes/basic_auth.csv \
    --min-request-timeout=300 \
    --tls-cert-file=/srv/kubernetes/server.cert \
    --tls-private-key-file=/srv/kubernetes/server.key \
    --token-auth-file=/srv/kubernetes/known_tokens.csv \
    --allow-privileged=true \
    --v=2 \
    --logtostderr=true

controller-manager
------------------
It will be running the plugin for LoadBalancer service type in the future.
Example parameters for running the hyperkube container::

    --master=127.0.0.1:8080 \
    --service-account-private-key-file=/srv/kubernetes/server.key \
    --root-ca-file=/srv/kubernetes/ca.crt \
    --min-resync-period=3m \
    --v=2 \
    --logtostderr=true

scheduler
---------
To do the hard job of scheduling to the only kubelet. Example parameters for
running the hyperkube container::

      --master=127.0.0.1:8080 \
      --service-account-private-key-file=/srv/kubernetes/server.key \
      --root-ca-file=/srv/kubernetes/ca.crt \
      --min-resync-period=3m \
      --v=2 \
      --logtostderr=true

Kubelet
-------
It needs to run in privileged mode and with the following volumes::

    --volume=/:/rootfs:ro \
    --volume=/sys:/sys:ro \
    --volume=/var/lib/docker/:/var/lib/docker:rw \
    --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
    --volume=/var/run:/var/run:rw \
    --volume=/var/log/kuryr:/var/log/kuryr \
    --net=host \
    --privileged=true \
    --pid=host

It also needs the CNI driver, which should be mounted as a volume from the
current /opt/stack/kuryr-kubernetes source. However, the container is not
likely to have Python, so I propose to build a CNI binary with python-install
and mount just the binary.

The example parameters to run it would be::
    --allow-privileged=true \  # this we can probably omit for tests
    --api-servers="http://127.0.0.1:8080" \
    --v=2 \
    --address='0.0.0.0' \
    --enable-server \
    --containerized \
    --network-plugin=cni

Why Hyperkube and compose instead of minikube?
==============================================

Hyperkube provides us with more flexibility to run the building blocks that we
need for the integration, like not running kube-proxy. It also can run easily
without modification in the OpenStack CI jenkins worker with little resources.

Minikube spawns a Virtual Machine using Docker Machine. This means that it
would need more resources and make its Keystone/Neutron usage more complicated.
It could prossibly be hacked to use the Docker Machine generic SSH driver and
point it to the same machine, but I find it going to too much trouble compared
to the simplicity of the solution above.


Why Hyperkube and compose instead of just running kubernetes from src
=====================================================================
Kubernetes building would take a high amount of resources ~8GiB and more time
than pulling the hyperkube containers. However, this is a decision that we may
want to revisit once we start contributing the kuryr cloud provider to
Kubernetes (for the loadbalancer service type).

-----------------------
-----------------------
-----------------------

Please, all Kuryrs, feel welcome to dispute the proposal and claims above and
to propose alternatives. After a bit of discussion we can propose a blueprint
and start implementing.


Regards,

Toni



More information about the OpenStack-dev mailing list