Open Stack

Michał, thank you very much for the reply!

> On Oct 27, 2021, at 2:54 AM, Michał Dulko <mdulko at redhat.com> wrote:
> 
> On Tue, 2021-10-26 at 22:27 +0000, Jason Anderson wrote:
>> Hello all,
>> 
>> I’m interested in letting Neutron provide the network configuration
>> frontend for networks realized on k8s kubelets. I have been reading a
>> lot about kuryr-kubernetes and it looks like it fits the bill, but
>> some of the older architecture diagrams I’ve seen indicate that OVS
>> and the neutron-openvswitch-agent (or similar) must also run on the
>> kubelet node. Is this still accurate? I am hoping to avoid this
>> because my understanding is that running the OVS agent means giving
>> the kubelet node access to RabbitMQ and potentially storing admin
>> keystone creds on the node as well.
>> 
>> Can kuryr-kubernetes work without such an agent co-located?
> 
> Hi,
> 
> So short answer is - yes it can. And the long answer is that there are
> some requirements for that to work.
> 
> It's called the nested mode [1] and currently we treat it as the major
> way to run K8s with kuryr-kubernetes. The assumption is that the
> Kubernetes nodes run as VMs on OpenStack and Kuryr services will run as
> Pods on those nodes. Kuryr requires the main ports of the VMs to be
> Neutron trunk ports and will create the ports for the Pods as subports
> of these trunk ports. This removes the need for neutron-openvswitch-
> agent to exist on the K8s node as Kuryr can bind such ports on its own.
> 
> The requirements are as follows:
> * K8s nodes run as VMs on OpenStack.
> * Trunk extension is enabled in Neutron.
> * VMs have access to OpenStack API endpoints.
> * You need Octavia to support K8s Services.
> 
> In terms of admin credentials - those should not be needed in nested
> mode, just regular tenant credentials should be fine.
> 
> If your K8s nodes are required to be baremetal, then maybe using OVN as
> a Neutron backend instead of OVS will solve the RabbitMQ problem? I
> think you'll still need the ovn-controller to run on the K8s nodes to
> bind the Neutron ports there. And I think this mode might actually
> require admin credentials in order to attach ports to nodes.

I will have to explore this a bit more, I think. Running the nested configuration
unfortunately will not work for me. For context, I’m exploring how to configure
a “public” cluster where worker nodes are enrolled over a WAN. K8s can
accommodate this better than OpenStack due to the drastically reduced attack
surface on the kubelet compared to, e.g., a Nova compute node. Yet, OpenStack
does have some very attractive “frontend” systems and interfaces (I count
Neutron among these) and it’s attractive to somehow allow connectivity between
VM instances launched on a “main” centrally-controlled cloud and container
instances launched on the workers exposed over a WAN. (Edge computing)

OVN may help if it can remove the need for RabbitMQ, which is probably the
most difficult aspect to remove from OpenStack’s dependencies/assumptions,
yet also one of the most pernicious from a security angle, as an untrusted
worker node can easily corrupt the control plane.

Re: admin creds, maybe it is possible to carefully craft a role that only works
for some Neutron operations and put that on the worker nodes. I will explore.

Cheers!

> [1]
> https://docs.openstack.org/kuryr-kubernetes/latest/nested_vlan_mode.html
> 
> Thanks,
> Michał
> 
>> Thanks!
>> Jason Anderson
>> 
>> ---
>> 
>> Chameleon DevOps Lead
>> Department of Computer Science, University of Chicago
>> Mathematics and Computer Science, Argonne National Laboratory
>> 
> 
> 
>