Open Stack

On Wed, 2021-10-27 at 16:00 +0000, Jason Anderson wrote:
> Michał, thank you very much for the reply!
> 
> > On Oct 27, 2021, at 2:54 AM, Michał Dulko <mdulko at redhat.com> wrote:
> > 
> > On Tue, 2021-10-26 at 22:27 +0000, Jason Anderson wrote:
> > > Hello all,
> > > 
> > > I’m interested in letting Neutron provide the network configuration
> > > frontend for networks realized on k8s kubelets. I have been reading a
> > > lot about kuryr-kubernetes and it looks like it fits the bill, but
> > > some of the older architecture diagrams I’ve seen indicate that OVS
> > > and the neutron-openvswitch-agent (or similar) must also run on the
> > > kubelet node. Is this still accurate? I am hoping to avoid this
> > > because my understanding is that running the OVS agent means giving
> > > the kubelet node access to RabbitMQ and potentially storing admin
> > > keystone creds on the node as well.
> > > 
> > > Can kuryr-kubernetes work without such an agent co-located?
> > 
> > Hi,
> > 
> > So short answer is - yes it can. And the long answer is that there are
> > some requirements for that to work.
> > 
> > It's called the nested mode [1] and currently we treat it as the major
> > way to run K8s with kuryr-kubernetes. The assumption is that the
> > Kubernetes nodes run as VMs on OpenStack and Kuryr services will run as
> > Pods on those nodes. Kuryr requires the main ports of the VMs to be
> > Neutron trunk ports and will create the ports for the Pods as subports
> > of these trunk ports. This removes the need for neutron-openvswitch-
> > agent to exist on the K8s node as Kuryr can bind such ports on its own.
> > 
> > The requirements are as follows:
> > * K8s nodes run as VMs on OpenStack.
> > * Trunk extension is enabled in Neutron.
> > * VMs have access to OpenStack API endpoints.
> > * You need Octavia to support K8s Services.
> > 
> > In terms of admin credentials - those should not be needed in nested
> > mode, just regular tenant credentials should be fine.
> > 
> > If your K8s nodes are required to be baremetal, then maybe using OVN as
> > a Neutron backend instead of OVS will solve the RabbitMQ problem? I
> > think you'll still need the ovn-controller to run on the K8s nodes to
> > bind the Neutron ports there. And I think this mode might actually
> > require admin credentials in order to attach ports to nodes.
> 
> I will have to explore this a bit more, I think. Running the nested configuration
> unfortunately will not work for me. For context, I’m exploring how to configure
> a “public” cluster where worker nodes are enrolled over a WAN. K8s can
> accommodate this better than OpenStack due to the drastically reduced attack
> surface on the kubelet compared to, e.g., a Nova compute node. Yet, OpenStack
> does have some very attractive “frontend” systems and interfaces (I count
> Neutron among these) and it’s attractive to somehow allow connectivity between
> VM instances launched on a “main” centrally-controlled cloud and container
> instances launched on the workers exposed over a WAN. (Edge computing)

Hm, so a mixed OpenStack-K8s edge setup, where edge sites are
Kubernetes deployments? We've took a look at some edge use cases with
Kuryr and one problem people see is that if an edge site becomes
disconnected from the main side, Kuryr will not allow creation of new
Pods and Services as it needs connection to Neutron and Octavia APIs
for that. If that's not a problem had you gave a thought into running
distributed compute nodes [1] as edge sites and then Kubernetes on top
of them? This architecture should be doable with Kuryr (probably with
minor changes).

> OVN may help if it can remove the need for RabbitMQ, which is probably the
> most difficult aspect to remove from OpenStack’s dependencies/assumptions,
> yet also one of the most pernicious from a security angle, as an untrusted
> worker node can easily corrupt the control plane.

It's just Kuryr which needs access to the credentials, so possibly you
should be able to isolate them, but I get the point, containers are
worse at isolation than VMs.

> Re: admin creds, maybe it is possible to carefully craft a role that only works
> for some Neutron operations and put that on the worker nodes. I will explore.

I think those settings [2] is what would require highest Neutron
permissions in baremetal case.

[1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/distributed_compute_node.html
[2] https://opendev.org/openstack/kuryr-kubernetes/src/branch/master/kuryr_kubernetes/controller/drivers/neutron_vif.py#L125-L127

> Cheers!
> > [1]
> > https://docs.openstack.org/kuryr-kubernetes/latest/nested_vlan_mode.html
> > 
> > Thanks,
> > Michał
> > 
> > > Thanks!
> > > Jason Anderson
> > > 
> > > ---
> > > 
> > > Chameleon DevOps Lead
> > > Department of Computer Science, University of Chicago
> > > Mathematics and Computer Science, Argonne National Laboratory
> > > 
> > 
> > 
> > 
>