[kuryr] Using kuryr-kubernetes CNI without neutron agent(s)?
Hello all, I’m interested in letting Neutron provide the network configuration frontend for networks realized on k8s kubelets. I have been reading a lot about kuryr-kubernetes and it looks like it fits the bill, but some of the older architecture diagrams I’ve seen indicate that OVS and the neutron-openvswitch-agent (or similar) must also run on the kubelet node. Is this still accurate? I am hoping to avoid this because my understanding is that running the OVS agent means giving the kubelet node access to RabbitMQ and potentially storing admin keystone creds on the node as well. Can kuryr-kubernetes work without such an agent co-located? Thanks! Jason Anderson --- Chameleon DevOps Lead Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory
On Tue, 2021-10-26 at 22:27 +0000, Jason Anderson wrote:
Hello all,
I’m interested in letting Neutron provide the network configuration frontend for networks realized on k8s kubelets. I have been reading a lot about kuryr-kubernetes and it looks like it fits the bill, but some of the older architecture diagrams I’ve seen indicate that OVS and the neutron-openvswitch-agent (or similar) must also run on the kubelet node. Is this still accurate? I am hoping to avoid this because my understanding is that running the OVS agent means giving the kubelet node access to RabbitMQ and potentially storing admin keystone creds on the node as well.
Can kuryr-kubernetes work without such an agent co-located?
Hi, So short answer is - yes it can. And the long answer is that there are some requirements for that to work. It's called the nested mode [1] and currently we treat it as the major way to run K8s with kuryr-kubernetes. The assumption is that the Kubernetes nodes run as VMs on OpenStack and Kuryr services will run as Pods on those nodes. Kuryr requires the main ports of the VMs to be Neutron trunk ports and will create the ports for the Pods as subports of these trunk ports. This removes the need for neutron-openvswitch- agent to exist on the K8s node as Kuryr can bind such ports on its own. The requirements are as follows: * K8s nodes run as VMs on OpenStack. * Trunk extension is enabled in Neutron. * VMs have access to OpenStack API endpoints. * You need Octavia to support K8s Services. In terms of admin credentials - those should not be needed in nested mode, just regular tenant credentials should be fine. If your K8s nodes are required to be baremetal, then maybe using OVN as a Neutron backend instead of OVS will solve the RabbitMQ problem? I think you'll still need the ovn-controller to run on the K8s nodes to bind the Neutron ports there. And I think this mode might actually require admin credentials in order to attach ports to nodes. [1] https://docs.openstack.org/kuryr-kubernetes/latest/nested_vlan_mode.html Thanks, Michał
Thanks! Jason Anderson
---
Chameleon DevOps Lead Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory
Michał, thank you very much for the reply!
On Oct 27, 2021, at 2:54 AM, Michał Dulko <mdulko@redhat.com> wrote:
On Tue, 2021-10-26 at 22:27 +0000, Jason Anderson wrote:
Hello all,
I’m interested in letting Neutron provide the network configuration frontend for networks realized on k8s kubelets. I have been reading a lot about kuryr-kubernetes and it looks like it fits the bill, but some of the older architecture diagrams I’ve seen indicate that OVS and the neutron-openvswitch-agent (or similar) must also run on the kubelet node. Is this still accurate? I am hoping to avoid this because my understanding is that running the OVS agent means giving the kubelet node access to RabbitMQ and potentially storing admin keystone creds on the node as well.
Can kuryr-kubernetes work without such an agent co-located?
Hi,
So short answer is - yes it can. And the long answer is that there are some requirements for that to work.
It's called the nested mode [1] and currently we treat it as the major way to run K8s with kuryr-kubernetes. The assumption is that the Kubernetes nodes run as VMs on OpenStack and Kuryr services will run as Pods on those nodes. Kuryr requires the main ports of the VMs to be Neutron trunk ports and will create the ports for the Pods as subports of these trunk ports. This removes the need for neutron-openvswitch- agent to exist on the K8s node as Kuryr can bind such ports on its own.
The requirements are as follows: * K8s nodes run as VMs on OpenStack. * Trunk extension is enabled in Neutron. * VMs have access to OpenStack API endpoints. * You need Octavia to support K8s Services.
In terms of admin credentials - those should not be needed in nested mode, just regular tenant credentials should be fine.
If your K8s nodes are required to be baremetal, then maybe using OVN as a Neutron backend instead of OVS will solve the RabbitMQ problem? I think you'll still need the ovn-controller to run on the K8s nodes to bind the Neutron ports there. And I think this mode might actually require admin credentials in order to attach ports to nodes.
I will have to explore this a bit more, I think. Running the nested configuration unfortunately will not work for me. For context, I’m exploring how to configure a “public” cluster where worker nodes are enrolled over a WAN. K8s can accommodate this better than OpenStack due to the drastically reduced attack surface on the kubelet compared to, e.g., a Nova compute node. Yet, OpenStack does have some very attractive “frontend” systems and interfaces (I count Neutron among these) and it’s attractive to somehow allow connectivity between VM instances launched on a “main” centrally-controlled cloud and container instances launched on the workers exposed over a WAN. (Edge computing) OVN may help if it can remove the need for RabbitMQ, which is probably the most difficult aspect to remove from OpenStack’s dependencies/assumptions, yet also one of the most pernicious from a security angle, as an untrusted worker node can easily corrupt the control plane. Re: admin creds, maybe it is possible to carefully craft a role that only works for some Neutron operations and put that on the worker nodes. I will explore. Cheers!
[1] https://docs.openstack.org/kuryr-kubernetes/latest/nested_vlan_mode.html
Thanks, Michał
Thanks! Jason Anderson
---
Chameleon DevOps Lead Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory
On Wed, 2021-10-27 at 16:00 +0000, Jason Anderson wrote:
Michał, thank you very much for the reply!
On Oct 27, 2021, at 2:54 AM, Michał Dulko <mdulko@redhat.com> wrote:
On Tue, 2021-10-26 at 22:27 +0000, Jason Anderson wrote:
Hello all,
I’m interested in letting Neutron provide the network configuration frontend for networks realized on k8s kubelets. I have been reading a lot about kuryr-kubernetes and it looks like it fits the bill, but some of the older architecture diagrams I’ve seen indicate that OVS and the neutron-openvswitch-agent (or similar) must also run on the kubelet node. Is this still accurate? I am hoping to avoid this because my understanding is that running the OVS agent means giving the kubelet node access to RabbitMQ and potentially storing admin keystone creds on the node as well.
Can kuryr-kubernetes work without such an agent co-located?
Hi,
So short answer is - yes it can. And the long answer is that there are some requirements for that to work.
It's called the nested mode [1] and currently we treat it as the major way to run K8s with kuryr-kubernetes. The assumption is that the Kubernetes nodes run as VMs on OpenStack and Kuryr services will run as Pods on those nodes. Kuryr requires the main ports of the VMs to be Neutron trunk ports and will create the ports for the Pods as subports of these trunk ports. This removes the need for neutron-openvswitch- agent to exist on the K8s node as Kuryr can bind such ports on its own.
The requirements are as follows: * K8s nodes run as VMs on OpenStack. * Trunk extension is enabled in Neutron. * VMs have access to OpenStack API endpoints. * You need Octavia to support K8s Services.
In terms of admin credentials - those should not be needed in nested mode, just regular tenant credentials should be fine.
If your K8s nodes are required to be baremetal, then maybe using OVN as a Neutron backend instead of OVS will solve the RabbitMQ problem? I think you'll still need the ovn-controller to run on the K8s nodes to bind the Neutron ports there. And I think this mode might actually require admin credentials in order to attach ports to nodes.
I will have to explore this a bit more, I think. Running the nested configuration unfortunately will not work for me. For context, I’m exploring how to configure a “public” cluster where worker nodes are enrolled over a WAN. K8s can accommodate this better than OpenStack due to the drastically reduced attack surface on the kubelet compared to, e.g., a Nova compute node. Yet, OpenStack does have some very attractive “frontend” systems and interfaces (I count Neutron among these) and it’s attractive to somehow allow connectivity between VM instances launched on a “main” centrally-controlled cloud and container instances launched on the workers exposed over a WAN. (Edge computing)
Hm, so a mixed OpenStack-K8s edge setup, where edge sites are Kubernetes deployments? We've took a look at some edge use cases with Kuryr and one problem people see is that if an edge site becomes disconnected from the main side, Kuryr will not allow creation of new Pods and Services as it needs connection to Neutron and Octavia APIs for that. If that's not a problem had you gave a thought into running distributed compute nodes [1] as edge sites and then Kubernetes on top of them? This architecture should be doable with Kuryr (probably with minor changes).
OVN may help if it can remove the need for RabbitMQ, which is probably the most difficult aspect to remove from OpenStack’s dependencies/assumptions, yet also one of the most pernicious from a security angle, as an untrusted worker node can easily corrupt the control plane.
It's just Kuryr which needs access to the credentials, so possibly you should be able to isolate them, but I get the point, containers are worse at isolation than VMs.
Re: admin creds, maybe it is possible to carefully craft a role that only works for some Neutron operations and put that on the worker nodes. I will explore.
I think those settings [2] is what would require highest Neutron permissions in baremetal case. [1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features... [2] https://opendev.org/openstack/kuryr-kubernetes/src/branch/master/kuryr_kuber...
Cheers!
[1] https://docs.openstack.org/kuryr-kubernetes/latest/nested_vlan_mode.html
Thanks, Michał
Thanks! Jason Anderson
---
Chameleon DevOps Lead Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory
Hi Michał, I continue to appreciate the information you are providing. I’ve been doing some more research into the landscape of systems and had a few follow-up questions. I’ve also left some clarifying remarks if you are interested. I’m currently evaluating OVN, haven’t used it before and there’s a bit of a learning curve ;) However it seems like it may solve a good part of the problem by removing RabbitMQ and reducing the privileges of the edge host w.r.t. network config. Now I’m looking at kuryr-kubernetes. 1. What is the difference between kuryr and kuryr-kubernetes? I have used kuryr-libnetwork before, in conjunction with kuryr-server (which I think is provided via the main kuryr project?). I am using Kolla Ansible so was spared some of the details on installation. I understand kuryr-libnetwork is basically “kuryr for Docker” while kuryr-kubernetes is “kuryr for K8s”, but that leaves me confused about what exactly the kuryr repo is. 2. A current idea is to have several edge “compute” nodes that will run a lightweight k8s kubelet such as k3s. OVN will provide networking to the edge nodes, controlled from the central site. I would then place kuryr-k8s-controller on the central site and kuryr-cni-daemon on all the edge nodes. My question is: could users create their own Neutron networks (w/ their own Geneve segment) and launch pods connected on that network, and have those pods effectively be isolated from other pods in the topology? As in, can k8s be told that pod A should launch on network A’, and pod B on network B’? Or is there an assumption that from Neutron’s perspective all pods are always on a single Neutron network? Cheers, and thanks! /Jason On Oct 27, 2021, at 12:03 PM, Michał Dulko <mdulko@redhat.com<mailto:mdulko@redhat.com>> wrote: Hm, so a mixed OpenStack-K8s edge setup, where edge sites are Kubernetes deployments? We've took a look at some edge use cases with Kuryr and one problem people see is that if an edge site becomes disconnected from the main side, Kuryr will not allow creation of new Pods and Services as it needs connection to Neutron and Octavia APIs for that. If that's not a problem had you gave a thought into running distributed compute nodes [1] as edge sites and then Kubernetes on top of them? This architecture should be doable with Kuryr (probably with minor changes). Sort of! I work in research infrastructure and we are building an IoT/edge testbed for computer science researchers who wish to do research in edge computing. It’s a bit mad science-y. We are buying and configuring relatively high-powered edge devices such as Raspberry Pis and Jetson Nanos and making them available for experimentation at a variety of sites. Separately, the platform supports any owner of a supported device to have it managed by the testbed (i.e., they can use our interfaces to launch containers on it and connect it logically to other devices / resources in the cloud.) Distributed compute node looks a bit too heavy for this highly dynamic use-case, but thank you for sharing. Anyways, one might ask why Neutron at all. I am hopeful we can get some interesting properties such as network isolation and the ability to bridge traffic from containers across other layer 2 links such as those provided by AL2S<https://internet2.edu/services/layer-2-service/>. OVN may help if it can remove the need for RabbitMQ, which is probably the most difficult aspect to remove from OpenStack’s dependencies/assumptions, yet also one of the most pernicious from a security angle, as an untrusted worker node can easily corrupt the control plane. It's just Kuryr which needs access to the credentials, so possibly you should be able to isolate them, but I get the point, containers are worse at isolation than VMs. I’m less worried about the mechanism for isolation on the host and more the amount of privileged information the host must keep secure, and the impact of that information being compromised. Because our experimental target system involves container engines maintained externally to the core site, the risk of compromise on the edge is high. I am searching for an architecture that greatly limits the blast radius of such a compromise. Currently if we use standard Neutron networking + Kuryr, we must give RabbitMQ credentials and others to the container engines on the edge, which papers such as http://seclab.cs.sunysb.edu/seclab/pubs/asiaccs16.pdf have documented as a trivial escalation path. For this reason, narrowing the scope of what state the edge hosts can influence on the core site is paramount. Re: admin creds, maybe it is possible to carefully craft a role that only works for some Neutron operations and put that on the worker nodes. I will explore. I think those settings [2] is what would require highest Neutron permissions in baremetal case. Thanks — so it will need to create and delete ports. This may be acceptable; without some additional API proxy layer for the edge hosts, a malicious edge host could create bogus ports and delete good ones, but that is a much smaller level of impact. I think we could create a role that only allowed such operations and generate per-host credentials. [1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features... [2] https://opendev.org/openstack/kuryr-kubernetes/src/branch/master/kuryr_kuber... Cheers! [1] https://docs.openstack.org/kuryr-kubernetes/latest/nested_vlan_mode.html Thanks, Michał Thanks! Jason Anderson --- Chameleon DevOps Lead Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory
On Fri, 2021-11-05 at 22:54 +0000, Jason Anderson wrote:
Hi Michał,
I continue to appreciate the information you are providing. I’ve been doing some more research into the landscape of systems and had a few follow-up questions. I’ve also left some clarifying remarks if you are interested.
I’m currently evaluating OVN, haven’t used it before and there’s a bit of a learning curve ;) However it seems like it may solve a good part of the problem by removing RabbitMQ and reducing the privileges of the edge host w.r.t. network config.
Now I’m looking at kuryr-kubernetes.
1. What is the difference between kuryr and kuryr-kubernetes? I have used kuryr-libnetwork before, in conjunction with kuryr-server (which I think is provided via the main kuryr project?). I am using Kolla Ansible so was spared some of the details on installation. I understand kuryr-libnetwork is basically “kuryr for Docker” while kuryr-kubernetes is “kuryr for K8s”, but that leaves me confused about what exactly the kuryr repo is.
In openstack/kuryr we have kuryr.lib module, which is hosting a few things shared by kuryr-libnetwork and kuryr-kubernetes. Nothing to worry about really. ;)
2. A current idea is to have several edge “compute” nodes that will run a lightweight k8s kubelet such as k3s. OVN will provide networking to the edge nodes, controlled from the central site. I would then place kuryr-k8s-controller on the central site and kuryr-cni-daemon on all the edge nodes. My question is: could users create their own Neutron networks (w/ their own Geneve segment) and launch pods connected on that network, and have those pods effectively be isolated from other pods in the topology? As in, can k8s be told that pod A should launch on network A’, and pod B on network B’? Or is there an assumption that from Neutron’s perspective all pods are always on a single Neutron network?
Ha, that might be a bit tough one. Basically you can easily set Kuryr to create separate subnets for each of the K8s namespaces, but then you'd need to rely on NetworkPolicies to isolate traffic between namespaces which might not exactly fit your multitenant model. The best way to implement whatever you need might be to write your own custom subnet driver [1] that would choose the subnet e.g. based on a pod or namespace annotation. If there's a clear use case behind it, I think we can include it into the upstream code too. [1] https://opendev.org/openstack/kuryr-kubernetes/src/branch/master/kuryr_kuber...
Cheers, and thanks! /Jason
On Oct 27, 2021, at 12:03 PM, Michał Dulko <mdulko@redhat.com> wrote:
Hm, so a mixed OpenStack-K8s edge setup, where edge sites are Kubernetes deployments? We've took a look at some edge use cases with Kuryr and one problem people see is that if an edge site becomes disconnected from the main side, Kuryr will not allow creation of new Pods and Services as it needs connection to Neutron and Octavia APIs for that. If that's not a problem had you gave a thought into running distributed compute nodes [1] as edge sites and then Kubernetes on top of them? This architecture should be doable with Kuryr (probably with minor changes).
Sort of! I work in research infrastructure and we are building an IoT/edge testbed for computer science researchers who wish to do research in edge computing. It’s a bit mad science-y. We are buying and configuring relatively high-powered edge devices such as Raspberry Pis and Jetson Nanos and making them available for experimentation at a variety of sites. Separately, the platform supports any owner of a supported device to have it managed by the testbed (i.e., they can use our interfaces to launch containers on it and connect it logically to other devices / resources in the cloud.)
Distributed compute node looks a bit too heavy for this highly dynamic use-case, but thank you for sharing.
Anyways, one might ask why Neutron at all. I am hopeful we can get some interesting properties such as network isolation and the ability to bridge traffic from containers across other layer 2 links such as those provided by AL2S.
OVN may help if it can remove the need for RabbitMQ, which is probably the most difficult aspect to remove from OpenStack’s dependencies/assumptions, yet also one of the most pernicious from a security angle, as an untrusted worker node can easily corrupt the control plane.
It's just Kuryr which needs access to the credentials, so possibly you should be able to isolate them, but I get the point, containers are worse at isolation than VMs.
I’m less worried about the mechanism for isolation on the host and more the amount of privileged information the host must keep secure, and the impact of that information being compromised. Because our experimental target system involves container engines maintained externally to the core site, the risk of compromise on the edge is high. I am searching for an architecture that greatly limits the blast radius of such a compromise. Currently if we use standard Neutron networking + Kuryr, we must give RabbitMQ credentials and others to the container engines on the edge, which papers such as http://seclab.cs.sunysb.edu/seclab/pubs/asiaccs16.pdf have documented as a trivial escalation path.
For this reason, narrowing the scope of what state the edge hosts can influence on the core site is paramount.
Re: admin creds, maybe it is possible to carefully craft a role that only works for some Neutron operations and put that on the worker nodes. I will explore.
I think those settings [2] is what would require highest Neutron permissions in baremetal case.
Thanks — so it will need to create and delete ports. This may be acceptable; without some additional API proxy layer for the edge hosts, a malicious edge host could create bogus ports and delete good ones, but that is a much smaller level of impact. I think we could create a role that only allowed such operations and generate per-host credentials.
[1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features... [2] https://opendev.org/openstack/kuryr-kubernetes/src/branch/master/kuryr_kuber...
Cheers!
[1] https://docs.openstack.org/kuryr-kubernetes/latest/nested_vlan_mode.html
Thanks, Michał
Thanks! Jason Anderson
---
Chameleon DevOps Lead Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory
participants (2)
-
Jason Anderson
-
Michał Dulko