Hi Michał, I continue to appreciate the information you are providing. I’ve been doing some more research into the landscape of systems and had a few follow-up questions. I’ve also left some clarifying remarks if you are interested. I’m currently evaluating OVN, haven’t used it before and there’s a bit of a learning curve ;) However it seems like it may solve a good part of the problem by removing RabbitMQ and reducing the privileges of the edge host w.r.t. network config. Now I’m looking at kuryr-kubernetes. 1. What is the difference between kuryr and kuryr-kubernetes? I have used kuryr-libnetwork before, in conjunction with kuryr-server (which I think is provided via the main kuryr project?). I am using Kolla Ansible so was spared some of the details on installation. I understand kuryr-libnetwork is basically “kuryr for Docker” while kuryr-kubernetes is “kuryr for K8s”, but that leaves me confused about what exactly the kuryr repo is. 2. A current idea is to have several edge “compute” nodes that will run a lightweight k8s kubelet such as k3s. OVN will provide networking to the edge nodes, controlled from the central site. I would then place kuryr-k8s-controller on the central site and kuryr-cni-daemon on all the edge nodes. My question is: could users create their own Neutron networks (w/ their own Geneve segment) and launch pods connected on that network, and have those pods effectively be isolated from other pods in the topology? As in, can k8s be told that pod A should launch on network A’, and pod B on network B’? Or is there an assumption that from Neutron’s perspective all pods are always on a single Neutron network? Cheers, and thanks! /Jason On Oct 27, 2021, at 12:03 PM, Michał Dulko <mdulko@redhat.com<mailto:mdulko@redhat.com>> wrote: Hm, so a mixed OpenStack-K8s edge setup, where edge sites are Kubernetes deployments? We've took a look at some edge use cases with Kuryr and one problem people see is that if an edge site becomes disconnected from the main side, Kuryr will not allow creation of new Pods and Services as it needs connection to Neutron and Octavia APIs for that. If that's not a problem had you gave a thought into running distributed compute nodes [1] as edge sites and then Kubernetes on top of them? This architecture should be doable with Kuryr (probably with minor changes). Sort of! I work in research infrastructure and we are building an IoT/edge testbed for computer science researchers who wish to do research in edge computing. It’s a bit mad science-y. We are buying and configuring relatively high-powered edge devices such as Raspberry Pis and Jetson Nanos and making them available for experimentation at a variety of sites. Separately, the platform supports any owner of a supported device to have it managed by the testbed (i.e., they can use our interfaces to launch containers on it and connect it logically to other devices / resources in the cloud.) Distributed compute node looks a bit too heavy for this highly dynamic use-case, but thank you for sharing. Anyways, one might ask why Neutron at all. I am hopeful we can get some interesting properties such as network isolation and the ability to bridge traffic from containers across other layer 2 links such as those provided by AL2S<https://internet2.edu/services/layer-2-service/>. OVN may help if it can remove the need for RabbitMQ, which is probably the most difficult aspect to remove from OpenStack’s dependencies/assumptions, yet also one of the most pernicious from a security angle, as an untrusted worker node can easily corrupt the control plane. It's just Kuryr which needs access to the credentials, so possibly you should be able to isolate them, but I get the point, containers are worse at isolation than VMs. I’m less worried about the mechanism for isolation on the host and more the amount of privileged information the host must keep secure, and the impact of that information being compromised. Because our experimental target system involves container engines maintained externally to the core site, the risk of compromise on the edge is high. I am searching for an architecture that greatly limits the blast radius of such a compromise. Currently if we use standard Neutron networking + Kuryr, we must give RabbitMQ credentials and others to the container engines on the edge, which papers such as http://seclab.cs.sunysb.edu/seclab/pubs/asiaccs16.pdf have documented as a trivial escalation path. For this reason, narrowing the scope of what state the edge hosts can influence on the core site is paramount. Re: admin creds, maybe it is possible to carefully craft a role that only works for some Neutron operations and put that on the worker nodes. I will explore. I think those settings [2] is what would require highest Neutron permissions in baremetal case. Thanks — so it will need to create and delete ports. This may be acceptable; without some additional API proxy layer for the edge hosts, a malicious edge host could create bogus ports and delete good ones, but that is a much smaller level of impact. I think we could create a role that only allowed such operations and generate per-host credentials. [1] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features... [2] https://opendev.org/openstack/kuryr-kubernetes/src/branch/master/kuryr_kuber... Cheers! [1] https://docs.openstack.org/kuryr-kubernetes/latest/nested_vlan_mode.html Thanks, Michał Thanks! Jason Anderson --- Chameleon DevOps Lead Department of Computer Science, University of Chicago Mathematics and Computer Science, Argonne National Laboratory