Hm,
so a mixed OpenStack-K8s edge setup, where edge sites are
Kubernetes
deployments? We've took a look at some edge use cases with
Kuryr
and one problem people see is that if an edge site becomes
disconnected
from the main side, Kuryr will not allow creation of new
Pods
and Services as it needs connection to Neutron and Octavia APIs
for
that. If that's not a problem had you gave a thought into running
distributed
compute nodes [1] as edge sites and then Kubernetes on top
of
them? This architecture should be doable with Kuryr (probably with
minor
changes).
Sort of! I work in research infrastructure and we are building an IoT/edge testbed for computer science researchers who wish to do research in edge computing. It’s a bit mad science-y. We are buying and configuring relatively high-powered edge devices
such as Raspberry Pis and Jetson Nanos and making them available for experimentation at a variety of sites. Separately, the platform supports any owner of a supported device to have it managed by the testbed (i.e., they can use our interfaces to launch containers
on it and connect it logically to other devices / resources in the cloud.)
Distributed compute node looks a bit too heavy for this highly dynamic use-case, but thank you for sharing.
Anyways, one might ask why Neutron at all. I am hopeful we can get some interesting properties such as network isolation and the ability to bridge traffic from containers across other layer 2 links such as those provided by
AL2S.
OVN may help if it can remove the need for RabbitMQ, which is probably the
most difficult aspect to remove from OpenStack’s dependencies/assumptions,
yet also one of the most pernicious from a security angle, as an untrusted
worker node can easily corrupt the control plane.
It's
just Kuryr which needs access to the credentials, so possibly you
should
be able to isolate them, but I get the point, containers are
worse
at isolation than VMs.
I’m less worried about the mechanism for isolation on the host and more the amount of privileged information the host must keep secure, and the impact of that information being compromised. Because our experimental target system involves container engines
maintained externally to the core site, the risk of compromise on the edge is high. I am searching for an architecture that greatly limits the blast radius of such a compromise. Currently if we use standard Neutron networking + Kuryr, we must give RabbitMQ
credentials and others to the container engines on the edge, which papers such as
http://seclab.cs.sunysb.edu/seclab/pubs/asiaccs16.pdf have documented as a trivial escalation path.
For this reason, narrowing the scope of what state the edge hosts can influence on the core site is paramount.
Re: admin creds, maybe it is possible to carefully craft a role that only works
for some Neutron operations and put that on the worker nodes. I will explore.
I
think those settings [2] is what would require highest Neutron
permissions
in baremetal case.
Thanks — so it will need to create and delete ports. This may be acceptable; without some additional API proxy layer for the edge hosts, a malicious edge host could create bogus ports and delete good ones, but that is a much smaller level of impact. I
think we could create a role that only allowed such operations and generate per-host credentials.