Open Stack

Excerpts from Tristan Cacqueray's message of 2015-02-04 09:02:19 -0800:
> On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
> > On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
> >> What solutions do we have ?
> >>
> >> (1) we could get our act together and audit and fix those filter
> >> definitions. Remove superfluous usage of root rights, make use of
> >> advanced filters for where we actually need them. We have been preaching
> >> for that at many many design summits. This is a lot of work though...
> >> There were such efforts in the past, but they were never completed for
> >> some types of nodes. Worse, the bad filter definitions kept coming back,
> >> since developers take shortcuts, reviewers may not have sufficient
> >> security awareness to detect crappy filter definitions, and I don't
> >> think we can design a gate test that would have such awareness.
> >>
> >> (2) bite the bullet and accept that some types of nodes actually need
> >> root rights for so many different things, they should just run as root
> >> anyway. I know a few distributions which won't be very pleased by such a
> >> prospect, but that would be a more honest approach (rather than claiming
> >> we provide efficient isolation when we really don't). An added benefit
> >> is that we could replace a number of shell calls by Python code, which
> >> would simplify the code and increase performance.
> >>
> >> (3) intermediary solution where we would run as the nova user but run
> >> sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
> >> That would leave it up to distros to choose between a blanket sudoer or
> >> maintain their own filtering rules. I think it's a bit hypocritical
> >> though (pretend the distros could filter if they wanted it, when we
> >> dropped the towel on doing that ourselves). I'm also not convinced it's
> >> more secure than solution 2, and it prevents from reducing the number of
> >> shell-outs, which I think is a worthy idea.
> >>
> >> In all cases I would not drop the baby with the bath water, and keep
> >> rootwrap for all the cases where root rights are needed on a very
> >> specific set of commands (like neutron, or nova's api-metadata). The
> >> daemon mode should address the performance issue for the projects making
> >> a lot of calls.
> > 
> > 
> > (4) I think that ultimately we need to ditch rootwrap and provide a proper
> > privilege separated, formal RPC mechanism for each project.
> > 
> > eg instead of having a rootwrap command, or rootwrap server attempting
> > to validate safety of
> > 
> >     qemu-img create -f qcow2 /var/lib/nova/instances/instance00001/disk.qcow2
> > 
> > we should have a  nova-compute-worker daemon running as root, that accepts
> > an RPC command from nova-compute running unprivileged. eg
> > 
> >     CreateImage("instane0001", "qcow2", "disk.qcow")
> > 
> > This immediately makes it trivial to validate that we're not trying to
> > trick qemu-img into overwriting some key system file.
> > 
> > This is certainly alot more work than trying to patchup rootwrap, but
> > it would provide a level of security that rootwrap can never achieve IMHO.
> > 
> 
> This 4th idea sounds interesting, though we are assuming this new service
> running as root would be exempt of bug, especially if it uses the same
> libraries as non-root services... For example a major bug in python would
> give attacker direct root access while the rootwrap solution would in
> theory keep the intruder at the sudo level...
> 

I don't believe that anyone assumes the new service would be without
bugs. But just like the OpenSSH team saw years ago, privilege separation
means that you can absolutely know what is running as root, and what is
not. So when you decide to commit your resources to code audits, you
_start_ with the things that run with elevated privileges.

> 
> For completeness, I'd like to propose a more long-term solution:
> 
> (5) Get ride of root! Seriously, OpenStack could support security mechanism
> like SELinux or AppArmor in order to properly isolate service and let them
> run what they need to run.
> 
> For what it worth, the underlying issue here is having a single almighty super
> user: root and thus we should, at least, consider solution that remove the
> need of such powers (e.g. kernel module loading, ptrace or raw socket).
> 

We don't need a security module to drop all of those capabilities
entirely and run as a hobbled root user. By my measure, this process for
nova-compute would only need CAP_NET_ADMIN, CAP_SYS_ADMIN and CAP_KILL.
These capabilities can be audited per-agent and even verified as needed
simply by running integration tests without each one to see what breaks.

> Beside, as long as sensitive process are not contained at the system level,
> the attack surface for a non-root user is still very wide (e.g. system calls,
> setuid binaries, ipc, ...)
> 
> 
> While this might sounds impossible to implement upstream because it's too
> vendor specific or just because of other technicals difficulties,
> I guess it still deserves a mention in this thread.
> 

I think OpenStack can do its part by making privilege separation a
reality for the security sensitive parts of OpenStack, and that will
ease pressure to implement strong controls in any security modules the
Linux distros ship with.