Open Stack

On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
> On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
>> The first one is performance -- each call would spawn a Python
>> interpreter which would then call the system command. This was fine when
>> there were just a few calls here and there, not so much when it's called
>> a hundred times in a row. During the Juno cycle, a daemon mode was added
>> to solve this issue. It is significantly faster than running sudo
>> directly (the often-suggested alternative). Projects still have to start
>> adopting it though. Neutron and Cinder have started work to do that in Kilo.
>>
>> The second problem is the quality of the filter definitions. Rootwrap is
>> a framework to enable isolation. It's only as good as the filters each
>> project defines. Most of them rely on CommandFilters that do not check
>> any argument, instead of using more powerful filters (which are arguably
>> more painful to maintain). Developers routinely add filter definitions
>> that basically remove any isolation that might have been there, like
>> allowing blank dd, tee, chown or chmod.
> 
> I think this is really the key point which shows rootwrap as a concept
> is broken by design IMHO. Root wrap is essentially trying to provide an
> API for invoking privileged operations, but instead of actually designing
> an explicit API for the operations, we done by implicit one based on
> command args. From a security POV I think this approach is doomed to
> failure, but command arg strings are faaaar to expressive a concept
> to deal with.
> 
>> What solutions do we have ?
>>
>> (1) we could get our act together and audit and fix those filter
>> definitions. Remove superfluous usage of root rights, make use of
>> advanced filters for where we actually need them. We have been preaching
>> for that at many many design summits. This is a lot of work though...
>> There were such efforts in the past, but they were never completed for
>> some types of nodes. Worse, the bad filter definitions kept coming back,
>> since developers take shortcuts, reviewers may not have sufficient
>> security awareness to detect crappy filter definitions, and I don't
>> think we can design a gate test that would have such awareness.
>>
>> (2) bite the bullet and accept that some types of nodes actually need
>> root rights for so many different things, they should just run as root
>> anyway. I know a few distributions which won't be very pleased by such a
>> prospect, but that would be a more honest approach (rather than claiming
>> we provide efficient isolation when we really don't). An added benefit
>> is that we could replace a number of shell calls by Python code, which
>> would simplify the code and increase performance.

I'm actually the biggest fan of this solution (even more than Daniel's
suggestion below) because it's the thing that is closest to reality.

Security isn't a useful concept in a vacuum - it's something we do to
prevent access to or damage resources that we don't want accessed by the
wrong people.

On compute nodes, the main valuable thing are the VMs themselves- and
I'd expect the most interested target of an attack to be interested in
manipulating, stealing data from or deleting the VMs.

No amount of rootwrap or privileges are going to prevent nova-compute
from performing unwanted actions on the VMs in its control - for the
reason that it's job in life is to manipulate those things.

Is it a security hole in the traditional distro sense - that we want to
be able to install all of these things with apt-get or yum on a single
server and have the actions of one service not affect the state of
another? Sure. Is it in the real world? No. You're not going to use this
to manage VMs on a laptop - you're going to use virtualbox or
virt-manager. You're going to use nova-compute to manage compute hosts
in a cloud - and in almost all circumstances the only thing that's going
to be running on your compute hosts is going to be nova-compute.


>> (3) intermediary solution where we would run as the nova user but run
>> sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
>> That would leave it up to distros to choose between a blanket sudoer or
>> maintain their own filtering rules. I think it's a bit hypocritical
>> though (pretend the distros could filter if they wanted it, when we
>> dropped the towel on doing that ourselves). I'm also not convinced it's
>> more secure than solution 2, and it prevents from reducing the number of
>> shell-outs, which I think is a worthy idea.
>>
>> In all cases I would not drop the baby with the bath water, and keep
>> rootwrap for all the cases where root rights are needed on a very
>> specific set of commands (like neutron, or nova's api-metadata). The
>> daemon mode should address the performance issue for the projects making
>> a lot of calls.
> 
> 
> (4) I think that ultimately we need to ditch rootwrap and provide a proper
> privilege separated, formal RPC mechanism for each project.
> 
> eg instead of having a rootwrap command, or rootwrap server attempting
> to validate safety of
> 
>     qemu-img create -f qcow2 /var/lib/nova/instances/instance00001/disk.qcow2
> 
> we should have a  nova-compute-worker daemon running as root, that accepts
> an RPC command from nova-compute running unprivileged. eg
> 
>     CreateImage("instane0001", "qcow2", "disk.qcow")
> 
> This immediately makes it trivial to validate that we're not trying to
> trick qemu-img into overwriting some key system file.
> 
> This is certainly alot more work than trying to patchup rootwrap, but
> it would provide a level of security that rootwrap can never achieve IMHO.