Open Stack

I suppose that the security argument against running the whole of
nova-compute as root is that a remote exploit in the service is much better
constrained when the thing isn't running as root - e.g. some input
validation fails and allows arbitrary shell in some (currently none-root)
command via an existing API call. This means that (2) is the worst possible
option - calling out to sudo at least limits the attach surface to those
code paths that do those specific exec calls.

I think Daniel's suggestion makes a great deal more sense from a security
point of view, but (3) is a significantly better architecture than (2) even
if it still has some holes.

On 4 February 2015 at 16:33, Monty Taylor <mordred at inaugust.com> wrote:

> On 02/04/2015 06:57 AM, Daniel P. Berrange wrote:
> > On Wed, Feb 04, 2015 at 11:58:03AM +0100, Thierry Carrez wrote:
> >> The first one is performance -- each call would spawn a Python
> >> interpreter which would then call the system command. This was fine when
> >> there were just a few calls here and there, not so much when it's called
> >> a hundred times in a row. During the Juno cycle, a daemon mode was added
> >> to solve this issue. It is significantly faster than running sudo
> >> directly (the often-suggested alternative). Projects still have to start
> >> adopting it though. Neutron and Cinder have started work to do that in
> Kilo.
> >>
> >> The second problem is the quality of the filter definitions. Rootwrap is
> >> a framework to enable isolation. It's only as good as the filters each
> >> project defines. Most of them rely on CommandFilters that do not check
> >> any argument, instead of using more powerful filters (which are arguably
> >> more painful to maintain). Developers routinely add filter definitions
> >> that basically remove any isolation that might have been there, like
> >> allowing blank dd, tee, chown or chmod.
> >
> > I think this is really the key point which shows rootwrap as a concept
> > is broken by design IMHO. Root wrap is essentially trying to provide an
> > API for invoking privileged operations, but instead of actually designing
> > an explicit API for the operations, we done by implicit one based on
> > command args. From a security POV I think this approach is doomed to
> > failure, but command arg strings are faaaar to expressive a concept
> > to deal with.
> >
> >> What solutions do we have ?
> >>
> >> (1) we could get our act together and audit and fix those filter
> >> definitions. Remove superfluous usage of root rights, make use of
> >> advanced filters for where we actually need them. We have been preaching
> >> for that at many many design summits. This is a lot of work though...
> >> There were such efforts in the past, but they were never completed for
> >> some types of nodes. Worse, the bad filter definitions kept coming back,
> >> since developers take shortcuts, reviewers may not have sufficient
> >> security awareness to detect crappy filter definitions, and I don't
> >> think we can design a gate test that would have such awareness.
> >>
> >> (2) bite the bullet and accept that some types of nodes actually need
> >> root rights for so many different things, they should just run as root
> >> anyway. I know a few distributions which won't be very pleased by such a
> >> prospect, but that would be a more honest approach (rather than claiming
> >> we provide efficient isolation when we really don't). An added benefit
> >> is that we could replace a number of shell calls by Python code, which
> >> would simplify the code and increase performance.
>
> I'm actually the biggest fan of this solution (even more than Daniel's
> suggestion below) because it's the thing that is closest to reality.
>
> Security isn't a useful concept in a vacuum - it's something we do to
> prevent access to or damage resources that we don't want accessed by the
> wrong people.
>
> On compute nodes, the main valuable thing are the VMs themselves- and
> I'd expect the most interested target of an attack to be interested in
> manipulating, stealing data from or deleting the VMs.
>
> No amount of rootwrap or privileges are going to prevent nova-compute
> from performing unwanted actions on the VMs in its control - for the
> reason that it's job in life is to manipulate those things.
>
> Is it a security hole in the traditional distro sense - that we want to
> be able to install all of these things with apt-get or yum on a single
> server and have the actions of one service not affect the state of
> another? Sure. Is it in the real world? No. You're not going to use this
> to manage VMs on a laptop - you're going to use virtualbox or
> virt-manager. You're going to use nova-compute to manage compute hosts
> in a cloud - and in almost all circumstances the only thing that's going
> to be running on your compute hosts is going to be nova-compute.
>
>
> >> (3) intermediary solution where we would run as the nova user but run
> >> sudo COMMAND directly (instead of sudo nova-rootwrap CONFIG COMMAND).
> >> That would leave it up to distros to choose between a blanket sudoer or
> >> maintain their own filtering rules. I think it's a bit hypocritical
> >> though (pretend the distros could filter if they wanted it, when we
> >> dropped the towel on doing that ourselves). I'm also not convinced it's
> >> more secure than solution 2, and it prevents from reducing the number of
> >> shell-outs, which I think is a worthy idea.
> >>
> >> In all cases I would not drop the baby with the bath water, and keep
> >> rootwrap for all the cases where root rights are needed on a very
> >> specific set of commands (like neutron, or nova's api-metadata). The
> >> daemon mode should address the performance issue for the projects making
> >> a lot of calls.
> >
> >
> > (4) I think that ultimately we need to ditch rootwrap and provide a
> proper
> > privilege separated, formal RPC mechanism for each project.
> >
> > eg instead of having a rootwrap command, or rootwrap server attempting
> > to validate safety of
> >
> >     qemu-img create -f qcow2
> /var/lib/nova/instances/instance00001/disk.qcow2
> >
> > we should have a  nova-compute-worker daemon running as root, that
> accepts
> > an RPC command from nova-compute running unprivileged. eg
> >
> >     CreateImage("instane0001", "qcow2", "disk.qcow")
> >
> > This immediately makes it trivial to validate that we're not trying to
> > trick qemu-img into overwriting some key system file.
> >
> > This is certainly alot more work than trying to patchup rootwrap, but
> > it would provide a level of security that rootwrap can never achieve
> IMHO.
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Duncan Thomas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150204/c80ae860/attachment.html>